The maths behind FM are not too complicated. FM is basically a super vibrato where the rate of vibrato is linked by (what can be a very simple) equation to the frequency of the note you're trying to play.
So you have a fundamental sine wave frequency oscillator that produces a sine tone. Then you have another oscillator that varies the frequency of your first tone like vibrato. At a modulation frequency of below the audible spectrum, (a vibrato of 20Hz or so) it sounds like anything from slight to very silly vibrato. Above a vibrato frequency of 20Hz though, strange things start to happen. Instead of vibrato you start to get additional frequencies related to the fundamental appearing that change the timbre instead of the pitch. What they are depends on the relationship between the fundamental frequency, and the modulation frequency. This relationship is called the modulation index.
So if you have a modulation index of 2, that means the mod frequency is always twice that of the fundamental. So when you play a note at 440Hz A on your keyboard, that would be being varied with a vibrato of 880Hz, which is ridiculous, but sounds awesome. In fact if you go through and try out all the basic mod indexes of 1, 2 3.... up to 10, you've basically heard all the sounds a Sega Genesis makes cause of it's simple FM synths .
Just try and keep all your modulation indexes integer numbers and the sounds will always be fairly harmonic. FM tends to produce spectra that only contain frequencies that are multiples of the fundamental, rather then standard harmonic series components. That's why the majority of first order FM sounds are similar in timbre to bells and chimes.
Obviously you can muck things about with FM8 and such by starting with a square wave instead of a sine wave. Or modulating the mod frequency with another oscillator. Try building a simple uber vibrato in Reactor if you have it. It really helps you understand whats happening.