Complicated.  And not necessarily the same from motor to motor.

The one I have on my desk has 12 magnets in the bell and 9 poles on the armature.  Having fewer armature poles than magnets makes it a bit harder to think about, but it can be done.  The magnets are arranged N, S, N, S, etc., so the magnetic field sort of loops in and out and in and out of those magnets.  The windings are arranged so that the pole pieces on the armature are wound with winding A, then B, then C, then A again, etc.

I don't have any pictures, but basically the way it works is that in operation the ESC will put power on two out of the three wires to the motor; this energizes one coil fully and puts half voltage on the other two, and gives the armature the appearance of having six magnetic poles.  These will try to line up with six of the 12 magnets in the bell, and the motor will turn.  The ESC uses the voltage present on the remaining wire to decide what to next, and based on that it switches what power is going where on the three motor wires.

I don't know if, without pictures, this is going to get you very far.  I have a very clear picture of how this works, but it's in my head.  Here, let me think it very hard at you . . . -- does that help?  I looked for an explanation on the web and didn't find anything  .

It's easy to tell what the magnet orientation is if you can get access to the bell and if you have a spare magnet or two (like a hatch magnet or a canopy magnet).  It's also easy to look at the armature and count poles -- and the armature poles are always wound A, B, C, etc.

Tim
"The ESC uses the voltage present on the remaining wire to decide what to next, and based on that it switches what power is going where on the three motor wires." 

I think I understand what you are saying, is the above statement referring to back EMF from the non-energized coils?  So it uses this information to decide which coils to energize next, I never would have guessed this process.
Andy

And it also explains why you haven't read an easy to understand article about it in any modelling magazine.
Dean

The magnets are arranged N, S, N, S, etc., so the magnetic field sort of loops in and out and in and out of those magnets  (quote)

This is the part I don't get.  If the magnets were widely spaced the above might work, but they are about as close as they can be assembled. In my experience, when you stick magnets together N-S,N-S  all you get is N---S and if they are in a circle, you don't get anything! With the close spacing,  I would think the magnets would have to be N-S,S-N,N-S to create the pole positions.   

Now that makes sense! Thanks.   

I still am not quite sure why one coil is fully energized and two are half energized.  Also what is the relative position of the two half energized coils to the one fully energized.

Actually I think I was half wrong when I said that.  If the motor is delta wound (which means that the head end of winding A goes to the tail end of winding B, and the head of B goes to the tail of C, and the head of C goes to the tail of A, and each intersection is connected to a motor lead), then what I said is correct.  If the motor is Y wound (which means that the tail end of all windings are common, and the head ends are connected to the motor leads) then only two coils are energized, each with half the available voltage.

In the delta-wound case, the two half-energized coils will have poles of the opposite sense as the one fully wound coil.  In the Y-wound case, the two energized coils will have poles of the opposite sense, and the last coil will just be going along for the ride.

ThanksTim, it's be nearly 50 years since I sat through "Servo and motors" class in Air Force Tech school, but I kind of remember that delta and "Y" windings were done for torque and revs.  One type delivering higher torque and the other delivering higher revs, I don't have a clue which was which.  As I said it was back in 1965, so I could have it all wrong.  Is there a standard type of winding, delta or "Y" used in the motors for our application?

I don't think there's a standard -- either way is just about as efficient, they just give different Kv and resistances depending on which one is chosen. 

I also remember that there was a way to connect the windings in a servo to servo connection that if you gave one of them a quick twist the other would accelerate to distruction.  We would let it wind up and then shut the power off before windings started to fly off, it was fun.

Probably by connecting the feedback of one to the controller for another, but that depends on the specific system.