Author Topic: Unexpected halts (Read 3244 times)

Brett Buck · « **on:** November 01, 2011, 10:09:48 PM »

Most people are aware that PTG's latest crashed on a practice flight at Golden State when the motor quit climbing in the wingover. I wasn't there for any forensics, so it could have been as simple as a connector pulling out (although knowing Phil, it's not likely). But it brought up the topic of unexpected motor halts and very soon there were reports of maybe 4-5 that locals had observed.

I will probably call Phil about his, but does anyone have any experience with this, and was the case a matter of wiring coming disconnected, or was it some sort of unexplained action of the timer or controller? I am looking to figure out if these are instances of uncommanded state changes (like the hundreds I have seen in other similar embedded processor or low-level logic circuits over the years), or some more mundane wiring or connector issues.

Brett

Tim Wescott · « **Reply #1 on:** November 01, 2011, 10:36:27 PM »

Keep us informed, Brett. A good part of my career has been involved in the care and feeding of itty bitty headless embedded systems (like, for example, timers and ESCs, although I haven't specifically done either of those). So I may have some insights to add along with yours. Life gets hard when they start doing intermittent odd things.

If you have an event and you report it to Brett, be sure to report on your whole setup: ESC brand and model, software version if you know it, timer ditto, motor brand and model, battery brand and model, connector brand and model, whether you use an arming plug and how it's mounted, how long your wires are from battery to ESC, ESC to timer and ESC to motor, how things are mounted, etc. In other words -- everything. Pictures wouldn't hurt. Also detail about what you were doing: where you were in the flight, what maneuver you were doing, whether it was cold, hot, windy, calm, humid, dry, etc.

Note if there are any "beeps" from the plane, too -- in particular an "I've just woken up and I'm happy" beep or a "hey, my receiver just crapped out" beep will narrow things down considerably.

If you have an ESC that logs events and the means to download it, by all means suck that puppy dry and see what it has to say for itself. That's your "black box" and should provide quite a bit more information than the indirect info available above.

If you're enough of an electronics geek to have a logic pulse detector in your tool kit and if you can do so without damage to yourself, check for pulses out of the timer, while you're at it.

Once the problem is figured out all that detail will seem excessive -- but you never know what seemingly insignificant detail is the one that means something.

Brett: how do you want folks to report in? Since I feel I have something to add I'd like folks to report to this thread, where I can keep tabs on it. But it's your baby.

Brett Buck · « **Reply #2 on:** November 01, 2011, 10:43:45 PM »

Quote from: Tim Wescott on November 01, 2011, 10:36:27 PM

Brett: how do you want folks to report in? Since I feel I have something to add I'd like folks to report to this thread, where I can keep tabs on it. But it's your baby.

Just note any relevant incidents here, with whatever details are available. I am not doing statistics, apocryphal reports are fine. But I was very surprised when I asked around and in about 3 emails found 5 distinct incidents, which surprised me given that I would have expected nearly zero. Seems to be more common that the very similar flame-out we occasionally get with ABC/AAC engines.

Brett

schuang · « **Reply #3 on:** November 01, 2011, 11:58:29 PM »

Brett,

I was the one lunched for Phil for that flight. Please read the main forum so that I don't need to repeat my post.

I work for a major semiconductor for 15 years as an integrated circuit designer so I should know few things about electronics. The idea of arming switch is good for safety but it might cause the model to fail more often than non-arming switch plane. The reason is easy-contact get worn over time. The fail mode is not totally lost the contact but instead increase the contact resistance over time. When the contact resistance increase over time, the trouble start coming.

Now let's put some numbers. Assuming the contact increase to 50mohm (0.05ohm), when motor pull 30amps, the IR drop is 1.5volt. When the contact resistance go up to 0.1 ohm, all hell gets loose. This can shut down ESC right away during the hard corner or vertical climb due to the auto low voltage cutoff function of ESC. I am not sure this is the fail mode for Phil, but this did come in to my mind.

Personally I think the arming device is a good idea for safety but we may want to chew on it for a while before putting this in the rule book.

Regards,

Sean

bob branch · « **Reply #4 on:** November 02, 2011, 05:46:20 AM »

I have expereinced this failure mode twice. Both times on rc planes where I was fortunate enough not to be hovering but at altitude. It did allow me to evaluate what happened and it was the same both times. The thing that led to the failure was the interesting part.

In both cases the deans connectors became disconnected. That was the simple part. As we all have expereinced, some of the things you almost need to pry apart and others go nicely, and some come apart real easily. In both cases it the connectors were ones that came apart very easily. The logical fix for this would be to spring open the tension spring on the male, but the male had remained the same in both cases... it was connected to the esc. It was the female connector on the battery that did not fit properly.

The underlying thing that led to the failures, however, was neatness. Having flown RC for a good while I try to have as much wiring done neatly as I can. I am not building planes that have every wire parallel and perfectly aligned but I try to be neat as any of us would. In the case of the plane in question the power leads were perfect. With the battery at the right cg position there was no tension on the power connection and no slack. Any of us would look at it and not give it a thought. But under g's of maneuvers the deans with little connection resistance came under enough load to pull loose. I was flying a move called a "wall" on each occasion of failure.... the first corner of our CL wing over!

To prevent this I tried spreading the retention spring on the males and that worked for the pack that failed but had of course too much tension on other packs. The solution was simple. I never allow a straight line run in my power lines any longer. I orient the battery so there is a 180 degree bend in the power cables with the connection in the middle. This provides adequate friction to keep the connectors together and assures there is some slack and nothing comes other tension if something shifts. Of course batteries never shift.... but in my experience in models... @#$% happens.

bob branch

Jason Greer · « **Reply #5 on:** November 02, 2011, 07:41:31 AM »

Could also have been a thermal or low voltage shutdown. Would there be any possibility that he may have accidentally flown with a battery that was not fully charged. It would be easy to do and I have to really watch myself when practicing to ensure that I always keep my discharged packs separate from the fully charged packs. This would explain why it shutdown while climbing due to the governor asking to much of the pack and the voltage dropping below the shutdown thresh hold setup in the ESC. Of course, this could easily be ruled out if the battery was not damaged during the crash and could be recharged.

If he was using a Castle ESC then both of these modes by default would shut the motor down to prevent further damage to the ESC (thermal) or battery (low voltage). It would be great if the data could be downloaded from the ESC, but if he is like me then he does not purge the logged data after each flight. There is just enough storage capacity on the ESC for about 6 minutes if the sample frequency is 10 Hz.

Jason Greer · « **Reply #6 on:** November 02, 2011, 08:23:10 AM »

My apologies. I just saw the thread in the open forum about verifying that a discharged battery was not to blame. Hopefully the data dump will provide some idea as to what happened.

Tim Wescott · « **Reply #7 on:** November 02, 2011, 09:26:55 AM »

Quote from: Jason Greer on November 02, 2011, 08:23:10 AM

My apologies. I just saw the thread in the open forum about verifying that a discharged battery was not to blame. Hopefully the data dump will provide some idea as to what happened.

It's a pretty common error with batteries to think that you can get a clue of their charge state by measuring their unloaded voltage.

I know that with NiCd cells, the open-circuit voltage is pretty much the same from about 95% charged to (if you give the pack time to recover), well past dead. This is why, if you want to assess the level of charge in a NiCd, you need to measure the thing under load (and even that isn't reliable). I understand that LiPo cells' charge level shows up more in their quiescent voltage -- but I don't think it shows up all that much, and did the fellow who measured the battery voltage know this when he made the decision that the pack was charged?

I just measured two of my RC electric packs (3S): pack 1 was used this past Saturday and I haven't gotten around to charging it. It has maybe 25% to 50% charge on it, and it measures 11.5V; pack 2 was charged the Saturday before last and has 12.4V on it.

So if someone just measured the pack and saw more than 3.7V/cell on it, particularly if it had had some recovery time, I'm not sure that would reliably indicate a charged pack.

Brett Buck · « **Reply #8 on:** November 02, 2011, 09:48:56 AM »

Quote from: Tim Wescott on November 02, 2011, 09:29:35 AM

Brett's effort here isn't necessarily going to have any effect on what goes into the rule book -- we don't, after all, know what's going on yet.

But it has great potential to reduce crashes with electric planes.

So why discourage him from doing a bunch of free work?

This has nothing to do with the rule book or the proposed rule change. Anything that could come out of this would already be covered by the existing proposal. And I remind everyone that the proposed rule is *already in place* for all intents and purposes, all I am doing is writing it down so it will be obvious to anyone who reads the rule book.

I am already fully up to speed on the mechanism of processor glitches, so it wouldn't surprise me to see a few of them. Given that I am (and almost everyone else is) going to wind up flying electric propulsion shortly I was very surprised at the high frequency of unexpected motor halts (and continue, I am up to about 10 apocryphal stories just since last night). I am curious about what the failure modes are and this the sort of thing I do all day long at work so I am taking some pre-emptive work for my own use, and hopefully others.

One of my jobs is to evaluate and analyze reaction wheel control systems for spacecraft attitude control. These are typically embedded processor code driving a regulator loop to control a brushless DC motor to a commanded RPM. Sound familiar?

Brett

Jason Greer · « **Reply #9 on:** November 02, 2011, 10:16:45 AM »

It was also my understanding that an arming device or plug would not be required as long as the model is positively restrained anytime the battery is connected.

I've only witnessed a few unexpected halts in the last 10 years of flying both RC and CL electric models. One event was my own RC model and was easily explainable. The ESC was a charred chunk of waste after landing and was traced back to a manufacturing defect after talking with the manufacturer. It was not a Castle ESC by the way. The other events I have witnessed have been linked to both improper cooling on the ESC and batteries that could not hold a sufficient voltage level at the discharge rate being asked of them. Of course I am not including the many events I have witnessed where the pilot flew to long and simply ran the battery down to a voltage below 3 volts per cell.

Being from south Arkansas it is not difficult to push an ESC to a thermal shutdown in the summer time, especially when pushing a system to its limits. These are the most common events that I have witnessed. Once the ESC is allowed to cool, power can be reestablished and the pilot can land the model.

Robertc · « **Reply #10 on:** November 02, 2011, 10:22:20 AM »

I've had two failures, both times at a contest and both times in Arizona. First one was magnets in the AXI motor coming loose. Motor lost all power as soon as I went
inverted, but was on grass. Second time, the Phoenix 45 esc died just after coming out of the inside squares. Castle Creations never did tell me what failed or why,
just replaced the esc.

Peter Nevai · « **Reply #11 on:** November 02, 2011, 12:16:35 PM »

Hi Brett, I don't envy you with this project. As the root cause for these failures can and probably come from wide variety of events. Historically with electronic devices the major cause for failures is Human Error, The second is environmental factors and the third is system design (Which ofter points back to the first cause) finally hardware failure (which may be aggravated by #2).

If we can eliminate the Human error factor then most likely it is going to be an environmental factor.

Electronics do not play well with wide swings in their operating environment.

Temperature and exposure to the elements plays havoc with electronics. For example anyone who wears glasses can tell you what happens when they go from a airconditioned room outside on a hot day or vice versa. (The main reason they used to put moisture indicators and safeties on camcorders)

Unspent Fuel, Oil, grit can also play a part. Solvents outgassing from the adhesives or finishes attacking insulation, oxidizing connections and solder joints, vibration, a combination thereof.

This is why the military subjects critical systems to a battery of tests such as Shock, Vibration, Humidity, Salt Spray, Temp, and more they demand MTBF for all commercial systems they implement. This is the only real way to get conclusive performance and survivability information. I'm afraid that results from the field, from uncontrolled situations is going to be problematic and unreliable at best.

Eric Viglione · « **Reply #12 on:** November 02, 2011, 01:43:39 PM »

I have witnessed several unexpected halts at my club field, and taken part of the diagnostics in the aftermath.

Reasons for halts so far:

Halt #1: Whoops, put in old battery that's not up to snuff for full pattern time, ran out of steam and ESC shut it down. Died before clover.

Halt #2: Whoops, put in a battery that wasn't marked properly. Wasn't re-charged. Didn't quite make it to the hourglass.

Halt #3 #4 & #5: Gain set too (low?) trying to avoid wind-up, and the high corner loading of the hourglass, and overhead eights on subsequent flights, told it to shut the motor off.

Halt #6: Retract gear hung up just a hair short of pulling the retract door all the way closed, and the door motor kept pulling juice while trying to close it the entire flight, draining the battery prematurely.

Halts #7 & #8 : Found out that if battery is installed, that you can start the timer even without the arming plug in. So, if you install the battery in the pit's, your helper bumps the little red start button by mistake, then the timer starts running. You put the plane on the circle, put in the arming plug, hit the start button and it starts the motor, but does not re-set the timer. So, however many minutes transpired, is subtracted from your expected flight time, and you shut down, probably in your overhead eights... which seems to be where Murphy likes it to happen in these guys case.

Halt #9: Bad deans plug connection on an R/C ship, shut down flying laps along runway. (mentioned the other day in other thread)

Edit: Halt #10: Almost forgot, motor winding partially fried, died middle of pattern, came back on in free-fall, and planted itself on other side of circle, narrowly avoiding pilot. Plane in lots of small pieces.

I'm guessing this probably doesn't even scratch the surface of reasons that an E Power plane can halt unexpectedly... and I've witnessed all these in just over a year in my club.

EricV

Jason Greer · « **Reply #13 on:** November 02, 2011, 02:17:42 PM »

"Halts #7 & #8 : Found out that if battery is installed, that you can start the timer even without the arming plug in. So, if you install the battery in the pit's, your helper bumps the little red start button by mistake, then the timer starts running. You put the plane on the circle, put in the arming plug, hit the start button and it starts the motor, but does not re-set the timer. So, however many minutes transpired, is subtracted from your expected flight time, and you shut down, probably in your overhead eights... which seems to be where Murphy likes it to happen in these guys case."

What kind of timer was this guy using? Did the timer have a dedicated battery? Every timer I've ever seen was powered through the BEC output from the ESC. It would not be possible to have power to the timer without power to the ESC. Assuming they do have a separate power source on the timer and if they are using a Hubin timer (which I think most people are) then nothing should have happened when they installed the arming plug and hit the button the second time on the circle. When they hit the button the second time then that would have stopped the timing sequence. It would be interesting to see how this guy has his components configured...I guess if he was the one running retracts in halt #6 then he could have a separate onboard battery to power the retracts and that could also be supplying power to the timer, but his motor should not have started after hitting the button the second time.

Eric Viglione · « **Reply #14 on:** November 02, 2011, 02:34:16 PM »

Jason - I don't know which one is being used at the moment because I have a hard time keeping track. It seems one of the constants with E-Power is "upgrades". I know they have used Hubin timers at times and still might be using one. If I remember correctly, the one that has the ability for retracts is "Special"... I know I've seen them using the LCD programing tool to set it. I know it's running on the main battery, no second battery. I remember them telling me it was possible to use a separate small battery for retracts, and they were contemplating putting one in the tip weight box for laughs...

As for how it should be wired, beats me... I just know that what was stated in my post was definitely found to be the fault of those halts. Come to think of it, one of their timers may have reacted as you mentioned in another failure, it wouldn't allow a start, because the timer was already running. It just wouldn't start at all, as you predicted how it should work. I didn't count it as a halt because it never got off the ground.

EricV

Jason Greer · « **Reply #15 on:** November 02, 2011, 02:39:30 PM »

Eric - No problem, I certainly wasn't doubting what you said in any way. There are definitely some creative minds out there that can really over complicate the setup with electrics.

Tim Wescott · « **Reply #16 on:** November 02, 2011, 03:00:03 PM »

Quote from: Eric Viglione on November 02, 2011, 01:43:39 PM

Halts #7 & #8 : Found out that if battery is installed, that you can start the timer even without the arming plug in. So, if you install the battery in the pit's, your helper bumps the little red start button by mistake, then the timer starts running. You put the plane on the circle, put in the arming plug, hit the start button and it starts the motor, but does not re-set the timer. So, however many minutes transpired, is subtracted from your expected flight time, and you shut down, probably in your overhead eights... which seems to be where Murphy likes it to happen in these guys case.

That's -- odd. That means that either the arming plug is in one of the motor wires (which is not where it belongs, you might let out the magic smoke), that the arming plug is in the signal lead from the timer (also not where it belongs, but at least there's less risk of letting the magic smoke out of the motor or ESC), or that the system somehow had a separate regulator or battery.

The arming plug should be in one of the leads from the battery to the rest of the system -- not anywhere else.

Do you happen to know where this particular arming plug was?

Eric Viglione · « **Reply #17 on:** November 02, 2011, 03:04:35 PM »

Tim - Like I told Jason - I have no idea on actual wiring spec, but I'm pretty sure I have heard them say it's in line with the motor in the past.
I do know one thing, it does let out a pretty agressive "zap" noise when it's being plugged in...
EricV

Larry Wong · « **Reply #18 on:** November 02, 2011, 03:23:26 PM »

Just a thought, how about to small of a battery? " C" or Volts ?

Dennis Adamisin · « **Reply #19 on:** November 02, 2011, 05:03:54 PM »

I'll wait until PTG checks in and shares what ever he discovers about his unfortunate event and/or ASKS for help.

Tim Wescott · « **Reply #20 on:** November 03, 2011, 11:54:21 AM »

I'm hoping (and kind of expecting) that what Brett will come up with is a laundry list of common pilot (really crew chief) errors that we can know to avoid. Having worked around electronics that are designed for high-vibe, high-shock environments I certainly think I know all the right things to do -- but that just means I'm an arrogant so-and-so. It doesn't mean that I really do know, nor does it mean that I'll be good about doing all of them without a checklist.

And if we do come up with some problems that can be traced back to ESCs and/or timers -- well, everything I've heard has indicated that Castle and Will Hubin are pretty responsive about fixes, and in the case of the cheap Chinese stuff we'll at least know what to do about things.

john e. holliday · « **Reply #21 on:** November 05, 2011, 08:17:27 AM »

Well, at almost 70 years of age I am still learning. When I quit learning is when they throw my ashes out over some control line model park. I have really learned from this thread that do not get in an argument/debate with Bobby Hunt or Derek Berry. They are way smarter than I am when it comes to debating. Now when is Phil going to give the details of the crash he had?

Igor Burger · « **Reply #22 on:** November 05, 2011, 10:29:52 AM »

From our experience here in middle Europe ... all of this happened, this is not "it could be" type of errors, I can imagine many others, this is practice

classic user mistakes:

- low voltage cut off - caused either by mistake when pilot used discharged battery, or old battery which does not have enough capacity. Since we use governors, motor does not show "weak" battery, it will simply one day stop before timer says so. Be sure that it will happen in figure - not just because of Murphy, because higher current draw in figures. Good prevention for this case, is ESC setting which stops motor softly. It gives enough time to recover model from danger position.

- overheating. Usually we do not check temperature of ESC. If it is on edge of allowed temperature, it will simply happen in hot day. We use ESCs in PWM regime, not for full throttle and ESC are rated from producer for full throttle. The current in FET transistors is higher than battery current, so it could be easily overloaded without knowing in PWM regime. The prevention (beside proper ESC selection) is also soft cut of. This happened to me, (in figure, where else? :-))) ) recovered without problems.

technical problems:
- wrong connectors - some types of connectors (especially those where spring part is also the contact) can make such surprise. I know positively about one crash caused by gold connector with spring part feely rotating also when connector was "in". Model entered wingover and stopped - lost model . Prevention is usage safe connectors. Classic "gold" connectors are not the best, deans "T" are much better, but they have only one segment contact and it can be dirty, one piece of "something" can change the area contact to one point what is visible before start, but everything changes under load of 30A. The best in my eyes are connectors with several segments on female side, with steel spring from outside.

- one crash was caused by erratic ESC. The resistor on lead which is sensing rotor position (on one phase leads) was broken because of something (this is information from producer of that ESC) and it caused several times unexpected stop of motor. It normally worked, but probably because of vibration, it simply sometimes lost the contact. Owner decided to test ESC more, it looked OK, but after few flights motor stopped = lost model.

ptg · « **Reply #23 on:** November 05, 2011, 02:13:39 PM »

Wow! The rigors or retirement and trying to decide when to get up in the morning, what to eat, whether to go flying or sit on the deck and drink margaritas etc., can be quite stressful so we went on vacation to San Diego for the past week. If you have never been to the Zoo there it is incredible, especially if you can go when the summer crowds have left, like October.

Anyway, I got an email from someone I have known for over 40 years and who also flies electrics in competition. He has had the same shut down experience twice using virtually the same equipment and we both suspect the same thing. We need to compare notes and consult with the producers of the electric components and see if what we think is really the culprit. Suffice it to say it is nothing that has been discussed here. I think we will have found the problem or we’ll still be scratching our heads by midweek.

‘Til then,
PTG

Dean Pappas · « **Reply #24 on:** November 06, 2011, 07:05:05 PM »

Quote from: ptg on November 05, 2011, 02:13:39 PM

Wow! The rigors or retirement and trying to decide when to get up in the morning, what to eat, whether to go flying or sit on the deck and drink margaritas etc., can be quite stressful so we went on vacation to San Diego for the past week. If you have never been to the Zoo there it is incredible, especially if you can go when the summer crowds have left, like October.

Anyway, I got an email from someone I have known for over 40 years and who also flies electrics in competition. He has had the same shut down experience twice using virtually the same equipment and we both suspect the same thing. We need to compare notes and consult with the producers of the electric components and see if what we think is really the culprit. Suffice it to say it is nothing that has been discussed here. I think we will have found the problem or we’ll still be scratching our heads by midweek.

‘Til then,
PTG

Hi Phil,
Nothing yet discussed here? You have my undivided attention.
Dean

Dean Pappas · « **Reply #25 on:** November 06, 2011, 07:13:01 PM »

Greetings Friends,
I just whittled this thread back to only the posts that are relevant to the subject at hand.
WOW! It got a lot shorter.

take care, All
Dean

EddyR · « **Reply #26 on:** November 07, 2011, 04:53:52 AM »

Some may have hears of electric starts or motor going to full power on RC planes. One well known brand of RC equipment sets the motor to on or full speed on loss of signal. Some have made the mistake of turning off the transmitter with the radio still on and the electric motor will start and go to full speed. This problem is not a CL problem,only RC. I just had a email from my RC group about this problem.
Ed

Dean Pappas · « **Reply #27 on:** November 07, 2011, 10:22:32 AM »

Hi Ed,
Yeah, that's a classic bit of pilot error: reverse the throttle channel and never bother to set the Fail-Safe appropriately.
Most RC-er with the Fail-Safe feature in their radios never even bother to look at that setup menu!
It's not the radio's fault.

Regards,
Dean

John Hammonds · « **Reply #28 on:** November 07, 2011, 04:32:11 PM »

Quote from: Dean Pappas on November 06, 2011, 07:13:01 PM

Greetings Friends,
I just whittled this thread back to only the posts that are relevant to the subject at hand.
WOW! It got a lot shorter.

take care, All
Dean

Well my post disappeared.

Was a motor bearing which seized causing the shut down of the controller not relevant?

TTFN
John.

Dean Pappas · « **Reply #29 on:** November 07, 2011, 06:25:58 PM »

If I got overzealous deleting blocks of posts, I hope you'll understand.
On the other hand, if there was something else in the post that fanned the flames, that would explain it.
Either way ...Sorry

Dean

John Hammonds · « **Reply #30 on:** November 08, 2011, 03:20:52 PM »

Nothing inflammatory that I was aware of.. Believe me, it would take a much braver man than me to step between Bob Hunt and Derek Barry on control line subject matter.

So.... The only "Unexpected Halt" I have every experienced was when half way through some inverted laps my Boxcar Chief just stopped, I was flying over grass so nothing more than a broken fin. The fault not immediately apparent was a seized bearing in the motor (Emax [Arrowind] 2815/09). Unplugging everything , re connecting and everything seemed fine, (Motor started up after the delay) but it was getting late so after a few seconds I shut it down and went home.

Once at home I connected a different motor and it all ran fine for a full 5 minutes, so out came the original and after removing the prop I let it start up in my hand, After about 30 seconds of running the stationary end of the motor (The bit I was holding) got very hot very quickly. Hitting the stop button I disconnected everything including the motor leads and found the motor very stiff to turn. (Rather like trying to turn it with the brake on). Dismantling the motor showed the bearing was toast. An easy replacement (Held in by a cir-clip and spacing washer). Afterwards it was as good as new.

That's it, pretty boring really.

TTFN
John.