Nats Stunt Seeding

[RC Storick]

"I am just discussing what I would like to see."

I am just discussing what I would like you to do.  There are many opportunities: 1) you could write the data-extraction macro for maneuver scores, 2) you could review the possibly bogus finals judge selection formula, or 3) you could discuss how to use the scores of multiple judges, as Mike did above.  I suggested the latter two because you mentioned being a Six Sigma Black Belt.  The process could use advice from people who know statistics.   

PW at one time suggested maybe throwing out high and low judges for the finals, but the discussion didn't get very far, probably because of what Mike showed above.  Sounds like a bad idea to me, but as Sparky says, other sports do it.  A little research might show why they do.  Then, armed with that information, we could see if it applies to our situation. 

I think it is done on a per maneuver attempt in large sporting events but I am not sure. If you did it that way I am sure it would not be the same judges score thrown out all the time. Makes me no difference how it is decided to be done. That said with the more I monitor this thread I think the idea of showing the scores by each judge for each flight is a even worse Idea than I had originally thought.

[Howard Rush]

Didn't Dave Cook suggest tossing the high and low score PER MANEUVER at one time? I thought that it had merit, reather than tossing the high and low score?

Based on what Mike shows above, I can't see that it has merit, but lets see some analysis. 

[Brett Buck]

Oh my GAWD, Brett...  get a hold of yourself.  Settle down.

I do not want to *start a war*.  The *war is won* as far as I am concerned.  I think what Paul Walker and Bill Rich have been doing with the Nats is just great.  I am just discussing what I would like to see.  This is still America.  I thought we could still do that....

    Sure you can, and you can expect someone else to discuss what THEY want to see, too. Who said you couldn't say it? I found it quite illuminating.

    And once again, you said the magic words yourself - "maybe those judges *should have been run off*" in response to my comments about NATs judges in the 90's. Anybody who thinks that was in any way justified has taken themselves out of the "rational analysis" realm and into the "harassment is just fine" realm.  If that's not what you meant, it was what you typed, and there is an "edit" function.

   The fact remains that there was a campaign of harassment against judges and the organizers based on the results of one analysis of one contest that resulted in several people getting fed up and refusing to judge anymore, and several others being threatened with physical violence.  That's not a point of debate, it's not an opinion, that actually happened. It's still happening, on a low level and on occasion. I for one find that appalling, I think any reasonable person would,  and one of the primary responsibilities I think we have as a group (stunt, PAMPA, however you want to do it) is to prevent that from happening again.

It's my opinion, strongly reinforced by this thread and the small amount of other information I have, the same thing would happen again if the data got released, at least without obscuring the mapping to pilot and judge to the point that the data became completely meaningless.

   It's also my opinion that any such analysis is patently pointless by it's nature. As I mentioned to several in 1994, there are no "facts" associated with judges scores. It's someone's opinion. Any attempt to glean "fact" by analysis is therefore doomed to failure. Any "outlier" could be the result of intentional bias, unintentional bias, bad but honest judging, or a genuinely different opinion. The methodology Paul and Howard came up with (essentially an automated version of what Warren/Shareen did for years, and were regularly pilloried for) will probably detect grossly incompetent judges in some cases, assuming there were any. And now that it is completely automated nobody can plausibly claim that it is being done to favor or harm anyone.  There's absolutely no way to distinguish the other three things afterwards, there is no math that can tell the difference, it's inherently not knowable. So any attempt to figure it out is doomed by definition.

   So, once again, I come to the conclusion that there is virtually no good that can come from publishing sufficient data to do an analysis, and plenty of documented evidence that such data was grossly misused to the severe detriment of the event in the past, and no reason to think it might be different this time around. Like I said, a no-brainer, not even a close call.

   I don't expect you to agree, but that's my case. Tear it apart if you want/can. I think you need to show that the potential advantages outweigh the potential for abuse and I think that bar is going to be VERY high to convince me or anyone else.

     Brett
     
     

[Howard Rush]

" I think what Paul Walker and Bill Rich have been doing with the Nats is just great. "

What has Bill Rich done?

[Brett Buck]

Based on what Mike shows above, I can't see that it has merit, but lets see some analysis. 

  I did that back in the day. What it does is select for a narrow scoring range, even if you normalize it first. If you don't normalize it first, it selects for narrow scoring range AND for how close the average scores are to each other. It's like codifying "bracket scoring".

    I think it's quite clear that ANY scheme that you might come up with would first have to normalize the scores, meaning it has to happen after all the scores were available, meaning you couldn't post the scores until the round was complete.   That's not necessarily a show-stopper but something that would have to be considered.

      Brett

[peabody]

All of these thoughts have some merit...but would not sway the results...

The real issue is that different regions produce different interpretations of the rule book, and fliers from each region exemplify the interpretation of that region..I would believe that the early results (when Navy judges were used) might reflect a more realistic view of who flew the best...

[Bob Reeves]

An Old Rancher's Advice:

* Your fences need to be horse-high, pig-tight, and bull-strong.

* Keep skunks, bankers (and generally lawyers) at a distance.

* Life is simpler when you plow around the stump.

* A bumble bee is considerably faster than a John Deere tractor.

* Words that soak into your ears are whispered...not yelled.

* Meanness don't jes' happen overnight.

* Forgive your enemies. It messes up their heads.

* Do not corner something that you know is meaner than you.

* It don't take a very big person to carry a grudge.

* You cannot unsay a cruel word.

* Every path has a few puddles.

* When you wallow with pigs, expect to get dirty.

* The best sermons are lived, not preached.

* Most of the stuff people worry about ain't never gonna happen anyway.

* Don't judge folks by their relatives.

* Remember that silence is sometimes the best answer.

* Live a good, honorable life. Then when you get older and think back, you'll enjoy it a second time.

* Don't interfere with somethin' that ain't botherin' you none.

* Timing has a lot to do with the outcome of a rain dance.

* If you find yourself in a hole, the first thing to do is stop diggin'.

* Sometimes you get, and sometimes you get got.

* The biggest troublemaker you'll ever have to deal with, watches you from the mirror every mornin'.

* Always drink upstream from the herd.

* Good judgment comes from experience, and a lotta that comes from bad judgment.

* Lettin' the cat outta the bag is a whole lot easier than puttin' it back in.

* If you get to thinkin' you're a person of some influence, try orderin' somebody else's dog around.

* Live simply. Love generously. Care deeply. Speak kindly.

[Bradley Walker]

" I think what Paul Walker and Bill Rich have been doing with the Nats is just great. "

What has Bill Rich done?

Nothing yet, I suppose.

[Bradley Walker]

   I don't expect you to agree, but that's my case. Tear it apart if you want/can. I think you need to show that the potential advantages outweigh the potential for abuse and I think that bar is going to be VERY high to convince me or anyone else.

Fine.  That is your opinion, I am not going to fault you for it.

Stop making everything into a "war".

[Mike Ferguson]

Didn't Dave Cook suggest tossing the high and low score PER MANEUVER at one time? I thought that it had merit, reather than tossing the high and low score?

But if you use the scores I posted earlier, that still means you keep the guys who score every single manuever a 33 or 34 for most of the scoring.  "Per manuever" doesn't necessarily mean you're keeping more accurate scoring, either.

Brett said it best, I think - you'd really need to normalize the scores, which couldn't be done until all the flights for a given round on a given circle were completed.  That'd certainly be different.  And I'd really need to run an analysis of many rounds of competition to see what difference (if any) it would make.

[Richard Grogan]

OK,We'll try this just one more time. If you have something intelligent to contribute to Howards thread, feel free to proceed in an civil manner. If personal attacks are on your agenda,and you still think you are intelligent, do not post. It is unfair to Howard and everyone else who are truly interested.

[Steve Fitton]

Just to be sure I understand this correctly, regardless of whatever computer information is or isn't released, the individual pilot can still get the copies of his scores, right?

[Howard Rush]

Yes.  I specifically asked the event director.  He said, "Yes, the individual score sheets will be available as we discussed.  Yes, the average score for each flight will go on the board. Has to!"  He doesn't want to release individual judge's information for all the fliers, though. 

Thanks for letting this continue, Richard.

[RandySmith]

Just to be sure I understand this correctly, regardless of whatever computer information is or isn't released, the individual pilot can still get the copies of his scores, right?

Hi Steve

The scores for the individual flyer has alway been given to the flyer, and will also be there for the flyers to pickup this year also.


Regards
Randy

[john e. holliday]

In the end and in reality isn't this all supposed be covered at the judges straining sessions at the NATS?  At the end of the session isn't the faults of the training flight brought to light?  Also the people that do the training flights are they trying for the perfect flight or their best flight?  In my opinion the training pilot should throw in a not so par maneuver to see if the judges are scoring accordingly or not.  Also do the training pilots come from the Intermediate class?  I hope they don't come from the group that is going to be judged.

In my opinion the current and past Stunt Event Directors have done their best with what they had.  How many whiners and gripers have stepped forward to help out?   

Now you know why I don't do the NATS event directing anymore.  Of course I have never tried running stunt, combat or speed.    DOC Holliday

[Paul Smith]

Maybe the seeding list *could* be made into a national ranking, and then it can be used on an ongoing basis. 

The problem with this is that the only people on the list are those who enter the AMA Nats.  No doubt, there are at least a few good flyers who do not chose the spend a week at Muncie and pay a rather high entry fee ten years in a row to move up on the totem pole.

It's a valid Nats seeding list, but that's all it is.

[Brett Buck]

The problem with this is that the only people on the list are those who enter the AMA Nats.  No doubt, there are at least a few good flyers who do not chose the spend a week at Muncie and pay a rather high entry fee ten years in a row to move up on the totem pole.

It's a valid Nats seeding list, but that's all it is.

    The question is, how would you use a National Ranking list, if one existed. Even if it was perfect - what would be the end goal or use for such a thing? Other than trying to figure out who went into Advanced or Intermediate (which, for the most part, is already taken care of), it doesn't leap out at me how it would matter. It's not like we are "seeding" anything in the sense that the higher-ranked fliers get an easier path, local or National, and I can't imagine that we would ever do that. Until you answer that question, I don't see how or even why you would bother with such a thing, aside from simply feeding one's ego.

    I want #800 to get the same chance and have jump through the same hoops as #1. If that's the plan, who cares where you rank.

   Alternately, if you aren't using it for anything, then it doesn't matter whether the ranking is right or not, so use whatever criteria you want.

      Brett

[bill marvel]

In the end and in reality isn't this all supposed be covered at the judges straining sessions at the NATS?  At the end of the session isn't the faults of the training flight brought to light?  Also the people that do the training flights are they trying for the perfect flight or their best flight?  In my opinion the training pilot should throw in a not so par maneuver to see if the judges are scoring accordingly or not.  Also do the training pilots come from the Intermediate class?  I hope they don't come from the group that is going to be judged.

The two Nats training sessions I attended (2006 & 2008) seemed to focus mostly on rules and interpretations.  The real value of judging at the Nats is the total immersion into the process.  When you are out there judging 6-8 hours (and more sometimes!) for 7-8 days with a variety of other judges and a broad spectrum of flyers and abilities, it is like graduate school.  As Randi Gifford told me, "You judge... you eat and you sleep... and you judge...!!!  The practical side of judging a practice flight was generally limited to a warm-up session each day.  Don't worry about throwing the judges a ringer, Doc.  There are enough unintendeded "not so par" maneuvers to keep the judges on their toes.
Most of the judges I have worked with seemed honest and were trying to do a good job.  In local contests, I am often paired with a flyer who has been drafted to judge.  I can't think of one who did not give it his or her best shot.  Some are hard and unforgiving.  Some are generous.  So are "Nats-Trained" judges.  Contrary to popular belief, most judges are pulling for you.  We have been standing out there in the sun (or rain!) all day.  One of the things that makes it worthwhile is to see a beautifully executed set of triangles or a square eight with real corners and no bounce in between.  I love to see 40 point maneuvers!
I have been judging only four years locally, regionally and nationally.  I have a long way to go before I will be very good at it.  I know how to be objective and I believe I am. I know how to be honest and I believe I am.  I am learning more and more about what a maneuver should look like.  I was taught from the beginning that 40 was the maximum score and zero was the bottom.  Judges can and should use any number in between to describe the maneuver he or she has just seen.  I think I have used most of the numbers.  Like every judge (and flyer!), I have seen some really gorgeous maneuvers!  And, like every judge (and flyer!), I have struggled to recognise enough elements to define the intended stunt.
I don't know if Paul Walker's Seeding  or Judge's Tracking works but I applaud him and Howard for trying to find a structure.
bill marvel