Question on top 20 judges

[Scott Richlen]

I'm not sure what you are saying.  A contestant's score for a flight is the average of the flight scores of all the judges on that circle.  Are you suggesting that we not average the scores, but adjust them for judging like the full-scale aerobatics guys do?  It would take a rules change, and would resist that unless I could see (and understand) proof of its statistical validity. 

Judge assessment is something different.  It's based on the order in which each judge ranks the contestants compared to the official ranking of the contestants (the one on the scoreboard).  Here is a description of our judge assessment, written when my dementia had not progressed to today's state: http://www.clstunt.com/htdocs/dc/dcboard.php?az=show_topic&forum=103&topic_id=327442&mesg_id=327442&listing_type=search .  I should mention that Paul intoduced the exceedance term to penalize favoritism.  He set it to zero, where it has remained, either because favoritism was not a problem or because we couldn't agree on how to spell exceedance. 

As I mentioned in the SSW writeup, we could change the judge assessment to maneuver-by-maneuver.  I'd do it and recommend that it be adopted if somebody can show me it's better than flight-by-flight. 

One interesting property of judging is that the guy who uses a narrow scoring band has less influence on the contest's outcome than a guy who uses the full range.  For example, if he gives all maneuvers a 34, he doesn't affect the outcome at all.

Howard:  Sorry that I didn't explain myself better.  I am not talking about changing the score.  But aren't we essentially using the scores to "adjust" the judges by eliminating those judges that do not conform to the "average"?  This, in effect, would bracket the scores simply by selecting the judges closest to average for judging the subsequent rounds.  I'm suggesting that we use maneuver-by-maneuver by first normalizing to average, then normalizing the individual maneuver scores.  Adding a "goodness" factor based on a judge's experience or acknowledged ability would then be used to weight individual scores in summing to an average maneuver score to which all scores would be compared.

The problem with using total scoring for comparison and selecting judges is embodied in your example: what happens when a judge scores 35 on every maneuver and coincidentally ends up with a total score that matches the average?  This judge would be considered the "best" judge.  Correct?

Scott    

[Howard Rush]

Um, it's clear, isn't it, that we don't use contestant scores at all to assess judges?  We look at the order that they rank contestants.  This avoids the high-low issue and the narrow-wide issue.    Please put your method in a formula.   I still don't understand it.

I don't see how to make an objective goodness factor.

[BillLee]

Um, it's clear, isn't it, that we don't use contestant scores at all to assess judges?  We look at the order that they rank contestants.  This avoids the high-low issue and the narrow-wide issue.    Please put your method in a formula.   I still don't understand it.

I don't see how to make an objective goodness factor.

Howard, the method is called a "nomalized cross correlation". It is a measure of how two series of measurements resemble each other. It is a common mathematical analysis tool, the "normalized" part removes the high-low concerns. I have used the technique to analyze F2B judging from past World Championships.

If you wish, we can talk about this later, after Poland, since we're both pretty tight for time right now.l

Bill

[Scott Richlen]

Howard: thanks for your patience with me.  I obviously am not doing a very good job of explaining this.  And Bill: thanks for saying what I was trying to say.

It seems to me that comparing total scores (and thus, resulting placement of pilots, as in 1st, 2nd, and 3rd) is more concerned with excluding favoritism by judges.  By comparing the judge's maneuver scores, one is trying to actually discern which judges are doing the best job of scoring.  I think that developing this method would benefit our judging core by helping show who is on the mark and who isn't.  I don't think that (within reason) it should matter that a particular judge scores high or low as long as they score consistently.  But, total score comparisons as done now, tends to drive towards an average.  If your goal is excellence, your target should not be "average".

Career-wise, I wish I had known this stuff about 20 years ago to use as a management tool.

Bill: I'm not sure how much I can help, but would be glad to go over anything you suggest.

I hope you guys do great in Poland!

Scott

[Howard Rush]

Scott,  thank you for your patience.  I'd like to discuss this in a month or so.

[Tim Wescott]

By comparing the judge's maneuver scores, one is trying to actually discern which judges are doing the best job of scoring.  I think that developing this method would benefit our judging core by helping show who is on the mark and who isn't.

Hi Scott:

I don't fly at the nats (yet), but I think I'd rather fly in a contest where the judges are allowed to be persnickety about different things, rather than one where they're all expected to judge exactly alike.  Judging is, of necessity, subjective -- asking the judges to be totally regimented in individual maneuver scores is going to suppress that.

And I must say, I'm quite happy about the fact that we don't pay the judges.  When you start paying people based on a performance metric, you usually end up with two things: an overall rise in that metric, and a deep appreciation about how raising that metric in unexpected ways can really mess up your overall productivity.

http://www.dilbert.com/strips/comic/1995-11-13/

[Trostle]

I don't fly at the nats (yet), but I think I'd rather fly in a contest where the judges are allowed to be persnickety about different things, rather than one where they're all expected to judge exactly alike.  Judging is, of necessity, subjective -- asking the judges to be totally regimented in individual maneuver scores is going to suppress that.

http://www.dilbert.com/strips/comic/1995-11-13/

Tim,

In all due respect to your analytical ability, I think you have missed the message here when you think that the Nats judges are "expected to judge exactly alike" or that the judges are to be "totally regimented in individual maneuver scores".

Yes, judges are expected to recognize errors, but they are also expected to assign their own point value to the maneuver based how well they saw the maneuver flown,.  Yes, it is recognized that the event us subjectively judged and yes, it is expected that each judge will have his/her own assessment of each maneuver.  If every judged scored each maneuver exactly the same and "correct" we would only need one judge.  When we see 3 or 5 or 6 judges on one circle, it is recognition that judging is far from a perfect process. 

Keith

[andrew stokey]

Interesting thread. As one of the judges this year I can say this. The training wasn't what you'd call "ground up", it was really more of a discussion session to point out safety and procedural issues with some discussion about where maneuvers begin and end, and other nuances. All the judges were experienced. The most interesting point was that in the consistency rating, there was a very small variation, the least consistent being a 2.25 which to my understanding is a very narrow range. This, also to my understanding would have qualified any of this years judges as top 20 last year which had a range nearing 5. The Head Judge is not necessarily the most qualified, he is an organizer, not a boss. Mark did an exceptional job in my opinion and himself admitted he is not an "expert", whatever that might be.

Personally it was a very rewarding however exhausting experience. the daily warm-up flights put everyone on the same page first thing. For me, it was a chance to compare my style and observations with judges other then our local guys. I was pleased to find that I fit in very closely with the group, seeing the same things an scoring very consistently.

Mr Ryan,
Mark IS an organizer, and not a boss, as you stated.  BUT, if he is not an expert judge, then NOBODY is.  He has been doing this for about 35 years.  Some may disagree with his scores, but thats human nature.  The "expert" reference may be the fact that he was talking about flying or something else.

[FLOYD CARTER]

I'm not responding to anyone in particular, nor am I advocating any of the proposals I've read here.

My observation with non-professional judges is , their first couple, or few flights in a day are judged differently from the remaining, after the judge set is "broken in".  If you are the first flight of the day, and the judges are cold, watch out!

Floyd