Question on top 20 judges

[Derek Barry]

I would expect the experienced judge to be more closely correlated with either newbie than the newbies would be with each other. 

Possibly, but not certainly.

Derek

[Brett Buck]

Nor should there be.  The high or low judge may be the best judge. 

  Agreed. Shareen and I ran down one of our walkie-talkie batteries arguing about this between Cheyenne, WY and North Platte, NE one year, and apparently it was a lively topic of discussion inside their van every year as well.  They selected for a total score range of around 535 for the best pilots. The guys averaging 570 were frequently excluded regardless of how well it correlated to the other rankings. I think this has the effect of selecting for narrow scoring range as well as a total score target, which is why we routinely had a few points total difference from 1-5th place.

   Brett

[Derek Barry]

  Agreed. Shareen and I ran down one of our walkie-talkie batteries arguing about this between Cheyenne, WY and North Platte, NE one year, and apparently it was a lively topic of discussion inside their van every year as well.  They selected for a total score range of around 535 for the best pilots. The guys averaging 570 were frequently excluded regardless of how well it correlated to the other rankings. I think this has the effect of selecting for narrow scoring range as well as a total score target, which is why we routinely had a few points total difference from 1-5th place.

   Brett

Which is not necessarily a bad thing. Top guys usually fly within a few points of each other. It really takes a stand out pattern to score 10 points higher than the competition in top 5 situations.

Derek

[peabody]

Dave Cook has some great thoughts on judging, and judging the computers of today....

As I recall, he advocated tossing the high and low judges' score per maneuver when more than three were used...

He has some cool ideas, although a bit contrary....

[RandySmith]

Dave Cook has some great thoughts on judging, and judging the computers of today....

As I recall, he advocated tossing the high and low judges' score per maneuver when more than three were used...

He has some cool ideas, although a bit contrary....

I do not think you will ever see more than 3 per circle until Saturday used, and dumping the high and low judge, or some of their scores is the wrong thing to do, many times you are dumping the best , most consistent judge, and keeping the one who is bracketing.

What does judging the computers mean? and what does that have do to with judges scoring a stunt contest? computers just do whatever they are told to do, and calculate really fast.

Randy

[Brett Buck]

Dave Cook has some great thoughts on judging, and judging the computers of today....

As I recall, he advocated tossing the high and low judges' score per maneuver when more than three were used...

He has some cool ideas, although a bit contrary....

  Tossing out high/low is never a good idea, it operates on the presumption that there is a correct score. It will automatically toss out judges that run "high" or "low" normally, whether they are doing a good job or not.  Buffalano's method (which involves normalizing the scores first, and is very similar to the methods I used to evaluate some previous results) solves this issue but has the fatal flaw of having to wait until all the scores are available. This method is more-or-less doing the same tracking method used to select judges like we have already done, but looks at an entire round and then excludes them for that round itself, at which point the others are used.

   Again, this might work for Top 20 and Top 5, but there aren't going to ever be enough judges per circle to do it for qualifying. And, when I have taken individual sets of results and applied various methods, it never once changed the winner, and it only occasionally changed any of the placements by one or two.

   If you process the scores with 3-4-5 different methods, and they all yield the same results, it tells you that all the fiddling or tweaking with format and processing is more-or-less irrelevant, AND, that you should probably choose the simplest version.

   Brett

[RC Storick]

Here is a question. How can you accurately track a judge until the contest is over? Or is it done on a whim and what is being tracked?

[Brett Buck]

I would expect the experienced judge to be more closely correlated with either newbie than the newbies would be with each other. 

  On average, that is probably true. It does raise the likelihood of occasional anomalies in the selection process. It seems unlikely to have any significant effect on the results.

    Brett

[Howard Rush]

Here is a question. How can you accurately track a judge until the contest is over? Or is it done on a whim and what is being tracked?

We track them during qualification rounds.  That ranking information can be used to assign judges on Friday.  Saturday judge assignments are (or can be) based on judge rankings through Friday.  

[Brett Buck]

Here is a question. How can you accurately track a judge until the contest is over? Or is it done on a whim and what is being tracked?

    You have the same sort of information at any point in the contest, including when it is over. So Top 20 selections are made based on the tracking from Qualifying, and Top 5 from the Top 20 day tracking.

    It is certainly not being done on a whim, it is entirely algorithmic and you can have the algorithm to examine for yourself. What is being tracked is as described, it is a comparison of each individual's flight score VS the others in the judging panel.

    Brett
      

[RC Storick]

So what is being tracked? If a judge gives a bad score to a top guy. If a Judge is too far from the rest? If a judge is not in a certain range?

So if 2 judges give a 32-35 and the low man gives a 22 ? Who's right? Is it not suppose to be subjective?

[Brett Buck]

So what is being tracked? If a judge gives a bad score to a top guy. If a Judge is too far from the rest? If a judge is not in a certain range?

   I don't know how to explain it any better, it compares each individual total flight score to the average of the others for the same flight, without regard to who it is, or the raw absolute value. Those with the best correlation to the others are ranked higher, those who deviate ranked lower. Basically it is who is furthest from the rest (after normalizing over the day).

     

    The algorithm is yours for the asking, in fact, Howard is anxious to provide the source code so you can see how it works in any detail necessary.

     Brett

     

[Brett Buck]

So what is being tracked? If a judge gives a bad score to a top guy. If a Judge is too far from the rest? If a judge is not in a certain range?

So if 2 judges give a 32-35 and the low man gives a 22 ? Who's right? Is it not suppose to be subjective?

  Its not done on a maneuver-to-maneuver basis, it's on the total flight score.

    Brett

[RC Storick]

It would not make much sense to me so no need to send it. I just see it as its suppose to be subjective until the math kicks in then its not.

So if one judge is 40 points off they do not make the cut? Who was right? The low or the high?

My new program awaits at the post office right now.

[Brett Buck]

I just see it as its suppose to be subjective until the math kicks in then its not.

   The current judge tracking algorithm is not subjective in any way. The scores themeselves are subjective but once the scores are entered, there is no subjectivity at all. That's the entire point of having an algorithm, as opposed to the very similar eyeball method used by Warren and Shareen.

     Brett

[Howard Rush]

It doesn't go by score.  It looks at the order in which each judge ranks contestants: 1st, 2nd, 3rd, and so forth.  Each time a judge's ranking of a guy is a notch off the average ranking (the one that appears on the scoreboard), he gets dinged a point.  There's some other stuff involved, but that's the main idea.  I've posted the formula here or on SSW.  It's done flight-by-flight, but it could also be done maneuver-by-maneuver.  I'll change if somebody shows me that maneuver-by-maneuver is better.  

[RC Storick]

I was just trying to understand how a subjective event can be done by a computer program that by its nature is not subjective. Just as always said the best will always win. I can see the extreme complaining if done by luck of the draw and the guys who are all the best wound up on one circle and only 5 got to move on.

When a Judge looks at a Vincent Van Gogh or a Michael Angelo and compair it to a Renoir or a Warhol  and the same set of judges look at those artists a year later not one of the judges will change his mind. While they are all great in their own right. Very similar to what we have today. Same judges ,judging the same fliers on a different day.

By they way you would not my critique on these painters. But as in any subjective event you will get different opinions.

[Trostle]

Dave Cook has some great thoughts on judging, and judging the computers of today....

As I recall, he advocated tossing the high and low judges' score per maneuver when more than three were used...

He has some cool ideas, although a bit contrary....

I am going to add to Brett's response to this idea of throwing out high and low scores.  As Brett said, it is not a good idea.  And it is not a good idea for a number of reasons.  You may be throwing out the scores of the judges who are more accurately assessing the relative merits of one flight over another.  Further more, it has  been my experience that some judges become concerned that their scores may be consistently thrown out for being high or low will tend to narrow their range of scoring when indeed the individual maneuvers being flown deserve a wider range of scoring.

In the analyses by Dr. Buffalano for the CIAM a number of years ago, he was asked to run the scores from the judges from the results of several World Championships, in one case averaging all of the judges' scores for each flight and in the second case discarding the high and low scores for each flight.  This was like more than 15 years ago in the day when there were 5 judges used in the finals (15 pilots, best 2 of 3 flights used to determine finals placement).  The results were basically the same with sometimes a change in who was awarded say a 7th or 8th place.  This was also for a period where there was not the blatant and obvious bias that has been experienced during certain years.

Keith

[Brett Buck]

I was just trying to understand how a subjective event can be done by a computer program that by its nature is not subjective. Just as always said the best will always win. I can see the extreme complaining if done by luck of the draw and the guys who are all the best wound up on one circle and only 5 got to move on.

   Now you are talking about the seeding. That, too, is entirely algorithmic, and not subjective. It uses selected contest results (NATS and TT) to come up with a seeding rank for each pilot who has a result in the contests. It is far from perfect as indicated by my ranking and Billy's ranking, since we were missing a fair number of these contests, and at least Billy hasn't forgotten how to fly.

    You are absolutely right about the random pilot group selection. The big problem would not be the top pilots failing to qualify, it would be with the guys on the edge getting locked out on one circle, and a free ride on a different circle. I thought it worked a lot better when we had two Open and two Advanced circles and took 10 from each, because the quantization effect is reduced.  But in any case the current system has far fewer complaints than anything before it.

     Brett

[john e. holliday]

See my post in debate section.

[RC Storick]

I am sure my next statement will not set well with the status quo but here goes. Take a look at how many NATS winner we have had sense moving to Muncie. Far less than when the NAVY NATS was in play. I have to ask is it normalization? Is it because it the same people judging the same fliers on different day? If this were not the case the worlds would be won by those same guys every year. But they are not, the reason is not because one set is less consistant than the other. Its because of different judges looking for a different set of mistakes.

There is always opposition to a new way of getting the judging pool to Judge because they don't know what they are looking for. I say good!
Or they don't know the pattern. I guess we were all born knowing the pattern. So among us we choose the same set of judges (most of which are our friends)to judge year after year and expect them to pick a winner. They do a good job as is but is it really? I made a statement that everyone see art differently just as everyone person sees the concours plane differently. That being said a pattern to one will not look the same to another just as it looks differently from the inside. I think it was Doug in this thread saying something about the NAVY way of judging being influenced by the paint job. OK ,Awhile back I said let the computer judge the flight and that was me with opposition as it would take the presentation out of the equation (white pants etc.) 25 years ago I might have persued this system but its too late.

The top guys should embrace outside judging to see who really is the best. But as is not much will change.

[Trostle]

I am sure my next statement will not set well with the status quo but here goes. Take a look at how many NATS winner we have had sense moving to Muncie. Far less than when the NAVY NATS was in play. I have to ask is it normalization? Is it because it the same people judging the same fliers on different day? If this were not the case the worlds would be won by those same guys every year. But they are not, the reason is not because one set is less consistant than the other. Its because of different judges looking for a different set of mistakes.

There is always opposition to a new way of getting the judging pool to Judge because they don't know what they are looking for. I say good!
Or they don't know the pattern. I guess we were all born knowing the pattern. So among us we choose the same set of judges (most of which are our friends)to judge year after year and expect them to pick a winner. They do a good job as is but is it really? I made a statement that everyone see art differently just as everyone person sees the concours plane differently. That being said a pattern to one will not look the same to another just as it looks differently from the inside. I think it was Doug in this thread saying something about the NAVY way of judging being influenced by the paint job. OK ,Awhile back I said let the computer judge the flight and that was me with opposition as it would take the presentation out of the equation (white pants etc.) 25 years ago I might have persued this system but its too late.

The top guys should embrace outside judging to see who really is the best. But as is not much will change.

Robert,

I guess I know what you mean by embracing "outside judging".  But before you criticize the judging corps at the Nats for the past 20 or 30 years, maybe you should first take the time and bother to identify all of the names who have judged the Open event at the Nats since the Nats stopped using Navy judges which was some time before moving to Muncie.  I think you will be astounded by the numbers involved.  Yet, over those years with all sorts of judging combinations from a relatively large pool, a fairly limited number of pilots have consistently found their way to the top scoring positions.   In your own way, you have condemned the efforts of a great number of honest and hard working volunteers.

Even during the Navy Nats when different judges were used each year, there was still a group of people who consistently moved to the top.  (Thinking in terms of Werwage, Gieseke, McFarland, Gialdini, Mathis.)

What is so hard to comprehend that some people just fly better than others and will consistently win, or place well, or QUALIFY at the Nats regardless of who is judging?

We have a happy situation now where there are some new faces appearing in the top 5 and top 20.  Can it really be that these new faces are flying well and that the old stodgy, set in their ways judging corps is really recognizing new talent?

Keith

[RC Storick]

Keith you won't be the only one to chastise me for my opinion. But switching one for the other year after year does not show much change to me. I in no way wish to discredit or take away any of the work done in the past. Its the future I am looking at.

[Randy Ryan]

I would expect the experienced judge to be more closely correlated with either newbie than the newbies would be with each other. 

Howard, could you go back to my post and comment please? To my understanding, the consistency of he judges was very close this year with the highest inconsistency 2.25 or 2.5 across the 12 judges. I keep reading about odd man out with some wildly varying scores but I don't think we had that at all. I apologize I have not read your description and don't remember ever seeing it, but perhaps you could layout some examples of the point spread of a judging panel and what is good or better, bad or worse.

[Mike Ferguson]

I would offer two other possible contributing reasons for the number of Nats champs lessening since the days of the "Navy Nats", besides just pointing at judging.

1) The switch to the Top 5 format. Instead of the finals being Top 20 day, with the best single flight winning, it became best 2 out of 3. That rewards consistency, rather than being able to "drill in" one single great pattern.

2) The switch to Muncie meant that the site conditions are more-or-less known. (Competitors who've flown there year after year know how the wind acts when it's blowing in certain directions, etc.) When the Nats moved from location to location, everyone was in the same boat - time to trim the plane to the conditions of the new location, and figure things out quickly. At Muncie, for a lot of the top guys, "trimming" is more making fine-tuned changes to a plane rather than significant changes.

And there's probably a lot of other factors as well. (Engines are better, flyers are better at trimming planes now than they were years ago ...) All I'm saying is that I don't think anyone can point at a single reason for "why less people win the Nats" now.

[Howard Rush]

Howard, could you go back to my post and comment please? To my understanding, the consistency of he judges was very close this year with the highest inconsistency 2.25 or 2.5 across the 12 judges. I keep reading about odd man out with some wildly varying scores but I don't think we had that at all. I apologize I have not read your description and don't remember ever seeing it, but perhaps you could layout some examples of the point spread of a judging panel and what is good or better, bad or worse.

My comment that you're quoting is about a hypothetical situation in which an experienced and presumably accurate judge gets on a circle with two rookie and presumably inaccurate judges.  Would that unfairly affect the experienced guy's ranking?  Unless the rookies make the same errors as each other consistently, I wouldn't think so, but I think that's amenable to analysis.  I don't have time now, but I'll do something after August.

I don't have the program with all the data in it from the 2014 Nats.  I have some from previous Nats, but don't have time to look at them now.  I would take the rankings from the last two Nats with a grain of salt.  I fear that adding "Expert" messed up the judge rating.  The jist of it is that a judge will have more contestant ratings that deviate from the official, averaged ratings the more contestants he judges: resolution gets finer with number of contestants.  We're now comparing judges from circles with greater differences in number of contestants than before.  Had Bob not fiddled with the program to put "Expert" on two circles instead of one, this might have distorted judge rankings even more.  So we need to find a way to normalize the judge scores better.  That's what I've been asking for help with.

To be fair, even if we recombine Expert and Advanced at the Nats, dwindling attendance may put them on fewer than four circles, and the normalization problem will still need to be addressed.  

[RC Storick]

My comment that you're quoting is about a hypothetical situation in which an experienced and presumably accurate judge gets on a circle with two rookie and presumably inaccurate judges.  Would that unfairly affect the experienced guy's ranking?  Unless the rookies make the same errors as each other consistently, I wouldn't think so, but I think that's amenable to analysis.  I don't have time now, but I'll do something after August.

I don't have the program with all the data in it from the 2014 Nats.  I have some from previous Nats, but don't have time to look at them now.  I would take the rankings from the last two Nats with a grain of salt.  I fear that adding "Expert" messed up the judge rating.  The jist of it is that a judge will have more contestant ratings that deviate from the official, averaged ratings the more contestants he judges: resolution gets finer with number of contestants.  We're now comparing judges from circles with greater differences in number of contestants than before.  Had Bob not fiddled with the program to put "Expert" on two circles instead of one, this might have distorted judge rankings even more.  So we need to find a way to normalize the judge scores better.  That's what I've been asking for help with.

To be fair, even if we recombine Expert and Advanced at the Nats, dwindling attendance may put them on fewer than four circles, and the normalization problem will still need to be addressed.  

Howard it takes no more work for EXPERT other than adding a E or a A on the score board when writing the score down.

[Robby Hunt]

And there's probably a lot of other factors as well. (Engines are better, flyers are better at trimming planes now than they were years ago ...) All I'm saying is that I don't think anyone can point at a single reason for "why less people win the Nats" now.

I'd venture to guess those few that have won also reflect a disproportionate ratio of handle time to keyboard time.

[Derek Barry]

Howard it takes no more work for EXPERT other than adding a E or a A on the score board when writing the score down.

I have to disagree, it is quite a bit more work than just writing a letter down. What if you end up with 2 experts on circle A, 2 on circle B, 0 on circle C, and 6 on circle D. You cannot take an equal number from each circle, same with Advance. The seeding has to be manually manipulated to get the correct number of guys on each circle. The only way to do this is by using math to figure out the seeding number for all of the Advance and Expert pilots. That is what has taken so long the past two years.

It has been suggested that we just use a different system to score Expert but this would add quite a bit of work for the ED. One of the really nice things about Howard's program is that it prints all the score sheets in order for each judge. The hours of printing, labeling, and stacking score sheets are eliminated with the program. If you try to run two separate programs you would have a lot of excess work trying to mix the Expert in with the pre-organized Open and Advanced pilots. Also the flight order would be all screwed up because after the ping pong draw the program takes care of the rest.

Derek

[Bill Morell]

I'd venture to guess those few that have won also reflect a disproportionate ratio of handle time to keyboard time.

The best answer so far.

[Derek Barry]

All I'm saying is that I don't think anyone can point at a single reason for "why less people win the Nats" now.

I can! Dave and Paul are very hard to beat.

Derek

[Howard Rush]

Howard it takes no more work for EXPERT other than adding a E or a A on the score board when writing the score down.

Well, you figure out the judge scoring normalization problem for starters.  I figure that integrating Expert into the program will take me 200-some hours.  Although it would be fun, I didn't do it this year because I was busy getting ready for a stunt contest.  Fortunately, Steve Yampolsky figured out how to accommodate all three events regardless of entry numbers, maintaining the same Nats format, ensuring a meaningful qualification round (not taking 20 out of 23), and balancing the circles.  Go back over the discussion two years ago and you can see what's involved.  

[Brett Buck]

Instead of the finals being Top 20 day, with the best single flight winning, it became best 2 out of 3. That rewards consistency, rather than being able to "drill in" one single great pattern.

  Can we retroactively switch to this format for 2014?  I like that idea for some reason...

   Brett

[Randy Ryan]

My comment that you're quoting is about a hypothetical situation in which an experienced and presumably accurate judge gets on a circle with two rookie and presumably inaccurate judges.  Would that unfairly affect the experienced guy's ranking?  Unless the rookies make the same errors as each other consistently, I wouldn't think so, but I think that's amenable to analysis.  I don't have time now, but I'll do something after August.

I don't have the program with all the data in it from the 2014 Nats.  I have some from previous Nats, but don't have time to look at them now.  I would take the rankings from the last two Nats with a grain of salt.  I fear that adding "Expert" messed up the judge rating.  The jist of it is that a judge will have more contestant ratings that deviate from the official, averaged ratings the more contestants he judges: resolution gets finer with number of contestants.  We're now comparing judges from circles with greater differences in number of contestants than before.  Had Bob not fiddled with the program to put "Expert" on two circles instead of one, this might have distorted judge rankings even more.  So we need to find a way to normalize the judge scores better.  That's what I've been asking for help with.

To be fair, even if we recombine Expert and Advanced at the Nats, dwindling attendance may put them on fewer than four circles, and the normalization problem will still need to be addressed.  

No Howard, my comment is not hypothetical, I was one of this years judges and I'm questioning the consistency spread.

[Doug Moon]

I can! Dave and Paul are very hard to beat.

Derek

You got that right! 

It's the same guys winning because stunt is odd in a way that you can dominate for 50 years. Billy is a great example. I think he has a Win in 5 different decades.  What other physical activity can you say that about?  And when you do find one, which I am sure there are some, you will probably see the same winners in that arena from year to year as well.  Being in Muncie every year has tailored who will show up and who wont.  The traveling nats would bring different people from different areas, but the hardcore psycho top guns would travel anyway and usually be in the equation for the win at the end anyway.  In 1994 when I attended Lubbock TX nats for my first time as a beginner Paul W and Bob H were in the flyoff.  Last year when I made the flyoff Bob H and Paul W were there as well.  A good pattern holds up over time. For those who think it should change more often get out your SN over the last 20+ years and look at the entries the same names keep on bringing it year after year.  I only hope my name is one they comment about in 20 years as always in the hunt and howcome....  3 2nd places will have to start translating to more wins sooner or later........I hope....A note to Dave and Paul, I think the nats will be in Kansas next year......

[RC Storick]

I am not saying who won was not the best pilots or the judging is unfair. I just cant for the life of me figure out why any mention of any of the subjects here cackeles feathers by the status quo. To all who have won in the past "YOUR THE GREATEST!" "Your the best at something that pays nothing." Thats a quote from a worlds champs wife.

[Howard Rush]

No Howard, my comment is not hypothetical, I was one of this years judges and I'm questioning the consistency spread.

I understand.  You quoted something I said, then went on to ask about this year's consistency spread.  I responded to both even though I suspected you were only interested in the latter.  I wasn't able to give you a satisfactory response because I don't have the data to see what happened at this year's Nats.  

[Doug Moon]

I am not saying who won was not the best pilots or the judging is unfair. I just cant for the life of me figure out why any mention of any of the subjects here cackeles feathers by the status quo. To all who have won in the past "YOUR THE GREATEST!" your the best at something that pays nothing. Thats a quote from a worlds champs wife.

Then what are you saying.  I dont think I follow.

Are you genuinley asking why do the same guys rise to the top and want to know their secrets?
Or
Are you asking that question because you think it should be otherwise?

PLEASE NOTE THIS IS FREINDLY DISCUSSION....Those are not pointed questions to try to stir it up so to speak.  Just discussion.

[RC Storick]

This thread started out as why do they put inexperienced judges on top 20 days and leave the judges with experience judging advanced. Anytime a question comes up the powers to be take great offence. I am not the only one who sees a problem with the status quo. I can not change anything on my own and in 10 years when there is only 20 guys to compete for top 20 it wont matter.

[Doug Moon]

This thread started out as why do they put inexperienced judges on top 20 days and leave the judges with experience judging advanced. Anytime a question comes up the powers to be take great offence. I am not the only one who sees a problem with the status quo. I can not change anything on my own and in 10 years when there is only 20 guys to compete for top 20 it wont matter.

Yeah I hear that too. And each year I wonder "Is this the year with the big drop in attendance?"  And each year is 36-38 guys battling it out....

[Derek Barry]

This thread started out as why do they put inexperienced judges on top 20 days and leave the judges with experience judging advanced. Anytime a question comes up the powers to be take great offence. I am not the only one who sees a problem with the status quo. I can not change anything on my own and in 10 years when there is only 20 guys to compete for top 20 it wont matter.

You are correct but other than a few comments in the very beginning of this thread it has been positive and informative. I do not see anyone getting their panties in a wad, just asking questions and debating ideas. That is what makes this forum what it is.

Derek

[Howard Rush]

This thread started out as why do they put inexperienced judges on top 20 days and leave the judges with experience judging advanced. Anytime a question comes up the powers to be take great offence. I am not the only one who sees a problem with the status quo. I can not change anything on my own and in 10 years when there is only 20 guys to compete for top 20 it wont matter.

The method used is completely public: at least as I wrote it.  I don't know whether Bob used it rigorously to pick the Top-20 judges.  Dig in and understand the method, then suggest an improvement, rather than just saying it's wrong.  I think it does have flaws, which I have described here.  

Lots of people have put in a lot of work to put on a Nats that's fair, objective, and picks the best as winners.  I think you owe it to them to understand the process before you criticize it or -- worse yet-- talk the PAMPA EC into making capricious changes to it.  If you do understand the process and propose and justify improvements, they will be welcomed, and I'll do my best to get them incorporated.  

[RC Storick]

I don't know what to say because all I get is BS from everyone so best not say anything about the sacred way things are done anymore just keep on what your doing.

[RC Storick]

Yeah I hear that too. And each year I wonder "Is this the year with the big drop in attendance?"  And each year is 36-38 guys battling it out....

So far your right with the same guys in the top 20 give or take 1 or 2 but its coming trust me.

[Scott Richlen]

Since some judges tend to consistently score high or low, I would think that the current method of normalizing the total score would create a bracketing effect.  It's unintended result would be to throw out total high score or total low score, despite those scores possibly being more accurate.  Is there a way to normalize the individual maneuver scores so you can compare how consistent the judges are, maneuver to maneuver (by normalizing the total score and then going back and on that basis normalizing each individual maneuver score)?  If most of the judges score a particular maneuver high (normalized score) and one judge scores significantly lower or higher  (normalized score) would that indicate that that particular judge was not as accurate as the others?  Suppose you weighted the normalized scores of the individual judges by a goodness factor (based on judging experience or by acknowledged expertise in judging) to give an indication of which normalized maneuver score should be considered "normative" or "basis."

Scott

[peabody]

My belief is that when, essentially, the same judges, judge essentially the same fliers, using the same system to "grade" the judges and fliers, the results will be predictable. I have no doubt that a lot of the same fliers would appear in the top 20 with an entirely new judging corps, but believe that all fliers all deserve to be viewed by fresh judges.

A flier that complains about the judging may be sour grapes.....if he/she goes off to the same contest and there are different judges and the results are the same, he/she should take up a stopwatch event.....but if the judges are not entirely different, then there is a shadow of doubt.

[Howard Rush]

Since some judges tend to consistently score high or low, I would think that the current method of normalizing the total score would create a bracketing effect.  It's unintended result would be to throw out total high score or total low score, despite those scores possibly being more accurate.  Is there a way to normalize the individual maneuver scores so you can compare how consistent the judges are, maneuver to maneuver (by normalizing the total score and then going back and on that basis normalizing each individual maneuver score)?  If most of the judges score a particular maneuver high (normalized score) and one judge scores significantly lower or higher  (normalized score) would that indicate that that particular judge was not as accurate as the others?  Suppose you weighted the normalized scores of the individual judges by a goodness factor (based on judging experience or by acknowledged expertise in judging) to give an indication of which normalized maneuver score should be considered "normative" or "basis."

I'm not sure what you are saying.  A contestant's score for a flight is the average of the flight scores of all the judges on that circle.  Are you suggesting that we not average the scores, but adjust them for judging like the full-scale aerobatics guys do?  It would take a rules change, and would resist that unless I could see (and understand) proof of its statistical validity. 

Judge assessment is something different.  It's based on the order in which each judge ranks the contestants compared to the official ranking of the contestants (the one on the scoreboard).  Here is a description of our judge assessment, written when my dementia had not progressed to today's state: http://www.clstunt.com/htdocs/dc/dcboard.php?az=show_topic&forum=103&topic_id=327442&mesg_id=327442&listing_type=search .  I should mention that Paul intoduced the exceedance term to penalize favoritism.  He set it to zero, where it has remained, either because favoritism was not a problem or because we couldn't agree on how to spell exceedance. 

As I mentioned in the SSW writeup, we could change the judge assessment to maneuver-by-maneuver.  I'd do it and recommend that it be adopted if somebody can show me it's better than flight-by-flight. 

One interesting property of judging is that the guy who uses a narrow scoring band has less influence on the contest's outcome than a guy who uses the full range.  For example, if he gives all maneuvers a 34, he doesn't affect the outcome at all.

[Howard Rush]

My belief is that when, essentially, the same judges, judge essentially the same fliers, using the same system to "grade" the judges and fliers, the results will be predictable. I have no doubt that a lot of the same fliers would appear in the top 20 with an entirely new judging corps, but believe that all fliers all deserve to be viewed by fresh judges.

A flier that complains about the judging may be sour grapes.....if he/she goes off to the same contest and there are different judges and the results are the same, he/she should take up a stopwatch event.....but if the judges are not entirely different, then there is a shadow of doubt.

Now Richard, you haven't looked at the judge roster lately.  I see a couple of new guys this year, a handful of guys who've only done it twice or so, and some guys who haven't done it for several years.  I only see three regulars.  And out of the hundreds of judges Bob had to pick from, I think he got a particularly competent group.  

[Derek Barry]

Now Richard, you haven't looked at the judge roster lately.  I see a couple of new guys this year, a handful of guys who've only done it twice or so, and some guys who haven't done it for several years.  I only see three regulars.  And out of the hundreds of judges Bob had to pick from, I think he got a particularly competent group.  

I agree, I could not have done a finer job myself...and I have tried.

Derek

[Brett Buck]

I am not saying who won was not the best pilots or the judging is unfair. I just cant for the life of me figure out why any mention of any of the subjects here cackeles feathers by the status quo.

    The quick answer is that you are trying to provoke this kind of response. The question is answered clearly and concisely but you don't bother to even read the response, or take the time to try to understand it. There have already been very extensive discussions about this for years now, you can have the exact source code yourself, put in your own numbers,  and see exactly what it is doing. You have had several qualitative descriptions.

    Of course, you don't actually care what the answer is, you want to grind your own axe. And bear in mind, I know what that is in detail since we talked about it for a while.  At one moment, you disingenuously ask about how the judge selection works. But you immediately respond with "why don't we bring in outside judges". Then later, you appear to complain that the judge selection algorithm fails to pick the most experienced judges - which is the diametric opposite of the "outside judges" theory, which ensures the least experienced judges we could possibly find.

    Then, you ask whether or not "normalization" is a factor. You have absolutely no idea what "normalization" means in this context, of course, and Howard stated earlier that it was not used. The ranking method amounts to the same thing as normalizing the scores, then choosing by the resulting normalized score, but it's not used directly.

   In short, you aren't listening to the answers. That's why people get upset, you are just tossing around crap about the work people have spent many careful hours/years trying to raft, and you have no idea or no interest in how it actually works. You just know it's garbage and whatever pops into your head is going to "fix" the issue.

  As I told you at the NATs,  there are plenty of us willing to help you - but not to get abused and denigrated for trying. As I mentioned at the NATs, the scoreboard is telling you something *very important*, and you aren't willing to listen. To get out of your current rut, you are going to have to do MANY things differently than you do now, and the first and most important is to accept that you are going to have to pay attention to other people's inputs to succeed. David does, and you don't - see the difference there?

    Brett