Impartial Judging

[john e. holliday]

Before I forget I thought I would start this.  Years ago we used to have discussions about stunt judging at our club meetings.  Racing was no problem as it was mainly the watch that told who won.  Combat was making the right judgment call at the right time.  But stunt was different.

The thing we thought about trying was having judges that were told they could not look at the pilot.  When a contestant was ready to move to the circle the judges would have their backs to it.  The score sheets would have a contestant number, no name.   When the contestant was ready to start he would notify the timer.  Once the contestant was at his handle the judges would be notified to start watching the airplane.  The contestant would the get his go signal for launch.

We even thought of robes for the contestant to put on just in case the judges did look at him.  The best scenario was like the Navy days.  About a dozen or half dozen people would volunteer to judge.  There are guys/gals willing to put up practice flights.  During the flights the coach of the judging team would be telling them what they are looking at.  After the flight then every one would talk about what was wrong and what the score might be.

In all actuallity we never got around to doing it.  Man power is the big hindrance.  In a way there is always the one or two judges that will let the halo affect get to them.  I have seen it done in which the perfect flyer blew a maneuver or got blowed out of one.  Then you look at their score sheet and see numbers alls the way down within a few points of each other. 

The way the NATS are run I think it would be hard for one maneuver to knock a person out of the running.  There is the two days of qualifying to try and make the top twenty.  I think they use the top score to decide who is moving on.  Then top twenty day in which you have two more officials.  I think one score gets you to the top five.  Hope Paul will straighten me out on this as I don't fly Expert or Open at the NATS.

I think someone pointed out that at VSC all the rounds are added together.  With four circles at the NATS it could work if every contestant got to fly in front of each set of judges.  No throw aways which means each and every flight you will be doing your best.  That means four perfect flights to make top twenty.  Two flights to make top five.  Then the important two best to make the Walker fly off and NATS Champion. 

Anyway this is my rant for the day.   DOC Holliday

[Bob Reeves]

I'm not sure the "halo" is really an issue at least I haven't seen it to a great extent.

I am convinced bracketing is a much bigger issue. When you pick up your score sheets and find a 6 point difference between lowest and highest knowing full well that you completely blew a couple manovers and did almost perfect on others something is wrong. Makes you think the judges weren't even watching, just writing down numbers. When I feel I flew a 12 on the triangle and a 38 on landing and the triangle gets a 28 and the landing gets a 29 you know the judge wasn't watching. Maybe the flight in the next circle but he couldn't have been watching my flight.


I believe if we can figure out how to fix the bracketing we can make giant steps toward scores that actually represent the flight.

[Jim Thomerson]

I agree with Bob on "bracketing".  When I get a scoresheet with all the maneuvers scored about the same, I don't feel the flight was well judged.  As a result of having this happen on several occasions, I have a personal rule that I will not write down the same score three maneuvers in a row. 

[Paul Smith]

This was the 13th Annual Brodak Fly In.  Pretty much, the same guys can count on winning.

But this year, a group of younger flyers from Toronto, who nobody had seen or heard of before, came to The B and won a goodly stack of stunt plaques, including at least two firsts,

You can't get more impartial than that.

[Shultzie]

Hummm?
Perhaps in truth...ANY SPORT THAT IS A JUDGED EVENT...whether it be an ice skating, dancing with the Stars, spring board diving, chatty-Kathy Dolly's, tittleeeewinkees' art... or  our beloved little toy airplane contests etc, etc.... HUMAN FACTORED JUDGED EVENTS will never render total satisfaction.
 Whether it is a local contest or a world cup event...bottom line still remains that ALL CONTESTS that utilize HUMAN BEINGS as judges are going to have issues  

[Bob Reeves]

Or go to a contest where a judge is selected in the middle of the pilots meeting because someone forgot and then see 103 points difference between the two judges.  ?

I don't see a problem with this as long as they are consistant. I judged at Brodaks two years ago, the other judge and I were almost exactly 100 points apart but it was on everybody that flew and both rounds. I get weird when my flight has a 60 point spread between judges but my buddy only has a 3 point spread. Something doesn't add up....

[phil c]

The thing I look for is how well the judges scores track from maneuver to maneuver.  If both judges are following the book then their scores should track up and down from maneuver to maneuver.  There is no way, following the rules, that one judge can justify scoring you up 3 pts when the other judge goes down 2.  When that happens you know they simply aren't scoring by the book.  I would have no problem if the judges looked at each other's scores after every flight and even discussed what they were doing if it helped them do a better job.

[Bob Reeves]

<snip> I would have no problem if the judges looked at each other's scores after every flight and even discussed what they were doing if it helped them do a better job.

Yep, never understood why it was concidered such a sin for the judges to talk.

[dale gleason]

Those dadgum !%&***$#@%^% judges again! Who needs 'em, anyway? I think they're trying to take over the event or something. Sittin' out there in the sun, taking it easy all the time. One time I got more points for a bad flight than I did for a good flight! Can you believe that? What were those guys thinkin'?  And after all we do for them yet. It's so bad now that they are giving halo points to everybody, except me, and I even dress like Keith and fly with solids!  Go figure.       dg

Man, it was hot at the DASH contest, I think the heat may have gotten to me a little bit......just a tad....maybe

[Howard Rush]

"The way the NATS are run I think it would be hard for one maneuver to knock a person out of the running.  "

Doc, you express your opinion on how stunt at the Nats should be run without looking at the procedure to see how it is run now.  One maneuver put PT Granderson in 20th on top-20 day, despite his 20 appearance points. 

I'm not sure of the reason for the Nats qualifying rounds format, but I think it was to compare contestants over a relatively short time to avoid judge score drift and weather changes.  I presume that people had bad experiences with the world championships format and think the current Nats process is better. 

There is a Nats process for rating judges to select finals judges.  It was posted here and on SSW.  I don't know if it makes sense statistically, but it's objective, and it's what we have.  If you want to help, you can do some statistical analysis to improve that process. 

[Matt Colan]

Yep, never understood why it was concidered such a sin for the judges to talk.

I was talking to banjock and Palko at Brodak one night, it might've been after appearance judging, anyway they ended up judging profile together, and they said, somebody did one maneuver, and Banjock looks over at Mike and Mike goes, "Don't look at me, I don't know what that was."  Then they said they came up with their judging system, CBJ (Cannot Be Judged), TS (You all know what that means).  Me, I don't have a problem with judges talking

[Jim Thomerson]

The broad problem with judges talking is that it takes their attention off watching the flight.  The narrower problem is that they might influence eachother's scoring. 

[Steve Helmick]

Judges need to communicate to some extent. "Was that a pair of triangles?" "Did I see four outside loops?" Etc. The judges need to agree on stuff like that, so I think some limited communication is justified.

What I don't like (as a Judge) is to hear the other judge say "Wow, that was almost perfect!". Still, it's difficult to not utter a grunt, "yikes!!!!" or "oooh" sometimes. I don't like to hear applause at the end of a flight, either, if I'm judging. If I'm flying, that's another thing altogether...but it hasn't happened yet.  Steve

[Leo Mehl]

Judges need to communicate to some extent. "Was that a pair of triangles?" "Did I see four outside loops?" Etc. The judges need to agree on stuff like that, so I think some limited communication is justified.

What I don't like (as a Judge) is to hear the other judge say "Wow, that was almost perfect!". Still, it's difficult to not utter a grunt, "yikes!!!!" or "oooh" sometimes. I don't like to hear applause at the end of a flight, either, if I'm judging. If I'm flying, that's another thing altogether...but it hasn't happened yet.  Steve

[Ted Fancher]

Yep, never understood why it was concidered such a sin for the judges to talk.

I think the underlying reason for judges saving their comparisons until after the contest is over is simple.  Two judges that compare scores after flyer "A" finishes his pattern aren't the same two judges that then judge flyer "B". 

The place for judges to compare opinions is during judging clinics and judges' warm-up flights.  When they go to work at the official circle all of their wisdom (as well as their personal prejudices) must remain intact and beyond outside influence until all of the official flights are complete.

The exception as noted above is the need to discuss and agree as to administrative pattern faux pas committed by a flyer ... i.e. too many or too few loops, not enough level laps, etc.

Ted

[Dennis Moritz]

The first thing one club member, a wise soul, says to anyone who has a beef with a judge is "judge." Sitting in the 80 plus heat for hours is humbling and real. Sure, when you're fresh the flights and maneuvers stand out, by hour three, it's much more difficult. Who of us have read the most recent rule book complete with changes, codicils, and flex point allocations. Bellow the Expert level, I expect some wandering and flubbing and slap dash moments. I actually looked over my PAMPA scores from Brodak (in Intermediate) and was surprised to find the overall scores similar. Even tho one judge had a greater tendency to Track. I believe weather conditions are not technically to be considered. But they are, aren't they. I know I was surprised to get the score I did for a Clover that was yanked and misshapen in order to save the plane, when the wind suddenly howled.

[phil c]

something like "wow, that second loop was huge" communicates a faux pas by the contestant too. P'rhaps if the judges communicated more they'd develop better agreement as to what they were seeing and how they were scoring it and prevent weird scoring scenarios such as going in diametrically different directions on two consecutive maneuvers.

[Ted Fancher]

something like "wow, that second loop was huge" communicates a faux pas by the contestant too. P'rhaps if the judges communicated more they'd develop better agreement as to what they were seeing and how they were scoring it and prevent weird scoring scenarios such as going in diametrically different directions on two consecutive maneuvers.

So the other judge says "yeah, but it was the only one of the three that was round".  And then the first judge says "well, if it's too big the shape doesn't matter".  And then the second judge says "B.S., without the right shape nothing else matters!"  Next time they look up neither one remembers a darn thing about what they had decided the score for the loops should be and the flyer is well into his second triangle.

I've done a lot of judging over the last several decades and can, without any hesitation, state the most aggravating thing another judge can do is to interject his/her opinion about a maneuver out loud during a flight.  Any judge that's doing a halfway serious job of judging takes the better part of the first lap after a maneuver finalizing exactly what number to write down on the score sheet.  Gratuitous advice from the guy next door isn't the least bit helpful or welcome.

A final thought.  The worst thing that can happen to a judge is to suddenly have second thoughts thrust into his mind about the competency of his own evaluations by somebody else.

There is plenty of opportunity to discuss judging technique when the lines are rolled up and the judges get together to review the results of their hard work.  If one is surprised by the outcome they can engage the other regarding the apparent (to one judge) anomaly.

Ted

[Doug Moon]

The judges cannot talk during the flight or the round or the contest.  Except for the on off for the inverted flight.  When I stand behind the judges at the nats there is usually one who says, quietly to the others, "on" at lap 3 and "off" at the end of lap 4.  If they feel the contestant left somthing out or did to many of something BUT they cant agree they should keep scoring until it can be reconciled at the end of the flight.  If they both agree right away then they make their scores and move on.

THE REST IS OFF LIMITS!!  The simple reason is you cant have a judge change the way he evaluates a manuever from flight to flight.  That is not fair to the contestants to come or the ones before.  It creates alot of inconsistency and more people wondering what the heck is going on.

Judges with a large difference, say 60 or more, in points total is also not such a good thing.  The old "If they do it for everybody it is ok" doesnt really work for me.  One pilot showing the low judge something he likes just one time can bring that score way up while the others are still way behind.  It can really skew the contest toward that one pilots favor.  Is there a way around this?  I dont think so.  I have seen it at the nats several times just like Bob said my buddy's score is showing 7 points difference between the 3 judges and mine is showing 65 points and my brothers is showing 60 points.  Makes for big questions marks.  BUT that is what you signed up for when you paid your money.

[RC Storick]

Judges with a large difference, say 60 or more, in points total is also not such a good thing.  The old "If they do it for everybody it is ok" doesnt really work for me.  One pilot showing the low judge something he likes just one time can bring that score way up while the others are still way behind.  It can really skew the contest toward that one pilots favor.  Is there a way around this?  I dont think so.  I have seen it at the nats several times just like Bob said my buddy's score is showing 7 points difference between the 3 judges and mine is showing 65 points and my brothers is showing 60 points.  Makes for big questions marks.  BUT that is what you signed up for when you paid your money.

Agreed! The low scoring judge decides who wins.

[peabody]

I disagree....poor old dead Lee Lorio probably never had his scores count in the finals...he was real low...
Given the 5 judge, high low out, the scores that count tend to be closer, but the second highest (or second lowest) can play heavily.

I also disagree about judges talking....not comparing scores or what they thought of maneuvers.

I also feel that a judge varying a lot in fine, as long as the scores are in line with the group as a whole.... i.e. a "bad loop" from judge A might be a 20, while B feels it is a 30....

[Doug Moon]

I disagree....poor old dead Lee Lorio probably never had his scores count in the finals...he was real low...
Given the 5 judge, high low out, the scores that count tend to be closer, but the second highest (or second lowest) can play heavily.

I also disagree about judges talking....not comparing scores or what they thought of maneuvers.

I also feel that a judge varying a lot in fine, as long as the scores are in line with the group as a whole.... i.e. a "bad loop" from judge A might be a 20, while B feels it is a 30....

Rich,

At most local contests there are no scores thrown out.  The low judge going VERY low all day only has to move half of the range he is normally low to decide the winner.

There is plenty of time to talk but during a flight is not a place for it unless it involves the flight, NOT Maneuvers and evaluation techniques.  No exclamations either.  A simple "oh no" heard by another judge can cause the other judge to wonder what he saw and or affirm what he saw.  It really isnt a good thing.

Varying scores are ok as long as they track up and down maneuver to maneuver.  But should there really be a difference of 100 points, common by the way.  In some cases that means one judge is seeing the maneuver as a 36 and one is a 29.  Really?  36 is very close to perfect and considered a high score.  29 is middle of the road advanced.  How can they be that far apart?  I know the answer, but what do you think?

[peabody]

There sure are a lot of "BRACKET" scorers around...

[Chuck Feldman]

Why not average the judges scores for each maneuver? Then add up the averages for the entire flight. Seems like that would make most of all your comments mute.

[Michael Floerchinger]

I think for the most part Judges are pretty honest about their evaluations, but just like anywhere else we are all human and there will be mistakes. I think there are very few people out there when they judge events that actually bias their score for a particular individual. I judged stunt for many years and did not let friendships bias my scoring. But it is an interpretive process, different people like different styles. I pretty much went by the book, right size, entrance and exit locations, crisp corners, correct intersections. Plus there is another factor that is hard to explain here. Everyone has days that they are on and off. I have witnessed average flyers turn in excellent results and I have seen experts have a crappy day. Just because someone that typically does not turn in a great score and suddenly has an expert outing does not mean a bias by the judge. Sometimes a person is in a grove, I have had those days, there would be no way my practice flights resulted in an exceptional performance. For whatever reason the engine was right on, my reactions were right on, it would be like the plane was on a rail, it was flying better than it had in the past or in the future. Like when I shoot pool, I am a mediocre pool player but once in a while I “SEE” shots, I instinctively know where to hit the ball. I will have a night like that once in a great while and then go back to my old self for a few months.

For the most part the judges get it right, there are rare cases of mistakes but I think all in all our “volunteer” judges get it right.

Thanks,

[Dennis Adamisin]

I should know better than to jump in but...!

BTW I am a great coach (can see every mistake) but a lousy judge (can't decide how many points to assign!) Judges have a tough job and I believe try to do there best - regardless of grumblings from flyers.  Here's a few points to ponder:

* Impartial judging?  Like US Navy Officers?  They did not know anyone, and did not know about anyone's halo, of course everybody tried to "create" their own halo in the 8 minutes alloted to them.  Consider this: in a golden age of spectacular flyers there were NO carried-over halos, and may be the reason there were darned few consecutive champions - in the only one I can think of is Bob Gialdini in 1964-65.  Hmmm, makes a case that while we now have more experienced judges - we LOST impartiality.

* Think about it: the above is why a great pattern with a Classic airplane is scored lower than a mediocre pattern with a "modern" bird.  If the judge doesn't realize the difference in eras, then he does not (unconsciously?) downgrade the older.

* Shapes, sizes, intersections, bottoms.  WHICH IS MOST IMPORTANT?  WHICH IS EASIEST TO SEE?  Which mistake(s) cost the most?  Intended impartiality is lost in the ratings.  OPINION: ALL judges bottoms, maybe half see intersections, a few see shapes, precious few see tops.

* Judges talking?  Only  on procedural issues.  For the same reason we should discourage using recorders.

* Bracket judges - the hardest to deal with.  If you had 5 or more judges their might come up with a statistical way of evaluating their scores on a manuver to manuver basis; thus finding and weeding out bracket scores from the MIDDLE (are you kidding me?)

* BTW averaging the individual manuver scores is the SAME as averaging the totals (think about it)

* I am not entirely crazy about automatically throwing out the high & low scores either - I think our sample size is too small.  Besides it tends to protect the dreaded bracket judges who will trend to the middle.  Yet NOT throwing out scores gives any particular judge the power (consciously or accidently) to control the outcome - being higher one flight and lower the next.


In the end, even the big guys like the Olympics have trouble with judging - folks, if you are going to fly CLPA ya better get used to results that seem odd sometimes.

[Bradley Walker]

Agreed! The low scoring judge decides who wins.

Agreed.  I have been saying this for years.  Low judges can go up into the "normal" range without anyone even noticing.

[Bradley Walker]

So the other judge says "yeah, but it was the only one of the three that was round".  And then the first judge says "well, if it's too big the shape doesn't matter".  And then the second judge says "B.S., without the right shape nothing else matters!"  

Wow, you made my whole point for separating size, shape, and corner/bobbles into separate judging duties (see CLW article below).

Due to this fact, the TOC would allow a much more decisive scoring system than the current system.  Much of the current scoring system encourages average scoring to eliminate the possibility of ballooning of the scores.  Since we do not care about the score ballooning effect any more, we are free to use more extreme measures to be decisive.

In an ideal setting, (4) judges would be used for each match.  (3) of the judges would be positioned upwind, and (1) would be positioned to the side. 

•   Two of the upwind judges will be judging maneuver shape.  The sole job of these two judges is to grade the shape of the maneuver as defined by the rulebook.  Is a square a square in shape?  Is a triangle triangular, are rounds round, etc.  These judges watch the entire maneuver for the correct shape.  Position in the hemisphere is irrelevant.  Size is irrelevant. Scores are from 1 to 10 for each maneuver. The (2) shape judges' scores are averaged to make one shape score (in a manpower pinch, (1) shape judge could be used instead of two).
•   The third upwind judge would be responsible for scoring the corner quality, tightness, bobbles, and airplane tracking.  The sole responsibility of this judge is to watch the airplane as it flies through the maneuver.  Scores are from 1 to 10 for each maneuver.
•   The final judge would be placed 90 degrees to the maneuver.  The 90 degree judge would be to stand to the side and grade 45-90 degree aspects of the maneuver, as well as the accuracy of the maneuver bottoms.  Scores are from 1 to 10 for each maneuver.

Now we have (3) sets of scores each derived from completely independent variables within the pattern.  We have (1) averaged shape score, (1) corner and bobble score, and (1) size, position, and bottom score.  To create spread between the flyers it is imperative that the scores be used as multipliers, and not averaged.  Using scores that are multiplied is the true secret to creating spread in the finish scores.

[Howard Rush]

Once again, here's the formula used to pick the finals judges at the Nats.  Low score is better.

For a given flight i, an exceedance is defined to be when |judge’s placing of pilot, flight i – consensus placing of pilot, flight i| > 3 X the average of |judge’s placing of pilot– consensus placing of pilot|, where consensus is the average placing by all the judges of a given pilot’s flight among all the flights in the round.  Paul now intends to set exceedance weighting to zero. 

It's too late to change this for the 2009 Nats.  If you have a better idea or want to discuss it, or-- better yet-- want to volunteer to judge at the Nats, tell Paul Walker. 

[Paul Walker]

The way the NATS are run I think it would be hard for one maneuver to knock a person out of the running. 

In qualifying, this may be true. There is a throw away flight each day. However, some days the weather changes and making up a bad score can get tough.

There is the two days of qualifying to try and make the top twenty. 
This is true.

I think they use the top score to decide who is moving on. 
The best score from Tuesday adds to the best score from Wednesday. This score determines the fliers to move to the semi-finals.

Then top twenty day in which you have two more officials.  I think one score gets you to the top five. 
That statement is NOT true. Two flights, and both scores are added. The top five move on.  Here is where one "bad" maneuver can keep a flier from moving on. This is by far, the most pressure packed day at the Nats, as any flier wishing to move on can't make a single mistake.

Hope Paul will straighten me out on this as I don't fly Expert or Open at the NATS.

Well, we don't have "expert" at the Nats. There is Open, Advanced, Intermediate, and Beginner. In Advanced, the qualifying rounds work the same as Open. In the Advanced finals, it is the sum of the two flights as well. Plenty of pressure that day also. Once again, one bad maneuver will cost you!






[/quote]