News:


  • March 28, 2024, 08:22:04 AM

Login with username, password and session length

Author Topic: Question on top 20 judges  (Read 11877 times)

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Question on top 20 judges
« on: July 22, 2014, 04:02:18 PM »
Why is it the program has the head judge and the experienced judges ,judging advanced instead of open? Seems to me there is amiss in the program. Not saying it would make any difference but it sure seems odd to me. Anyone else notice this?

Seems that the judges that track well and don't use the full range are rewarded and the ones who judge accordingly are penalized.

Too many anomalies for instance (not saying it would make a difference) But I started out in group D with Jose and then the re shuffle and ended up in group C to fly against all the NATS champs. Water under the bridge but I watch everything.
AMA 12366

Online Crist Rigotti

  • 23 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 3855
  • Electric - The future of Old Time Stunt
Re: Question on top 20 judges
« Reply #1 on: July 22, 2014, 08:40:03 PM »
Why is it the program has the head judge and the experienced judges ,judging advanced instead of open? Seems to me there is amiss in the program. Not saying it would make any difference but it sure seems odd to me. Anyone else notice this?

Seems that the judges that track well and don't use the full range are rewarded and the ones who judge accordingly are penalized.

Too many anomalies for instance (not saying it would make a difference) But I started out in group D with Jose and then the re shuffle and ended up in group C to fly against all the NATS champs. Water under the bridge but I watch everything.

It's always something, huh Bob?
Crist
AMA 482497
Waxahachie, TX
Electric - The Future of Old Time Stunt

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #2 on: July 22, 2014, 08:44:06 PM »
It's always something, huh Bob?

Yep. This was told to my by two different NATS Judges.
AMA 12366

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #3 on: July 22, 2014, 11:33:40 PM »
You'd need to ask Bob and Curt how they assigned judges. 

You seem to have an erroneous idea of how the program works.  Whether a judge uses the full range of score does not affect his ranking.  His ranking is determined by how well his ordering of contestants agrees with the consensus.

That program is public.  I don't know if I've sent you a copy, but I have sent it to a bunch of people, particularly critics, and I have explained many times on these fora how the judge ranking works.  I have also put out many requests to tell me if it is statistically valid and to review how scores are normalized with the new "How the Nats Will Be Ran" process introduced last year.  I have received no input.  Hey, I'm not asking folks to do the math, just to read what's there.  If you think something is wrong, at least look at what I've written.   
The Jive Combat Team
Making combat and stunt great again

Online Bill Morell

  • 21 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 953
Re: Question on top 20 judges
« Reply #4 on: July 23, 2014, 02:12:56 AM »
It would be so refreshing if just one time there could be a Nats without someone implying some sort of "conspiracy theory" regarding the outcome.  Wouldn't that be cool Robert?
Bill Morell
It wasn't that you could and others couldn't, its that you did and others didn't.
Vietnam 72-73
  Better to have it and not need it than it is to need it and not have it.

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #5 on: July 23, 2014, 05:56:44 AM »
You'd need to ask Bob and Curt how they assigned judges.  

You seem to have an erroneous idea of how the program works.  Whether a judge uses the full range of score does not affect his ranking.  His ranking is determined by how well his ordering of contestants agrees with the consensus.

That program is public.  I don't know if I've sent you a copy, but I have sent it to a bunch of people, particularly critics, and I have explained many times on these fora how the judge ranking works.  I have also put out many requests to tell me if it is statistically valid and to review how scores are normalized with the new "How the Nats Will Be Ran" process introduced last year.  I have received no input.  Hey, I'm not asking folks to do the math, just to read what's there.  If you think something is wrong, at least look at what I've written.  

Seems that every time something is questioned the guy who is doing Thar questioning is labeled a critic. I guess that's why the job landed on me. If no one speaks out on anything everything will remain the same. Thanks for your work on the program Howard not discounting that. But because this is a subjective process of judging should not the selection of the judges be subjective and at the Head Judges discretion?

Meaning the judges with the most experience to judge on top 20 day open and not advanced.

It would be so refreshing if just one time there could be a Nats without someone implying some sort of "conspiracy theory" regarding the outcome.  Wouldn't that be cool Robert?

Bill when was the last NATS you attended? I am not saying there is any conspiracy to the Judging. I guess no one is allowed to question status Que?

Definition of Insanity is doing the same thing over and over expecting different results.
« Last Edit: July 23, 2014, 06:29:56 AM by Robert Storick »
AMA 12366

steven yampolsky

  • Guest
  • Trade Count: (0)
Re: Question on top 20 judges
« Reply #6 on: July 23, 2014, 07:29:15 AM »
Seems that the judges that track well and don't use the full range are rewarded and the ones who judge accordingly are penalized.

Biggest pile of bonk! Instead of "I'm watching" or "some judge who wants to remain nameless told me" drivel, you should actually get trained and spend ONE, just ONE NATS as a judge.

For YEARS, I've been reading on SSW and SH about conspiracies and other drivel about NATS judging. I am a thinking man and I REFUSE to base my own perceptions on someone else's opinion. Instead, I spent FOUR years volunteering to judge every contest I could get to and then ACTUALLY volunteering to go judge at the NATS. I spent close to 3 THOUSAND dollars on that trip, stood for 8 hours in the blistering sun while pilots hung out in the shade, relaxing and talking. In the end, I learned for myself what NATS judging is about. What you are suggesting is COMPLETELY UNFOUNDED AND 100% HEARSAY.





Offline Doug Moon

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 2188
Re: Question on top 20 judges
« Reply #7 on: July 23, 2014, 07:31:09 AM »
(I use the word you in this post.  I am using that as a general term not directed at anyone in particular)

I think the program picks the judges correctly.  

It takes the judges who agreed with the majority and groups them.

You should want the judges who see the pattern the same or close to it judging you.  It will create consistency.  If you have judges who are all using different criteria then you will have no clue what is going on when you look at your scores/sheets.  Your frustration will only multiply at that point. Judges who are agreeing across the board with the overall outcome should be using similar criteria.  Their scoring range is not and should not be taken into account.  Their years of experience I would think would also not be taken into account as well.  At the point where the program is picking the judges for the next rounds they have all been through the nats judging meeting and warms ups and discussions etc.

I feel the judging was fair.  On Friday my first flight was so so.  My second flight may have been my best judged flight of the week and they scored me for it too.  

Thank you Howard for creating a program that can track all of that information and create a judging corp based on their production and not their name/experience.

Also as a judge working the nats you know you are selected based on your output and not your name and or years of experience in the game.  The program cannot be biased in that way. It only knows numbers and that's it.  As a judge you have to be honest and fair and call it like you see it and let the program decide the group of judges for each class of flying.  Its a great situation to be in if you ask me.
Doug Moon
AMA 496454
Dougmoon12@yahoo.com

steven yampolsky

  • Guest
  • Trade Count: (0)
Re: Question on top 20 judges
« Reply #8 on: July 23, 2014, 07:34:37 AM »
Definition of Insanity is doing the same thing over and over expecting different results.

Who said we need different results? There has not been ONE instance of the wrong pilot getting the win since the current judging program been instituted!

Please don't hide behind the innocent of "Am I not allowed to ask questions?". Your questions a similar to asking regular folk "How often do you expose yourself to little children?" or "When was the last time you beat up your wife?".

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #9 on: July 23, 2014, 07:39:08 AM »
 As a judge you have to be honest and fair and call it like you see it and let the program decide the group of judges for each class of flying.  Its a great situation to be in if you ask me.

I guess Mark, Steve and Dale (don't forget Bruce) must not have chosen the right numbers as opposed to the new judges. I can see this is leading nowhere. Status quo shall remain in play. I have learned my lesson.

Anytime someone questions status quo it goes against the Holy Grail of stunt.

Where the pilots chosen in the right order? Probably I was just speaking up on how they got there.
AMA 12366

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #10 on: July 23, 2014, 08:25:38 AM »
You guys are unfairly jumping on Sparky. I know what he is talking about but to even mention that something is amiss makes you look like a poor sport. There truly is a problem with a formula that puts the Head Judge anywhere except with the best pilots. The head judge is the person responsible for training all the other judges. If we cannot trust his judgment, then who can we trust?  

Derek

P.S.  Mark had to be entered manually into the finals on Saturday. Thankfully the ED felt that it was wrong to have him anywhere else.

I have also offered a solution to make the formula work better but it was met with resistance...


Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #11 on: July 23, 2014, 08:35:23 AM »
Who said we need different results? There has not been ONE instance of the wrong pilot getting the win since the current judging program been instituted!

Please don't hide behind the innocent of "Am I not allowed to ask questions?". Your questions a similar to asking regular folk "How often do you expose yourself to little children?" or "When was the last time you beat up your wife?".


I am stating FACTS!  Read this

You guys are unfairly jumping on Sparky. I know what he is talking about but to even mention that something is amiss makes you look like a poor sport. There truly is a problem with a formula that puts the Head Judge anywhere except with the best pilots. The head judge is the person responsible for training all the other judges. If we cannot trust his judgment, then who can we trust?  

Derek

P.S.  Mark had to be entered manually into the finals on Saturday. Thankfully the ED felt that it was wrong to have him anywhere else.

I have also offered a solution to make the formula work better but it was met with resistance...

I had no dog in the fight so as a outside observer I can only report what I saw.
AMA 12366

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #12 on: July 23, 2014, 08:46:44 AM »
I just want to state that I am not angry about anything that happened at the Nats, I could have flown a little better and made the cut. I just do honestly feel like the formula does not work as it was intended. You know what the say about good intentions...

Derek

Online Paul Smith

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 5793
Re: Question on top 20 judges
« Reply #13 on: July 23, 2014, 09:03:01 AM »
Just as an observer from the other events, Stunt is fortuante to have enough volunteers to even think about accepting some and rejecting others.  Most events are hard-pressed to fill the miniumum requirement.

Paul Smith

Offline Doug Moon

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 2188
Re: Question on top 20 judges
« Reply #14 on: July 23, 2014, 09:10:21 AM »
You guys are unfairly jumping on Sparky. I know what he is talking about but to even mention that something is amiss makes you look like a poor sport. There truly is a problem with a formula that puts the Head Judge anywhere except with the best pilots. The head judge is the person responsible for training all the other judges. If we cannot trust his judgment, then who can we trust?  

Derek

P.S.  Mark had to be entered manually into the finals on Saturday. Thankfully the ED felt that it was wrong to have him anywhere else.

I have also offered a solution to make the formula work better but it was met with resistance...



Derek is making a very good point!!!!!!
Doug Moon
AMA 496454
Dougmoon12@yahoo.com

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #15 on: July 23, 2014, 09:14:41 AM »
Derek is making a very good point!!!!!!

No one listens to me. It took a top 5 flier to chime in before it was a good point. If I am watching so is everyone else, Interesting.

I also find it interesting that some one said they had a idea to get Ball State students to judge (like it was a new idea) when I mentioned that 6 years ago. They would be good at it just for that fact they are 1/2 the age of most of the current field and can see twice as good. I had then heard they don't know what to look for (I say good) but they could be trained just like the judging pool we have now.

Could we get a pool from them? I don't know but if I got a GO I would be willing to make the phone calls.
AMA 12366

Offline Paul Walker

  • 23 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 1626
Re: Question on top 20 judges
« Reply #16 on: July 23, 2014, 09:19:17 AM »
Derek is making a very good point!!!!!!


Great!  Now all that is needed is a formula for the program. We went to a formula because people didn't like the subjective picks.

No matter what is done someone will complain.

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #17 on: July 23, 2014, 09:20:49 AM »

No matter what is done someone will complain.

It was a observation not a complaint . Here is a novel idea Use something like this http://www.random.org/integers/ then no one would complain. It would choose flight order and circle selection in a matter of seconds not hours. and its totally random.

Thanks to everyone who made this contest possible.
AMA 12366

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #18 on: July 23, 2014, 09:22:10 AM »
Just as an observer from the other events, Stunt is fortuante to have enough volunteers to even think about accepting some and rejecting others.  Most events are hard-pressed to fill the miniumum requirement.



You are absolutely correct! And we had a great pool of judges, great tabulators, great ED and assistant ED, and great pull testers. We could not have a contest without these people!

Derek

Offline Paul Walker

  • 23 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 1626
Re: Question on top 20 judges
« Reply #19 on: July 23, 2014, 09:23:59 AM »
Derek is making a very good point!!!!!!


Great!  Now all that is needed is a formula for the program. We went to a formula because people didn't like the subjective picks.

No matter what is done someone will complain.

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #20 on: July 23, 2014, 09:33:03 AM »

Great!  Now all that is needed is a formula for the program. We went to a formula because people didn't like the subjective picks.

No matter what is done someone will complain.

You are correct, someone will always complain.

A few years ago I offered Howard some suggestions but they didn't go very far.

Derek

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #21 on: July 23, 2014, 09:38:48 AM »
Let me make this Krystal Clear! I am not complaining I am only giving a observation on what goes on. Anytime someone questions status quo its against the HOLY GRAIL of stunt. http://www.random.org/integers/ If this system was used on the true luck of the draw the end results may or may not change. It does not matter to me how you want to play. My lessons have been learned.  Play with that link and envision how it would work. I am sure it all the top guys were put on one circle there would be some complaining then.

AMA 12366

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #22 on: July 23, 2014, 09:40:25 AM »
There truly is a problem with a formula that puts the Head Judge anywhere except with the best pilots. The head judge is the person responsible for training all the other judges. If we cannot trust his judgment, then who can we trust?  

P.S.  Mark had to be entered manually into the finals on Saturday. Thankfully the ED felt that it was wrong to have him anywhere else.
I have also offered a solution to make the formula work better but it was met with resistance...


As I remember, your suggestion had some subjective stuff in it.  My initial agreement with Paul when he ran the Nats was that I would help him with the tabulation program only if everything in it was open and objective, because I didn't want any of the noisy, chronic losers to have anything to complain about when I won.  I had seen a lot of trouble at previous Nats caused by those losers exploiting the subjective seeding and judge selection used at the time because objective formulas weren't available.    I think the objective judge selection we have could use improvement, but even a mediocre objective method beats any subjective method.  As I have said many times, the "How the Nats Will Be Ran" proposal that introduced "Expert" to the Nats may have created normalization errors in the judge-picking formula.  So lets see some math.  

If the head judge didn't rate as well as the others, I wouldn't have intervened.  
The Jive Combat Team
Making combat and stunt great again

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #23 on: July 23, 2014, 09:49:01 AM »
Let me make this Krystal Clear! I am not complaining I am only giving a observation on what goes on. Anytime someone questions status quo its against the HOLY GRAIL of stunt. http://www.random.org/integers/ If this system was used on the true luck of the draw the end results may or may not change.

That's how we pick the flight order for local contests.  Some may see this as mysterious and giving opportunity for manipulation, so we use pingpong balls at the Nats.  Paul didn't even want to use Betty Adamisin's method to map the first round draw into the other three rounds, even though it's open and objective.

Peabody thinks we should pick judges randomly, as I think you are suggesting.  I think that idea has merit. 
The Jive Combat Team
Making combat and stunt great again

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #24 on: July 23, 2014, 10:37:53 AM »
As I remember, your suggestion had some subjective stuff in it.  


You are probably correct. I understand and respect your stance on the matter.

And as Paul stated, whatever method we use there will be some that do not like it. All in all, what we have now is better than what we use to have...

Derek

steven yampolsky

  • Guest
  • Trade Count: (0)
Re: Question on top 20 judges
« Reply #25 on: July 23, 2014, 10:41:35 AM »
You guys are unfairly jumping on Sparky. I know what he is talking about but to even mention that something is amiss makes you look like a poor sport. There truly is a problem with a formula that puts the Head Judge anywhere except with the best pilots. The head judge is the person responsible for training all the other judges. If we cannot trust his judgment, then who can we trust?  

Derek,

I disagree. Last year Mark WAS NOT the most consistent judge. Granted, he was No.2 but not because he is a head judge. If Mark was off this year, the system would have identified it.

Offline Doug Moon

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 2188
Re: Question on top 20 judges
« Reply #26 on: July 23, 2014, 10:57:01 AM »

Great!  Now all that is needed is a formula for the program. We went to a formula because people didn't like the subjective picks.

No matter what is done someone will complain.

Let's not jump to conclusions here.  I am in no way shape or form complaining.  Just thought he made a good point. 

The contest was awesome!  The weather was great.  It was cool and I got to wear my favorite ElectriFly.com t-shirt many times!  :)  I flew good and others flew good and David flew best when it mattered.  This is all discussion stuff. 

I do find it odd when people come up with ideas that were already used and complained about in the past as a new way to do it. Robert says use Ball State students.  That would be similar to Navy judges back in the day yes?  I have heard they were partial to the kiddos or navy painted planes etc. etc. etc.... 

Its all just dicussion from here.  No complaining!!!!!!!  NONE ZIP ZERO NADDA!!!!
Doug Moon
AMA 496454
Dougmoon12@yahoo.com

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #27 on: July 23, 2014, 11:01:28 AM »
  That would be similar to Navy judges back in the day yes? 

No
AMA 12366

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #28 on: July 23, 2014, 11:08:04 AM »
Derek,

I disagree. Last year Mark WAS NOT the most consistent judge. Granted, he was No.2 but not because he is a head judge. If Mark was off this year, the system would have identified it.

Hey Steve,

I guess we will just have to agree to disagree. I don't care if he was not the most consistent if he was the most correct.

This is not worth arguing over anyways, the guys that come to judge every year always do their best and that is all we can ask of any of them.

Derek

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #29 on: July 23, 2014, 11:49:22 AM »
I don't care if he was not the most consistent if he was the most correct.

So come up with a formula that measures correctness.
The Jive Combat Team
Making combat and stunt great again

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #30 on: July 23, 2014, 12:25:51 PM »
So come up with a formula that measures correctness.

Correctness is subjective.

Derek

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #31 on: July 23, 2014, 12:50:15 PM »
I've looked at stuff like full-scale aerobatics scoring methods.  Folks seem to think that the method we use measures correctness.  I'm not so sure, and I haven't seen any convincing statistical justification of it.  Other sports use it, so you'd think somebody would have written a scholarly paper sometime.  I would be interested in discussing the math after August.
The Jive Combat Team
Making combat and stunt great again

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #32 on: July 23, 2014, 01:02:29 PM »
I've looked at stuff like full-scale aerobatics scoring methods.  Folks seem to think that the method we use measures correctness.  I'm not so sure, and I haven't seen any convincing statistical justification of it.  Other sports use it, so you'd think somebody would have written a scholarly paper sometime.  I would be interested in discussing the math after August.

Sounds good Howard, keep practicing and go kick some ass in Poland.

Derek

Offline RC Storick

  • Forum owner
  • Administrator
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12393
  • The finish starts with the first piece of wood cut
    • Stunt Hangar
Re: Question on top 20 judges
« Reply #33 on: July 23, 2014, 01:09:21 PM »
I've looked at stuff like full-scale aerobatics scoring methods.  Folks seem to think that the method we use measures correctness.  I'm not so sure, and I haven't seen any convincing statistical justification of it.  Other sports use it, so you'd think somebody would have written a scholarly paper sometime.  I would be interested in discussing the math after August.

Look at Olympics gymnastics or figure skating,diving or any of the 100 Olympic judging aspects. =640
AMA 12366

Offline Trostle

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 3338
Re: Question on top 20 judges
« Reply #34 on: July 23, 2014, 05:57:32 PM »
I've looked at stuff like full-scale aerobatics scoring methods.  Folks seem to think that the method we use measures correctness.  I'm not so sure, and I haven't seen any convincing statistical justification of it.  Other sports use it, so you'd think somebody would have written a scholarly paper sometime.  I would be interested in discussing the math after August.

Howard,

I am not arguing with you or anybody else here.  George Buffalano developed a methodology for our stunt contests based on the program used for full scale aerobatics.  Bob Baron was pushing hard for this.  It is based on normalizing scores by each judge for each maneuver.  The system can detect incorrect scoring by individual judges and to some degree, bias, and can offer a ranking of the judges. George performed analyses for the CIAM after several World Championships using his program.  On my recommendation and urging from Bob Baron, George wrote a paper in Stunt News several years ago that presented this concept.  I wrote an introduction for George on that article.  The article received absolutely zero response from the PAMPA membership.  I made reference to that article and when it was published on one of these forums a year or so ago and again, zero response.

One characteristic of this program, as I understand it, is that no rankings or scores are available until all of the flights are completed, the scores entered into the computer, the computer does its thing, and then the final scores are printed and the rankings of the pilots are available.  I do not know if our event is ready for that process.

The full scale aerobatic community has embraced this program as bias in the international aerobatic judging community was previously causing considerable strife and dissention to the point of eliminating the international program.

Keith

« Last Edit: July 23, 2014, 07:49:26 PM by Trostle »

Offline Randy Ryan

  • AMA Member
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 1767
Re: Question on top 20 judges
« Reply #35 on: July 23, 2014, 07:48:17 PM »
Interesting thread. As one of the judges this year I can say this. The training wasn't what you'd call "ground up", it was really more of a discussion session to point out safety and procedural issues with some discussion about where maneuvers begin and end, and other nuances. All the judges were experienced. The most interesting point was that in the consistency rating, there was a very small variation, the least consistent being a 2.25 which to my understanding is a very narrow range. This, also to my understanding would have qualified any of this years judges as top 20 last year which had a range nearing 5. The Head Judge is not necessarily the most qualified, he is an organizer, not a boss. Mark did an exceptional job in my opinion and himself admitted he is not an "expert", whatever that might be.

Personally it was a very rewarding however exhausting experience. the daily warm-up flights put everyone on the same page first thing. For me, it was a chance to compare my style and observations with judges other then our local guys. I was pleased to find that I fit in very closely with the group, seeing the same things an scoring very consistently.
Randy Ryan <><
AMA 8500
SAM 36 BO all my own M's

Offline Brett Buck

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 13716
Re: Question on top 20 judges
« Reply #36 on: July 23, 2014, 09:03:44 PM »
Let me make this Krystal Clear! I am not complaining I am only giving a observation on what goes on. Anytime someone questions status quo its against the HOLY GRAIL of stunt.

   Not at all, people question it all the time. Almost all of the time the questions are carefully and clearly answered, but do not resolve the issue, because the answer is at odds with the questioner's pre-concieved notions, or doesn't make mathematical sense. Usually because they try to determine what needs to be changed first, and then work backwards to come up with a problem and the justification for the change.

Quote
http://www.random.org/integers/ If this system was used on the true luck of the draw the end results may or may not change. It does not matter to me how you want to play. My lessons have been learned.  Play with that link and envision how it would work. I am sure it all the top guys were put on one circle there would be some complaining then.

    No one involved fails to understand random numbers, or how "random" they need to be.

    Random placement was done for years, there was bitter complaints afterwards. Manual seeding was used, there were fewer complaints, although it indirectly led to *death threats*. Now, automatic seeding is used, which more-or-less implements the manual method, and there are very few complaints. It was a little off this year because Billy and I had some gaps in our attendance, so seeded lower than normal.

    Same with judge placement, now no one can plausibly accuse anyone else of rigging the system to favor particular fliers. It wasn't happening with the manual method, and it is not happening now, so of course the results of the contest are largely as before.

   
    Brett

     

   

Offline bill rutherford

  • 2015
  • Trade Count: (0)
  • Lieutenant
  • *
  • Posts: 99
Re: Question on top 20 judges
« Reply #37 on: July 23, 2014, 09:30:05 PM »
I'm so thankful that we have Mark and the rest of the judges. I think the judging is now more consistent than ever.
     I judged classic with Steve Smith as my partner and after the contest was over I compared some of his scores with mine and was some what amazed at how close we were with our numbers., In the end I think we got it right.  Bill Rutherford

John Leidle

  • Guest
  • Trade Count: (0)
Re: Question on top 20 judges
« Reply #38 on: July 23, 2014, 10:45:12 PM »
  Hey Howard... as far as picking judges randomly  has anyone considered of having them grabbing a marked Ping Pong ball from a small paper bag as the pilots do? 
    John 8)

Offline Tim Wescott

  • 2016 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 12804
Re: Question on top 20 judges
« Reply #39 on: July 23, 2014, 11:24:17 PM »
  Hey Howard... as far as picking judges randomly  has anyone considered of having them grabbing a marked Ping Pong ball from a small paper bag as the pilots do?

As much work as it would be to judge the Nats, maybe it would be more appropriate to have the ED draw a ping-pong ball out of a paper bag and then chase the guy down before he much judge.
AMA 64232

The problem with electric is that once you get the smoke generator and sound system installed, the plane is too heavy.

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #40 on: July 24, 2014, 10:34:12 AM »
I am not arguing with you or anybody else here.  George Buffalano developed a methodology for our stunt contests based on the program used for full scale aerobatics.  Bob Baron was pushing hard for this.  It is based on normalizing scores by each judge for each maneuver.  The system can detect incorrect scoring by individual judges and to some degree, bias, and can offer a ranking of the judges. George performed analyses for the CIAM after several World Championships using his program.  On my recommendation and urging from Bob Baron, George wrote a paper in Stunt News several years ago that presented this concept.  I wrote an introduction for George on that article.  The article received absolutely zero response from the PAMPA membership.  I made reference to that article and when it was published on one of these forums a year or so ago and again, zero response.

One characteristic of this program, as I understand it, is that no rankings or scores are available until all of the flights are completed, the scores entered into the computer, the computer does its thing, and then the final scores are printed and the rankings of the pilots are available.  I do not know if our event is ready for that process.

The full scale aerobatic community has embraced this program as bias in the international aerobatic judging community was previously causing considerable strife and dissention to the point of eliminating the international program.

I think we had this conversation before.  I don't remember reading Dr. Buffalano's piece, but I got the reference to the full-scale aerobatics method, which is equivalent to what we now use except for the difference in how contestants are scored.  I remain sceptical about its statistical validity.  I had some correspondence with Dr. Buffalano.  As I recall, he was more interested in the problem of the differences between the top fliers flying being less than what judges can distinguish.  That's an interesting problem, too. Phil Cartier has also studied this.  The upshot is to use as many judges as possible on finals day.

I think our current judge ranking method is erroneous because of changes brought by the addition of Expert.  I continue to ask for help fixing it.

The Jive Combat Team
Making combat and stunt great again

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #41 on: July 24, 2014, 10:56:06 AM »
Look at Olympics gymnastics or figure skating,diving or any of the 100 Olympic judging aspects. =640

Send me their formulae.  What I've seen of them is that they emphasize minimizing the effect of dishonest judges.  We don't have that problem.
The Jive Combat Team
Making combat and stunt great again

Online Paul Smith

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 5793
Re: Question on top 20 judges
« Reply #42 on: July 24, 2014, 11:02:35 AM »
Look at Olympics gymnastics or figure skating,diving or any of the 100 Olympic judging aspects. =640

In the Olympics they use high-res stop action photography to micro-analyze every little move and second-guess the jury.
In model airplanes we don't have it, couldn't afford it, and are better off without it.  I expect the Olympic events have enough cash to make judging really worthwhile, unlike us.

Maybe the chance of catching a lucky break from the judges is what keeps some people interested.
Paul Smith

Offline Brett Buck

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 13716
Re: Question on top 20 judges
« Reply #43 on: July 24, 2014, 11:15:44 AM »
Send me their formulae.  What I've seen of them is that they emphasize minimizing the effect of dishonest judges.  We don't have that problem.

    There are some who don't buy that argument, as recently as last week. They are wrong, but sincere.

    Our problem is fundamentally different. We have the judges we have, and they are pretty uniformly good. But, you have to have some of them do Open and the others do Advanced (now that Expert will be gone again), Junior, and Senior. You have to decide somehow which do which. Picking them by hand using the tracking method led to false but persistent accusations of cheating, so we now accomplish the tracking with a computer algorithm that does mostly the same thing. The one consequential difference is that there is no check and no distinction that would select out "high" or "low" judges, which was definitely a feature of the old manual system.

    The one legitimate issue with the current system is that it counts on what amounts to majority voting logic to determine "consistency". But there are only 3 judges per circle when the important tracking decisions are made. It's entirely possible that you get a group with one highly experienced judge and two relative newbies. You might expect the experienced judge to frequently be inconsistent with the newbies, which will tend to defeat the tracking system. I think this explains Derek's problem. You really need a lot of people judging the same flights to make it work, but we can't do that. I think the manual tracking method, with the added personal judgement possible, might have been less prone to these sorts of seeming anomalies. And got someone to threaten to shoot Gary McClellan.


    Brett

Offline FLOYD CARTER

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 4458
    • owner
Re: Question on top 20 judges
« Reply #44 on: July 24, 2014, 11:30:59 AM »
all deleted.  the discussion really doesn't concern or involve me.
Floyd
« Last Edit: July 25, 2014, 01:11:52 PM by FLOYD CARTER »
89 years, but still going (sort of)
AMA #796  SAM #188  LSF #020

Offline Derek Barry

  • Trade Count: (0)
  • Admiral
  • ******
  • Posts: 2830
Re: Question on top 20 judges
« Reply #45 on: July 24, 2014, 11:45:32 AM »

    The one legitimate issue with the current system is that it counts on what amounts to majority voting logic to determine "consistency". But there are only 3 judges per circle when the important tracking decisions are made. It's entirely possible that you get a group with one highly experienced judge and two relative newbies. You might expect the experienced judge to frequently be inconsistent with the newbies, which will tend to defeat the tracking system. I think this explains Derek's problem. You really need a lot of people judging the same flights to make it work, but we can't do that. I think the manual tracking method, with the added personal judgement possible, might have been less prone to these sorts of seeming anomalies. And got someone to threaten to shoot Gary McClellan.


    Brett

Thank you Brett, that is exactly my problem with the formula.

No Howard, I do not have a mathematical solution to fix it either. The only thing I could come up with was to remove the top two pilots from each circle. This would take the "Big Names" out of the equation just in case there was any unintentional bias. It still does not fix the problem that Brett described which, in my opinion, is the real problem.

Derek



Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #46 on: July 24, 2014, 12:16:45 PM »
Anything objective that you can do manually you should be able to write a formula for.
  
The Jive Combat Team
Making combat and stunt great again

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #47 on: July 24, 2014, 12:20:06 PM »
It's entirely possible that you get a group with one highly experienced judge and two relative newbies. You might expect the experienced judge to frequently be inconsistent with the newbies, which will tend to defeat the tracking system.

I would expect the experienced judge to be more closely correlated with either newbie than the newbies would be with each other. 
The Jive Combat Team
Making combat and stunt great again

Offline Howard Rush

  • 22 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 7805
Re: Question on top 20 judges
« Reply #48 on: July 24, 2014, 12:25:21 PM »
The one consequential difference is that there is no check and no distinction that would select out "high" or "low" judges, which was definitely a feature of the old manual system.

Nor should there be.  The high or low judge may be the best judge. 
The Jive Combat Team
Making combat and stunt great again

Offline Doug Moon

  • 24 supporter
  • Trade Count: (0)
  • Admiral
  • *
  • Posts: 2188
Re: Question on top 20 judges
« Reply #49 on: July 24, 2014, 12:26:22 PM »
I think the word "newbie" sounds funny....   :)

Doug Moon
AMA 496454
Dougmoon12@yahoo.com


Advertise Here
Tags:
 


Advertise Here