Keith, could you explain the process used in previous years to select judges for the finals. Was it free from personal bias by those who did the selection?
I would think random selection has significant merit and would preclude any sort of "conspiracy nonsense". To those who express discontent you simply say..."Hey, that's the luck of the draw!" A rather simple yet elegant solution.
Dick,
There is an Event Director's Guide that has been developed by the Event Directors over the years. There are basically three topics that are used to assign judges. One of those is to remove the appearance of any particular region being overly represented by any set of judges assigned to a specific circle. In other words, having a preponderance of judges from any particular region in a group is to be avoided. This is not always easy to do because sometimes a significant portion of the judges come from the midwest (or more specifically, they come from an area "convenient to the Nats site") though this is not always the case. At some Nats, there have been a fair number of judges from the West Coast. As far as I know, for the Nats that I judged for Warren Tiahrt, there has never been more than one West Coast judge used in the finals or the semifinals rounds of the Nats. In spite of the extraordinary effort by Warren to eliminate even the appearance of regionalism in the judge assignments, there have been accusations, always unwarranted, to the contrary. Is the elimination of regionalism a good thing? Maybe not, because when this procedure is followed, sometimes the most competent group of judges is not selected. (From my own personal experience, I have not been selected to judge the Open semifinals and finals because there was another West Coast judge for each semifinals circle as well as for the finals circle. I did not mind judging the Advanced Finals as that was part of what was volunteered for.
Another factor considered in judge assignments is to reduce/eliminate the assignment of any judge that has a standing relationship with one or several of the fliers in those semifinals and finals rounds. This is not to say that the judges cannot know any of the individual competitors. It would be impossible to assign any group of judges at the Nats where no judge knew any of those being judged. However, where known personal relationships, where individual judges often fly with the competitors or even have coached any of the competitors, these combinations are reduced to some practical limit. Here again, Warren has gone to great lengths to minimize the appearance of allowing personal friends/coaches of individual competitors to be assigned to those groups of judges who will be judging the semifinals and finals.
Even during the qualification rounds, efforts are made to make sure that each judge grouping is not overly represented by any single region and to separate those judges who are know to have particular affiliations with any individual or group of competitors.
The third area is (horrors) the rating of individual judge performance as the Nationals proceeds from day to day. Ballooning has been a problem and probably always will be. Some judges appear to be more prone to this. Ballooning is fairly easy to detect when all of the scores are compiled at the end of the round. Those judges who show ballooning tendencies are simply not assigned to judge during the semifinals and finals rounds. There is another process that has been used over the years to evaluate judge performance. It is probably not a statically sound as what a normalization process would provide. The process is based on the assumption that the average score for each competitor for each of his flights determined by the combination of each judge evaluating that flight is a valid indication of how well that individual flew. This assumption essentially needs to be made and apperas to be valid as it is those average scores that eventually determine the placing of each individual after the completion of the Nats, the placings are recorded and most seem content to accept those placings. Now, after a round is completed, the individual competitors are ranked from the best average score to the lowest average score. Compared to that "waterfall" chart that shows the highest to lowest average scores, individual judges scores are also plotted. There is normally some variance in the ranking of individual competitors by each judge. But using the assumption that the average score is a fair assessment of the individual's flight, there will be judges who are way off of the scale of even being close to assessing individual flight scores and/or individual placings. Ballooning will show it ugly head here also because it can be shown what time of day (or portion of the round) higher scores that appear out of sync start to appear. There have been cases where there have been 5 or more judges on a circle. Most judges will generally have a similar order of how the competitors are placed. Yet, in one situation, one judge had the order completely reversed. Now, do you want to have already randomly assigned that judge with that kind of performance to judge the finals? Or to have already assigned a judge who is prone to ballooning? I think not.
This evaluation process is probably against any pure statistician's thinking. (Obviously, I am not a statistician.) Generally, by the time the semifinals and finals are reached, several of the corps of judge volunteers are not needed and some will not be assigned to the finals round. The process used over the years is a tool, in my opinion as accurate as the scoring system we use to determine placings of competitors, to help assign a group of judges that will yield the best possible assessment of each individual competitor.
I do not think selection of our National Stunt Champion and the placing at our Nats should be written off as simply the luck of the draw. Yes, weather can play a part and that can become a significant fact involving the luck of the draw. However, the weather is something the ED cannot control. The ED can control various aspects of the competition that does minimize to some extent how the luck of the draw might impact the final results. Of those things the ED can control is the seeding process and how the judges are assigned to various circles. Warren Tiahart and those preceding him should all be applauded for trying to make the competition field as level as possible while minimizing the luck of the draw to some acceptable level. (Let the naysayers howl, but they are the ones who chose either not to understand or are incapable of understanding the efforts taken over long periods of time to make the Nats as fair as possible for every individual competitor.)
I have judged at 11 Nats and Team Trials. Based on my knowledge of our stunt event and experience with being the ED at previous Nats and Team Trials as well as being a judge at 3 World Championships, I have had the privilege to be asked by Warren to be involved with sorting through the judge assignment process. Admittedly, it is not a perfect process, but I can say that there is no doubt in my mind that the process used to select and assign the judges thoughout each entire Nats and Team Trials program is absolutely free from any personal bias that could in any way be a negative impact on the placing of any individual competitor. Contrary to statements from the myth makers, the people involved with these processes are well above the actions of those myth makers. Many understand that the myth makers have no credibility. Unfortunately, there those who do not have the experience or knowledge to doubt the subterfuge that is generated by the continued harangue that comes from that small but destructive group.
This has all been explained before. But, thanks for asking.
Keith Trostle
Topic: Seeding (Read 11206 times)