Hugo stats: where are we today

camestrosfelapton

Jan 22

Oh boy, it simply hasn't got better has it.

The data does not look organic, there are confirmed (but maybe superificial) errors in the first version of the results and there are poorly explained disqualification of nominee who may have won in the final stage (or affected the outcome) and there is poor transparency/explanation from Chengdu Worldcon.

there could be a whole bunch of genuine mistakes in the data e.g. I think the simplest explanation for the totals not adding up right that Marshall has documented is that the published totals have typos in them. Maybe that explains Babel's weird stats as well. There were confirmed errors in the stats (duplicate names). If there are lots of transcription/cut-and-paste errors then we are all working off faulty information and drawing wrong conclusions about reality. The released data looks rushed and I'm not confident that it reflects the actual data. Conclusion 1: The released data has definite transcription/copy-paste errors
Even so, to explain how the data look requires a lot of errors and the patterns Heather Rose Jones showed do not look like errors from publishing the data but rather errors in the data itself. The strange ratios in the EPH data for Best Novel make little sense as a typo or transcription error. Conclusion 2: As well as errors in how the data is presented, the underlying data appears odd.
Maybe EPH was just run wrong. Without access to raw data it is hard to check. Still, it doesn't look like that. Access to the raw data is unlikely. The Hugo Committee for 2023 hasn't been transparent at the best of times but sharing data from China outwards is legally complex and sharing identified data between Worldcons is legally complex under EU law (and similar legislation) Conclusion 3: Without access to the raw data we will never know for sure what all the issues are.
The simplest, hardest-to-check way to skew the results is to stuff the ballot. That could be done externally to the Hugo committee and there's no way to tell. That would explain some of the patterns but not other issues (eg Babel's EPH values not changing). Conclusion 4: Even with access to the raw data, statistics/auditing can only tell us that the data looks strange and that the strangeness is in the ballots themselves it can't confirm a slate or ballot stuffing.
However...one thing a Hugo committee could do if they thought (but couldn't prove) that the nominations were being ballot stuffed is disqualify a bunch of stuff they thought was dodgy without saying why...except the stuff they did disqualify makes no sense in that way. Conclusion 5: I think the weird disqualifications are a separate issue to the other irregularities.
The ballot data oddities and the disqualification oddities do not work in tandem eg Babel has weird numbers and was disqualified but all the Best Novel finalists had weird numbers and weren't disqualified. Paul's numbers weren't weird and wasn't disqualified. Conclusion 6: There are two unrelated things going on which suggests that no single person or cohesive group did all these things.
The Hugo nomination process was particularly complex this year, with works in two different languages, many new voters unfamiliar with eligibility rules, and technical/legal challenges with dealing with data across China's internet borders. Seperate from the nomination stats, there were delays at each stage of the process and other mistakes (e.g. the release of an incorrect set of finalists). Conclusion 7: circumstances with the Worldcon itself made errors more likely but also made it likely that there would be weird (but legitimate) data that did not follow previous patterns.
The process was complex but also the Hugo committee was inexperienced. Putting aside Dave McCarty's personality, he was clearly busy with many other aspects of the convention (with all its scale and complexity) and much of the work must have had to be delegated. In particular, I suspect the data cleaning stage and eligibility checks would have been challenging (point 7). Major stuff ups with the data may have occurred such as accidental duplication of data, accidental deletion of data or accidental disordering of data. These three kinds of issues do happen when dealing with a data-heavy project (trust me - I've made all three kinds of errors in the past at work but luckily either I spotted it in time or somebody else did). Time pressure, inexperience, weak supervision, novel issues and unsystematic QA checks are all circumstances that will lead to substantial mistakes that poison the whole process. The solution when that happens is to go back to the raw data and start all over again...but people (in my experience) try to "fix" things instead. Conclusion 8: Cock-up and conspiracy aren't alternate explanations. Sometimes, the conspiracy is an attempt to fix a cock-up. We can't tell from the data but the Hugo Committee can and if this what happened can tell us. We also know for certain that errors of various kind were made and have been admitted to (point 2).
The weird eligibility decisions are, regardless of everything else, overt decisions made by somebody who had actual reasons for those decisions. Conclusion 9: We can't know everything that went on but these decisions can and should be explained by the people who made them or by officials of the Chengdu Worldcon committee.
I suspect there may be explosive aspects to those decisions and maybe explaining those decisions create personal or legal liability issues for officials or the Worldcon. Conclusion 10: We still need to know if only to plan to avoid such issues in the future.
Conclusion 11: I do not trust the data and, as awful it is to say this, I don't trust the results of the 2023 Hugo Awards.