The original calibration algorithm was pretty good but, after nearly 20 years of results, it was showing some areas that clearly needed some investigation. In particular:
- The pro men were too low
- The pro women were too high
- The historic levels were too high (those more than 10 years old)
- Some counties were too low
- Some clubs were too high
- There were some oddities in the junior levels
We had also received quite a bit of feedback on the better players being penalised for holding back on their lesser opponents and, although we’d allowed for that, it seemed we needed to do more.
This summer, we spent a great deal of time and effort updating the SquashLevels calibration engine to address these problems and also beef up our behavioural modelling for even greater accuracy. We’ve had 1.6 million results to learn from and it’s been a fascinating learning, all of which has been dialled into the algorithm. There really isn’t a calibration engine like this for squash, or any other racket sport for that matter.
We have been busy:
- Developed behavioural profiles for both effort and weighting for players of all levels and all match types. This allowed us to use many more cross-pool results and be better able to filter out rogue results.
- Developed the calibration level pump that allows a more dynamic adjustment of players and pools over time
- Using our behavioural modelling and level pump we have been able to develop a near ‘lossless’, drift-free algorithm that lets us compare levels over very long periods of time.
- Improved the accuracy of calibrating results with no points scores though it’s still significantly better if you have them.
- Specifically detect and allow for players on meteoric rises so that we can reduce the impact on their opponents as they work their way up.
- More accurate (typical) assumptions around what happens to juniors if there are periods (sometimes multi-year) where the system has no results for them. Also for masters and retired pros.
- Updated the amount of change any player can have per match and per tournament based on level and match history.
- Created a provisional level for those players with just a few results in their history and are indicated with a (P) in their profile and rankings. The purpose of this is to allow players with a provisional level to play friendly matches with players who have a non-provisional level in order to establish their own level but, very importantly, the match does *not* affect the level of their opponent. We’ve had feedback that these matches are avoided to protect their level! Not so now. Please support new players if they ask for a match to get them started. Note - you need to play ‘properly’ in order to give them an accurate level!
- A more lenient approach to better players giving their lesser opponents a runaround. With the improved behavioural modelling and analysis from thousands of example matches we have split the outcomes into four bands:
- The better player actually played better than expected - they go up as normal, their opponent goes down.
- The better player played down as expected - no change for either player.
- The better player played at a lower level than expected but within an acceptable range. No change for the better player but their opponent gets something for pushing the better player hard.
- The better player played below the ‘acceptable range’. Their level goes down a little (it’s still damped) and their opponent is rewarded.
- Related to the above point is the detection of exhibition matches. This is focused on PSA players playing league matches - even against other PSA players - as well as other cases. These matches are nearly always exciting 3-2 encounters to keep the crowds entertained but it doesn’t help the accuracy of the system! Levels are not changed for exhibition matches unless their opponent is lower level (it’s all relative!) and they do particularly well in which case they are rewarded at least a little.