The calibration engine has code that works on all of the calibration types pretty much at the same time making small adjustments and moulding the data almost like clay to get it as close as possible to the minimum overall calibration error.
There is no right answer as we’re dealing with human behaviour and you only have to watch yourself and your teammates varying wildly from week to week to realise it’s not an exact science! The goal is to get as close as we can.
The engine separates its efforts into:
- Player calibration - ensuring that each player is correctly calibrated within their pool
- Pool calibration - ensure that each pool is correctly calibrated compared to the other pools
In both cases the engine attempts to set a starting level and then adjust dynamically over time. Note that, as a pool is made up of players, it is the player levels that we are adjusting; both for the starting level and also over time.
Fundamentally, there are three processes running as we work through the results
- Identify the pools - allowing for them to change season by season
- Compare the pools with all other pools
- Adjust the pools - i.e. set the starting level as soon as we can and then adjust dynamically
Identifying the pools is not straightforward as they come and go and change over time. Some of the bigger pools like the county leagues are good examples of pools but they also vary quite a bit over time with new players coming in, others leaving or retiring or, simply, the county deciding to restructure their leagues! It is a complex task to identify all the pools, which players are core to those pools (as opposed to occasional players, visitors etc.) and where the pool boundaries are. As mentioned earlier, clubs are interesting because often the top players play in the county leagues so we have to decide if they are representative of the club or the county… This all has to be done automatically.
Comparing the pools is also a challenge. One of the great things about squash is that there are many players who play in more than one pool such as the county players who play in multiple counties and also the tournament players who travel around the region or further. The trick is to use these players to compare their respective pools.
It turns out that these multi-pool players, being human, are somewhat variable in their playing level as they play from pool to pool but, through all the behavioural analysis we have done, we have worked out which players we can use, which matches are representative. We can even predict their expected effort allowing us to use a lot more matches as, previously, we’d only been able to use the subset of matches that we felt full effort had been applied.
After all this analysis we end up with a sparsely populated matrix, several hundred pools across and down with their pool differences recorded where we have them. This is effectively a massive simultaneous equation with semi-random numbers to work from. The trick is not to attempt to solve it but to minimise the overall error - small adjustments at a time. Large adjustments can cause calibration shock which is to be avoided!
Once we have the pool differences we then need to adjust the pools as best we can. A pool’s starting level is made up of all the players in the pool but as those players come and go over time, we have to be careful which ones we adjust. We even need to assess whether a pool can be adjusted as it may be derived from other pools that have already been calibrated.
The starting levels of both players and pools are worked out and set but after that they can only be adjusted a very small amount each match. So, in order to be able factor in these dynamic adjustments needed over time, we created a level pump. This effectively runs throughout the calibration process adding or reducing each player’s level after each match to make the overall adjustments needed; whether for the player’s themselves, the pool they are playing in or the system as a whole. They must be small enough to be virtually unnoticed but large enough that they have enough effect. A difficult balance. If you look at the match review page you will see, at the bottom, an allowance for this dynamic adjustment.
For a full calibration, once the new results have been read in, the engine is fed all the results from day one, in order, all the way through to the very latest results. This is needed as new feeds connect and their results can go back 5-10 years or more. For a partial calibration, just the last few seasons are re-run ensuring any recent historic result changes are taken into account (e.g. merging player histories). Some form of calibration is run every night to keep up to date with the latest results and updates.