Machine learning and archery talent: Musa 2019 study

Fifty young Malaysian archers, 37 boys and 13 girls aged between 13 and 20, in development programmes aimed at state and national competition, completed six standard fitness tests and a shooting test: six arrows at 50 metres, after four practice shots, on a range set up as for competition. The tests were handgrip on a dynamometer, vertical jump, standing broad jump, single-leg balance, press-ups in one minute and sit-ups in one minute.

Put together in a multiple linear regression, only two tests are associated with the score: the vertical jump (standardised beta 0.391, p = 0.018) and the sit-ups, which the authors call core muscle strength (beta 0.299, p = 0.026). Grip, press-ups, standing broad jump and single-leg balance come out unassociated. The authors then used hierarchical agglomerative cluster analysis on score, vertical jump and sit-ups to split the fifty into «high potential archers» and «low potential archers», and trained six variations of k-nearest neighbour and a logistic regression to tell the two groups apart.

Comparing the two groups so constructed, the difference is clear on the score (t = 5.870, p = 0.0001) and on the vertical jump (t = 5.725, p = 0.0001), but absent on the sit-ups (t = 0.307, p = 0.760): the core measure, significant in the regression, does not separate the two groups. Among the classifiers the weighted k-NN reaches 82.5 ± 4.75% accuracy, followed by logistic regression at 80.0 ± 5.45%, with Matthews correlation coefficients of 0.49 and 0.40. That same weighted k-NN, however, has a precision of 50%: half the archers it calls high potential are not, by its own label.

The participants came from various development programmes in Malaysia, with written consent from them and from parents for those under 18, and ethical approval from the Terengganu sports institute. A warm-up of 5 to 10 minutes of jogging and stretching preceded the fitness tests. The shooting test was carried out before the fitness tests, not after.

The six tests follow standard protocols. Press-ups and sit-ups, on a mat, for one minute each, alternately, once only: these are what the authors call upper body strength and core muscle strength. Balance is the standing stork test, timed, shoeless, hands on hips, on the ball of one foot with the other foot against the inner knee, until the position is lost. Grip is measured with an adjustable Takei dynamometer, standing, shoulder adducted and in neutral rotation, elbow extended. The standing broad jump is a two-footed take-off onto a landing mat, three attempts, best of three. The vertical jump is measured with a Vertec, with the vanes set to the archer's standing height, three attempts, best of three.

The analysis has three steps. First, a multiple linear regression with the score as dependent variable and the six tests as independent, to select the variables. Second, a hierarchical agglomerative cluster analysis, run in XLSTAT 2014 on the three remaining variables (score, vertical jump and sit-ups), which produces the two classes of high and low potential. Third, the training of the classifiers in MATLAB 2016a: six k-NN variations (fine with k = 1, coarse with k = 100, and medium, cosine, cubic and weighted with k = 10, using Euclidean, cosine or cubic Minkowski distance) and a logistic regression, assessed by fivefold cross-validation on 40 observations, plus ten cases held out of training. The quality measures reported are accuracy, sensitivity, specificity, precision, error rate and Matthews correlation coefficient.

The title speaks of prediction, but nothing is predicted in the study. The high and low potential labels are built by the authors themselves with a cluster analysis that includes the shooting score, and the classifiers are then trained to recover those labels. No archer was followed over time, and no external measure, such as a later result, was used to check who actually had more potential. What the models recognise is a label derived from that day's score, so 82.5% accuracy measures the internal consistency of a procedure rather than any ability to identify talent.

The shooting test is thin: six arrows at 50 metres, a single end, after four practice shots. Over six arrows an archer's score moves a great deal for reasons that have nothing to do with their level, and it is on those six arrows that the regression, the clusters and all the classifiers rest.

The figures carry inconsistencies worth noting. Age is given as 17.0 ± 0.56 years for a sample stated as 13 to 20: a standard deviation of half a year is not compatible with that range. In the regression table the sit-ups have t = 1.726 with p = 0.026, while a t of that size on about 43 degrees of freedom corresponds to a probability around 0.09; the first row of the same table carries the label of the dependent variable instead of the constant. The comparison between the two groups gives a single column of means rather than one per group, and the score value is written as «47.95±5.10.58». The text states that the high potential group's means are higher on every variable, sit-ups included, while the table beside it gives the sit-ups at p = 0.760.

On the modelling side, coarse k-NN is set with k = 100 on 40 training observations, more neighbours than data: its specificity and Matthews coefficient are reported as unavailable, the sign that it put everyone in one class, and its 100% precision has to be read in that light. The model declared best, the weighted k-NN, has 82.5% accuracy but 50% precision and 71.4% sensitivity: with precision at one half, calling it best depends on which measure you look at. The ten held-out cases are almost all high potential, so a classifier that always said «high potential» would get nearly all of them right: ten cases that unbalanced validate nothing. Neither the composition of the two classes in the training sample nor any correction for multiple comparisons is reported.

What remains is a sample of fifty, with 13 girls out of 50, measured once, in one country and one development system. The discipline is not stated. The raw data are attached to the paper, which is open access on PLOS ONE: anyone wanting to redo the calculations can.

For an archer, the useful result is not the one announced in the title but the one that falls away. Of six common fitness tests, handgrip, press-ups, standing broad jump and single-leg balance are not associated with the score once they are looked at together. Single-leg balance in particular is the test many would take for granted in an aiming sport: here it separates nothing. It is the most honest measurement this study offers, because a null result on fifty archers stays a null result even when the rest of the analysis is fragile.

Of the two tests that do associate, the vertical jump is the one that also holds up in the comparison between groups; the one-minute sit-ups appear in the regression and then fail to separate the two groups. The vertical jump measures leg power, and the authors read it as reflecting the capacity to stand still on the shooting line. That is a reading, not a measurement: nobody recorded how steady these archers were while shooting.

None of this serves to select or exclude anyone. The «high potential» and «low potential» labels were built from the score over six arrows shot in one afternoon, by youngsters between 13 and 20, and nobody was followed over the following years to see how they turned out. A model that recognises those labels 82% of the time does not say who will become a good archer: it says that whoever scored more that day also jumped higher. Anyone looking for talent identification tools finds here an example of how easily cross-sectional data can be made to look predictive.

What remains is a point of common sense for anyone assembling a fitness test battery at club level: six arrows are not enough to measure an archer's level, and a fitness test that does not associate with the score does not become useful because it is convenient to administer.