Neurotracker and archers' scores: Komarudin 2021

Twenty archers from an Indonesian university club were split at random into two groups of ten. For four weeks, across twelve sessions, one group worked with NeuroTracker, a computer task in which several objects moving in three dimensions have to be tracked with the eyes; the other followed the club's usual training, meaning a warm-up, shadow shooting without a bow, concentration drills staring at a candle flame and following a movement, and shooting at the distance set by the programme.

Measured by a 30 metre shooting test before and after, both groups improved. The NeuroTracker group went from 265.70 points, with a standard deviation of 24.85, to 298.60, standard deviation 21.77, a gain of 32.90; the control group went from 259.80, standard deviation 13.84, to 279.50, standard deviation 12.18, a gain of 19.70. Within each group the improvement is significant, with a t of 11.961 in the experimental group and 9.229 in the control. The comparison between groups gives a t of 3.791 with a reported significance of 0.01, and the authors conclude that NeuroTracker worked better than the usual training.

A pre-test post-test design with a control group. The university club had forty archers; twenty were selected at the researchers' judgement, on two criteria, having shooting skill and having competed, and were then assigned at random to one of the two groups. The treatment ran for four weeks, twelve sessions, with one shooting test at the start and one at the end.

The experimental group worked in a quiet and nearly dark room, seated upright on a chair 1.33 metres from a 50 inch 3D television, wearing 3D glasses, following the NeuroTracker manual's protocol: sessions of eight seconds, twenty repetitions per meeting. The control group did the club's routine training. Performance was measured by a 30 metre shooting test, and the data were handled with t-tests in SPSS 24, after checks for normality and homogeneity: a paired test within each group and an independent samples test to compare them.

Ten people per group, no sample size calculation, no confidence intervals and no effect size: only t values. As far as the text goes, whoever administered the shooting test knew which group each archer belonged to, and nothing is said about the 30 metre test itself: how many arrows, at what target face, over how many ends, under what conditions. A total between 260 and 300 suggests thirty-six arrows, but that is an inference rather than anything the paper states.

The two treatments are not comparable. The NeuroTracker group did twenty repetitions of eight seconds, which is a few minutes of a seated task in the dark; the control group did a full session that included shooting. Whether the experimental group continued its own shooting training alongside NeuroTracker is not stated, and without that we do not know what was compared with what. The final comparison is unspecified too: we do not know whether the between-groups t concerns post-test scores or gains, and since the two groups started from different scores, 265.70 against 259.80, the two readings do not give the same answer.

Nothing is measured of what NeuroTracker is meant to train, that is attention or visual tracking: only the score on the target, before and after. There is no follow-up. The choice of twenty out of forty was made at judgement, so the randomisation covers only the split into groups, and the archers all come from one university club. The p value for the final comparison is given as 0.01, whereas a t of 3.791 on eighteen degrees of freedom corresponds to a smaller figure.

For an archer this study is not enough to support the claim that a visual tracking exercise improves scores. The signal runs in the direction the authors describe, but it comes from twenty people, with a control group doing something else for a different length of time, with no detail about the shooting test and no measurement of the attention the method is supposed to train.

One thing, though, shows clearly in the numbers and holds beyond this study: the control group improved too, by 19.70 points at 30 metres over four weeks. Club archers who train and are measured twice get better, and anyone wanting to show that a method works has to beat that improvement, not their own starting point. Here the difference between the groups is 13.20 points, less than the standard deviation of the experimental group's starting scores: a difference of that size, with ten archers per group, is consistent with two or three arrows landing elsewhere.