Talent test for archery: Nugroho 2022 study

The authors assemble a battery of tests for spotting potential in archery, and check two things about it: whether a panel of experts considers it adequate, and whether its items hold together as a scale. The battery has eleven items: height, weight and arm span, a kinaesthetic perception test, sit and reach, standing balance, a wall sit, a side-support hold, grip on a dynamometer, VO2max estimated from a shuttle test, and a shooting test.

Seven experts, four coaches and three university lecturers, rated five aspects of the battery on a four-point questionnaire running from "very relevant" to "irrelevant". Agreement, calculated with Aiken's V, was 0.857 for the suitability of the content, 0.952 for the way the individual items are built, 0.809 for the administration procedures, 0.857 for safety and 0.857 for ease of execution. With seven raters and a four-point scale, Aiken's table requires a value between 0.76 and 0.86, so all five figures sit above the threshold.

The battery was then administered to 61 archers from three clubs in Yogyakarta, and on those data Cronbach's alpha across the eleven items was 0.635. The authors conclude that the instrument is valid and reliable. No score appears anywhere in the paper: not the results of the individual tests, not the thresholds that would separate a promising youngster from another, and no comparison with what those 61 archers can actually do on the shooting line.

The study combines a qualitative and a quantitative part in four steps. First a narrative review of papers and coaching books is used to define what is meant by talent in archery and which tests would measure it. The battery then goes to the seven experts through the Delphi method: none of them meets the others during the assessment, their comments go back to the authors, the battery is revised, and the cycle repeats until the experts accept it with no further changes. The final ratings feed into Aiken's formula, which compares the score each rater gives with the maximum the scale allows.

The last step is administration: 28 archers from Selabora FIK UNY, 26 from Blangkon Archery and 7 from Bamboe Roentjing Arc, 61 in all, took the tests, and their scores went into an internal-consistency calculation with Cronbach's alpha, run in SPSS 20. The authors adopt a threshold of 0.60 for calling an instrument reliable and 0.50 for calling it unreliable, and they also cite a source for which 0.5 is enough for research purposes.

About the testing itself the paper says almost nothing: not the archers' age, sex or level, not the protocol for each test, not how many attempts were allowed, not who administered them. The shooting test is listed as a "skill test" with no distance, arrow count or target face.

The 61 archers come from three clubs in a single city, and nothing else about them is reported. Without age, sex and level it is unclear whom the battery is calibrated for, which is a serious problem in an instrument meant to select children.

The figure that gets quoted most, the alpha of 0.635, answers a different question from the one that matters. Cronbach's alpha tells you whether the items of a scale measure the same thing, and the items here are height, flexibility, balance, grip, VO2max and a shooting score: qualities that have no reason to agree with one another, and that are in fact put together precisely because they differ. Even taken on its own terms, 0.635 sits below the thresholds normally used when a decision about a person hangs on a score, and the authors call it good by relying on a threshold of 0.60 while citing another at 0.50.

The validity part is content validity only, that is, seven people agreeing that the tests make sense. There is no predictive validity: we do not know whether those who do well on this battery shoot better now, or will become better archers later, which is the one thing a selection battery ought to demonstrate. There is no follow-up, no retest a few days apart, no norms or cut-off scores, and the paper does not say whether the experts were independent of the group that built the battery. Without scores and without thresholds, the battery as described cannot be applied. These are also conference proceedings, and peer review is not stated.

For someone who shoots, this study changes nothing in training: it does not measure performance and it never relates any of its tests to a score. It stays in the registry as a document on how junior selection is designed and justified in a setting where archery is a flagship sport.

What an attentive reader can take from it is the distance between two meanings of the word "validated". Here it means that seven competent people looked at a battery of tests and judged it relevant, safe and practicable. It does not mean the battery identifies who will become a good archer, because nobody checked. For a coach presented with a list of tests described as scientific, the useful question is the one this paper never asks: has anyone verified that the people who score well on these tests go on to shoot better than the rest?