Self-reflection diary and archery: Tan 2016 study

Eight archers from the Singapore national team kept a guided self-reflection diary for five weeks, before and after every training session, and ended up shooting worse than before. They were four men and four women, two from each of the four categories (recurve men, recurve women, compound men, compound women), aged 20 to 24, and were preparing for the South East Asian Games held in Singapore in June 2015.

The comparison is made on the mean of six rounds of 36 arrows before and after: at 70 metres for the recurve archers and 50 for the compound archers, with a maximum of 360 points. Two archers improved, by 0.83 and 1.5 points; the other six got worse, from half a point to 6.83. In percentage terms the range runs from plus 0.45% to minus 2.31%, with five of the eight already starting between 320 and 342 points out of 360. No statistical test was carried out: the authors compare the numbers by eye.

The second half of the study is an interview with each archer, and there the picture is different. The archers rated the diary 7.63 out of 10 for usefulness, and named more points in its favour than against. Among the first, its use as a reminder of their own goals and the things to work on (the most recurrent theme, 35 mentions), the chance to describe the feel of their shooting (31) and its effect on motivation (27), followed by planning ahead, keeping a record, describing their thoughts and having a tool to use straight after training. Among the second, only three: that it took time (35), that it had too many questions (29) and that the questions were repetitive (19).

The authors explain the fall in scores with what came out of the interviews: during the five weeks a new national coach had arrived, and all four recurve archers were adjusting their technique and equipment, as were two of the four compound archers, while another was recovering from an injury. The archer with the largest drop said she had been attending to her shooting form rather than to her scores.

The diary was designed for archery, drawing on comparable studies in other sports and on input from archers and coaches. It is an A5 notebook with thirty sessions, each with five sections: goals (long-term, weekly, daily and those set by the coach), a rating of how the day went on technique, mind and physical condition, a detailed description of movement and thoughts, the coach's input, and finally a reflection on the outcome and on what to do next. The archers were to fill them in daily, with the freedom to decide how much to write.

Before starting there was a workshop of about two hours on planning and using self-reflection, and the first author followed the participants over the first few days to check they had understood. Permission came from the high performance committee of the Archery Association of Singapore.

Baseline performance is the mean of each archer's six most recent rounds of 36 arrows; the final figure is six rounds the participants submitted during the last week of the intervention. The scores come from training and are reported by the archers themselves, and the comparison is made in Excel using mean and percentage differences, with no test.

After the intervention each archer gave a semi-structured interview following a common guide, recorded, in which they described their experience with the diary, scored its usefulness from 1 to 10, proposed changes and said whether they meant to go on using it. The recordings were transcribed (56 single-spaced A4 pages), sent back to the participants for verification and analysed thematically: the text was divided into meaning units, labelled and grouped into themes, with a review by a colleague experienced in qualitative research and a second independent review by the second author.

Eight archers, no control group, five weeks and no statistical test: the authors themselves write that the sample is small and the method of analysis very simple. The most compromised figure, though, is the score, because seven of the eight archers were changing something during those five weeks. A new national coach had arrived at the start of the intervention, the four recurve archers were adjusting their form and equipment, two of the compound archers were too, and one was coming back from an injury. In that setting a drop of one or two points out of 360 cannot be put down to the diary.

The measurement is fragile in its own terms as well. The scores come from training, the archers reported them, and six rounds of 36 arrows are not enough to separate differences of this order: the two positive changes are 0.83 and 1.5 points out of 360, which is less than half a hundredth of a point per arrow. It is not stated whether the rounds before and after were shot in the same conditions, or how far apart.

Compliance is a problem too. The diary had thirty sessions, and the participants completed between nine and twenty-three: nobody used it as intended, and the authors acknowledge that the effort put into the reflection was not the same across the archers. The five weeks, they write, may not have been enough, and they suggest six to eight weeks as a better duration.

On the qualitative side, who conducted the interviews matters. The first author is himself a recurve archer representing Singapore, with nine years of practice and a former vice-president of the university archery club; he designed the diary and interviewed the participants, and the favourable ratings of the tool should be read knowing that.

There are then three things in the paper that do not add up. The abstract speaks of three facilitating factors, while the results list seven. The discussion says the two archers who improved gained 1%, whereas the table gives 0.28% and 0.45%, and says that both the men and the women got worse, whereas one man and one woman improved. The participants are described as having more than five years of elite competition behind them, yet the range given in brackets starts at 2.5 years; and the mean interview length is given as 30 minutes with a range of 31 to 36.

Anyone keeping a training diary will find two things here worth knowing, and neither is a strong finding. The first is that in five weeks, across eight national team archers, a structured diary produced no points: six of the eight lowered their average and two raised it by little more than a point out of 360. The second is that the likely reason is not the diary, because in those same five weeks seven of the eight were changing technique or equipment or coming back from injury. The study does not allow anyone to say the diary made anything worse, and it does not allow anyone to say it made anything better.

What remains is the archers' judgement of the tool, which is interesting because it is specific. They found it useful mainly as a reminder, for picking up the goals they had set and the things they had been working on, and as a place to put the feel of their shooting into words. One archer said the diary meant he did not depend on the coach on the days the coach was away; another said the section on thoughts was most use to him in competition.

Their reasons for not filling it in are just as concrete: it was long, it had too many questions and the questions repeated. Nobody completed more than twenty-three of the thirty sessions, and one of the archers said that when filling it in starts to irritate her the tool loses its effect; another said some questions led her to dwell too much on her shooting problems. These are observations from eight people rather than a design rule, but they are the kind of thing an archer recognises straight away in their own notebook.