Arrow quality and neural networks: Jeong 2014 study

Two electronic engineers at Pusan National University take on a factory problem: how to judge, at the end of a production line, whether an arrow is good. Today, they write, it is done with the experience of the hunter, the technician or the expert, and more objectively with the density of impact points obtained by shooting many times. What is missing is the link between a shaft's measurable characteristics and how the arrows behave on the target.

So they build two things. The first is a frame that reads where the arrow passes: two line lasers on two sides of a square frame, rows of photodiodes on the opposite sides, and when the arrow passes its shadow lowers the voltage of some photodiodes. From the centroid of the fallen voltages the shadow's centre on the two axes is recovered, and from similar triangles the position. The frame is calibrated with a plate drilled at one-centimetre spacing, and the mean error falls from about three millimetres to half a millimetre.

The second is a model predicting an arrow's performance from two of its characteristics, straightness and spine, both measured over a twenty-eight-inch span, the spine with an 880-gram weight at the centre. Seventy-nine arrows, a neural network with two inputs, two hidden nodes and one output, and a test root-mean-square error between 1.51 and 1.71 millimetres.

The output, though, is the weak point, and the article says so itself. Each arrow is shot three times at twenty-five metres with the nock turned a hundred and twenty degrees, the three impacts form a triangle, and the number derived from it is the mean absolute deviation between the lengths of the sides. One reads that this number is zero when the three points coincide or when they form an equilateral triangle. Those are two very different things: three arrows in one hole and three arrows thirty centimetres apart arranged as an equilateral triangle get the same perfect score. The index does not measure how tight the group is, it measures how regular it is.

The measuring system. Two line lasers are mounted on two sides of a square frame and opposite each is a row of photodiodes, converting the red light into voltage. When the arrow crosses the frame it casts a shadow on each row, the voltage of the covered photodiodes falls, and the shadow's centre is computed as a centroid weighted by the voltage drops and the photodiode index. The shadows on the two axes give two coordinates, and by similar triangles the arrow's position follows.

The electronics: twenty-four channels, 74HC4051 eight-input analogue multiplexers driven by three digital bits from a National Instruments 9401 module, and acquisition with a 9205 module at twenty samples per millisecond per channel. A trigger signal from the launch stand starts one second of acquisition, and with the multiplexer each photodiode is read two thousand five hundred times a second. A half-metre arrow at three hundred kilometres an hour crosses the plane in six milliseconds, so each photodiode is sampled about fifteen times during the passage.

The calibration is the best-made part: a plate with holes a centimetre apart mounted on the frame, the arrow inserted in each hole, and a neural-network correction between the holes' coordinates and the measured ones. The mean error goes from about three millimetres before correction to half a millimetre after.

The model's inputs are two. Straightness is how far the shaft's centre lies off the straight line, measured by resting the arrow on two points twenty-eight inches apart. Spine is the deflection under load, measured the same way with an 880-gram weight hung at the centre. In the data table straightness runs from 0.2567 to 0.5942 and spine from 0.1797 to 0.9297, with no units declared.

The output is the mean absolute deviation between the sides of the triangle formed by the three impacts, with the nock turned a hundred and twenty degrees between shots so that the direction in which the shaft flexes changes. The measuring frame is twenty-five metres from the stand. The values run from 1.53 to 10.8 millimetres.

The model is a feed-forward network with a two, two, one structure and a learning rate of 0.01, and the seventy-nine data points are divided into three cases between training and test. The results are three response surfaces and a table of root-mean-square error: in training 1.68, 1.77 and 1.66 millimetres, in testing 1.71, 1.51 and 1.52.

The problem lies in the variable the model tries to predict, and the article writes it out without noticing. The mean absolute deviation is computed between the lengths of the triangle's sides, not between the points and their centre: so it is zero both when the three impacts coincide and when they form an equilateral triangle of any size. Three arrows in one hole and three arrows at the corners of a thirty-centimetre equilateral triangle have the same value. Since what matters to anyone buying arrows is how tight the group is, the index does not measure the thing it was built for, and the rest of the work, the neural network and its response surfaces, predicts that index.

Within the same article the quantity has two different names. The English abstract calls it mean absolute distance, the body calls it mean absolute deviation, and those are two different measures: the first, the mean of the distances between the three points, really would measure the group's size, the second does not. The declared property, zero for an equilateral triangle, identifies the second.

Then the comparison that would give the numbers meaning is missing. The test root-mean-square error lies between 1.51 and 1.71 millimetres, on an output running from 1.53 to 10.8 in the data. To know whether the network predicts anything, one would need the output's dispersion across the seventy-nine cases, which is not reported, or a comparison with always predicting the mean, or a coefficient of determination, or a linear regression set against it. None of the four is there. From the table's ten visible rows, which are the first five and last five of a list sorted by spine and so the extremes, the output's standard deviation is around 2.9 millimetres: if that value holds for the whole set, an error of 1.6 leaves out an important part of the variability, but on a subset chosen for its extremes the estimate cannot be used.

Of the seventy-nine data points, how they were divided is not known. One reads that they were classified into three cases between training and test, without saying how many to each part or by what criterion, and no cross-validation scheme is described: with only three results and an undeclared procedure, the possibility that the split was chosen remains open.

Two measures out of five. The introduction lists length, weight, spine, overlap and straightness among the characteristics that matter, and laments that the link with the distribution of impacts is little studied; then only straightness and spine enter the model. Length, weight and overlap drop out with no explanation, and since spine varies across the whole commercial range it is very likely that mass varies too, that is that an undeclared confounder sits in the model.

The units are absent. In the data table straightness and spine are bare numbers, and the only column with a unit is the output, in millimetres. Spine measured as deflection under an 880-gram weight is usually expressed in thousandths of an inch, and values from 0.18 to 0.93 would in that case cover the whole range from very stiff to very soft; commercial straightness is quoted in thousandths of an inch and values from 0.26 to 0.59 in inches would be visibly bent shafts. Without the unit there is no knowing what arrows are being talked about, and in particular no knowing whether they are arrows an archer would find in a shop.

Three shots per arrow, and who shot them is not known. Each arrow's triangle rests on three impacts, which is the minimum for a triangle and very little for estimating anything; the sequence is not repeated, so the repeatability of the output measure is unknown, and it is the same missing comparison, between the test's noise and the differences between arrows. Of the bow nothing is written, neither draw weight nor type, and whether the shots were taken by a machine or a person is not said: there is a launch stand with a trigger signal, which suggests a machine, but it is not described.

Finally the conclusion goes beyond the model. One reads, in the results and again in the conclusions, that with the proposed model one can choose the arrow with the spine suited to one's own bow. In the model, though, there is no bow: the inputs are straightness and spine, the bow is one, fixed and undescribed, and a model trained on one bow cannot indicate the spine for a different bow. What the work can do is rank the arrows coming off a production line with respect to that bow, which is another thing and no small one.

This is factory work, not field work, and read that way it has two things to give.

The first is the measuring frame, which is the good idea: two line lasers and two rows of photodiodes on a frame, and the position of the arrow in flight recovered from its shadow. Calibrated with a plate drilled at one centimetre, it reaches half a millimetre of mean error, starting from three. For a club or a maker wanting to measure where arrows pass without a high-speed camera, the scheme is complete and the components are common.

The second is the method of shooting with the nock turned. Shooting the same arrow three times, turning the nock a hundred and twenty degrees, changes the direction in which the shaft flexes, and so tests the arrow in three orientations instead of one. It is a test an archer can do unaided, with no electronics, and it says something different from shooting three different arrows: if the three impacts move a lot as the nock turns, that shaft is not symmetrical.

What should not be taken is the index with which the article sums up those three impacts. It is the deviation between the lengths of the triangle's sides, so it is zero even for a large equilateral triangle: it rewards regularity, not precision. Anyone wanting to redo the test should do the right calculation, the mean distance of the three points from their centre, which is the group's mean radius and takes two lines to work out.

Nor should the conclusion's headline be taken, that the model allows one to choose the spine suited to one's own bow. The bow does not enter the model: it is always the same one, and it is not described. The link the article finds between straightness and performance, that the closer straightness gets to zero the better the result, is plausible and serves as a reminder: among a shaft's characteristics, straightness is the one that cannot be tuned and cannot be compensated, it can only be chosen good.

A last note for anyone reading production data: in this batch the measured spine varies across the whole range, from very stiff to very soft, and straightness over a wide range. That is why the article exists, because a line produces shafts that differ from one another, and why a set of arrows should be chosen by measuring it rather than by reading the number printed on it.