STATISTICS - ES, SD, Testing - Understanding the numbers

[DaveMc]

There has been a great deal of discussion lately on the pros and cons of ES, SD, stats etc. The main issue I see is that a lot of shooters aren't statisticians (or rocket scientists or ballisticians) and yet we are constantly exposed to statistical "results". Whilst we may be great shots, have great loads, do everything else right there can be a lot of misinterpretation of the data. This is often based on experience that we (or our friends) have had (sometimes extensive). Rather than argue this out backwards and forwards I thought I would throw up some readings and show the possible interpretations that can come from it.

Here is a string of 30 velocity readings. Without going into detail lets just say every effort was made to remove barrel temp, fouling and other issues so as to provide good readings of what the load is really doing TOTAL ES is 39 and SD 9.8. This is not a bad load but certainly way off holding the super V at 1000!

2950, 2953, 2950, 2966, 2931, 2962, 2958, 2947, 2960, 2950, 2944, 2944, 2951, 2948, 2970, 2934, 2937, 2962, 2945, 2949, 2953, 2933, 2945, 2954, 2937, 2968, 2942, 2954, 2964, 2943

Here are the results of ES every 3 shots (3, 34, 12, 6, 22, 28, 8, 21, 31, 20)

every 5 shots (35, 15, 27, 28, 21, 26) and every 10 (35, 36, 35)

Before discussing SD (more complicated) Lets dwell on the benefits of testing ES and what our sampling means.

First of all lets deal with the 3 shot groups (and I will point out this same theory applies to group testing as well as velocities). Please note that our ES every 3 shots are extremely variable. I have seen many many times where shooters jump up and down about an ES of 2-5 on 3 shots - does it mean the load is brilliant??? - absolutely not. This is the same load that also produced an ES of 34 over 3 shots. Just the random fall of the numbers allows a few "middle ones" to align fairly regularly. The worst readings you get will be closer to what you can expect. Ignore the best

Now look at the 5 shot groups. - ES ranges from 15-35. Better but same applies. You would still be thinking that 1 or 2 of your loads could hold super v height (far from the case)

10 shots are much better but still slightly underestimate where your "wild ones can go".

I will leave this here for a while for people to dwell, discuss and then come back later to discuss sd.

[IanP]

http://riflemansjournal.blogspot.co.uk/ ... oters.html

I dont know if you saw this post but it explains it all pretty well!

Ian

[DaveMc]

Thanks Ian, Yes I have seen it. Most people in the sport glance over it and can't get the detail even though it is essentially what we have here. I thought I would just simply give an example that people can see the issues rather than try and explain the detail behind it. The shorter the better.- it might help.....

[Brad Y]

I can see the point very clearly. As with any testing, repetition is key. Shooting 3 shots doesnt constitute repetition. Shooting 10 or 15 gives a better overall representation.

The link below was a little long winded for simple folk like me

[dmitri]

http://riflemansjournal.blogspot.co.uk/2010/07/statistics-for-rifle-shooters.html

I dont know if you saw this post but it explains it all pretty well!

Ian

I'm following up your discussion re projectile speed variation where you are trying to analyse the projectile speed variation within significantly better than 1% accuracy (better than 30fps at 3000fps velocity).

I have a feeling that shooters assume that their measuring equipment is absolutely accurate. I believe it’s absolutely essential to understand the technical limits of the instruments used for this exercise, the principles they work on, the methods employed for the measurements and how external conditions and variation in measurement subject can affect the measurements. Also how calibration of the unit may change with time, light, muzzle blast shockwave and why.

For this particular test you are doing one chronograph characteristic of absolute importance is the measurement repeatability.

I tried to find any technical characteristics which specify the measurement accuracy, resolution, drift and repeatability of the projectile speed measurement for a number of devices available on the market but was unable to.

Based on the principles of operations of these devices, available on the consumer market, I do not believe they can reliably provide repeatability better than a few percent of measurement value. (remember you want to measure projectile speed distribution with significantly better than this, accuracy)

I would recommend before you make any conclusions about you projectiles, powder load, cases etc. check the measurement repeatability and accuracy with manufacturer of your chronograph. You may be in for a nasty surprise.

Dmitri

[DaveMc]

Dmitri,
We have plotted heights against measured velocity readings at 1000 yards for various chronographs and with some simple ballistics have been able to deduce that the magnetospeed readings we have been getting on some tested samples have an sd <1.5 fps (so expect readings to generally be within 3-6 fps - sometimes out to 9-10). This also includes (but not limited to) complications such as angular accuracy of the rifle and BC inconsistency etc so suspect the chrono to be even better than this. The PVM is not far behind.

I will not name and shame other brands of chronies but I have seen error values in the hundreds of fps in more than one other brand. Even with IR screens, background light intensity and gain settings can have a huge effect on reading reliability.

Whilst it is a worthwhile point I ask if we can reserve this discussion for another topic as I don't want to complicate the simple message of probability and sampling error (regarding sample size) here please. And this message can be transferred directly to sampling and measuring group size, height or whichever other measuring system you wish to use - not just velocity spreads.

[dmitri]

Dmitri, Believe it or not we have done exactly that (to a point) by reverse engineering plot points vs velocity readings and angular accuracy. Within reason I can say the Magnetospeed variation +/- bullet bc inconsistency effects comes in as a normal distribution with sd <1.5fps!!! (so expect readings to generally be within 3-6 fps - sometimes out to 9-10. considering this is a pretty good expectancy for bullet variation then I suspect the chronograph to be much better than this. The PVM is not far behind!

I will not name and shame other brands of chronies but I have seen error values in the hundreds of fps in more than one other brand. Even with IR screens, background light intensity and gain settings can have a huge effect on reading reliability.

This is really good result. However even on their website http://www.magnetospeed.com/pages/reviews-tests I can see much larger errors (12fps in one out of 7 shots) at given conditions... Base on this data every 7th shots already can be measured with +/-12fps error. At 1 sigma level the variation will be approx +/- 5 fps (spread of 10 fps). So even with this optimistic results I would not relay on this device to measure the projectile speed variation (!!!) as the measurement equipment uncertainty contribution is already almost the same as the expected measurement value (variation of projectile speed).
It can be perfectly used, for example, for average speed measurement vs. powder load those...

Again people must be aware about the measurement equipment limitations.

P.S. I'm not using chronographs so I'm not advocating any of them - just expressing my personal opinion.

added later: Just in simple terms: for example if shooter expect to see reduction of projectile velocity variation by selecting case to batches, and this selection will reduce variation by lets say 5fps, he will be unable to see it on background noise of the equipment repeatability error by simple method - firing a few shots.

[DaveMc]

Dmitri - they compared this with the oehler system - the Oehler is not bad but not as accurate (trust me) so I would suggest most of the error is in the measurement of the measurement! Go down the page another line and check out the high speed video measurement.

BUT CAN WE PLEASE RESERVE chronograph inaccuracies for another thread. The point I am trying to make here is not the readings I got or how I got them. The readings are made up of errors, whether they be measurement errors or other is irrelevant. The point I am making is random chance will mean sampling errors that have nothing to do with measurement accuracy. A clear point that has been discussed 100's of times over but not understood by many - Why you cannot take much out of 3 point chronograph measurements, group sizes or any other sampling method. It just doesn't work. It can only indicate that something might be there - that will need further exploring. For statisticians this is called false positives and are very common in shooting and load development.

e.g. the "optimum charge weight" load development technique. 3 shot round robin load development - looking for one tight group immediately after a scatter. or similarly with chronograph readings. You can do it but there is a great chance any individual tight group you find is not necessarily the answer. It needs further confirmation. Ladder tests, individual or three shot or even 5 shots are just "exploring the water". A whole different story if you have 3 or four tight groups in a row.

On the added later comment. You would not see this anyway (even if measurement was perfect -which I am not advocating it is) because the measured normal variation we have as a final effect is a complicated overlay of various normal distributions of many factors, some real, some in measurement. The reduction in sd of one particular element of this will not correspond to a similar reduction in overall sd. It is not cumulative. This is an important point and one in which many people disregard the "testing" of a particular element as - "It did not reduce my group size (or velocity variation) so has no effect". The particular example you mention is pertinent as we have done a great deal of testing on this in a very scientific manner and hope to write something up on it after Raton.

[saum2]

DaveMc
What effect would the ambient temperature & pressure have on ES & SD.
Say for instance, a cold morning of, 8C and afternoon of 20C when testing.
Geoff

[DaveMc]

Geoff That is a good question and I am no expert on this at all but will give you my take (again a little off topic - maybe Alan might move some of these?).

I have lots of barrels here and they all behave differently. Some like to run hot, others go well up to shot 10 and fall apart. Some will start off well in the morning - others I have to "preheat" (along with the ammo).

All powders are temperature sensitive - some more so than others. All barrels have different coefficients of friction. This can be affected by number and size of grooves, finish, age etc. They also have differing diameters (some start off a few ten thousands larger at chamber than at muzzle - others are similar all the way through) and these things can change over the age of a barrel and at different ambient temperatures. All these (plus more) seem to affect the shape of the pressure curve and the way the powder burns. Thus some will burn more consistently at low temps, others hold together well at the high end.

So to answer your question - yes I believe temperature can play a big part, as can the timing between shots and each barrel will be different. BUT I don't think atmospheric pressure has much to do with internal ballistics. Most (light screen) chronys however do suffer from ambient light affects which vary greatly during the day. Be wary of using one when sun is directly overhead or alternatively in low light or changing light conditions.

[Craig McGowan]

Dave with reference to your original post, from my limited knowledge on the subject, ES is only two data points. When compared to SD, ES is statistically irrelevant.


Craig.

[DaveMc]

Yes Craig, I agree.

And whilst we are using velocity readings here - I point out again the same principle goes to group size or elevation measurements.

The issue is pretty obvious above. ES only represents the worst shots so far and tells you very little about what to expect in the next shot or indeed next 10 unless you use a very large sample size. ie ES of small samples is ALWAYS smaller than the population or total group velocity spread - sometimes (actually quite often due to simple probability that most lie near the centre) very significantly. ES of 3 shot groups in velocity spread, group size or elevation is basically useless unless it is accompanied by adjacent groups doing same thing (ie resolving power comes in having 6, 9 or 12 tight shots)

e.g. In the above 3/10 samples had 3, 6, 8 fps spread when the true expected velocity spread in competition would be more like 35-40 fps. MILES OUT AND VERY MISLEADING.

Imagine if you were doing a three shot round robin load development (with varying loads) - The same thing would happen and you would have thought the "TUNED LOAD" was at those points. Whether you were measuring group size or velocities or elevation the principle remains the same. Similarly when doing a ladder - groups can coincide very easily by pure random "normal" variation.

My favourite saying when someone has a high or low 5 is to say "that is "normal"'.

If you have 5 or 6 shots per range holding super v elevation then you will always see one or two venture into the 5 ring. If you have 8-9/10 holding x ring then correspondingly far fewer points will be lost to elevation.

Righto back to the packing

[Norm]

Dave,
Is there any point burning a barrel out trying to obtain ES and SD with a gun shop quality chronograph?

Maybe just use the chronograph to determine your average velocity and then hit the range.

Maybe powder would be better spent practicing at long range and checking for vertical dispersion on the target?

A sloppy waterline = more load developement.
Tight waterline = right to go.

[AlanF]

Norm,

I'll jump in with my 2 cents worth here.

Its good insurance to minimise your velocity variation in case your tune goes out of its node. Normally a tuned setup will tolerate reasonable velocity variations. However velocity sweet spots are only so wide and they can change up or down for various reasons, so if your velocity variation is low, you're less likely to go outside your sweet spot.

I actually test velocity variation at home without a target. When testing on the range I usually have a chrono but only to check whether high or low fliers coincide with high or low velocity.

But I look forward to what Dave has to say. His ability to tune a rifle has been speaking for itself lately.

Alan

[DaveMc]

Mostly agree with you Norm - but I believe it is not worth using them to find your average velocity either - cheap light screen chronies can vary hugely when trying to determine average velocity. From middle of day to late afternoon or cloud cover or even different days I have seen variations up to 150fps on something that claims 1% accuracy (30 fps). You might think you are doing 3000 fps when it is really only 2850-2900. I found with mine that they worked best in afternoon with no direct sunlight but not too dark. Some are using lit boxes etc to maintain consistency which does help immensely. I am not arguing in any way that the best (and cheapest all round) confirmation of loads isn't on the target. The magnetospeed and PVM have raised the bar here though. If you actually want to save your barrel you should "explore" your loads with small samples and then hone in on the promising ones for 10 shot groups - these may as well be done at the range on club days but beware the range elevation effects of the day messing with your mind. What can be equally wasteful of a barrel is people just going out with "any load" and shooting a whole day of rubbish. There goes 25-50 wasted rounds too.

BUT what I am trying to do is explain this theory in terms of targets as well. Understanding group sizes and distribution on the target is EXACTLY the same principle as ES and SD with chrony. Whether you look at ES and SD of group or ES and SD of vertical in mm, inches or minutes it is exactly the same as dealing with ES and SD of velocity readings.

If your 10 shot range group has 6-7 in x ring height and the others in 6 ring elevation (bar the odd flyer) then this is equivalent to having a vertical sd of around 0.25moa. If 8-9 out of 10 hold the x you may have vertical sd of 0.15-0.2 moa. If almost all hold the x ring it should be better than 0.125 moa sd.

There is BIG difference in the above in the scoring ability of these rifles although it may not seem it at the range. Also it is unlikely you will tell the difference with a single repetition of 3 shot groups measuring ES (group size or elevation) although you may be able to rule out some shockers.

Of course the more accurate the rifle/barrel is in general the lower the sd and the easier it is to hone in on the "tuned" areas.