Measuring Improvements in Failure Rates
For example, instead of measuring in improvement in the strength of material used for bolts you need to measure an improvement in the rate of bolts that pass a destructive test.
The old rate might be clear from past data collected over some time (maybe years, and the actual failure rate is known with considerable accuracy).
However, with only a relative small sample from after changes have been made it will be difficult to determine the new failure rate.
If you test 100 pieces and none fail, then what is the new rate? If one had failed, would that mean it was 0.
01? The fact is that there will always be uncertainty in any failure rate that is based on testing.
The only way to reduce this uncertainty is to run more tests.
But how much uncertainty is there and how many tests are required before the uncertainty is low enough? Ideally, you want to know the distribution of the failure rate given the number of pieces tested and the number that failed.
The failure rate is like the risk we wish to understand and the distribution is like the uncertainty associated with that understanding.
There is a distribution that can be used for just this.
It is the Beta distribution.
This is how you use the Beta distribution with experimental data to determine the distribution of (and changes in) the failure rate.
Steps to generating a distribution for failure rate:
- Run tests on your N pieces
- Record the number of failures n
- Set the alpha parameter of the Beta distribution to n+1
- Set the beta parameter of the Beta distribution N-n+1
- Plot the distribution
- Generate distribution for each set of data (for example one set before a change was made and one for after or one for each of a number of proposed treatments)
- Plot each distribution on the same set of axis
- Use a subjective assessment to decide if one distribution is different from the other
However, be aware that this assumes a Normal distribution for each failure, and the calculated outcome is an approximation.
By using the above, you are able to easily evaluate if any action that has been taken has an affect upon failure rate.