Confidence-Based Skip-Lot Sampling

In the US Army, current ammunition production policies require¹ that a sample out of every lot be tested to assess both the performance and safety of the lot. Due to the long history of sustained quality in ammunition development and production, we seek to explore the practice of skip-lot sampling as part of small-caliber ammunition production. Skip-lot sampling allows an organization to reduce the testing burden by not testing every produced lot and should be considered when sustained, high-quality production is observed³. The overall objective of this work is to implement skip-lot sampling to create as much savings as possible while controlling the risk we are willing to accept.

To solve this problem, we first looked at current skip-lot testing procedures commonly used, specifically SkSP-2². This process functions by starting with every-lot testing, where a sample from every lot in production is tested. After i consecutive lots pass a particular test, we then enter skip-lot sampling for that test. During this phase, each lot has some probability f of being tested. This process will continue until a lot is observed as a failure. Once this occurs, we reenter every-lot testing until i consecutive lots pass again, reverifying a sufficiently high level of production quality. We then enter the skipping phase again, and this iterative cycle repeats indefinitely.

While the SkSP-2 process has been successfully used for some time, we wanted to look at whether there might be a better way to both control risk and maximize savings with a different skip-lot sampling procedure. In SkSP-2, there is a constant probability (f) that any given lot is tested in the skip-lot sampling phase of the procedure. We felt that there should be a way to optimize the probability of testing vs. skipping based on the current belief surrounding estimated production quality. Specifically, we believe it matters how close the predicted true non-conforming rate (TNCR – or percentage of bad rounds in a lot) is to the acceptable non-conforming rate (ANCR – or the percentage of bad rounds in a lot that is acceptable). A core premise of our approach is that both the willingness to skip testing a lot and the probability with which to do so must be tied to our belief about the quality of the manufacturing process at a given time. Accordingly, we need a way to predict the TNCR of the next lot in production and determine an appropriate probability for skipping the lot. A second core premise of our approach is the practical consideration that recent test results are more informative than old test results; the quality of a lot produced yesterday is a better reflection of the current quality of our manufacturing process than the quality of a lot produced six months ago.

To calculate the probability that a lot should be skipped, we must first estimate the probability that a lot is bad. We do this by estimating the TNCR with a beta distribution as inspired by the Bayesian beta-binomial approach. In a classic beta-binomial model, we would estimate a binomial proportion (success rate or failure rate) by starting with some beta distribution as a prior. We choose to start with an uninformative prior (α=1, β=1) and then adjust α and β as we encounter successes and failures in testing. The α and β parameters can be thought to nearly represent the number of observed successes (α=successes+1) and observed failures (β=failures+1), such that (α-1)/(α+β-2) is the success rate. A larger sum for α and β denotes a sharper peak in the beta distribution (high confidence about where the true success rate lies), and a smaller sum corresponds to a flatter distribution (less confidence about where the true success rate lies). An assumption of this model is that the quality of each cartridge produced is related to a fixed but unknown parameter governing process quality. We seek a more flexible framework that allows the possibility of an evolving probability due to machine wear and tear, catastrophic component failures, etc. We accomplish this by introducing a decay variable, a notable departure from the traditional beta-binomial approach, to allow this method to give more recent test results greater weight than older information. This appears in Equation 1 where we calculate α from n prior test results (including skipped lots). The β parameter is governed by an identically structured equation that tracks the number of rounds that failed during testing. This variable governs how quickly previous test results vanish from consideration. A simulated example can be seen in Table 1 using a decay rate of 0.8. This shows how the α and β parameters change over time, which determines our confidence in our TNCR estimate. Additionally, the decay variable is the driver behind how skipping lots decreases our confidence in our estimate, ultimately leading us to resume testing at some point.

Table 1: Example data used to calculate the α and β parameters of the beta distribution used in the beta-binomial-inspired approach we employ to estimate the TNCR. Testing a lot increases the α and β parameters, which concentrates the mass of the beta distribution into a smaller interval and thus decreases our uncertainty about the quality of the manufacturing process. This example shows how the decay component of our model effectively decreases the α and β parameters when a lot is skipped. When a lot is skipped, the distribution flattens as both parameters are decreased (see lots 3 and 4 above). A flattening of the distribution yields the desired increase in uncertainty following a skipped test, ultimately increasing the probability of testing the next lot.

Once we have estimated the TNCR (in the form of a beta distribution, not as a point estimate), we break down the risk into two components: the first is the risk that a bad lot passes testing, and the second is the risk that we skip a bad lot. There will always be risk associated with accepting a bad lot based on a sample, as the lot in its entirety might exceed the ANCR while still producing a favorable sample. The probability that a bad lot produces a passing sample is something that we can compute using the binomial distribution. Panel A in Figure 1 shows the probability that we will pass a lot at many TNCRs given certain lot and sample sizes and an ANCR of 0.1. Using the probability distribution for estimated lot quality obtained through our proposed approach, we can combine the two sources of risk to estimate the overall risk of accepting a bad lot as a function of the probability of skipping a lot. The risk that we skip a bad lot is a product of the probability that the lot is bad, which we estimated, and the probability that we do not test it, which we are trying to solve for. By combining these risks as in (2), we get the overall risk of accepting a bad lot.

Figure 1: (A.) Probability of accepting a lot across TNCRs with an ANCR of 0.1. We use the binomial distribution to directly compute the probability of obtaining a sample from a given lot non-conforming rate that would pass a test at the ANCR threshold of 0.1. At any reasonable sample size, there is still risk of accepting a lot that exceeds the acceptable non-conforming rate (ANCR). (B.) Probability distribution for estimated lot quality. Prior test results inform the location and shape of the distribution that defines our estimate of the true non-conforming rate (TNCR). Here, we show how our distribution estimate of the TNCR (with the actual TNCR shown by the dashed line) helps us determine a risk estimate of producing substandard lots.

In Equation 2 we have (1) the probability that a lot is bad, which we are estimating using a modification to the beta-binomial approach (prior successes and failures plus decay), (2) the probability that a bad lot passes the test for quality at a given TNCR, which we know via the binomial distribution, and (3) the probability that we do not test, which we solve for given some level of allowable risk. Thus, the customer determines the maximum allowable risk of accepting a bad lot, and the process described here algebraically updates the testing vs. skipping probability to stay below that risk level. The risk-focused approach presented in Equation 2 requires the novel approach of an evolving skipping probability, and we call this new method SkSP-Bayesian.

To calculate these risks, we take the TNCR (estimated as a beta distribution through a beta-binomial-like approach) and discretize it. While theoretically there is a continuous range of possibilities of the TNCR in a sample, there is a limited number of TNCR estimates that would ever result from a finite sample. This discretization process transforms our probability density function (PDF) into a probability mass function (PMF) with mass assigned to each attainable test outcome given the sample size. With our PMF, we now have the probability that the lot is bad by looking at the percentage of the PMF that is greater than the ANCR (red area in Figure 1, Panel B).

Once this is calculated, we need to find the probability that a bad lot will still pass the test. We use the binomial distribution to calculate the probability that a lot of any given TNCR will pass the test, as shown in Figure 1, Panel A. With the PMF describing our belief about the TNCR, we have the probability that the next lot is a specific TNCR for all TNCRs that are possible based on the sample size. We then take the probability that a lot has a specific TNCR and multiply it by the probability that a lot with that TNCR would pass if tested. This will give us the probability that a single lot at a particular TNCR (specifically, a TNCR greater than the ANCR) will pass the test. We can then sum these probabilities for all possible TNCRs greater than the ANCR to estimate the overall probability that a lot is bad and will pass testing (see Equation 3).

From these calculations, we now have (1) the probability that the lot is a bad lot, and (2) the probability that a bad lot will pass if it is tested. From this point, we can take the customer’s stated risk tolerance for accepting a bad lot and algebraically solve for the probability that we should skip the next lot in production to maximize savings while controlling risk, as seen in Equation 4. This process is then repeated for every single lot in production as new test results are considered (and old results decay further), a new distribution for the TNCR is estimated, a new skipping probability is calculated, and a random skipping decision is made.

We directly compared the results of SkSP-2², SkSP-Bayesian (or SkSP-B, our proposed method), and every-lot sampling across a variety of TNCR profiles. A TNCR profile is a series of moving TNCRs over time to simulate a time series of production quality, perhaps constant and perhaps variable, to learn where different methods might be preferable. The TNCR profiles we provide in the main text as well as in the appendix consider such things as the lot-to-lot variability of production quality (referred to as noise below), a slow degradation of production quality (perhaps as production line components slowly wear out), catastrophic failures in production line components (lots see a jump in poor quality), etc. The possibility of skipping lots makes it important that we quickly catch on if quality has degraded. Under each TNCR profile, we simulated 1000 samples in a process we repeated 500 times to best understand the expected behavior of the different testing procedures. We varied the parameters of SkSP-2 (i,f) based on the most commonly selected values seen in previously published work². In SkSP-B, we kept a constant decay rate of 0.8, and we varied the selected maximum total risk that we consider allowing (risk of accepting a bad lot). We are particularly interested in the average sample number (ASN) as a measure of savings and the Type II error as a measure of risk (accepting bad lots via passed or skipped tests). These results can be seen below in Tables 2 through 4. The TNCR profiles used in these simulations are depicted in Figures 2, 3, and 5 and correspond to the tables above them. We also produced operating characteristic (OC) curves for selected parameter settings for each TNCR profile for a better visual representation of the results. Results for additional TNCR profiles appear in Appendix A, including “Trending TNCR without noise” and “Jumping TNCR with and without noise.”

TNCR Profile 1: Stable, Acceptable Quality

Figure 2: Stable, acceptable quality. Here we compare methods when production quality is both stable from lot to lot and acceptable (TNCR < ANCR).

Table 2: Results from Stable TNCR Profile with No Noise. SkSP-2 achieves greater savings and fewer Type 1 errors than SkSP-B at the indicated parameter settings. SkSP-B commits more Type 1 errors because it tests more lots, and while all lots are acceptable, some test unfavorably. Both skip-lot sampling methods introduce significant savings.

TNCR Profile 2: Variable Quality Without Trend

Figure 3: Variable Quality Without Trend. Here we compare methods when production is on average acceptable, but somewhat erratically produces lots with unacceptable quality.

Figure 4: OC Curve for Variable TNCR Profile Without Trend. Contains results for Every-Lot, SkSP-2 (i = 6, f = .1), and SkSP-B (Risk = .1) evaluating how the probability of acceptance changes as the TNCR changes. This shows the savings when lot quality is better than the ANCR and the significant risk above the ANCR for Skip-Lot testing procedures.

Table 3: Results from Variable TNCR Profile Without Trend. At the indicated parameter settings, neither skip-lot sampling method can effectively prevent Type II errors (accepting bad lots) when there is no temporal trend explaining at least some of the variation in lot quality. Skip-lot sampling introduces unacceptable risk when recent test results are not sufficiently predictive of future test results.

TNCR Profile 3: Variable Quality with Trend

Figure 5: Variable Quality with Trend. Here we introduce a temporal drift in production quality while maintaining a degree of noise in production quality from lot to lot.

Figure 6: OC Curve for Variable Quality with Trend. Contains results for Every-Lot, SkSP-2 (i = 10, f = .2), and SkSP-B (Risk = .01) evaluating how the probability of acceptance changes as the TNCR changes. This shows the savings when lot quality is better than the ANCR and minimal increased risk above the ANCR for SkSP-B as it converges to Every-Lot testing.

Table 4: Results from Variable Quality with Trend. Here we see that all skip-lot sampling methods achieve lower average sample numbers (testing savings), and SkSP-B more aggressively avoids Type II errors by returning to every-lot testing more quickly in the region where the TNCR exceeds the ANCR.

When looking at the results above, it becomes clear that there are significant advantages and disadvantages to the use of SkSP-Bayesian. Overall, it appears that SkSP-Bayesian has slightly less savings in terms of ASN than SkSP-2, but also less increased risk of Type II error. This means that in the two most significant performance measures for skip-lot sampling procedures, SkSP-Bayesian falls in between every-lot testing and SkSP-2.

While this overall generalization appears true, there are some cases in which SkSP-Bayesian clearly is not the best solution, and yet in other scenarios it appears like it may be the best option. Under an initial assumption that all lots will always be good as shown in Figure 2, minimizing testing is the best option as shown in Table 2 where SkSP-2 (i = 6, f = .1) has relative savings of 86.77% compared to SkSP-Bayesian (risk = 0.1), which has relative savings of 76.95%. In cases where our method fails, such as the Variable Quality Without Trend TNCR profile (Figure 3), other skip-lot sampling approaches will also have trouble. The Bayesian-inspired estimator attempts to predict the quality of the next lot in production, so any excessive variance in production quality (no matter the average quality across many lots) will make it very difficult to prevent accepting bad lots. In such a scenario, as seen in Table 3, this can lead to a lot of savings in terms of ASN, but also a substantial increase in Type II error (179.52% relative increase in Type II error with a risk parameter of 0.1, as shown in Table 2). In the Variable Quality with Trend TNCR profile, Figure 5, our method appears to perform very well, with the noise not causing too much trouble in detecting bad lots. Our method, with a risk parameter of 0.01, only has a 2.96% relative increase in Type II error from every-lot testing, and only a 0.48% absolute increase in Type II error. This may prove a reasonable tradeoff for a producer who values the corresponding 22.5% reduction in testing.

Our method, SkSP-Bayesian, allows for the introduction of skip-lot testing while prioritizing a minimal increase in Type II error (bad lots are accepted, either through testing or skipping). This prioritization can be desirable for cases where there is sustained, high-quality production and the potential for significant savings, but at the same time, a high cost associated with Type II errors. In such cases, the implementation of skip-lot testing is reasonable, but a significant increase in Type II error, which may be seen in SkSP-2, might prove unacceptable. The SkSP-Bayesian method also allows for a more comprehensive understanding on the part of the customer, who can set a risk tolerance rather than having to find optimal values for f and i to get their desired balance of savings and increased risk.

Despite this improvement in the reduction of Type II errors, it must be understood that the TNCR profile of production quality is crucial in understanding the best method of testing that should be implemented, as shown in the varying results across different TNCR profiles. There must also be a good understanding of how increases in Type II error can affect the end user, which in our application is the soldier. If any increase in Type II error can have significant negative repercussions for the end user, then the implementation of skip-lot testing in all forms should be avoided.

In further research, it could be important to investigate optimizing the decay rate we incorporated into this Bayesian-inspired estimator with consideration given to the amount of noise observed in the TNCR profile. Profiles with more noise might benefit from having historic data decay faster when there is a more rapid change in production quality. It may also be helpful in future research to generate simulated results using historic test data to validate the advertised benefits and risks to gain buy-in from potential end users.

We’d like to thank our project sponsors at PM-MAS and Lake City Army Ammunition Plant for their continued support.

1. Department of Defense, MIL-STD-1916: Department of Defense Preferred Methods for Acceptance of Product (Washington, DC: Department of Defense, 1996), https://variation.com/wp-content/uploads/standards/mil-std-1916.pdf.

2. Koatpoothon, Pholkris, and P. S. Ayudthya. 2014. “Comparison of Skip-Lot Sampling Plans (SkSP-V vs. SkSP-2).” Songklanakarin Journal of Science and Technology 36 (4): 465–469.

3. Montgomery, Douglas C. 2019. Introduction to Statistical Quality Control. 8th ed. Hoboken, NJ: Wiley.

Here we walk through three additional TNCR profiles and highlight the performance differences between every-lot testing, SkSP-2, and our proposed SkSP-Bayesian method. Specifically, in TNCR Profile A1 we show how each method responds to slowly evolving production quality with minimal variation between consecutive lots. Gradual wear and tear could introduce a slow but consistent degradation in quality. Seasonal weather patterns could potentially introduce a cyclical pattern in some manufacturing processes. In TNCR Profile A2, we look at sudden jumps in production quality, which could result from a catastrophic part failure or a follow-on repair. In this setting, production quality is consistent before and after the jumps (minimal variability). Finally, TNCR Profile A3 shows sudden jumps combined with highly variable production. In this setting, the challenge is to determine whether a particularly bad lot is reasonable given the observed variability or is perhaps an indicator of something more serious.

TNCR Profile A1: Trending TNCR

Figure 7: Trending TNCR with No Noise. Here we introduce a temporal drift in production quality without adding any noise.

Figure 8: OC Curve for Trending TNCR Profile with No Noise. Contains results for Every-Lot, SkSP-2 (i = 10, f = .2), and SkSP-B (Risk = .01) evaluating how the probability of acceptance changes as the TNCR changes. This shows the savings when lot quality is better than the ANCR and the minimal increased risk above the ANCR for both Skip-Lot testing procedures as they converge to Every-Lot testing.

Table 5: Results from Trending TNCR Profile with No Noise

TNCR Profile A2: Sudden Jumps

Figure 9: Jumping TNCR with No Noise

Table 6: Results from Jumping TNCR Profile with No Noise

TNCR Profile A3: Sudden Jumps with High Variability

Figure 10: Sudden Change in TNCR with Noise

Figure 11: OC Curve for Jumping TNCR Profile with Noise. Contains results for Every-Lot, SkSP-2 (i = 6, f = .2), and SkSP-B (Risk = .01) evaluating how the probability of acceptance changes as the TNCR changes. This shows the dramatic savings when the TNCR is below the ANCR. This also demonstrates the significantly reduced risk of Type II Error that SkSP-B has over SkSP-2.

Table 7: Results from Jumping TNCR Profile with Noise