OIV-MA-AS1-08 Reliability of analytical results

Data concerning the reliability of analytical methods, as determined by collaborative studies, are applicable in the following cases:

1. Verifying the results obtained by a laboratory with a reference method
2. Evaluating analytical results which indicate a legal limit has been exceeded
3. Comparing results obtained by two or more laboratories and comparing those results with a reference value
4. Evaluating results obtained from a non-validated method

1. Verification of the acceptability of results obtained with a reference method

 

The validity of analytical results depends on the following:

• the laboratory should perform all analyses within the framework of an appropriate quality control system which includes the organization, responsibilities, procedures, etc.
• as part of the quality control system, the laboratory should operate according to an internal Quality Control Procedure
• results should be obtained in accordance with the acceptability criteria described in the internal Quality Control Procedure

Internal quality control shall be established in accordance with internationally recognized standards, such those of the IUPAC document titled, "Harmonized Guidelines for Internal Quality Control in Analytical Laboratories."

Internal Quality Control implies an analysis of the reference material.

Reference samples should consist of a template of the samples to be analyzed and should contain an appropriate, known concentration of the substance analyzed which is similar to that found in the sample.

To the extent possible, reference material shall be certified by an internationally recognized organization.

However, for many types of analysis, there are no certified reference materials.  In this case, one could use, for example, material analyzed by several laboratories in a competence test and considering the average of the results to be the value assigned to the substance analyzed.

One could also prepare reference material by formulation (model solution with known components) or by adding a known quantity of the substance analyzed to a

sample which does not contain (or not yet contain) the substance by means of a recovery test (dosed addition) on one of the samples to analyze.

Quality Control is assured by adding reference material to each series of samples, and analyzing these pairs (test samples and reference material).  This verifies correct implementation of the method and should be independent of the analytical calibration and protocol as its goal is to verify the aforementioned.

Series means a number of samples analyzed under repeatable conditions.  Internal controls serve to ensure the appropriate level of uncertainty is not exceeded.

If the analytical results are considered to be part of a normal population whose mean is m and standard deviation is s, only around 0.3% of the results will be outside the limits m ± 3s.  When aberrant results are obtained (outside these limits), the system is considered to be outside statistical control (unreliable data).

The control is graphically represented using Shewhart Control Graphs.  To produce these graphical results, the measured values obtained from the reference material are placed on the vertical axis while the series numbers are placed on the horizontal axis.  The graph also includes horizontal lines representing the mean, m, m ± 2 (warning limits) and m ± 3 (action limits) (Figure 1).

To estimate the standard deviation, a control should be analyzed, in pairs, in at least 12 trials.  Each analytical pair shall be analyzed under repeatable conditions and randomly inserted in a sample series.  Analyses will be duplicated on different days to reflect reasonable changes from one series to another.  Variations can have several causes: modification of the reactants composition, instrument re-calibration and even different operators.  After eliminating aberrant data using the Grubbs test, calculate the standard deviation to construct the Shewhart graphs.  This standard deviation is compared to that of the reference method.  If a published precision level is not obtained for the reference method, caused should be investigated.

The precision limits of the laboratory should be periodically revised by repeating the indicated procedure.

Once the Quality Control graph is constructed, graph the results obtained from each series for the control material.

A series is considered outside statistical control if:

(I)a value is outside the action limit,

(II)the current and previous values are situated outside the attention limits even in within the action limits,

III) nine successive values lie on the same side of the mean.

The laboratory response to "outside control" conditions is to reject the results for the series and perform tests to determine the cause, then take action to remedy the situation.

A Shewhart Control Graph can also be produced for the differences between analytical pairs in the same sample, especially when reference material does not exist.  In this case, the absolute difference between two analyses of the same sample is graphed.  The graph's lower line is 0 and the attention limit is 1.128 while the action limit is 3.686Sw where = the standard deviation of a series.

This type of graph only accounts for repeatability.  It should be no greater than the published repeatability limit for the method.

In the absence of control material, it sometimes becomes necessary to verify that the reproducibility limit of the reference method is not exceeded by comparing the results obtained to those of obtained by an experimental laboratory using the same sample.

Each laboratory performs two tests and the following formula is used:

= Critical difference (P=0,95)

= Means of 2 results obtained by lab 1

= Means of 2 results obtained by lab 2

R = Reproducibility of reference method

r = Repeatability of reference method

If the critical difference has been exceeded, the underlying reason is to be found and the test is to be repeated within one month.

2. Evaluation of analytic results indicating that a legal limit has been exceeded

When analytical results indicated that a legal limit has been exceeded, the following procedure should be followed:

In the case of an individual result, conduct a second test under repeatable conditions.  If it is not possible to conduct a second test under repeatable conditions, conduct a double analysis under repeatable conditions and use these data to evaluate the critical difference.

Determine the absolute value of the difference between the mean of the results obtained under repeatable conditions and the legal limit.  An absolute value of the difference which is greater than the critical distance indicates that the sample does not fit the specifications.

Critical difference is calculated by the formula:Mean of results obtained

= Limit

n=Number of analyses

R=reproducibility

r=repeatability

In other words, this is a maximal limit where the average of the results obtained should not be greater than:

If the limit is a minimum, the average of the results obtained should not be less than:

3. Comparing results obtained using two or more laboratories and comparing these results to a reference value

 

To determine whether or not data originating in two laboratories are in agreement, calculate the absolute difference between the two results and compare to the critical difference:= Mean of 2 results obtained by lab 1

=Mean of 2 results obtained by lab 2

= number of analyses in lab 1 sample

=number of analyses in lab 2 sample

R=Reproducibility of reference method

r=Repeatability of reference method

If the result is the average of two tests, the equation can be simplified to:

If the data are individual results, the critical difference is R.

If the critical difference is not exceeded, the conclusion is that the results of the two laboratories are in agreement.

Comparing results obtained by several laboratories with a reference value:

Suppose p laboratories have made n1 determinations, whose mean for each laboratory is y₁ and whose total mean is:

The mean of all laboratories is compared with the reference value.  If the absolute difference exceeds the critical difference, as calculated using the following formula, we conclude the results are not in agreement with the reference value:

)

=Critical difference, calculated as indicated in point 2, for the reference method.

For example, the reference value can be the value assigned to a reference material or the

value obtained by the same laboratory or by a different laboratory with a different method.

 

4. Evaluating analytical results obtained using non-valitated methods

A provisional reproducibility value can be assigned to a non-validated method by comparing it to that of a second laboratory:

= Mean of 2 results obtained by lab 1

= Mean of 2 results obtained by lab 2

r = Repeatability of reference method

Provisional reproducibility can be used to calculate critical difference.

If provisional reproducibility is less than twice the value of repeatability, it should be set to 2r.

A reproducibility value greater than three times repeatability or twice the value calculated using the Horwitz equation is not acceptable.

Horwitz equation:

%=Standard deviation for reproducibility(expressed as a percentage of the mean)

C= concentration, expressed as a decimal fraction (for example,   10g/100g = 0.1)

This equation was empirically obtained from more than 3000 collaborative studies including a diverse group of analyzed substances, matrices and measurement techniques.  In the absence of other information, RSD_R values that are lower or equal to the RSD_R values calculated using the Horwitz equation can be considered acceptable.

 values calculated by the Horwitz equation:

 

Concentration	%
10-9	45
10-8	32
10-7	23
10-6	16
10-5	11
10-4	8
10-3	5,6
10-2	4
10-1	2,8
1	2

If the result obtained using a non-validated method is close to the limit specified by legislation, the decision on the limit shall be decided as follows (for upper limits):

and, for lower limits,

S = decision limit

= legal limit

= provisional reproducibility for non-validated method

=reproducibility for reference method

= critical difference, calculated as indicated in point 2, for the reference method

The result which exceeds the decision limit should be replaced with a final result obtained using the reference method.

Critical differences for probability levels other than 95%

This difference can be determined by multiplying the critical differences at the 95% level by the coefficients shown in Table 1.

Table 1 - Multiplicative coefficients allowing

the calculation of critical differences for

probability levels other than 95%

Probability level P	Multiplicative coefficient
90	0,82
95	1,00
98	1,16
99	1,29
99,5	1,40

Shewhart control graph

 

Bibliography

• "Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories". IUPAC. Pure and App. Chem. Vol 67, nº 4, 649-666, 1995
• "Shewhart Control Charts" ISO 8258. 1991.
• "Precision of test methods - Determination of repeatability and reproducibility for a standard test method by inter-laboratory tests". ISO 5725, 1994.
• "Draft Commission Regulation of establishing rules for the application of reference and routine methods for the analysis and quality evaluation of milk and milk products". Commission of the European Communities, 1995.
• "Harmonized protocols for the adoption of standardized analytical methods and for the presentation of their performance characteristics". IUPAC. Pure an App. Chem., Vol. 62, nº 1, 149-162. 1990.