Statistics in Method Validation (Analytical and Clinical Performance)

Analytical performance in accordance with CLSI guidelines in accordance with CLSI guidelines

The intended use (intended use) is to be proven in a method validation. In the biomedical environment, depending on the intended purpose, this includes the measurement properties, this is what this section is about:

 

  • Accuracy,
  • intra- and inter-serial precision
  • Precision profile
  • Limit of detection, limit of quantification (LoD, LoQ)
  • Measuring range
  • Calibration curve
  • Proof of linearity
  • Specificity / selectivity
  • robustness
  • Stability.

 

ACOMED statistik supports your method validation according to CLSI guidelines (Clinical and Laboratory Standards Institute, formerly NCCLS) with services regarding the design (incl. sample size calculation) and analysis of the experiments. In addition, corresponding Statistics training is offered.


A common task is to adapt the experiments proposed in the guideline to the needs of the manufacturer, e.g. there is only a small sample volume or samples cannot be stored. Here the experiments and statistical analysis methods proposed in the guideline are to be adapted appropriately. ACOMED statistik supports you in this.

 

  • CLSI guideline EP5 - precision
  • CLSI guideline EP6 - linearity
  • CLSI Guideline EP7 - Interference
  • CLSI guideline EP9 - method comparison, bias.

 

Dr. Keller has been a member of the CLSI Subcommitteé for the creation of the new version (EP9-A3) of this guideline (observer status).

 

  • CLSI Guideline EP10 - Preliminary Evaluation
  • CLSI Guideline EP12 - Performance Qualitative Tests
  • CLSI Guideline EP14 - Matrix Effects / Commutability
  • CLSI Guideline EP17 - Detection Limit (LoD),
    Limit of quantification (LoQ)
  • CLSI Guideline GP10 - Diagnostic quality, ROC analysis
  • CLSI Guideline C28 - reference range

 

Dr. Keller has been a member of the CLSI Subcommitteé for the creation of the new version of the EP25 guideline (observer status).


Furthermore, Dr. Keller is a member of the working group "Commutability in Metrological Traceability (WG-CMT)" of the IFCC and member of the Decision Limits / Reference Limits section of the DGKL (link).

Clinical performance

If the method is to be used not only for scientific purposes but also for clinical diagnostics on the patient, the diagnostic properties depend on the intended application situation (diagnostics, screening, ...)

 

  • Sensitivity,  Specificity
  • Predicted values PPV / NPV
  • ROC curve
  • alternatively if necessary: positive and negative percentage agreement with the comparison method
  • Cut-off value
  • possibly up to therapeutic relevance (efficacy studies)..

 

One has to decide, whether the study is undertaken to estimate the diagnostic accuracy measures or to prove them vs. predefined acceptance criteria.


The nature of the method used to describe the clinical reality is decisive for the design. A distinction is made (according to CLSI guideline EP12 or FDA guidance) between a diagnostic quality criterion or Reference standard on the one hand (allows estimation of sensitivity, specificity etc.) or a Comparison method on the other hand (allows determination of positive and negative percent agreement PPA and NPA). [This is simply a matter of terminology; the calculation formulas are identical.] 


A final major point is, whether a case control design (known reference standard result is used as inclusion criterion) or a cohort design (inclusion refers to the intented diagnosis) is chosen. Whereas the case control design is easier to implement, it typically leads to a spectrum bias connected with an overestimation of the diagnostic accuracy measures: This is the bias with the largest impact in diagnostic studies.

Decision and reference limits

Knowing the cut-off values is a prerequisite for determining the diagnostic quality as well as for clinical use in general. The determination of the clinically used cut-off values is a complex topic, since their determination is complex and also depends on the clinical application. A basic distinction is made between decision limits (definition based on clinical aspects, eg therapeutic consequences) and reference intervals consisting of two or one reference limit. The reference limits are determined using the "outer limit (s)" (usually percentiles) of a well-defined and carefully selected group of healthy people. This determination of the reference intervals is described in the CLSI guideline C28. There are also alternative concepts that are examined in the AG Reference Limits of the DGKL, in which Dr. Keller cooperates. Literature:
  • Haeckel R, Wosniok W, Arzideh F. A plea for intra-laboratory reference limits. Part 1. General considerations and concepts of determination. Clin Chem Lab Med 2007; 45: 1033-42,
  • Arzideh F, Wosniok W, Gurr E, Hinsch W, Schumann G Weinstock N, Haeckel R. A plea for intra-laboratory reference limits. Part 2. A bimodal retrospective concept for determining reference limits from intra-laboratory databases demonstrated by catalytic activity concentrations of enzymes. Clin Chem Lab Med 2007; 45: 1043-57

Examples of statistical support for method validations

Method comparison according to CLSI-EP9 with different data scaling, especially counting data (hematology, Eli-Spot, ...)

When comparing the methods of analyzers in hematology, there are usually a large number of analytes to be examined. These also have different properties, in particular a number of analytes are counted data. Other methods, such as the analysis of tumor cell numbers, flow cytometric data or Elispot data, also result in count data. These have the peculiarity that they are Poisson distributed. For Poisson-distributed data, homogeneity of variance is achieved, as it is e.g. in the plot of differences, neither by showing the absolute nor the relative differences. This can also lead to problems with precision tests, since neither the standard deviation nor - alternatively - the coefficient of variation are independent of the concentration (correct: count).

ACOMED statistics has developed procedures to deal with these issues.

Experimental setup of precision experiments according to CLSI-EP05 with little sample material

Precision experiments according to the CLSI EP05 guideline, in which 2 factors are examined, e.g. lot and day, require approx. 75 .. 80 measurements, ie a corresponding number of aliquots must be prepared from one sample. However, there is often not enough sample material available.


The way out is to carry out the experiments on several samples and to statistically pool the results.


ACOMED statistics plans your precision experiments under these conditions and evaluates them.


Example of use: Varga Z, Lebeau A, Bu H, Hartmann A, Penault-Llorca F, Guerini-Rocco E, Schraml P, Symmans F, Stoehr R, Teng X, Turzynski A, von Wasielewski R, Gürtler C, Laible M, Schlombs K, Joensuu H, Basement T , Sinn P, Sahin U, Bartlett J, Viale G (2017). An international reproducibility study validating quantitative determination of ERBB2, ESR1, PGR, and MKI67 mRNA in breast cancer using MammaTyper®.Breast Cancer Research. Breast Cancer Res. 2017 May 11; 19 (1): 55. doi: 10.1186 /s13058-017-0848, link

Qualitative detection limit according to CLSI-EP12 for measurements in difficult matrices

The detection limit of such assays is usually evaluated using the probit method. In the case of difficult matrices or small numbers of cases, an estimation using the probit method is difficult. Roche Molecular Diagnostics (USA) statisticians have developed a different evaluation method [Vaks JE (2017). New Method of Evaluation of Limit of Detection in Molecular Diagnostics, link ; Canchola JA, Vaks JE, Tang S (2016): Limit of Detection (LoD) Estimation Using Maximum Likelihood from (Hit) Rate Data: The LoD_MLE SAS Macro, link ] .

We have programmed this method from JE Vaks in SAS, but without IML, and are happy to analyze your data with this innovative method.