Validation of Biochip Products Based on Microarray Platform

Validation of Biochip Products

based on Microarray Platform

Jen-pei Liu, PhD

Division of Biometry, Department of Agronomy National Taiwan University

and

Division of Biostatistics and Bioinformatics National Health Research Institutes

at

Outline

„

Current Issues

„

Validation and Analytical

Performance

„

MAQC Project

Two-gene expression ratio predicts

clinical outcome (Ma, et al, 2004)

„ Tamoxifen for breast cancer

„ A competitive inhibitor of estrogen binding to estrogen

receptor (ER)

„ Reduction of 40%-50% in annual risk of recurrence „ 5.6% improvement in 10-year survival

„ ER and progesterone receptor (PR, an indicator of a

functional ER pathway) currently the best predicator of tamoxifen response

„ 25% of ER+/PR+, 66% of ER+/PR-, and 55% of

ER-/PR- fail to respond

„ To predict tamoxifen treatment outcome in early-stage

Issues

„ Reid et al. (2005, JCLO) reported that they

can not reproduce the Ma’s results

„ Simon (2005, JCLO) criticized the MA’s

methodology

„ False + by chance = 5475x0.001 = 5.4 „ Observed + = 9 ⇒ 60% false + rate „ Unlikely to have an accurate predictor

Cross-platform and intra- and

inter-laboratory reproducibility

„

Different designs

Different concepts

„

Different referenced samples

„

Different procedures for sample

acquisition

„

Different experimental protocols

Cross-platform and intra- and

inter-laboratory reproducibility

„ Recognition and evaluation of reproducibility

Only Recently:

„ Dobbin et al. (Clinical Cancer Research, 2005) „ Larkin et al. (Nature Methods, 2005)

„ Irizarry et al. (Nature Methods, 2005)

„ Members of the Toxicogenomic Research Consortium

(Nature Methods, 2005)

„ Tan, et al. (Nucleic Acids Research, 2003) „ Yauk, et al. (Nucleic Acids Research, 2004)

Evaluation of Commercial

Platforms (Tan et al, 2003)

„

Platforms

„ Affymetrix (U95Av2, GeneChips, 25mer oligo probe sets)

„ Agilent (Human 1, cDNA probes)

„ Amersham (Codelink UniSet Human I Bioarrays, 30mer Oligo probes)

„ A group of 2009 common genes present on three platforms

Crossplatform and intra- and

inter-laboratory reproducibility

„

Conclusions from cross-laboratory and

cross-platform studies are meaningless

when there are fundamental problems in

achieving acceptable intra-laboratory

reproducibility

„

Conclusions from microarray data analysis

are meaningless when a minimum

requirement for data quality is not met

and provided

Quality Assurance and

Quality Control

„

Four types of factors affecting

microarray experiments

„ Technical

„ Instrumental „ Computational „ Interpretive

A single hidden and uncontrolled factor can completely negate a microarray experiment

Quality Assurance and

Quality Control

Technical Factors

„ Labeling with fluorescent dye „ Hybridization

Quality Assurance and

Quality Control

Microarray QC Metrics and

Thresholds (MAQC) Project

„

Sponsored by the US FDA

„

Establish QC metrics and thresholds to

objectively assessing the performance

of microarray platforms and merits of

various data analysis methods

„

Two RNA samples from three species

Microarray QC Metrics and

Thresholds (MAQC) Project

„

Assess the precision and cross-platform

comparability

„

Nature and magnitude of systematic

bias assessed by QT-PCR

Microarray QC Metrics and

Thresholds (MAQC) Project

„

Four government agencies

10 platform providers

„

3 RNA sample provider

27 test sites

„

10 data analysis sites

„

200 people from more than 70

Microarray QC Metrics and

Thresholds (MAQC) Project

„ Completion of MAQC main study – Oct. 2005 „ Submission of manuscript and release of

MAQC datasets – Feb. 2006

„ Publications – July-Sept. 2--6

„ Public meeting on microarray quality control

and data analysis – Dec. 2006

„ Guidance on microarray quality control and

Performance Evaluation

„

Analytical Performance

„ Accuracy „ Precision

„

Clinical Effectiveness

„ Diagnostic accuracy and variability

„

Clinical Utility

Analytical Performance

Validation assessed functional performance

„ Precision (Reproducibility)

„ Assay Sensitivity (Limit of Detection) —ability to

accurately identify positive samples

„ Assay specificity (Accuracy)

„ Interfering substances (endogenous and

exogenous)

„ Validation of cut-off, reference range, or medical

decision point

Roche AmpliChip CYP450 2C19 Test

Analytical Performance

Precision (Reproducibility)

CYP2C19

genotype No. Tested

Genotype Calls Correct Calls Correct Call Rate (95% CI) *1 / *1 134 134 (100.0) 133 0.99 (0.97) *1 / *2 135 135 (100.0) 135 1.00 (0.98) *1 / *3 135 135 (100.0) 135 1.00 (0.98) *1 / *1 135 135 (100.0) 135 1.00 (0.98) *1 / *2 135 134 (99.3) 134 1.00 (0.98) *1 / *2 135 134 (99.3)) 134 1.00 (0.98) Total 809 807 (99.8) 806 1.00 (0.99)

Roche AmpliChip CYP450 2C19 Test

Analytical Performance

Roche AmpliChip CYP450 2C19 Test

Analytical Performance

Roche AmpliChip CYP450 2C19 Test

Analytical Performance

Limit of detection

Lowest and highest concentration of input sample that yields a consistent and accurate result

DNA Amount (ng) Number of Arrays Number of Correct Calls Positive Rate 95% Confidence Limit 50 144 144 100% 97.5 – 100% 25 144 144 100% 97.5 – 100% 2.5 144 144 100% 97.5 – 100% The lowest level of genomic DNA at which a ≥ 95% positive rate was obtained for correct detection of the CYP2C19 gene was 2.5 ng of input DNA.

DNA Amount (ng) Number of Array s # Correct Calls Positivity Rate 95% CI 50 144 144 100% 97.5 – 100% 25 144 144 100% 97.5 – 100% 2.5 144 134 93.1% 87.6 – 96.6%

Roche AmpliChip CYP450 2D6 Test

Analytical Performance

„ The lowest level of genomic DNA at which a >= 95%

positivity rate was obtained for correct detection of CYP2D6 (*4DxN/*41 and *4/*5 samples) was 25ng.

Potential Interferences

„ Endogenous and exogenous common substances „ Commonly prescribed drugs

„ Molecules similar to the analyte

AmpliChip CYP450 Test

z 10 unique patient samples were tested with and without spiking

of albumin, bilirubin and triglycerides.

z Albumin – 6000 mg/dL; Bilirubin - 60 mg/dL; Triglycerides –

3000 mg/dL (approximately 10-fold greater than normal).

z Elevated levels of lipids, bilirubin and albumin in specimens did

Five-Year View

„ Calibration methods to systematically correct

ratio under-estimation

„ Minimization of technical variation to

reproducibly detect subtler changes (e.g. 1.4-fold)

„ High-throughput tools for a large number of

samples (instead of large number of genes) to identify a small set of biomarkers for drug discovery and development and patient

Five-Year View

„ A technology platform for assay a small to

medium number (dozens to hundreds) of established biomarker genes in a

high-throughput fashion in terms of samples is needed for diagnostic purposes (Genomic Composite Biomarker Classifier)

„ A balance between high-density microarrays

Measures of Similarity

„

Correlation coefficient

„ A measure for association

„ Not a measure for similarity (or agreement)

„

Euclidean distance

„ A measure for agreement

Measures of Similarity

„ Example

Case I Case II Case III

X1 X2 X1 X2 X1 X2

1 1 1 2 1 4

2 2 2 4 2 8

3 3 3 6 3 12

Methodological Research

„

Differentially expression genes

„ Statistical significance based on hypothesis of difference does not take into

consideration of biological significance „ Fold change does not take into

Methodological Research

„ Statistical hypothesis for identification of differentially expression genes should take into consideration biological significance

H_o: μ_T - μ_C ≥ δ_L and μ_T - μ_C ≤ δ_U Vs.

Methodological Research

„

Hypothesis of no correlation can not

prove the consistency

„

With 5000 genes, a correlation of 0.05

is statistically significant at 1% level

„

Use of concordance correlation for the

consistency

Methodological Research

„

Genomic Composite Biomarker (GCB)

Classifier

„ The number of genes in GCB classifier is a random variable

„ The expression level for each selected gene in GCB classifier is also a random variable

Methodological Research

„

Selection of threshold

„

Evaluation of systematic bias at

threshold

„

Evaluation of random error

Measurement error model

„

Expression levels of genes are not

Courses

„ Design and Analysis of Microarray

Experiments – Spring, 2006

„ Statistical Genomics – Spring, 2006 „ Biological Assays – Every two years „ Statistical Methods for Biotechnology

Products – Every two years

„ I – QC/QA of Biotechnology products

„ II – Clinical Trials, Target Trials, BA/BE studies,

Research Products Developm ent

$

$

?

Realty or Dreams