| n | P(1) | P(2) | P(3) |
|---|---|---|---|
| 1 | 0.046 | 0.001 | 4.9e-08 |
| 10 | 0.37 | 0.01 | 4.9e-07 |
| 20 | 0.61 | 0.021 | 9.9e-07 |
| 40 | 0.85 | 0.041 | 2.0e-06 |
The first principle is that you must not fool yourself — and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists.
— Richard Feynman (1918-1988)
1 Preface
THIS monograph concerns how to determine whether an assay is suitable to support the development of structure-activity relationships (SAR) by medicinal chemists for their design and optimization of small-molecule probes and drug candidates. It is crucial to confirm that assays used for SAR-building are fit for this purpose because reliance on unqualified assays undermines compound advancement and can cause costly delays and missed opportunities. The responsibility for confirming assay fitness belongs to the scientists who run them. Without attempting to be exhaustive, the topics in this monograph are those which experience has taught me are important and relevant to qualify assays to support SAR development, some of which are occasionally neglected under the pressure of project demands for data. And while most content will merely underscore what seasoned pharmaceutical researchers fully appreciate, I hope that some usefulness may be gleaned by scientists newly engaged in the worthy pursuit of inventing drugs needed by patients with serious medical conditions.
Biological assays in vitro are fundamental tools of scientific research that have broad utility ranging from determination of catalytic constants in purified-component enzyme reactions to modeling bodily systems using innovative cell and organoid cultures. Assays used to understand drug function are controlled experimental systems in which the effect(s) of perturbagens, such as small molecules, antibodies, and antisense oligonucleotides, can be measured to predict their activity in vivo. The output of a physiologically relevant assay translates to humans, often to a specific disease or disorder, and the link from in vitro system to targeted disease has—for phenotypic assays—been dubbed the chain of translatability (3).
Voluminous information about assays can be found in many scientific publications. Specialty journals like SLAS Discovery and Assay and Drug Development Technologies, as well as several books, such as 4, 5, and 6, are devoted to developing, optimizing, and running reproducible, high-quality cell-based and biochemical assays for drug development. A distinctive resource in this category is the NIH Assay Guidance Manual, a freely available eBook that is updated periodically (7).
As illustrated metaphorically in Figure 1, three foundational elements support the quality and reproducibility of an assay’s output, which is often the IC50 or EC50 of a test substance. All three legs of assay infrastructure must be well-constructed to support a testing platform capable of facilitating development of sound SAR by medicinal chemists. Major constituents of each supporting leg are listed below.
Design and optimization: \(~\)assay technology and format, microplate type, active component concentrations, pH, buffer, carrier protein concentration, temperature, reaction volume, assembly order, timing, cell selection and density, signal window, DMSO tolerance.
Execution: \(~\)reagent dispensing and mixing, microplate layout, protein handling procedures, cell culture protocol, test-compound dissolution, storage, and dilution methods.
Data analysis & quality control: \(~\)reference compound \(~\)IC50 run charts (8) and distribution histograms (Figure 5, CRC quality metrics), variance of positive and negative control signals and signal window, outlier handling, reproducibility of test-compound potency measurements (Figure 4).
A model often fitted to concentration-response data to determine a test-compound’s IC50 is the 4‑parameter logistic model (16, p 330): \[ y = LA + \frac{UA - LA}{1 + 10^{(-log_{10}(x) - pIC_{50}) \cdot slope}} \tag{1}\]
where \(x\) = inhibitor molar concentration, \(y\) = signal or normalized signal (POC), and the four fitted parameters are:
\(LA\) lower asymptote
\(UA\) upper asymptote
\(slope\) slope of curve at its inflection point (the midpoint)
\(pIC_{50}\) negative log10 molar inhibitor concentration at the midpoint.
Shown in Figure 2, this 4-parameter logistic model describes a symmetrical S‑shaped curve of activity as a function of log10 inhibitor concentration that is applicable to many concentration-response relationships in biochemical and cell-based assay systems. Its etymology and relationship to the Hill equation and other models are described in 17 and 18, Chapter 10.
The sensitivity, accuracy, and throughput required of an assay are dictated by its intended purpose. Ideally, the capacity and throughput of primary SAR-driving assays supporting the compound optimization cycle (design, synthesize, test, analyze) allow weekly turnaround of concentration-response testing that matches or exceeds weekly compound output of the medicinal chemistry team working on the project.
Above all, any assay—whatever its purpose or throughput—must be dependable to be useful; that is to say, its output must be reproducible, meaning it is characterized by precision. Methods to quantify this precision are described in Section 3.
2 Resolution Categories
BIOLOGICAL assays in vitro are essential to every phase of drug invention, and they are often the gatekeepers that provide front-line potency assessment of hit and lead molecules. As touched upon in the Preface, one of the most demanding uses for an assay is to provide high-quality activity data which allow medicinal chemists to develop the SAR that informs and guides design and optimization of candidate drug molecules.
Two hallmark attributes of such “SAR-driving” assays are physiological relevance (predictive power, translatability) and high reproducibility (precision). Establishing an assay’s physiological relevance is out of scope here. To achieve suitable precision typically requires an assay to be robust—that is, tolerant of minor changes in conditions—and together with appropriate sensitivity, its degree of precision establishes in which resolution category it belongs:
- Quantitative assays include those suitable to enable development of exploitable SAR;
- Semi-quantitative assays provide only rank-order binning of compound activity;
- Qualitative assays are limited to a binary (yes/no) readout of compound activity.
Notwithstanding this monograph’s exclusive focus on quantitative assays that can support SAR development, semi-quantitative and qualitative assays make important contributions in other areas of drug development.
3 Qualifying Metrics
ASSAY performance can be evaluated in several ways. Many years in the pharmaceutical industry developing and running assays to support compound optimization by drug-invention teams have taught that three practical metrics described below, taken together, reliably indicate whether an assay can effectively support SAR development. Although these three metrics assess interrelated aspects of an assay’s precision, any one by itself can mislead. Each metric consists of an intuitive graphical-statistical analysis that views data quality from a different angle, and their combined perspective paints an accurate picture of assay performance.
1) Quality of individual concentration-response curves (CRC) represented by the 95% confidence limits ratio of the IC50 parameter (hereafter abbreviated CLR) determined by nonlinear least-squares fitting of the 4‑parameter logistic model (Equation 1) to concentration-response data sets produced by the assay (Metric 1).
2) Agreement of n=2 IC50 measurements of ≥ 20 active compounds tested in two independent assay runs, quantified using a Tukey mean-difference plot (Metric 2).
3) Distribution of repeated IC50 values over time measured in ≥ 20 independent assays of a single active reference compound (Metric 3).
The following subsections present examples of each of these three analyses and provide—beneath subheadings in Green Typeface—guideline criteria for quantitative assay qualification.
3.1 CRC Quality
The dimensionless confidence limits ratio (CLR) was introduced by Charles Poole as a useful measure of precision in the context of sample statistics (20). First applied to concentration-response curves in our labs (to my knowledge), the 95% confidence limits ratio of the best-fitting IC50 parameter of the 4‑parameter logistic model (Equation 1), is calculated as shown in Equation 2: \[ \text{CLR} = \frac{\text{95\% } CL_{upper} \text{ of fitted } IC_{50}\text{ parameter}}{\text{95\% } CL_{lower} \text{ of fitted } IC_{50} \text{ parameter}} \tag{2}\]
Calculating confidence limits for the parameters generated by nonlinear least-squares curve fitting is more complicated than calculating confidence limits of a population mean, difference, or proportion (21). Fortunately, commercial and open-source software programs, including GraphPad Prism, R (22), and specialized R packages (23, 24), employ nonlinear fitting algorithms that compute both the symmetric asymptotic confidence limits and the asymmetric profile likelihood confidence limits of each optimized model parameter. Asymptotic confidence limits were more frequently used used historically because their calculation is less computationally intensive, and for this reason they were used to establish the CLR guideline threshold for SAR-driving suitability described below.
The mathematics and algorithms used to calculate profile likelihood confidence limits (25) are outside the present scope; the simpler asymptotic confidence limits of fitted parameters, including the IC50, are calculated using the following equations (26, 27, 28):
\[CL_{lower} = P_i-t \cdot SE_{P_i} \tag{3}\] \[CL_{upper} = P_i+t \cdot SE_{P_i} \tag{4}\]
where the abbreviations are:
\(P_i\) \(i\)th parameter obtained by nonlinear fitting
\(DF\) degrees of freedom, the number of data points minus the number
of parameters fit by nonlinear regression
\(t\) t distribution value for 95% confidence and \(DF\) degrees of freedom
\(SE_{P_i}\) standard error of \(P_i=\sqrt{(SSR \div DF)\cdot cov(i,i)}\)
\(SSR\) sum of squared residuals \(=\sum_{i=1}^{n}(y_i-f(x_i))^2\)
\(cov(i, i)\) \(i\)th diagonal element of the covariance matrix.
Their different computational methods aside, it is worth noting that as concentration-response data quality increases, asymptotic and profile likelihood confidence limits—and CLRs—converge, as can be shown by fitting the 4‑parameter logistic equation to many simulated concentration-response data sets of varying quality.
The log10 of the CLR represents the width of the fitted IC50 confidence interval plotted on a logarithmic concentration scale, with CLR = 10 spanning one log of compound concentration. The calculated CLR can assume values from
1\({\text{ to } \infty}\), with high-quality curve fits returning CLRs < 5 (see Figure 3 and CRC Quality Criteria below).
(a) CRC with CLR > 5, outside the acceptable curve quality range.
CLlower = 5.9 nM (pConc = 8.23)
CLupper = 40 nM (pConc = 7.39)
(b) CRC with CLR < 5, within the acceptable curve quality range.
CLlower = 16 nM (pConc = 7.79)
CLupper = 46 nM (pConc = 7.33)
The CLR in this context is a concise indicator of CRC quality that integrates multiple facets of the model’s fit to the data and, by so doing, accurately represents the quality of the concentration-response data set itself. Because it is unitless, the CLR simplifies comparison of IC50 confidence across different potency levels and assays. Moreover, the CLR relates directly to the potency index of interest—the IC50 or EC50—making it more intuitive than commonly used goodness-of-fit statistics.
Another assay metric, the Z’-factor, not to be confused with the z-statistic, was designed to flag performance dips in large high-throughput screening campaigns by aggregating signal window and control signal variability in a single unitless number (29). Sometimes employed outside its domain, the Z’‑factor is not useful for evaluating or troubleshooting the relatively small number of microplates and control wells in typical CRC assays, for which run charts (8) are better suited, and it is immaterial to the evaluation of CRC quality. Other methods to measure the variability of biological assays and their acceptance are desscribed in references 30 and 31.
At this point in the discussion of quality metrics it is prudent to remember a cardinal precept of data analysis—namely, there is no substitute for visualization. No summary statistic(s) can eliminate the need for careful inspection of IC50 curves by an experienced assay scientist. Quality metrics and statistics, while informative, are best used in concert with plot examination to confirm the credibility of each CRC in the context of the assay’s particular biochemistry and biology.
—— CRC Quality Criteria ——
[1] Greater than 80% of independent IC50 values determined in ≥ 5 test occasions return untuned CLR values < 5, where untuned means none of the four model parameters is fixed and no outlier data points are removed to improve the fit. Fixing one of the model parameters or removing multiple errant data points will result in underestimation of the CLR so that it no longer accurately reflects CRC quality and, by extension, assay data quality.
The CLR quality threshold of 5 was derived empirically using asymptotic confidence intervals of fitted curves to 10- or 11-point duplicate concentration-response data at 3-fold or 3.1623-fold (half log) dilution steps (see CRC fine points below). Use of profile likelihood confidence intervals, different inhibitor concentration number or spacing, or a different number of replicates will affect the IC50 CLR, but it will nonetheless remain a dependable aggregate indicator of both concentration-response data quality and the model’s goodness of fit to the data.
[2] The absolute value of the \(slope\) parameter of Equation 1 fitted to the concentration-response data is between 0.6 and 3.0. Slopes outside of this zone can be caused by several factors and are often associated with large IC50 CLRs. For instance, low slopes can be the result of compound carryover during serial dilution, or of a complex non-monotonic MOA. High CRC slopes can signal cytotoxicity or the involvement of amplified pathways, and can also be caused by a potent inhibitor whose true IC50 is less than one half of its target’s concentration in the assay system—a resolution limit sometimes called the “wall” of the assay.
3.2 pIC50 Agreement
The Tukey mean-difference plot was invented by John W. Tukey in the 1950s to visualize reproducibility and quantify agreement of multiple pairs of measurements (32, pp. 22-23, pp. 150-151; 33). The plot was popularized by Bland and Altman in 1986 to assess concordance of two different clinical assays measuring the same analyte (34); later Eastwood et al. (35) used the geometric mean (Equation 11) and ratio of IC50 values (congruous with the arithmetic mean and difference of pIC50 values) to generate a plot identical in form to a mean-difference plot of pIC50 values. A Tukey mean-difference plot of pIC50 values generated in two runs of a cell-based assays is shown in Figure 4.
The Tukey mean-difference plot is superior to the more common n1 vs n2 scatter plot and squared correlation coefficient R2 (Equation 7) to assess agreement because it allows easier visualization and quantitation of deviation from n1 = n2, and because it is not skewed by the range or distribution of pIC50 values. Nevertheless, to ensure that mean-difference quality metrics reflect assay performance across the potency range of interest and detect any systematic changes across this range, it’s best if the IC50 values of test compounds span two or more orders of magnitude. This may necessitate repeating the analysis when compounds spanning a suitable potency range have been identified.
—— pIC50 Agreement Criteria ——
[1] The mean difference of > 20 pairs of independently determined pIC50 values is within 0.0 ± 0.1 (antilog ratio range 0.79 to 1.26).
[2] The limits of agreement (± 2-SD) of the mean of pIC50 differences are less than ± 0.8 (antilog ratio range 0.16 to 6.31).
3.3 pIC50 Distribution
The distribution of many IC50 determinations for a test compound over time provides a high-fidelity picture of an assay’s performance, though practical considerations limit this analysis to one or two reference compounds that are tested in every run of the assay.
The three histograms in Figure 5 show the distribution of accumulating pIC50 values of a reference compound determined in a cell-based assay over several years. To reflect assay performance accurately, histogram analysis should include no fewer than 15 pIC50 values, preferably 20 or more. Although more values better resolve the distribution’s normality as assessed by its skewness and kurtosis, 20 values estimate well the mean pIC50 returned by sample sizes 5 and 25 times larger.
—— pIC50 Distribution Criteria ——
[1] The distribution of pIC50 values approximates a Gaussian (normal) distribution (Equation 5).
[2] A 2-SD interval < 1 for the distribution of > 20 pIC50 values
(2-SD IC50 ratio < 10).
When a suitable reference compound is not available in the early phase of a drug invention project, this analysis will necessarily be postponed until such a compound has been identified and tested multiple times.
Though the distribution, mean, and 2-SD interval of the mean of 20 pIC50 values (top plot) are concordant with those of the larger samples (middle and bottom plots), sample skewness and kurtosis values of the larger samples better approximate those of an ideal normal distribution: 0 and 3, respectively (36).
4 Operational Advice
SECTION 4 highlights five considerations for the practice of performing SAR-driving assays and analyzing their data. The first two are exhortations about variability; the last three are practical points about CRCs, activator analysis, and comparing potency values from different assays.
4.1 Criticality of n ≥ 2
Perhaps the preeminent imperative for testing compounds to support the development of SAR by medicinal chemists is DO NOT MISLEAD. Erroneous IC50 values risk costly pursuit of mirages, promising avenues missed, and competitive position squandered. To avoid these pitfalls, even laboratories with experience running high-quality assays are wise to consider a single potency value derived from one CRC a preliminary result until it is confirmed by a second independent assay run on a different day, or on the same day using freshly prepared reagents, cells, and compound dilutions. Most assay variability is caused by unavoidable day-to-day variance in reagents, cells, timing, assay assembly, and—sometimes principally—compound handling and dilution. For this reason, repeating potency measurements on fully independent test occasions is essential, both to reflect an assay’s true overall variability and to ensure that it satisfies quality metrics 1 and 2. Practicing n ≥ 2 also protects against rare protocol-deviation errors that produce dramatically inaccurate IC50 values.
Among several reasons for adhering to a best-practice standard of n=2 IC50 determinations, the most compelling may be a statistical argument based on the normal distribution of repeated pIC50 values, shown in idealized form in Figure 6 and in a real example above in Figure 5. Many quantities in nature in addition to IC50 values distribute log-normally (37), which means their logarithms follow a normal (Gaussian) distribution described by the probability density function (21, p 45),
\[ y=\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-\mu)^2 \div 2\sigma^2} \tag{5}\]
where
\(y\) = \(f\), the frequency of occurrence,
\(\mu\) = population mean,
\(\sigma\) = population standard deviation, and
\(x\) = pIC50.
Given this distribution, the probability that a single pIC50 measurement for a compound will poorly represent the true mean by falling in the blue areas of Figure 6 beyond 2-SD from the true mean is 4.56%, or 1 in 22, which is not a particularly rare event. The probability of a second measurement of the same compound returning a pIC50 in the same tail region of the distribution as the initial pIC50, and consequently not moving the mean of the two values closer to the true mean, is 4.56% \(\times\) 2.28% \(=\) 0.104%, or 1 in 962, which is a rare event. Table 1 shows the probability of a misleading pIC50 value in a single test or mean pIC50 of two or three independent tests, as a function of the number of compounds tested in the assay instances.
Table 1. The probability, in a test set of n compounds, of observing one, P(1); two, P(2); and three, P(3) misleading pIC50 values for one compound (> 2-SD in the same direction from the theoretical true mean) is \(P_n=1-(1-P)^n\).
\(P\) was determined from the Gaussian distribution as described in the text.
As this statistical argument makes clear, it is highly likely that the mean of two pIC50 values (or the geometric mean of two IC50 values) will be reasonably close to the true mean, and therefore will not confound SAR generation or mislead chemists to abandon promising compounds or pursue dead-end structures. Additional independent IC50 measurements will further increase confidence in the proximity of the geometric mean IC50 value to the true mean. In practice, multiple IC50 determinations are called for when the first two values appear unusually disparate or when potency in vitro will be a determining factor in setting a compound’s dosing schedule in vivo.
In conclusion, the substantially decreased risk of misleading medicinal chemists and confounding their SAR is well worth the relatively modest cost, time, and effort required to determine IC50 values in two or more independent assay runs.
4.2 Survivorship bias
This monograph’s leitmotif is the importance of carefully qualifying an assay’s suitability to support SAR development by medicinal chemists. Needless to say, this assay qualification process must be objective to be valid. One practice that undermines objectivity is the imposition of a potency validation gate for each IC50 determination based solely on its proximity to a previous value or mean of values (to be within 3‑fold, e.g.). This misguided practice is similar to masking outlier concentration-response data points to improve CRC fit statistics except that the masked data points remain visible on the plot, whereas these invalidated IC50 values vanish without trace.
Whereas some military analysts suggested that the damaged areas should be armored more heavily before sending planes on bombing runs, the statistician Abraham Wald included survivorship bias in his calculations about how to minimize bomber losses to enemy fire. Wald proposed reinforcement of areas where the returning aircraft were unscathed, inferring that planes hit in those areas were the ones most likely to be lost (39).
Potency gating, however well intentioned, should be avoided because it leads inevitably to two undesirable consequences. First, since the reproducibility of surviving IC50 values returned to the database will, by self-fulfilling prophecy, be whatever the gating criterion imposes, the apparent variability of the assay—an artifact arising from artificial selection—will fully accomplish the assay scientist’s self-fooling via survivorship bias (Figure 7). Second, on the predictably probable occasions when the first-determined IC50 of a compound is ≥ 2-SD from the true mean (see above), the gating process will become unbalanced and install a misleading IC50 value in the database. By masking the true variability of the assay and—even worse—displaying values for some compounds far from their true means, potency gating poses a real danger that contemporaneous and future assay scientists, medicinal chemists, and other data miners will be misled.
The best approach when faced with disparate qualified IC50 values is to make additional independent determinations (see above section about the Criticality of n ≥ 2). With more determinations, the pIC50 distribution will better reflect the assay’s variability and the geometric mean will approach its true IC50.
4.3 CRC fine points
Execution
A CRC with \(slope\) = 1 spans 5% to 95% inhibitory activity across approximately 2.5 logs of inhibitor concentration (Figure 2). A compound dilution series that covers twice this range or more will accommodate well-defined fitted curves across a reasonably wide range of IC50 values before re-testing at a lower starting compound concentration becomes necessary. A suitable concentration range can be achieved using ten serial or direct 3-fold dilutions to make 11 solutions spanning nearly 5 logs of compound concentration. Concentration spacing larger than 4-fold is not recommended because of diminished IC50 precision.
Acoustic pipetting is the optimal technology to prepare compound dilutions in microplates (40, 41). Acoustic droplet ejection is superior to air- or positive-displacement pipetting because of its unsurpassed accuracy and precision, and because it makes practical the generation of direct dilutions from a single stock solution, thereby avoiding known liabilities of serial dilution, such as carryover and lack of independence of diluted solutions.
Replicates
There are several levels of replication (42), but typical bioassay replicates consist of the same assay reaction assembled in separate microplate wells of the same assay run—often referred to as technical replicates. For assays qualified to support SAR, two replicate wells at each test compound concentration are sufficient for high-quality CRC fitting. The best practice is to perform curve fitting to the replicate data values themselves, not to the mean values of replicates. While replicate independence is assumed by nonlinear least-squares curve fitting (16, p. 334), variability of replicate measuremets is a useful indicator of assay performance that also provides redundancy when an obvious outlier needs masking. In addition to obscuring replicate variability, use of the mean signal of replicates conceals the number of replicates at each compound concentration, and unweighted curve fitting to signal means from different replicate numbers can produce invalid fitted parameter confidence intervals (43).
ICZ for Z ≠ 50
An alternative potency index to the IC50—such as the IC75 or IC90—may sometimes be appropriate for compound optimization or to guide dosing in vivo. These indices and their confidence limits can be calculated by fitting Equation 6 to the concentration-response data (44). In this equation, the expression bracketed in blue type replaces \(pIC_{50}\) in Equation 1 and includes the potency parameter \(pIC_{Z}\), where \(Z\) is the alternative percent inhibition.
\[
y = LA + \frac{UA - LA}{1+exp10 \left( \left[ -log_{10}(x) -
\color{blue}{ \left\{ pIC_{Z} - \frac{1}{slope}
\cdot log_{10}(\frac{Z}{100-Z}) \right\} } \color{black} \right]
\cdot slope \right) }
\tag{6}\]
It is worth noting that the CLR of a variant potency index will be larger than the IC50 CLR in proportion to the variant index’s distance from the IC50, as shown in Figure 8.
A different method to calculate activity levels other than 50% is to use a fixed multiple of the IC50. This method should be used cautiously, however, because the same IC50 multiplier will represent different levels of inhibition for compounds whose CRC slopes differ signficantly.
Activity scale
Although CRC plots of inhibitor data often employ a percent inhibition scale on the ordinate, a POC scale has the intuitive advantage that curves slope downward with increasing inhibitor concentration and upward with increasing activator concentration. And unlike percent inhibition, the POC scale accommodates both inhibitor and activator curves, simplifying graphical analysis across many assays and where either type of modulator may be encountered in the same assay.
Significant figures
IC50 variability arises from measurement (instrument) imprecision and biological variation, and their combined magnitude is such that bioassay activity measurements rarely support more than two significant figures. Roughly speaking, two significant figures means that confidence is high for the first digit and low for the second, and IC50 and EC50 values should be rounded accordingly before reporting and storing them in a database. A pIC50 with two-digit mantissa faithfully produces, by antilog and rounding, the corresponding two-significant figure IC50. Even better would be to standardize on two-digit pIC50 values to eliminate log-antilog interconversion and rounding, and to simplify statistical aggregation and reporting, among other advantages (45).
Signal normalization
Normalization of assay signal to POC, percent inhibition, or fractional activity is a common practice because it facilitates comparison of inter-assay and intra-assay CRCs. Signal normalization, however, comes at a cost of making important assay information invisible on CRC plots. For example, normalization hides both the baseline signal level—typically subtracted before normalization—and the absolute signal window size. Moreover, signals in control wells on outer edges of a microplate can be affected by evaporation and temperature gradients that skew the normalized activity of internal wells. Nonetheless, provided that absolute signals on assay microplates are unfailingly monitored, the convenience of normalization outweighs its drawbacks in most circumstances.
Confounding compounds
Not only unqualified assays but troublesome test compounds themselves can confound the generation of valid SAR. Such compounds may interfere with assays because they are cytotoxic, affect many targets, or perturb assay technology, among other characteristics, and they may even produce reasonably well-formed CRCs. These chemical trouble makers, commonly referred to as pan-assay interference compounds (PAINS), have been well documented in the scientific literature (e.g., 46, 47, 48). An especially insidious source of assay interference is the presence of one or more metal ions carried through compound synthesis and purification. Such metal ions lurking in compound stock solutions can affect cells or assay technologies and produce artifactual CRCs, even at low concentrations (49).
Coherent SAR is itself a revelatory diagnostic for assay readout artifacts caused by test compounds, as highlighted by Dahlin et al. (48): “SAR is one of the most important yet least appreciated arbiters of nuisance activity. The most frustrating cellular nuisance compounds usually exhibit the following behaviors: weak-to-moderate activities, partial selectivity, imperfect SAR, and equivocal counter-screens.”
4.4 Activator analysis
The chief distinguishing feature of activator concentration-response data analysis compared to inhibitor analysis is the uncertainty about an assay system’s maximal attainable effect level. Whereas maximal inhibition typically equals the activity of the assay’s negative control (inverse agonism or cell growth effects excepted), the highest possible activation level of an assay system is often not known definitively, and this uncertainty can complicate accurate determination of test compound potency (EC50 or midpoint) and efficacy (maximal effect or UA), particularly for low-potency compounds. Test compound activation levels can be normalized to a reference agonist, but this does not solve the problem posed by partial curves of test compounds whose maximal activation may differ from that of the reference agonist.
When the upper asymptote of an activator CRC is not well defined by the data, uncertainty of both the EC50 and efficacy are elevated, as illustrated in Figure 9. The extent to which upper asymptote ambiguity affects the reliability of computed efficacy and potency values can be gauged by the EC50 CLR and the confidence limits difference (CLD) of the UA, fitted without parameter constraint, as illustrated by comparing upper and lower plots of Figure 9. One practical option in this circumstance of uncertain UA is to use an absolute potency index at a selected activity above the control ligand (see horizontal purple dashed line in Figure 9b). This method can be especially useful for less potent compounds being optimized early after their identification by a high-throughput screen.
(a) Activator data that well define the UA (efficacy), as indicated by the small UA CLD and EC50 CLR.
(b) Activator data that ill define the UA (efficacy), having large UA CLD and EC50 CLR values. The dashed purple lines show the concentration resulting in absolute activity 50% above the baseline activation.
4.5 IC50 comparison
Compound optimization triage funnels usually include both purified-component (biochemical) and cell-based assays, and—inevitably—potency measurements of compounds in these assay types of vastly different complexity are compared. Correspondence of test compound potency in biochemical and cell-based assays can provide indirect evidence of target engagement in the immensely more complex cell-based system, but a lack of perfect correspondence should be interpreted carefully. The complexity gap can cause potency shifts in cell-based assays due to differential target exposure caused by membrane permeability, off-target binding opportunities, metabolism, efflux pumps, compound partitioning in organelles, or by a cell-specific conformation of the molecular target. Moreover, nonspecific binding to proteins and/or assay microplates in either cell-based or biochemical assays increases with compound lipophilicity, and can result in variable underestimation of true compound potency that will confound SAR if not accounted for (50).
If an x-y scatter plot is used to compare the potency results of compounds in two assays, the logarithms of molar IC50 (or pIC50) values should be plotted and both the correlation coefficient (R) and the coefficient of determination (R2) should be interpreted cautiously as reliable indicators of correspondence because of their sensitivity to both the range and distribution of values (51).
Another way to compare pIC50 values (or their means) from different assays is to plot compound ID on a categorical x-axis and pIC50 on the y‑axis, with compounds ordered by pIC50 values in one of the assays, as shown in the example in Figure 10. In this example, as is often but not always the case, most compounds appear less potent in the cell-based assay. In this example, most compounds’ IC50 values in the cell-based assay are higher by a factor of 4 to 16 than their IC50 values in the biochemical assay. That some compounds, such as C-13, C-3, and C-22, appear nearly equipotent in the two assays should be interpreted with care, and may be due to one or more of the factors arising from the assay complexity differential mentioned above. An additional consideration, not shown in Figure 10, is that means of small samples typically have wide confidence intervals and confidence intervals for the ratios of such means can be very large indeed.
5 Closing
O F necessity, consumers of potency data generated in physiologically relevant bioassays—especially medicinal chemists building SAR—will use the IC50 values provided to them. This inescapable reliance affixes the onus on assay scientists to ensure in a rigorous way that their assays relied upon to develop SAR are up to the task. The methods and metrics described above can confirm SAR-driving suitability so that assay scientists avoid the adverse consequences of failing to follow Feynman’s first principle.
6 Acknowledgements
RENDERING this document in HTML was done with Quarto (52) and R (22). The following R packages were also used: ggplot2 (53) to generate plots, minpack.lm (23) for nonlinear least-squares curve fitting using the Levenberg-Marquadt algorithm, nlstools (24) for parameter confidence limits, and moments (54) to calculate skewness and kurtosis.
The Richard Feynman quotation epigraph is from his 1974 commencement address at Caltech.
I am indebted to Dr. Daniel Goldberg at the Bill & Melinda Gates Foundation, Dr. Peter White at Paraza Pharma, and Dr. John Proudfoot at Discoverybytes for kindly offering to read drafts of this monograph and for their thoughtful comments and suggestions, which led to a much improved final product. Of course any mistakes are entirely my own.
Finally, I am immensely grateful for numerous former colleagues at Cutter Laboratories, Amgen, Boehringer Ingelheim, and Biogen, in whose excellent company I was privileged to pursue drug invention over the course of my career in the pharmaceutical industry.
7 References
1.
Crum-Brown, A., and Fraser, T.R. (1865) The connection of chemical constitution and physiological action. Trans R Soc Edinb. 25, 257
2.
Wassermann, A. M., Wawer, M., and Bajorath, J. (2010) Activity Landscape Representations for Structure−Activity Relationship Analysis. Journal of Medicinal Chemistry. 53, 8209–8223
3.
Moffat, J. G., Vincent, F., Lee, J. A., Eder, J., and Prunotto, M. (2017) Opportunities and challenges in phenotypic drug discovery: An industry perspective. Nature Reviews Drug Discovery. 16, 531–543
4.
Minor, Lisa K. (ed.) (2006) Handbook of Assay Development in Drug Discovery, Taylor & Francis Group
5.
Wu, G. (2010) Assay development: Fundamentals and practices, Wiley, Hoboken, NJ
6.
Kenakin, T. P. (2022) A pharmacology primer: Techniques for more effective and strategic drug discovery, Sixth edition, Academic Press, Amsterdam
7.
Markossian, S., Grossman, A., Brimacombe, K., Arkin, M., Auld, D., Austin, C. P., Baell, J., Chung, T. D. Y., Coussens, N. P., Dahlin, J. L., Devanarayan, V., Foley, T. L., Glicksman, M., Hall, M. D., Haas, J. V., Hoare, S. R. J., Inglese, J., Iversen, P. W., Kales, S. C., Lal-Nag, M., Li, Z., McGee, J., McManus, O., Riss, T., Saradjian, P., Sittampalam, G. S., Tarselli, M., Trask, O. J., Wang, Y., Weidner, J. R., Wildey, M. J., Wilson, K., Xia, M., and Xu, X. (eds.) (2004) Assay Guidance Manual, Eli Lilly & Company; the National Center for Advancing Translational Sciences, Bethesda (MD), [online] http://www.ncbi.nlm.nih.gov/books/NBK53196/ (Accessed June 26, 2022)
8.
Perla, R. J., Provost, L. P., and Murray, S. K. (2011) The run chart: A simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety. 20, 46–51
9.
Hafner, M., Niepel, M., Chung, M., and Sorger, P. K. (2016) Growth rate inhibition metrics correct for confounders in measuring sensitivity to cancer drugs. Nature Methods. 13, 521–527
10.
Niepel, M., Hafner, M., Mills, C. E., Subramanian, K., Williams, E. H., Chung, M., Gaudio, B., Barrette, A. M., Stern, A. D., Hu, B., Korkola, J. E., Shamu, C. E., Jayaraman, G., Azeloglu, E. U., Iyengar, R., Sobie, E. A., Mills, G. B., Liby, T., Jaffe, J. D., Alimova, M., Davison, D., Lu, X., Golub, T. R., Subramanian, A., Shelley, B., Svendsen, C. N., Ma’ayan, A., Medvedovic, M., Feiler, H. S., Smith, R., Devlin, K., Gray, J. W., Birtwistle, M. R., Heiser, L. M., and Sorger, P. K. (2019) A Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines. Cell Systems. 9, 35–48.e5
11.
Singh, J., Petter, R. C., Baillie, T. A., and Whitty, A. (2011) The resurgence of covalent drugs. Nature Reviews Drug Discovery. 10, 307–317
12.
Hu, Y., and Bajorath, J. (2012) Growth of Ligand–Target Interaction Data in ChEMBL Is Associated with Increasing and Activity Measurement-Dependent Compound Promiscuity. Journal of Chemical Information and Modeling. 52, 2550–2558
13.
Landrum, G. A., and Riniker, S. (2024) Combining IC50 or Ki Values from Different Sources Is a Source of Significant Noise. Journal of Chemical Information and Modeling. 10.1021/acs.jcim.4c00049
14.
Schabenberger, O., Tharp, B. E., Kells, J. J., and Penner, D. (1999) Statistical Tests for Hormesis and Effective Dosages in Herbicide Dose Response. Agronomy Journal. 91, 713–721
15.
Di Veroli, G. Y., Fornari, C., Goldlust, I., Mills, G., Koh, S. B., Bramhall, J. L., Richards, F. M., and Jodrell, D. I. (2015) An automated fitting procedure and software for dose-response curves with multiphasic features. Scientific Reports. 5, 14701
16.
Motulsky, H. (2014) Intuitive biostatistics: A nonmathematical guide to statistical thinking, 3. ed, Oxford Univ. Press, New York, NY
18.
Bindslev, N. (2017) Drug-Acceptor Interactions: Modeling Theoretical Tools to Test and Evaluate Experimental Equilibrium Effects, CRC Press, London, 10.4324/9781315159782
19.
Bevington, P. R., and Robinson, D. K. (2003) Data reduction and error analysis for the physical sciences, 3rd ed, McGraw-Hill, Boston
20.
Poole, C. (2001) Low P-Values or Narrow Confidence Intervals: Which Are More Durable? Epidemiology. 12, 291–294
21.
David J. Sheskin (2007) Handbook of Parametric and Nonparametric Statistical Procedures, 4th Ed., Chapman & Hall/CRC
22.
R Core Team (2023) R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria
23.
Elzhov, T. V., Mullen, K. M., Spiess, A.-N., and Bolker, B. (2023) Minpack.lm: R Interface to the Levenberg-Marquardt Nonlinear Least-Squares Algorithm Found in MINPACK, Plus Support for Bounds. [online] https://cran.r-project.org/web/packages/minpack.lm/index.html (Accessed July 20, 2023)
24.
Baty, F., Ritz, C., Charles, S., Brutsche, M., Flandrois, J.-P., and Delignette-Muller, M.-L. (2015) A Toolbox for Nonlinear Regression in R : The Package nlstools. Journal of Statistical Software. 10.18637/jss.v066.i05
25.
D. J. Venzon, and S. H. Moolgavkar (1988) A Method for Computing Profile-Likelihood-Based Confidence Intervals. Applied Statistics. 37, 87
28.
Walk, R. A. R. (2021) Asymptotic confidence intervals for NLS regression in R R-bloggers. [online] https://www.r-bloggers.com/2021/07/asymptotic-confidence-intervals-for-nls-regression-in-r/ (Accessed December 31, 2023)
29.
Zhang, J.-H., Chung, T. D. Y., and Oldenburg, K. R. (1999) A Simple Statistical Parameter for Use in Evaluation and Validation of High Throughput Screening Assays. SLAS Discovery. 4, 67–73
30.
Li, H., Witkos, T. M., Umlauf, S., and Thompson, C. (2024) Potency Assay Variability Estimation in Practice. Pharmaceutical Statistics. 10.1002/pst.2408
31.
C. Jane Robinson, Michael Sadick, Stanley N. Deming, Sian Estdale, Svetlana Bergelson, and Laureen Little (2014) Assay Acceptance Criteria for Multiwell-Plate–Based Biological Potency Assays. BioProcess International. 12, 30–41
32.
Cleveland, W. S. (1993) Visualizing Data, AT&T Bell Laboratories
34.
Bland, J. M., and Altman, D. G. (1986) Satistical methods for assessing agreement between two methods of clinical measurement. The Lancet. 327, 307–310
35.
Eastwood, B. J., Farmen, M. W., Iversen, P. W., Craft, T. J., Smallwood, J. K., Garbison, K. E., Delapp, N. W., and Smith, G. F. (2006) The minimum significant ratio: A statistical parameter to characterize the reproducibility of potency estimates from concentration-response assays and estimation by replicate-experiment studies. Journal of Biomolecular Screening. 11, 253–261
36.
Cain, M. K., Zhang, Z., and Yuan, K.-H. (2017) Univariate and multivariate skewness and kurtosis for measuring nonnormality: Prevalence, influence and estimation. Behavior Research Methods. 49, 1716–1735
37.
Limpert, E., Stahel, W. A., and Abbt, M. (2001) Log-normal Distributions across the Sciences: Keys and Clues: On the charms of statistics, and how mechanical models resembling gambling machines offer a link to a handy way to characterize log-normal distributions, which can provide deeper insight into variability and probability—normal or log-normal: That is the question. BioScience. 51, 341–352
38.
Wikipedia (2023) Survivorship bias. Wikipedia. [online] https://en.wikipedia.org/w/index.php?title=Survivorship_bias&oldid=1164560893 (Accessed July 11, 2023)
39.
Wald, A. (1980) A Method of Estimating Plane Vulnerability Based on Damage of Survivors (reprint), CENTER FOR NAVAL ANALYSES ALEXANDRIA VA OPERATIONS EVALUATION GROUP, [online] https://web.archive.org/web/20190713234909/https://apps.dtic.mil/docs/citations/ADA091073 (Accessed July 11, 2023)
40.
Ellson, R., Mutz, M., Browning, B., Leejr, L., Miller, M., Papen, R., and Picoliterinc (2003) Transfer of low nanoliter volumes between microplates using focused acoustics: Automation considerations. Journal of the Association for Laboratory Automation. 8, 29–34
41.
Olechno, J., Shieh, J., and Ellson, R. (2006) Improving IC \(_{\textrm{50}}\) Results with Acoustic Droplet Ejection. JALA: Journal of the Association for Laboratory Automation. 11, 240–246
42.
Blainey, P., Krzywinski, M., and Altman, N. (2014) Points of significance: replication. Nature Methods. 11, 879–880
46.
Baell, J., and Walters, M. A. (2014) Chemistry: Chemical con artists foil drug discovery. Nature. 513, 481–483
47.
Baell, J. B., and Nissink, J. W. M. (2018) Seven Year Itch: Pan-Assay Interference Compounds (PAINS) in 2017—Utility and Limitations. ACS Chemical Biology. 13, 36–44
48.
Dahlin, J. L., Auld, D. S., Rothenaigner, I., Haney, S., Sexton, J. Z., Nissink, J. W. M., Walsh, J., Lee, J. A., Strelow, J. M., Willard, F. S., Ferrins, L., Baell, J. B., Walters, M. A., Hua, B. K., Hadian, K., and Wagner, B. K. (2021) Nuisance compounds in cellular assays. Cell Chemical Biology. 28, 356–370
49.
Gerstberger, T., Berger, H., Büttner, F. H., Gmachl, M., Kessler, D., Koegl, M., Lucas, S., Martin, L. J., Mayer, M., McConnell, D. B., Mitzner, S., Scholz, G., Treu, M., Wolkerstorfer, B., Zahn, S., Zak, K. M., Jaeger, P. A., and Ettmayer, P. (2024) Chasing Red Herrings: Palladium Metal Salt Impurities Feigning KRAS Activity in Biochemical Assays. Journal of Medicinal Chemistry. 67, 11701–11711
50.
Miller, R. R., Madeira, M., Wood, H. B., Geissler, W. M., Raab, C. E., and Martin, I. J. (2020) Integrating the Impact of Lipophilicity on Potency and Pharmacokinetic Parameters Enables the Use of Diverse Chemical Space during Small Molecule Drug Optimization. Journal of Medicinal Chemistry. 63, 12156–12170
51.
Anscombe, F. J. (1973) Graphs in Statistical Analysis. The American Statistician. 27, 17–21
52.
Allaire, J. J. (2022) Quarto: R Interface to ’Quarto’ Markdown Publishing System
53.
Wickham, H. (2016) ggplot2: Elegant Graphics for Data Analysis, Springer-Verlag New York
54.
Komsta, L., and Novomestky, F. (2022) Moments: Moments, Cumulants, Skewness, Kurtosis and Related Tests. [online] https://cran.r-project.org/web/packages/moments/index.html (Accessed February 12, 2024)
55.
Spiess, A.-N., and Neumeyer, N. (2010) An evaluation of R2 as an inadequate measure for nonlinear models in pharmacological and biochemical research: A Monte Carlo approach. BMC Pharmacology. 10, 6
56.
Minitab Blog Editor Why Is There No R-Squared for Nonlinear Regression? [online] https://blog.minitab.com/en/adventures-in-statistics-2/why-is-there-no-r-squared-for-nonlinear-regression (Accessed September 20, 2023)
8 Appendix
8.1 Commonly used goodness-of-fit statistics
The coefficient of determination, R2, a unitless statistic often calculated for nonlinear models, is strictly valid only for linear models (55, 56). Where \(TSS\) and \(RSS\) are the total and residual sum of squares, \(n\) is the total number of observations, \(y_i\) is the observed value, \(\bar{y}\) is the mean of observed values, and \(f(x_i)\) is the model’s predicted (fitted) value (55): \[R^2=1-\frac{TSS}{RSS}=1-\frac{\sum_{i=1}^{n}(y_i-\bar{y})^2}{\sum_{i=1}^{n}(y_i-f(x_i))^2} \tag{7}\]
The Chi-squared statistic, where \(y_i\) is the observed value and \(f(x_i)\) is the model’s predicted value (16, p 238): \[\chi^2 = \sum_{i=1}^{n}\frac{(y_i-f(x_i))^2}{f(x_i)}\hspace{1cm}\]
The root mean square error, RMSE, is the standard deviation of the residuals, in units of \(y\) (57): \[RMSE=\sqrt{\frac{\sum_{i=1}^{n}(y_i-f(x_i))^2}{n-1}}\hspace{-1cm}\]
The Sy.x statistic, similar to RMSE but with denominator \(n-K\), where \(K\) is the number of parameters in the fit model (57): \[Sy.x=\sqrt{\frac{\sum_{i=1}^{n}(y_i-f(x_i))^2}{n-K}}\hspace{-1cm}\]
8.2 Descriptive versus inferential statistics
Sample standard deviation (SD), or \(s\), to distinguish it from the population standard deviation, \(\sigma\), is the appropriate descriptive statistic to indicate variance of ≥ 3 pIC50 values (21, p 13). Where \(n\) is the total number of observations, \[s=SD =\sqrt{{\frac{1}{n-1}}\sum_{i=1}^{n}(x_i-\bar{x})^2} \tag{8}\]
Standard error of the mean (SEM) is the sample standard deviation divided by the square root of \(n\) (21, p 145), an inferential statistic that indicates how well the sample mean approximates the true population mean. Not only does the SEM not reflect the dispersion of multiple IC50 values, it decreases with increasing \(n\)—a property with the potential to seriously mislead when mistaken for an indicator of sample variability. \[ SEM=\frac{SD}{\sqrt{n}} \tag{9}\]
8.3 Means arithmetic and geometric
There are several statistical equations for estimating central tendency of samples drawn from populations having different distributions, two of which are relevant to pIC50 and IC50 values: the arithmetic mean and the geometric mean (21, p 9).
The arithmetic mean of a sample applies to populations that follow a normal (Gaussian) distribution, as do pIC50 values: \[ \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i \tag{10}\]
The geometric mean is a measure of central tendency for populations that follow a log-normal distribution, as do IC50 values. The geometric mean can be calculated as the antilog of the arithmetic mean of the logarithms of values, or as the nth root of the product of values:
\[\begin{aligned} geomean(x) & = exp10 \! \left( \frac{1}{n}\sum_{i=1}^{n}log_{10}(x_i) \right) \\ \\ &= \left( {\prod_{i=1}^n x_i}\right )^\frac{1}{n} \\ \end{aligned} \tag{11}\]
9 Abbreviations
| \(\divideontimes\) | multiply/divide symbol, an analog of the ± symbol; described in this note and exemplified in the subtitles of plots in Figure 5 |
| CL | confidence limit |
| CLD | 95% confidence limits difference, \(\text{CL}_{upper}-\text{CL}_{lower}\), of the fitted UA parameter, generated by nonlinear least-squares fitting of the 4-parameter logistic model (Equation 1) to a concentration-response data set |
| CLR | 95% confidence limits ratio, \(\frac{\text{CL}_{upper}}{\text{CL}_{lower}}\), of the fitted IC50 parameter, generated by nonlinear least-squares fitting of the 4-parameter logistic model (Equation 1) to a concentration-response data set |
| CRC | concentration-response curve, generated by fitting the 4-parameter logistic or other model to a concentration-response data set (see Metric 1 for examples) |
| DMSO | dimethyl sulfoxide, the solvent of choice for small molecules |
| EC50 | effective concentration 50%, usually refers to the concentration of test compound that produces 50% of maximal activation in an assay system but can refer to both inhibition and activation effects |
| IC50 | concentration of test compound that inhibits assay activity by 50%; can be calculated in two ways (see this note) |
| pIC50 | −log10(molar IC50) |
| LA | lower asymptote parameter of the 4-parameter logistic model (Equation 1) |
| MOA | mechanism of action |
| pConc | −log10(molar concentration) |
| POC | percent of control, the percent of signal an assay produces in the presence of a perturbagen relative to signal measured in the presence of DMSO alone |
| SAR | structure-activity relationship(s); see definition |
| SD | standard deviation(s), Equation 8 |
| UA | upper asymptote parameter of the 4-parameter logistic model (Equation 1) |