Errors-in-variables regression: is it valid to pool data from three sites?

Question

I recently had a client come to me to do a bootstrap analysis because an FDA reviewer said that their errors-in-variables regression was invalid because when pooling data from sites the analysis include pooling data from three sites where two sites included some samples that were the same.

BACKGROUND

The client had a new assaying method they wanted to show was "equivalent" to an existing approved method. Their approach was to compare the results of both methods applied to the same samples. Three sites were used to do the testing. Errors-in-variables (Deming regression) was applied to the data at each site. The idea is that if the regression showed the slope parameter to be close to 1 and the intercept near 0 this would show that the two assaying techniques gave nearly the same results and hence the new method should be approved. At site 1 they had 45 samples giving them 45 paired observations. Site 2 had 40 samples and site 3, 43 samples. They did three separate Deming regressions (assuming a ratio of 1 for the measurement errors for the two methods). So the algorithm minimized the sum of squared perpendicular distances. Separate regressions were done at each site and a pooled regression was also done using all the data from all three sites.

In their submission the client pointed out that some of the samples used at sites 1 and 2 were the same. In the review the FDA reviewer said that the Deming regression was invalid because common samples were used which causes "interference" that invalidates the assumptions of the model. They requested that a bootstrap adjustment be applied to the Deming results to take account of this interference.

At that point since the client did not know how to do the bootstrap I was brought in. The term interference was strange and I was not sure exactly what the reviewer was getting at. I assumed that the point really was that the because the pooled data had common samples there would be correlation for the common samples and hence the model error terms would not all be independent.

THE CLIENT'S ANALYSIS

The three separate regressions were very similar. Each had slope parameters close to 1 and intercepts near 0. The 95% confidence interval contained 1 and 0 for the slope and intercept respectively in each case. The main difference was a slightly higher residual variance at site 3. Furthermore they compared this to the results from doing OLS and found them to be very similar (in only one case did the confidence interval for the slope based on OLS not contain 1). In the case where the OLS CI for the slope did not contain 1 the upper bound of the interval was something like 0.99.

With the results being so similar at all three sites pooling the site data seemed reasonable. The client did a pooled Deming regression which also lead to similar results. Given these results I wrote a report for the client disputing the claim that the regressions were invalid. My argument is that because there are similar measurement errors in both variables the client was right to use Deming regression as a way to show agreement / disagreement. The individual site regressions had no problems of correlated errors because no samples were repeated within a given site. Pooling data to get tighter confidence intervals. The pooling which included use of the the common samples twice might produce positive correlation between the residuals for those common samples which would mean that the confidence intervals for the regression parameters would be too narrow (the estimated model residual variance biased on the low side).

This difficulty could be remedied by simply pooling the data with the common samples from site 1 say left out. Also the three individual site models do not have the problem and are valid. This seems to me to provide strong evidence of agreement even without the pooling. Furthermore the measurements were taken independently at sites 1 and 2 for the common sites. So I think that even the pooled analysis using all the data is valid because the measurement errors for a sample at site 1 are not correlated with the measurement errors in the corresponding sample at site 2. This really just amounts to repeating a point in the design space which should not be a problem. It does not create correlation / "interference".

In my report I wrote that a bootstrap analysis was unnecessary because there is no correlation to adjust for. The three site models were valid (no possible "interference" within sites) and a pooled analysis could be done removing the common samples at site 1 when doing the pooling. Such a pooled analysis could not have an interference problem. A bootstrap adjustment would not be necessary because there is no bias to adjust for.

CONCLUSION

The client agreed with my analysis but was afraid to take it to the FDA. They want me to do the bootstrap adjustment anyway.

MY QUESTIONS

A) Do you agree with (1) My analysis of the client's results and (2) my argument that the bootstrap is unnecessary.

B) Given that I have to bootstrap the Deming regression are there any procedures SAS or R that are available for me to do the Deming regression on the bootstrap samples?

EDIT: Given the suggestion of Bill Huber I plan to look at bounds on the errors-in-variables regression by regression both y on x and x on y. We already know that for one version of OLS the answer is essentially the same as errors-in-variables when the two error variances are assumed to be equal. If this is true for the other regression then I think that will show that the Deming regression gives an appropriate solution. Do you agree?

In order to meet the client's request I need to do the requested bootstrap analysis that was vaguely defined. Ethically I think it would be wrong to just provide the bootstrap because it doesn't really solve the client's real problem, which is to justify their assay measuring procedure. So I will give them both analyses and request at least that they tell the FDA that in addition to do the bootstrap I did inverse regression and bounded the Deming regressions which I think is more appropriate. Also I think that analysis will show that their method is equivalent to the reference and the Deming regression is therefore adequate also.

I plan to use the R program that @whuber suggested in his answer to enable me to bootstrap the Deming regression. I am not very familiar with R but I think I can do it. I have R installed along with R Studio. Will that make it easy enough for a novice like me?

Also I have SAS and am more comfortable programming in SAS. So if anyone knows a way to do this in SAS I would appreciate knowing about it.

Answer 1

This is a mutual calibration problem: that is, of quantitatively comparing two independent measurement devices.

There appear to be two principal issues. The first (which is only implicit in the question) is in framing the problem: how should one determine whether a new method is "equivalent" to an approved one? The second concerns how to analyze data in which some samples may have been measured more than once.

Framing the question

The best (and perhaps obvious) solution to the stated problem is to evaluate the new method using samples with accurately known values obtained from comparable media (such as human plasma). (This is usually done by spiking actual samples with standard materials of known concentration.) Because this has not been done, let's assume it is either not possible or would not be acceptable to the regulators (for whatever reason). Thus, we are reduced to comparing two measurement methods, one of which is being used as a reference because it is believed to be accurate and reproducible (but without perfect precision).

In effect, the client will be requesting that the FDA allow the new method as a proxy or surrogate for the approved method. As such, their burden is to demonstrate that results from the new method will predict, with sufficient accuracy, what the approved method would have determined had it been applied. The subtle aspect of this is that we are not attempting to predict the true values themselves--we don't even know them. Thus, errors-in-variables regression might not be the most appropriate way to analyze these data.

The usual solution in such cases is "inverse regression" (as described, for instance, in Draper & Smith, Applied Regression Analysis (Second Edition), section 1.7). Briefly, this technique regresses the new method's results $Y$ against the approved method's results $X$, erects a suitable prediction interval, and then functionally inverts that interval to obtain ranges of $X$ for any given values of $Y$. If, for the intended range of $Y$ values, these ranges of $X$ are "sufficiently small," then $Y$ is an effective proxy for $X$. (In my experience this approach tends to be conservatively stringent: these intervals can be surprisingly large unless both measurements are highly accurate, precise, and linearly related.)

Addressing duplicate samples

The relevant concepts here are of sample support and components of variance. "Sample support" refers to the physical portion of a subject (a human being here) that is actually measured. After some portion of the subject is taken, it usually needs to be divided into subsamples suitable for the measurement process. We might be concerned about the possibility of variation between subsamples. In a liquid sample which is well-mixed, there is essentially no variation in the underlying quantity (such as a concentration of a chemical) throughout the sample, but in samples of solids or semisolids (which might include blood), such variation can be substantial. Considering that laboratories often need only microliters of a solution to perform a measurement, we have to be concerned about variation almost on a microscopic scale. This could be important.

The possibility of such variation within a physical sample indicates that the variation in measurement results should be partitioned into separate "components of variance." One component is the variance from within-sample variation, and others are contributions to variance from each independent step of the subsequent measurement process. (These steps may include the physical act of subsampling, further chemical and physical processing of the sample--such as adding stabilizers or centrifugation--, injection of the sample into the measuring instrument, variations within the instrument, variations between instruments, and other variations due to changes in who operates the instrument, possible ambient contamination in the laboratories, and more. I hope this makes it clear that in order to do a really good job of answering this question, the statistician needs a thorough understanding of the entire sampling and analytical process. All I can do is provide some general guidance.)

These considerations apply to the question at hand because one "sample" that is measured at two different "sites" really is two physical samples obtained from the same person and then split among laboratories. The measurement by the approved method will use one piece of a split sample and the simultaneous measurement by the new method will use another piece of the split sample. By considering the components of variance these splits imply, we can settle the main issue of the question. It should now be clear that differences between these paired measurements should be attributed to two things: first, actual differences between the measurement procedures--this is what we are trying to assess--and second, differences due to any variation within the sample as well as variation caused by the physical processes of extracting the two subsamples to be measured. If physical reasoning about the sample homogeneity and the subsampling process can establish that the second form of variance is negligible, then indeed there is no "interference" as claimed by the reviewer. Otherwise, these components of variance may need explicitly to be modeled and estimated in the inverse regression analysis.