1

I would like to ask for help in choosing the correct statistical test for analyzing experimental data on small samples (4 repeats)

THE EXPERIMENT: the experiment simulates development of a deposit on a sample. It allows for evaluation of different deposit sources and factors affecting its growth. Each combination of source & factor is a treatment. The main response is the deposit extent recorded 4 times over the experiment duration.

THE DESIGN OF EXPERIMENT:

  • 18 treatments
  • 4 repeats for each treatment
  • All the setups are tested at the same time (18 treatments x 4 = 72 setups), under the same, controlled conditions
  • the test goes for 4 weeks and responses are collected 4 times for every single setup

I choose 4 repeats, because the reference setups show high repeatability, and because of practical limitations. However, for a few of the treated setups I have identified single outliers, based on visual data analysis. So for some setups I have only 3 repeats available.

THE DATA ANALYSIS: The main aim is to compare the final deposit extent and its growth trend (e.g. inhibited after 1 week vs still developing). I was considering the following approaches:

  • DOE in SAS JMP (regression on dummy categorical variables), but it always asks for randomization, while all my setups are run at the same time
  • ANOVA, but I was advised to use it only for much larger sample sizes (>30)
    • Recent idea: compare data visually based on graph with 95% confidence intervals using a Student's t distribution (CONFIDENCE.T function in Excel)
jacek
  • 11
  • 3
  • 1
    (1) So for some treatments you've discarded **one quarter** of your observations just because they were higher or lower than you thought they should be compared to the other observations for that treatment? (2) "Randomization" refers to the allocation of treatments, to whatever "set-ups" are, & doesn't necessarily imply a sequence in time. (3) It's unclear what the difference is between the first two approaches you've considered - ANOVA's equivalent to a regression on categorical predictors represented by dummy variables. (4) Some explanation of the nature of the experiment & the ... – Scortchi - Reinstate Monica Nov 17 '15 at 09:37
  • ... measurements, & a more precise statement of the goals of your analysis might help potential answerers. – Scortchi - Reinstate Monica Nov 17 '15 at 09:58
  • h) (1) I couldn't apply any mathematical formula for outliers treatment due to such a small sample size. Nevertheless I have discarded single results only in case where 3 setups show very high repeatability and a single one was is way off, what I attribute to e.g. a setup flaw. (2) I didn't identify any step in the allocation of treatment, where order could be of importance, and often treatments were simultaneous – jacek Nov 17 '15 at 10:13
  • (3) I almost agree with you, however after hours spent on reading on "regression vs anova" I find that in many instances people address these methods to different problems, and state than one fits better than another in certain situations, (some refer to different ways of null hypothesis definition). I must admit that I was first tempted by DOE, as I find that most case studies which we analyzed during a DOE course were based on single measurements or just 2 repeats per treatment (only later I started to learn on sample size calculation). – jacek Nov 17 '15 at 10:13
  • (1) It'd be advisable to, at the least, (a) compare results with & without those "outliers" included. (b) examine the set-ups for evidence of a flaw rather than assuming it. (2) Again, "randomization" isn't just, or even mainly, concerned with order in time, but with any potential differences between set-ups that might confound estimation of treatment effects. (3) Try both & reconcile the results. And "DOE" stands for "Design of Experiments", experiments typically analysed by regression/ANOVA. – Scortchi - Reinstate Monica Nov 17 '15 at 10:30
  • First of all, thanks for this insightful feedback. 1a) certainly I will try so 2b) that's obviously the most reasonable way, but in this case it would be likely a separate research project, which I cannot afford at this moment (2) I get the idea, nevertheless I didn't identify the potential differences which I could somehow rank & randomize, to my understanding it is a simultaneous experiment (if such one can be defined) and I wonder then whether it can be addressed to the DOE (e.g. JMP) software – jacek Nov 17 '15 at 10:46
  • (3) Could you also share you opinion on data analysis based on 95% CI ? I asked about DOE JMP, only because I am familiar with this software, and considered that the correct way to input my data is via DOE menu, but somehow without the randomization idea. – jacek Nov 17 '15 at 10:47
  • (3) See [Relation between confidence interval and testing statistical hypothesis for t-test](http://stats.stackexchange.com/q/18215/17230). But the extra information you've added (thanks & +1) suggests that you don't have four replicates per treatment - *time* is a factor in your experiment. You might consider a linear model with a simplified interaction structure to allow some estimate of the error variance. For example, time could be represented as linear, quadratic, & cubic, orthogonal polynomial contrasts & interaction limited to the lower-order contrasts - & need ... – Scortchi - Reinstate Monica Nov 17 '15 at 11:06
  • ... the effects of "deposit source" & "factors affecting growth" be modelled non-additively? If you've a growth curve model in mind though - some of your language hints at deposit extent approaching an asymptote more or less slowly depending on treatment - , it might be better to use it. There's no getting around the fact that treatment is confounded with set-up. – Scortchi - Reinstate Monica Nov 17 '15 at 13:09
  • Scortchi - thanks a million for your feedback! Besides asking these important questions, you have provoked many more. I will try to apply your ideas when analyzing the data and see how it works. Thanks again! – jacek Nov 23 '15 at 08:17
  • According to your latest edit you've got four setups per treatment *and* four measurements on each set-up - that's good news & of course you can ignore some of my previous comments. – Scortchi - Reinstate Monica Nov 30 '15 at 18:07

0 Answers0