How to deal with arbitrary choices in statsitics? (avoiding data-manipulation)

Question

While I mostly have a good understanding for the mathematical aspects of statistics, I find myself struggling with seemingly arbitrary choices:

For example I have a 2D matrix of chemical compounds showing whether they co-locate or not. (binary 0/1)

To reduce the noise, I want to remove outcomes (0/1) that have a probability over 90% at random. (for example based on concentrations and random movement)

Now how can I ever justify why I decided for a 90% cut-off and not 75% or 99% instead? Do you have any resources that help with such "arbitrary" choices and how to defend against potential claims of data-manipulation? I wonder how you statistics veterans deal with such decisions!

(Basically whatever I do, it will look like I screened through every cut-off and selected whichever looks best for the scientific publication.)

One way to avoid accusations like this would be to show that the results are robust with respect to these choices — nope, Oct 21 '19 at 13:32
What do they do in the chemistry literature? Can you write something like "Following Dave et al we removed outcomes at the 90% random probability threshold"? — Dave, Oct 21 '19 at 14:16
**Blind analysis**: See my answer: https://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection/466423#466423 — kjetil b halvorsen, Jul 08 '20 at 00:42

score 2 · Accepted Answer · answered Oct 21 '19 at 14:06

I would like to provide my take on this from academic experience. You are absolutely right that such criteria are often arbitrary. However, over time I observed two popular ways of dealing with this.

First, you could search relevant literature on the topic, and see what criteria researchers in your field use. Usually, such choices come with justifications, which can help you to think about the appropriateness of their choices. You could also use statistical textbooks as an alternative.

The second method might simply be to develop a theoretical justification for your choice. I will provide an example from psychometric research. Often when modeling latent constructs, two choices are possible: a) to assume that the latent construct causes specific observed behaviors; b) in the opposite, observed behaviors cause latent construct. Depending on a theoretical justification in favor of the either choice, one would proceed with a different statistical model.

Note: @nope provided a very sensible idea, and you should, of course, see how robust your results are based on removing outcomes with varying degrees of probability.

How to deal with arbitrary choices in statsitics? (avoiding data-manipulation)

1 Answers1