What test has this null hypothesis?

Question

A given, unknown function takes as arguments a random number, a finite numerical sequence and a numerical value, and outputs a numerical result. Known is a set of triples of sequence, value and result. What test tests the null hypothesis that the function ignores the sequence?

Answer 1

There is no test that will do so, at least in part because the space of possible functions is too large.

For example, consider sequences of length 1 on the real line (i.e., a real number), and a space of functions on the sequences that are equal to $0$ everywhere except on a single interval of the form $[a, a+1]$, where $a$ is any real number (different for each function), extended by adding the function $f^*(\cdot) = 0$ to it. If the chosen sequence is in $[a,a+1]$, the function returns $1$, else it returns $0$. Clearly no finite sample size will enable you to state with any statistically-based confidence that your function is in fact $f^*$.

However, you can, in a subjective way, form some belief about whether the function ignores the sequence. One such way is as follows.

1. Generate a lot of random numbers,
2. Get a lot of values,
3. Get a lot of sequences,
4. For each random number - value pair that is feasible, for each sequence: calculate $f(\dots)$

If $f$ is the same for every sequence given each (random number, value) pair, that lends credence to the idea that $f$ ignores the sequence.

If $f$ changes even once, you know it responded to the sequence, so the hypothesis that it ignores the sequence can be rejected.

Note that this procedure would have to be modified in fairly obvious ways if the members of the tuple were not independent.

Now, if you limited yourself to functions drawn from some "small" space with an associated probability distribution, you might be able to construct a formal statistical test.

Answer 2

I agree that jbownan that a practical test covering all possible functions is nearly impossible. There are a few methods that could approach generality as long as:

• you have sufficient sample size
• the ranges of the numbers is sufficiently small (for example only 100, 1000... values)
• the sequences are short enough

All the methods I can think of require research.

First of all, I 'll describe a simpler problem: I remove the additional number argument $A$ for an easier discussion. In this simpler problem, the input variable $X$ is only the sequence (the random number argument does not need to be an argument, it can be generated inside the function), the output is called $Y$. Basically you want to test if $X$ and $Y$ are independent.

A first idea is using machine learning + independence test. Train an "all purpose" model (like random forest) that creates an estimate $f(X)$ of $Y$. Note that the random forest needs to handles sequences which probably needs some research. Then, if $X$ and $Y$ are independent, so are $f(X)$ and $Y$. If $Y$ has few bins, you can use a $\chi^2$ test of independence. A similar method is described here: Investigating differences between populations.

If $Y$ is a contiuous variable, it is possible to make many independance test of $Y\in I$ vs $f(X)\in J$ with sets $I$ and $J$ being intervals or unions of intervals. I think, since $f(X)$ is supposed to vary like $Y$, it is enough to test things like $Y\in I$ vs $f(X)\in I$ only. It has to be done in a multiple test framework (https://en.wikipedia.org/wiki/Multiple_comparisons_problem).

An other idea is to use information theory. The independence if equivalent to $H(X,Y)=H(X)+H(Y)$ or equivalently $I(X,Y)=0$. You can try a powerful entropy estimation technique that estimates $I(X,Y)$. If the technique provides a confidence interval (or maybe a credible interval if Bayesian), then this interval can be used for a test: estimate a confidence (or credible?) interval $[a;b]$ for $I(X,Y)$ and reject if 0 is not in it. Entropy estimation techniques are reviewed here: https://math.stackexchange.com/questions/604654/estimating-the-entropy.

The latest method can be adapted to the additional argument $A$ "easily": you want to test $I((X,A),Y)=I(A,Y)$.