3

Suppose there are $n$ different types of events, $i=1,2,...,n$. In each experiment, a subset of two or more distinct events occurs, with no repeated events.

My data consists of counts $f_{ij}$ $(i\ne j)$, where $f_{ij}$ is the number of experiments where events $i,j$ occurred together. The total number of experiments is unknown. Also, the total number of occurrences of event $i$ is unknown. I do know, however, that an event never occurs alone. In each experiment, at least two distinct events occur.

I need to infer the correlation/independence between events, as best as I can, from this data. Ideally, I want to obtain a new matrix giving some measure or estimate of the pairwise correlations between the events.

What methods/correlation coefficients, are appropriate for this problem?

Related: Fix dominant columns/rows in symmetric data matrix?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
becko
  • 3,298
  • 1
  • 19
  • 36
  • 1
    Interesting. The main problem, undoubtedly, is that you don't know $f_{ii}$, the occurence of each event. If you knew you could compute almost any similarity measure for binary data, between $i$ and $j$. Observe, however, that since at each experiment there is at least 2 events occuring, $max \le f_{ii} \le \Sigma$, where $max$ is the maximal off-diagonal element and $\Sigma$ is the sum of off-diagonal elements, of column (or row) $i$ in your nXn frequency table. (If only I'm correct.) Then you could assume some value for $f_{ii}$, - say, arithmetic or geometric average of the two ultimates. – ttnphns Dec 11 '14 at 18:17
  • @ttnphns I think $max$ should be $max_i$, the maximal off-diagonal element of row i. – becko Dec 11 '14 at 19:33
  • Of course, I meant it - both ultimates of the expression are for i. Your task seems to be the task of estimating the diagonal out of off-diagonal elements. The nXn co-frequency matrix is the [SSCP matrix](http://stats.stackexchange.com/a/22520/3277) for binary data, and it must be positive semidefinite. You could assume some initial diagonal entries and then bring the matrix to be psd (if it isn't) by adding small constant to the diagonal or to the whole matrix. – ttnphns Dec 11 '14 at 19:52

0 Answers0