For a research project, I've got a bunch (N=507) of 20-second VR tracking data clips (6DOF x head and hands), each from a different participant. My goal is to predict the participant using a small subset of each data stream (say 3-5 seconds of each 20-second clip). Obviously, one of the main influencers of the algorithm's accuracy is how many participants it needs to discern between - it's a lot easier to get things right with 3 than 50 or 500.
The number of participants we have is arbitrary, so I don't feel like it makes sense to report accuracy at any particular level. I did notice, though, that the accuracy in my baseline model (kNN on only height) has a linear pattern in log-odds to log-participants space, and that it's a mostly-constant offset above chance. I like this measure of "odds ratio above change" because it seems to be relatively consistent as the number of participants varies. On the other hand, it's pretty unnecessary as a measure if there are other ways to express this fact, especially ones that are validated by previous literature.
My first question is: are there more standard ways to report some analog to 'accuracy' across a varying number of classes?
If not, does this measure work? Why? [I ran some simulated data that involved normal samples from "participants" with a normal mean, and by varying how spread out these participant's individual means were relative to the noise they got, the odds-ratio-over-chance changed, but at each level it was mostly consistent among different numbers of participants.]
I am a PhD student, but machine learning isn't my primary field, so links to research papers are especially welcome!
