I'm modelling an event with two outcomes, a and b. I have created a model which estimates the probability that either a or b will happen (i.e. the model will calculate that a will happen with 40% chance and b will happen with 60% chance).
I have a large record of outcomes of trials with the estimates from the model. I would like to quantify how accurate the model is using this data - is this possible, and if so how?
Suppose your model does indeed predict A has a 40% chance and B has a 60% chance. In some circumstances you might wish to convert this into a classification that B will happen (since it is more likely than A). Once converted into a classification, every prediction is either right or wrong, and there are a number of interesting ways to tally those right and wrong answers. One is straight accuracy (the percentage of right answers). Others include precision and recall or F-measure. As others have mentioned, you may wish to look at the ROC curve. Furthermore, your context may supply a specific cost matrix that rewards true positives differently from true negatives and/or penalizes false positives differently from false negatives.
However, I don't think that's what you are really looking for. If you said B has a 60% chance of happening and I said it had a 99% chance of happening, we have very different predictions even though they would both get mapped to B in a simple classification system. If A happens instead, you are just kind of wrong while I am very wrong, so I'd hope that I would receive a stiffer penalty than you. When your model actually produces probabilities, a scoring rule is a measure of performance of your probability predictions. Specifically you probably want a proper scoring rule, meaning that the score is optimized for well-calibrated results.
A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.
Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.
I'll add a caveat that regardless of what you do, when assessing your model this way I suggest you look at your metric on out-of-sample data (that is, data not used to build your model). This can be done through cross-validation. Perhaps more simply you can build your model on one dataset and then assess it on another (being careful not to let inferences from the out-of-sample spill into the in-sample modeling).
