2

When we train a model, we usually use pairs of $(x_n,\ label_n)$ as training data. In my case, I have pairs of $(x_n, \ label_n)$ and also pairs of $(x_n,\ wrong\_label_n)$, where both $label$ and $wrong\_label$ come from the same set of labels. Note: the wrong labels are known, not noise.

Is there any way to train on samples with known wrong and correct labels instead of training on correct labels only? Maybe a modified loss function I can use? I don't really mind what kind of model it works for (regressions, CART, ANNs etc).

Nigel Ng
  • 97
  • 5
  • There's certainly information in known-to-be-wrong labels; a Bayesian approach could incorporate information like that fairly readily for example. – Glen_b May 03 '17 at 12:25
  • Possible duplicate of https://stats.stackexchange.com/questions/156281/is-there-a-standard-way-for-training-neural-networks-with-negative-labeled-data (even though there's no answer there) – galoosh33 May 03 '17 at 13:28
  • @Glen_b can you elaborate more on the Bayesian approach please? Like a basic model specification would help. Pretty interested in this – Nigel Ng May 04 '17 at 08:53
  • @galoosh33 yes that question does look similar, though my question is more general in that it doesn't have to be neural network-based. – Nigel Ng May 04 '17 at 09:01

2 Answers2

2

It's a good question. There are similar works in the field of clustering (aka constrained clustering, where you provide "must link" and "cannot link" connections as input to the algorithm), but I'm not aware of anything along these lines for classification using neural networks.

As a first attempt, I would probably do something along these lines:

First, convert your labels to the format $([true\_labels], [false\_labels])$. For example, if I do animal classification:

  • Image1, which I know is of a cat, would be labeled $([cat], [])$
  • Image2, which I'm sure is not a whale would be $([], [whale])$

I would then adapt the loss function appropriately. For example, standard 0-1 loss could be extended to this case by paying a penalty of 1 for either of the two following scenarios: Either classifying an animal differently than its' true label (classifying Image1 as whale) or classifying an animal to be a something which I know is not the true label (classifying Image2 to be a whale).

For example, in the simple case in which we allow a single label for each case (true label and false label), then the regular 0-1 loss, $\sum _{x,y} {1[h(x)\neq y ]} $, will become:

$\sum _{x,y=(y_{true},y_{false})} {1[h(x)\neq y_{true} \lor h(x)=y_{false}]} $

Nigel Ng
  • 97
  • 5
galoosh33
  • 2,202
  • 13
  • 20
1

The key is to notice that your problem is equivalent to machine learning with overlapping classes.

Given item $x_n$ and information that it does not belong to class $L_i$, notice that

$x_n \notin L_i \iff x_n \in (L \setminus L_i)$

and, for another class $L_j$,

$L_i \cap L_j = \emptyset \Rightarrow L_j \in (L \setminus L_i)$

so

$x_n \in L_j, x_n \notin L_i \Rightarrow x_n \in L_j \subseteq (L \setminus L_i)$

You can thus define $L'_i:= L \setminus L_i$. These new classes work just like regular classes, and can contain the regular classes defined earlier.

Unfortunately, this does not mean that the classification is hierarchical (which would simplify matters), since for two classes $L'_i$ and $L'_j$ these classes might have a non-zero intersection. Consider $x_1, x_2 \notin L_1$ and $x_2, x_3 \notin L_2$. Then, $L'_1$ and $L'_2$ overlap, but one is not contained in the other.

I don't know much about ML with overlapping classes; I know the problem is considered to be tough and that there are definitely publications on this. Search for multi-label classifications (e.g. the mldr package in R). Hope this helps.

January
  • 6,999
  • 1
  • 32
  • 55
  • Thanks for this! I see two difficulties with this method. 1- Say I have a sample $x_n \in L_1$. When feeding it into the model, I have to reparameterized the sample to $(x_n, L'_2), (x_n, L'_3)...$. 2- Inference (predictions) from the model will give a multilabel output e.g. $(L'_1, L'_3, L'_5)$. This just means the sample is classified as not label 1, not label 3, not label 5, and I don't think there's a straightforward way of converting $L'$s back to $L$, i.e. the model can't tell me what label an observation belongs to. Just what it doesn't – Nigel Ng May 04 '17 at 08:44
  • 1
    OK, you have a point. I think that nonetheless you still have the same problem -- depending on the particular setup of your data. For example, consider what happens if there any labels which are exclusively defined by negative relationships. Its just that with using $L'_i$ you make this problem explicit. – January May 04 '17 at 09:21