Is there any difference between 'transfer learning' and 'domain adaptation'?
I don't know about context, but my understanding is that we have some dataset 1 and train on it, after which we have another dataset 2 for which we want to adapt our model without retraining from scratch, for which 'transfer learning' and 'domain adaptation' help solve this problem.
According to the field of Convolutional Neural Networks:
It seems that there are some disagreement between researchers on what the difference between 'transfer learning' and 'domain adaptation' is.
From {0}:
The notion of domain adaptation is closely related to transfer learning. Transfer learning is a general term that refers to a class of machine learning problems that involve different tasks or domains. In the literature, there isn't yet a standard definition of transfer learning. In some papers it's interchangeable with domain adaptation.
From {1}:
References:
{0} Li, Qi. "Literature survey: domain adaptation algorithms for natural language processing." Department of Computer Science The Graduate Center, The City University of New York (2012): 8-10. https://scholar.google.com/scholar?cluster=2828982016930721315&hl=en&as_sdt=0,22 ; https://pdfs.semanticscholar.org/532e/3d5b1b5807771b77cac60fe8594b506fcff9.pdf ; http://nlp.cs.rpi.edu/paper/qisurvey.pdf (mirror)
{1} Pan, Sinno Jialin, and Qiang Yang. "A survey on transfer learning." IEEE Transactions on knowledge and data engineering 22, no. 10 (2010): 1345-1359. https://scholar.google.com/scholar?cluster=17771403852323259019&hl=en&as_sdt=0,22 ; http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.158.4126&rep=rep1&type=pdf (mirror) (2.6k citations)
From Hal Daume's article [1]:
The standard classification setting is a input distribution p(X) and a label distribution p(Y|X). Domain adaptation: when p(X) changes between training and test. Transfer learning: when p(Y|X) changes between training and test.
In other words, in DA the input distribution changes but the labels remain the same; in TL, the input distributions stays the same, but the labels change.
Throughout the literature on transfer learning, there is a number of terminology inconsistencies. Phrases such as transfer learning and domain adaptation are used to refer to similar processes. Domain adaptation is the process of adapting one or more source domains for the means of transferring information to improve the performance of a target learner. The domain adaptation process attempts to alter a source domain in an attempt to bring the distribution of the source closer to that of the target. In the Domain Adaptation setting the source and target domains have different marginal distributions p(X). According to Pan's survey, Transfer Learning is a broader term that can also include the case when there is a difference in the conditional distributions p(Y|X) of the source and target domains. In contrast, Daume discriminates the two terms [1], referring that Domain Adaptation is when p(X) changes between source and target and Transfer learning is when p(Y|X) changes between source and target domains.
I think that "Transfer Learning" is a more general term, and "Domain Adaptation" is a scenario of "Transfer Learning".
[1] Transferable Attention for Domain Adaptation. http://ise.thss.tsinghua.edu.cn/~mlong/doc/transferable-attention-aaai19.pdf
According to [1], domain adaptation is the transfer learning in NLP: "Transfer learning in the NLP domain is sometimes referred to as domain adaptation."
[1] Pan, S. J., and Q. Yang. “A Survey on Transfer Learning.” IEEE Transactions on Knowledge and Data Engineering 22, no. 10 (October 2010): 1345–59. https://doi.org/10.1109/TKDE.2009.191 or https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf
