Im trying to learn some hyper-parameters for SVM classifier,
I want to know if there is any correlation between the kernel parameters and the regularization parameter - C,. because if not i can then try optimizing the C parameter and only when one has being optimized start with the kernel parameter, which will save me alot of runtime.
In principle, no. One cannot optimize one parameter and then the other.
There is (at least) one paper that proposes a method to optimize first the C (using a Linear SVM) and then the gamma.
http://www.mitpressjournals.org/doi/abs/10.1162/089976603321891855#.WE3VlpJrWLA
but I tried this and it did not work well on many datasets. Two problems (a) the selection it makes is not that great and (b) it takes a surprising long time - because the linear SVM is not that fast (I did not use the LibLinear implementation - I used libSVM with the linear kernel).
They are indeed correlated, the complexity of the model can be increased by making the kernel more sensitive, which may mean more regularisation is required. Alternatively if you use a "bland" kernel, there is usually less need for regularisation. In short, the optimal value of the regularisation parameter depends on the values of the kernel parameters, and vice versa, so you need to tune both (I tend to use the Nelder-Mead simplex method, which is generally cheaper than grid-search).
