When using libsvm, the parameter $\gamma$ is a parameter for the kernel function. Its default value is setup as $$\gamma = \frac{1}{\text{number of features.}}$$
Is there any theoretical guidance for setting up this parameter besides existing methods, e.g., grid search?
I would suggest the following theoretical guidance. When you are using Gaussian RBF kernel, your separating surface will be based on a combination of bell-shaped surfaces centered at each support vector. The width of each bell-shaped surface will be inversely proportional to $\gamma$. If this width is smaller than the minimum pair-wise distance for your data, you essentially have overfitting. If this width is larger than the maximum pair-wise distance for your data, all your points fall into one class and you don't have good performance either. So the optimal width should be somewhere between these two extremes.
No, it is essentially data dependent. Grid search (over log-transformed hyper-parameters) is a very good method if you only have a small number of hyper-parameters to tune, but don't make the grid resolution too fine or you are likely to over-fit the tuning criterion. For problems with a larger number of kernel parameters, I find the Nelder-Mead simplex method works well.
