Parameter 'C' cannot be optimized for 'nu-svr'? mlr3 with kernlab

Question

I am trying to optimize an SVR model within the mlr3 ecosystem with the kernlab package and I am getting the following error:

The parameter 'C' can only be set if the following condition is met 'type <U+2208> {eps-svr, eps-bsvr}'. Instead the current parameter value is: type=nu-svr.

I find it very weird that cost parameter C cannot be optimized for type 'nu-svr'.

This is a part of my code:

library(mlr3tuning)

learner_ksvm$param_set

search_space = ps(
  C = p_dbl(lower = 0.01, upper = 1),
  type = p_fct(levels = c("eps-svr", "nu-svr")),
  epsilon = p_dbl(lower = 0.01, upper = 1)
)

measure = msr("regr.rmse")

terminator = trm("evals", n_evals = 10)

instance = TuningInstanceSingleCrit$new(
  task = task_train_prerp,
  learner = learner_ksvm,
  resampling = rsmp_cv,
  measure = measure,
  search_space = search_space,
  terminator = terminator
)

tuner = tnr("random_search")

library(progressr)
handlers(global = TRUE)
handlers("rstudio")

tuner$optimize(instance)

Is it true that parameter C cannot be optimized for 'nu-svr' or is it an error? — Eduardo A. Sánchez Torres, Sep 25 '21 at 08:01
Questions that are only about software (e.g. error messages, code or packages, etc.) are generally off topic here. If you have a substantive machine learning or statistical question, please edit to clarify. — kjetil b halvorsen, Sep 25 '21 at 16:56
I'm not sure where it belongs better (maybe it depends on the answer!), but here's the cross-post to SO: https://stackoverflow.com/q/69334453/10495893 — Ben Reiniger, Sep 27 '21 at 14:33

score 1 · Answer 1 · answered Sep 26 '21 at 16:55

From the sklearn User Guide (I know you're asking about an R package, but sklearn's user guide is pretty great about introducing topics IMO):

The $\nu$-SVC formulation [15] is a reparameterization of the $C$-SVC and therefore mathematically equivalent.

We introduce a new parameter $\nu$ (instead of $C$) which controls the number of support vectors and margin errors: $\nu\in(0,1]$ is an upper bound on the fraction of margin errors and a lower bound of the fraction of support vectors. A margin error corresponds to a sample that lies on the wrong side of its margin boundary: it is either misclassified, or it is correctly classified but does not lie beyond the margin.

The linked reference (Schölkop et al) introduces $\nu$-SVC in section 7, and I believe Proposition 6 is the result justifying the claim "reparameterization". So indeed, while varying $\nu$ and $C$ in the two problems has the same end effect, their interpretations and implementations are different. If you want to vary $C$, you should use specifically $C$-SVC; if instead you want to use $\nu$-SVC, then you really do need to specify $\nu$ and not $C$.

From the kernlab documentation of the ksvm function:

C: cost of constraints violation (default: 1) this is the ‘C’-constant of the regularization term in the Lagrange formulation.

nu: parameter needed for nu-svc, one-svc, and nu-svr. The nu parameter sets the upper bound on the training error and the lower bound on the fraction of data points to become Support Vectors (default: 0.2).

Thanks for your answer. However, your argument is based on SVM for classification. My problem is focused on regression. Thus, from the reference of Schölkop et al, it can be observed that the parameter C does not disappear for nu-SVR. In this sense, in the abstract of this work it is already indicated that the nu parameter replaces epsilon, not the C parameter, as it does for C-SVC. — Eduardo A. Sánchez Torres, Sep 27 '21 at 09:47
Right, sorry, I got too focused on the classification version. I've looked through the kernlab source and can't find where your error message would arise from. The [code that sets up the problem](https://github.com/cran/kernlab/blob/51ed48cb146e998994911708ff1323c3ae0b91fa/src/svm.cpp#L4123) even includes `C`... — Ben Reiniger, Sep 27 '21 at 14:30

Parameter 'C' cannot be optimized for 'nu-svr'? mlr3 with kernlab

1 Answers1