What will change is the interpretation of the coefficients in the model. Suppose that there is no cointegration between your variables and you estimate a bivariate VAR(1) model with the first difference of the log of prices:
$$
\Delta Y_t = a_0 + a_1 \Delta Y_{t-1} + a_2 \Delta X_{t-1} + \epsilon_{1,t} \\
\Delta X_t = b_0 + b_1 \Delta Y_{t-1} + b_2 \Delta X_{t-1} + \epsilon_{2,t}
$$
In this model, you know that an increase of 1 (100%) in asset $X$ is usually followed by an increase of around $a_2$ in asset $Y$ (in terms of continuously compounded returns).
But what if you run the same model with both series in the second differences? Well, no problem, but you need to remember that you are working with the difference of the continuously compounded returns. Now $a_2$ is measuring something different (and more unusual).
The same reasoning applies to the case where one variable is I(1) and the other I(2). There is no problem in running a VAR with one variable in the first-difference and the other in the second-difference, but the interpretation of the coefficients will be harder. Also, if you want to predict values of $Y$, you need to know if the algorithm you are applying is predicting $Y$, $\Delta Y$ or $\Delta^2 Y$.
It is unusual that your returns are not stationary, but it may happen with some commodity prices. If you are working with monthly data, make sure there is no seasonality in it. Here is a suggestion for you:
- Run the model in first differences;
- Check if residuals are not correlated;
- If they are, try to add more lags;
- If they are still correlated, then try to run the model with the second differences.