I am doing a survival analysis using Cox's proportional hazards model with several predictive variables. Some of the estimated hazards ratio goes in an implausible direction, which contradicts what we know from the literature. For example, smoking is protective !!
I was wondering if I can force my statistical program (in this analysis SAS) to estimate HR in one direction or in a certain range.
Even if you could force a Cox regression hazard ratio in one direction or over a range, it would almost certainly be unwise. What you describe sounds like a common phenomenon in multiple regression, where including additional variables in a model changes the magnitude or even the direction of a coefficient of a particular variable from a previously expected direction. This is often called Simpson's paradox. Several questions linked from that page, for example this question, and the Wikipedia page provide several examples.
This holds for Cox regressions as well as for other types of regression. In clinical studies, with so many predictor variables typically correlated with each other and with potentially different types of underlying biologic phenomena, there can be several types of problem.
Here's one potential example. Let's say that you are modeling cancer survival outcome and have included both tumor size (T) and smoking status in your model. T size is typically related to shorter survival. Say that some cases of that type of cancer have a non-smoking-related cause that leads to a particularly aggressive form of the cancer, with more rapid death, at the same initial T size, than seen in cases caused by smoking.
Multiple regression coefficients are the relations to outcome with the other predictors held constant, so in this scenario an apparent "protective" effect of smoking would be correct. If, however, those with the non-smoking-related cause typically present with lower T initially than do those with smoking-related disease, a regression omitting T could well show "what you expect from the literature," lower survival for those who smoked.
So instead of trying artificially to limit the range of acceptable hazard ratios, look at your data and think carefully about what your apparently paradoxical hazard ratios are telling you about the underlying biology and clinical situations.
