I am attempting to provide an answer to a question I think you are asking, but you should know that similar questions to yours were already asked, and some excellent answers have already been provided: look here, here, and here.
There are conflicting views about this, but generally, the selection of predictors included in your model should be theory driven, and as such you should not remove predictors just because they are not significant or the removal of them improves the model fit; removing any variables from your model, without a good reason, can be considered as data snooping. Moreover, the effect of a predictor can be high, even if it is not significant -- so don't look at p values alone.
Coefficients of your predictors in your model are conditional upon the remaining variables in your model. As such, the removal of any predictors will likely affect the remaining covariates; it can inflate your coefficients and bias AIC, not to mention it will likely alter the associated p values -- see this about some of the problems associated with stepwise regression.
I would recommend thinking carefully about which variables are of theoretical importance in your research. If you are performing a confirmatory study, with variables of theoretical interest, then I would advise against the removal of any covariates from your model. If you are conducting an exploratory study with many variables, and if some of them are of no theoretical importance, were not specified in your literature review and hypothesis, and have marginal effect on the outcome and the model, then some would say that it would be acceptable to remove them.
