If you are willing to assume that the non-outlier series is uncorrelated then you could use https://cran.r-project.org/web/packages/tsoutliers/tsoutliers.pdf as it will flag the anomalies thus providing you with the facility to obtain a mean of the outlier-free values. If however the non-outlier series is autocorrelated then things are a tad more complicated . AUTOBOX , a time series analysis package is designed to simultaneously identify both the ARIMA structure and the form of the anomalies.
Unusual values provide an insight into possible drivers and are ignored at the user's peril. If they are identified they should be allowed for in the forecast.

EDITED TO PRESENT HOW PULSE IDENTIFICATION WORKS.
Consider you have 60 values and are trying to find the most "unusual value"
run 61 ols models to search for whether or not there is an unusual value and where it is.