3

I have pollutant concentration for long period I want to determine trend. I have read some of the question answers in this blog regarding same . I have few queries.

  1. What is most accepted theory to calculate trend which is mostly used by climate scientists. Could someone suggest/provide any research paper.
  2. My data also have some missing data sometimes for days and sometime for some month due to instrument error or other reason . So how the missing data is deal in trend analysis ? Interpolating data in parameters like temperature is not that much big issue because temperature does not show large fluctuations. But pollutant concentration which shows large fluctuation from day to day so I think its not a good idea to interpolate the concentration. So how can we deal with missing data 3) I will depersonalized the data can we check piece-wise trend in time series. By piece-wise I mean to say suppose my data shows enhancement for 10 years from starting and after that the concentrations now decreases. So how to deal with that. 4) I have to report the significance in the trend as well.

I am using MATLAB for dealing and analysis of the data. Could someone suggest/help me in this regard

I shall be extremely thankful.

Thanks Gracy

Gracy
  • 31
  • 1
  • 1
    Not sure how many climate scientists are here to answer your first question but for the second one you will certainly find something about imputation of missing values on this site given that similar questions are asked frequently. – Andy May 26 '15 at 08:12
  • 1
    I don't know about the most accepted theory and I'm not aware of the general characteristics or patterns in pollutant concentration time series data, but I think you may be interested in [this discussion](http://jalobe.com/blog/temperature-trends-in-alaska/). It compares trends in temperature data obtained upon a structural time series model with the common approach of fitting a deterministic linear trend. – javlacalle May 26 '15 at 10:10
  • As mentioned by @Andy, you may find in this site several posts related to imputation of missing values in time series, see for example [this list](http://stats.stackexchange.com/search?q=time+series%3B+missing+data), [this other](http://stats.stackexchange.com/search?q=time+series%3B+imputation) or an example in [this post](http://stats.stackexchange.com/questions/104565/how-to-use-auto-arima-to-impute-missing-values/). – javlacalle May 26 '15 at 10:12

1 Answers1

1

First of all, I would like to say that I have no weather prediction education. In my experience missing data is huge problem, especially when you don't have whole month or just a day (I was working on 1-hour samples, one day = 24 missing rows, one month = 750 missing rows). When it is very important feature, e.g. value which you are predicting, you should drop these rows, to avoid further problems. There is many approaches to cope with missing data.

Second issue, which you should do is remove trends from your data. It is crucial step.

Third issue, as you may know some of data may be chaotic, with high variance. E.g. 1-hour PM10 in some day may vary from 10 μg/m3 to 150 μg/m3. So it's good idea to work also on moving averages, to decrease variance: http://www.sciencedirect.com/science/article/pii/S1352231002004193 - it is here described a little - I cannot find free version of this article.

4th issue, if you have data from last 10 years, then you should choose last 3-4 for learning and evaluation. Because city was changing a lot during this whole time. Pollution from 2004 are really different than pollution in 2015.

According to your first point, I was using neural networks(MLP, RBF net, SVM, linear regressions, random forest tree). I didn't use stats model like ARIMA. I had plans for using CRF or SSVM.

I started working on data with MATLAB - but it was pain to fill missing values, to print charts, to describe the data. So I change enviroment to Python, and it was good choice. I had used Pandas (http://pandas.pydata.org/) for working with missing data, for ploting data, for extracting data for learning. Then I start using sklearn (http://scikit-learn.org/stable/) for non-neural nets methods: SVM, trees, Naive Bayes, LASSO, linear regressions. At the end I choose pybrain (http://pybrain.org/ you can look at keras.io) for learning neural nets: MLP and RBF net. Finally I connect some methods in ensemble learning.

I will don't share my master thesis, because it is not written in english.

404pio
  • 204
  • 3
  • 12