3

I have multiple dataframes each representing traffic speed for each day of the year (366 dataframes for 366 days of the year). The raws of the dataframe are timestamp from 00:00 to 23:55 at 5 minute intervals and the columns are mileposts at 0.5 mile intervals and the entries are speed of traffic corresponding to the specific time and milepost.

I want to group days of similar traffic conditions to examine daily traffic patterns/variations, which is standard for traffic analysis at a macro level, e.g., examining traffic patterns during weekdays and weekends.

To do this, I will have to measure similarity of the dataframes and apply clustering algorithms. Any idea on how to calculate similarity of dataframes and cluster them? Any R package that can do this?

Thanks

Filly
  • 173
  • 7
  • Your data maybe is stored as multiple data frames but from your description it rather sounds like you have a very long (one year) time series data with 5min intervals. Think of it rather as a single dataset. – Tim Jan 22 '15 at 17:35
  • Yes, you are correct that the data can be considered as a single dataset. But the clustering has to be carried out based on the information corresponding to one complete day which contains 288 time series (there are 288 intervals of 5 minutes in one day). My question is how will you compare or meaa set of 288 time series with another set of 288 time series datasets? – Filly Jan 22 '15 at 18:12
  • Why you need it to be analyzed using daily data? – Tim Jan 22 '15 at 18:13
  • To examine daily traffic patterns/variations, which is standard for traffic analysis applications at a macro level, e.g., examining traffic patterns during weekdays and weekends. – Filly Jan 22 '15 at 18:18
  • yes, you can do this using Dynamic time wrapping to determine similarity measures and then use a clustering algorithm. Here is a [blog post](https://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r/) that shows how to do this in R and here is another [post](http://stats.stackexchange.com/questions/131281/dynamic-time-warping-clustering/131284?noredirect=1#comment249959_131284). Also try searching `time series clustering` in this site, there is plenty of useful posts. – forecaster Jan 22 '15 at 21:33
  • I have the same problem : https://stats.stackexchange.com/questions/326848/r-clustering-on-my-data-based-on-variable-id-and-its-daily-behavior Can you help me? – user147460 Feb 05 '18 at 23:20

1 Answers1

0

Rather than look for a specialized similarity, you could represent the data in a way to use ordinary distance measures. For instance, for each day, you could think of this a set of n time series (speed vs. time of day), one for each half mile marker. Then rearrange the day's data as one large vector (of size 288*n). Then you have one long vector for each day that you can analyze just like any other vector for pattern recognition, distance measures, clustering, etc. You can use weighted Euclidean or any other ordinary distance measure. Then, you can process the days sequentially with a clustering technique suited for real time use. Pictorially, if you have time series each day for 5 mile markers, the combination looks like:

You'd probably want to add an extra feature with values 1-7 for day of the week to each day's data, to account for the different conditions on weekends and nearby days, which should also aid in filtering out special events. You could track changes over time continually updating the clustering each day.

This approach to time series analysis was demonstrated for process control and estimation &prediction at https://gregstanleyandassociates.com/whitepapers/BDAC/bdac.htm
and an associated paper that is in press.

Efficient approaches to real-time clustering are discussed there also, in particular, RTEFC.

gms
  • 331
  • 2
  • 6