7

I am interested in tools/techniques that can be used for analysis of streaming data in "real-time"*, where latency is an issue. The most common example of this is probably price data from a financial market, although it also occurs in other fields (e.g. finding trends on Twitter or in Google searches).

In my experience, the most common software category for this is "complex event processing". This includes commercial software such as Streambase and Aleri or open-source ones such as Esper or Telegraph (which was the basis for Truviso).

Many existing models are not suited to this kind of analysis because they're too computationally expensive. Are any models** specifically designed to deal with real-time data? What tools can be used for this?

* By "real-time", I mean "analysis on data as it is created". So I do not mean "data that has a time-based relevance" (as in this talk by Hilary Mason).

** By "model", I mean a mathematical abstraction that describe the behavior of an object of study (e.g. in terms of random variables and their associated probability distributions), either for description or forecasting. This could be a machine learning or statistical model.

Shane
  • 11,961
  • 17
  • 71
  • 89
  • Bayesian methods are theoretically well suited for real time analyses because the essential Bayesian method is to take a prior belief, and using a bit of data, compute a posterior belief -- which can become the prior for a new cycle. While that part sounds good, there are practical difficulties and significant hurdles in specifying interesting models on real, changing data that are tractable and easily computable. – Paul Jul 28 '10 at 21:35

6 Answers6

3

This area roughly falls into two categories. The first concerns stream processing and querying issues and associated models and algorithms. The second is efficient algorithms and models for learning from data streams (or data stream mining).

It's my impression that the CEP industry is connected to the first area. For example, StreamBase originated from the Aurora project at Brown/Brandeis/MIT. A similar project was Widom's STREAM at Stanford. Reviewing the publications at either of those projects' sites should help exploring the area.

A nice paper summarizing the research issues (in 2002) from the first area is Models and issues in data stream systems by Babcock et al. In stream mining, I'd recommend starting with Mining Data Streams: A Review by Gaber et al.

BTW, I'm not sure exactly what you're interested in as far as specific models. If it's stream mining and classification in particular, the VFDT is a popular choice. The two review papers (linked above) point to many other models and it's very contextual.

ars
  • 12,160
  • 1
  • 36
  • 54
  • I was actually going to discuss this same thing. I had a job evaluating CEP/ESP tools, such as Aurora, STREAM, and Esper once... – Thomas Owens Jul 28 '10 at 20:40
2

It is going to depend a lot on what exactly you are looking for, but start at Data Streams: Algorithms and Application by Muthukrishnan .

There are many others that can be found by googling "data stream algorithms", or following the references in the paper.

PeterR
  • 1,712
  • 1
  • 16
  • 13
  • This is an interesting paper. Could you add to your answer with some more references and include (for instance) the paper's title and author? Are there any particular academic research labs working in this area that would be relevant? – Shane Jul 28 '10 at 15:37
1

Bayesian networks are perfect for online estimation, and offer a great diversity of models.

RockScience
  • 2,731
  • 4
  • 27
  • 46
1

I am not sure how far this would be relevant to what you want to do but see the paper on adaptive question design called FASTPACE. The goal of the algorithm is to ask the next question from a survey respondent based on his/her previous questions and answers.

The data does not arrive as fast as stock prices but nevertheless latency is an issue as most survey respondents expect the next question to appear within a few seconds.

0

I'm not sure what sort of data you need to process, but statistical algorithms are used quite frequently in robotics, media art, digital signal processing etc. It may be worthwhile to borrow from these domains.

terrace
  • 101
  • 3
0

you have postrank http://data.postrank.com/content

How do you sort what content is current and meaningful? We do it for you. PostRank gathers original content, as well as information at the feed and story level. This includes title, author, language, tags, and related links.

We analyze that data to deliver timely, socially relevant information from millions of online sources. Then we deliver it how you want it. Real-time, custom, filtered access to every piece of content we index, along with its rich metadata

mariana soffer
  • 1,091
  • 2
  • 15
  • 18