2

Assume we have a sensor field with dimension M*M. In order to apply any data compression technique, first I want to know what is the compression limit or minimum entropy of the entire sensor field. How could I compute the minimum entropy or compression limit for the sensor field?

or

Actually I want to have the theoretical compression limit. Let's put the problem for an image. I want to know whether there are any mathematical methods to calculate the theoretical compression limit. Please let me know or suggest any readings to formulate the problem.

Thanks

user2384
  • 21
  • 1

1 Answers1

2

The "entropy" is defined only within the context of a probabilistic model for the data. If you characterize the image as a set of $M^2$ distinct "characters" and assume the frequencies of those characters adequately reflect their probabilities, then you need only apply the formula

Entropy = Sum (over all characters $c$) of [-log(probability of $c$) * probability of $c$].

A standard (but by no means the only) estimate of the probability of a character in a set of $N = M^2$ characters is

Estimated probability of $c$ = (Number of occurrences of $c$) / $N$.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • Thanks for your quick reply. Yes I know this straight forward computation. But I am not sure is this the lower bound of the entropy or upper bound. – user2384 Dec 14 '10 at 20:44
  • Bound in what sense? You need a context here. What set of values are you trying to bound and how is it defined? Read the Wikipedia segment I linked to: it has some good examples. For instance, if you model the image as one realization of a set of possible images, then its entropy equals -1*log(1) = 0. You can't do better than that! – whuber Dec 14 '10 at 20:47
  • Let us take an example, we have a sensor filed of M*N dimension and sensors are uniformly distributed in grid architecture. let say they are sensing the temperature in the agriculture field. and it is obvious they have spatial correlated data. Assume the they are sensing 35,40,45 and 50 degree C. Now I want to calculate the minimum entropy, is this follow the same formula as you said earlier in your post? – user2384 Dec 14 '10 at 20:50
  • Nope: This is a different probability model. You need to specify the nature of the spatial correlation and estimate its parameters. – whuber Dec 14 '10 at 21:13
  • Entropy calculations for fully specified data have been used to get a theoretical bound on how much that data can be compressed. My specified filed is the data from sensor filed. And I want to calculate the minimum entropy (assuming lossless compression) for entire sensor field not for individual nodes. – user2384 Dec 15 '10 at 13:02