How to represent an unbounded variable as number between 0 and 1

Question

I want to represent a variable as a number between 0 and 1. The variable is a non-negative integer with no inherent bound. I map 0 to 0 but what can I map to 1 or numbers between 0 and 1?

I could use the history of that variable to provide the limits. This would mean I have to restate old statistics if the maximum increases. Do I have to do this or are there other tricks I should know about?

@spencer This forum welcomes (IMHO) everyone. You are perfectly fine. Can you please a bit more details about your data? Is your data non-negative? ... — suncoolsu, Mar 14 '11 at 20:58
Because any nondecreasing function from $[0, \infty) \to [0, 1]$ will do the trick, you have a lot of flexibility. But some methods will be better than others, depending on the application. What is your purpose in seeking such a re-expression? — whuber, Mar 14 '11 at 21:12
I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood. — Spencer, Mar 14 '11 at 21:14
@Spencer Exactly how are you measuring content and "relevance"? E.g., on arbitrary scales, as counts, proportions, frequencies of views, correlations with other content, etc. etc. Different types of measurements benefit from different kinds of re-expressions. — whuber, Mar 14 '11 at 21:29
I am measuring them on arbitrary scales. How old the content is. How many "points" a piece of content is received. Self-reported "interest" in the domain of the content. — Spencer, Mar 14 '11 at 21:38
One of the simplest transformations you could use is to convert your data into quantile scores. — charles.y.zheng, Mar 15 '11 at 01:28

score 38 · Accepted Answer · edited Aug 03 '10 at 00:57

A very common trick to do so (e.g., in connectionist modeling) is to use the hyperbolic tangent tanh as the 'squashing function". It automatically fits all numbers into the interval between -1 and 1. Which in your case restricts the range from 0 to 1. In r and matlab you get it via tanh().

Another squashing function is the logistic function (thanks to Simon for the name), provided by $ f(x) = 1 / (1 + e ^{-x} ) $, which restricts the range from 0 to 1 (with 0 mapped to .5). So you would have to multiply the result by 2 and subtract 1 to fit your data into the interval between 0 and 1.

Here is some simple R code which plots both functions (tanh in red, logistic in blue) so you can see how both squash:

x <- seq(0,20,0.001)
plot(x,tanh(x),pch=".", col="red", ylab="y")
points(x,(1 / (1 + exp(-x)))*2-1, pch=".",col="blue")

Thanks for your answer. That solves the problem of bounding. For my data it goes to 1 very quickly for my data so I guess the next thing I need to do is to scale this information to concentrate on the interesting range which I could do based on the history of it without fear of leaving the bound, just hitting the limit. — Russell Gallop, Aug 02 '10 at 15:19

onestop · Answer 2 · 2011-03-15T10:49:20.283

As often, my first question was going to be "why do you want to do this", then I saw you've already answered this in the comments to the question: "I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood."

There is no reason to normalize the data so that the max is 1 and the min is zero in order to achieve this, and my opinion is that this would be a bad idea in general. The max or min values could very easily be outliers that are unrepresentative of the population distribution. @osknows parting remark about using $z$-scores is a much better idea. $z$-scores (aka standard scores) normalize each variable using its standard deviation rather than its range. The standard deviation is less influenced by outliers. In order to use $z$-scores, it's preferable that each variable has a roughly normal distribution, or at least has a roughly symmetric distribution (i.e. isn't severely skew) but if necessary you can apply some appropriate data transformation first in order to achieve this; which transformation to use could be determined by finding the best fitting Box–Cox transformation.

so there is nothing to do with forcing the data to the $[0, 1]$, and in general I do agree with standardization if the author really looks for it :) I answered before the discussion appeared, so probably delete my answer if this one is correct :) — Dmitrij Celov, Mar 15 '11 at 10:27
If one is worried about outliers, one might consider using the median absolute deviation (from the median) instead of the standard deviation. In R, use the `mad()` function. And if one is worried about skewness, one can use the ranks of the data instead of the original observations. In R, this would be `rank()`, but if one’s going to use this on new data, `ecdf()` would be a better alternative (`ecdf(x)` returns a new function that *basically* gives the value $p$ to the $p$-quantile of $x$, e.g., 0 (really $1/n$) to the lowest value of $x$, $1$ to the highest value, 0.5 to the median etc.) — Karl Ove Hufthammer, Apr 01 '15 at 09:32

score 11 · Answer 3 · answered Aug 02 '10 at 15:20

11

Any sigmoid function will work:

The top half of the logistic function (multiply by 2, subtract 1)
The error function
tanh, as suggested by Henrik.

answered Aug 02 '10 at 15:20

Simon Byrne

3,336
15
29

erf is not a very handy function, provided you don't want to rather use it for its derivative. – Aug 02 '10 at 15:26
I ended up using a simple logistic function with some small tweaks: (1 / (1 + java.lang.Math.exp(-1 * (factor * i))) - 0.5) * 2. I picked a factor of 0.05 which seems to work well for i between 0 and a few hundred. – Jilles van Gurp Sep 22 '15 at 12:08
1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x))) is a close approximation to pnorm – Chris Aug 24 '16 at 09:51

score 4 · Answer 4 · answered Aug 02 '10 at 16:49

In addition to the good suggestions by Henrik and Simon Byrne, you could use f(x) = x/(x+1). By way of comparison, the logistic function will exaggerate differences as x grows larger. That is, the difference between f(x) and f(x+1) will be larger with the logistic function than with f(x) = x/(x+1). You may or may not want that effect.

aljabadi · Answer 5 · 2020-10-10T05:10:22.413

Another customisable approach that you can explore is to simply divide all values by the maximum value and take it to the power of a positive shape value ($\gamma$) that best satisfies your desired tranformation objectives. See example below in R in which the dashed line is the simple case of dividing x by max(x):

scaled_power_transform <- function(x, gamma=0.25)
{
    ## x must be nonnegative
    stopifnot(all(x >= 0))
    ## scale to [0, 1]
    x_scaled <- x / max(x)
    ## customise the shape
    x_scaled <- x_scaled^gamma
    return(invisible(x_scaled))
}



x <- seq(0, 1000)

plot(x = x,  y = scaled_power_transform(x, gamma = 0.1), col = 'blue',  
     type = 'l', lwd = 2, ylab = 'x transformed')
lines(x, x/max(x), lty = 2)
lines(x = x, y = scaled_power_transform(x, gamma = 0.5), col = 'green', 
      type = 'l', lwd = 2)
lines(x = x, y = scaled_power_transform(x, gamma = 2), col = 'red',   
      type = 'l', lwd = 2)

legend(x = 0.6*max(x), y=0.3, 
       legend = c(expression(paste(gamma,'= 0.1')), 
                  expression(paste(gamma,'= 0.5')), 
                  expression(paste(gamma,'= 2.0'))), 
       pch = rep('*', 3), col = c('blue', 'green', 'red'))

^{Created on 2020-10-10 by the reprex package (v0.3.0)}

score 1 · Answer 6 · edited Mar 20 '19 at 23:06

There are two ways to implement this that I use commonly. I am always working with realtime data, so this assumes continuous input. Here's some pseudo-code:

Using a trainable minmax:

define function peak:
    // keeps the highest value it has received

define function trough:
    // keeps the lowest value it has received

define function calibrate:
    // toggles whether peak() and trough() are receiving values or not

define function scale:
    // maps input range [trough.value() to peak.value()] to [0.0 to 1.0]

This function requires that you either perform an initial training phase (by using calibrate()) or that you re-train either at certain intervals or according to certain conditions. For instance, imagine a function like this:

define function outBounds (val, thresh):
    if val > (thresh*peak.value()) || val < (trough.value() / thresh):
        calibrate()

peak and trough are normally not receiving values, but if outBounds() receives a value that is more than 1.5 times the current peak or less than the current trough divided by 1.5, then calibrate() is called which allows the function to re-calibrate automatically.

Using an historical minmax:

var arrayLength = 1000
var histArray[arrayLength]

define historyArray(f):
    histArray.pushFront(f) //adds f to the beginning of the array

define max(array):
    // finds maximum element in histArray[]
    return max

define min(array):
    // finds minimum element in histArray[]
    return min

define function scale:
    // maps input range [min(histArray) to max(histArray)] to [0.0 to 1.0]

main()
historyArray(histArray)
scale(min(histArray), max(histArray), histArray[0])
// histArray[0] is the current element

This can all be implemented in Max/MSP/Jitter with the [peak] and [trough] objects for the first example and with [jit.3m] for the second example. — terrace, Aug 03 '10 at 15:40

Chris · Answer 7 · 2016-08-24T10:52:21.603

1

To add to the other answers suggesting pnorm...

For a potentially optimal method for selecting parameters I suggest this approximation for pnorm.

1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x)))

This is essentially Softmax Normalization.

Reference Pnorm in a pinch

edited Aug 24 '16 at 10:52

answered Aug 24 '16 at 10:04

Chris

1,151
9
26

score 1 · Answer 8 · edited Apr 13 '17 at 12:44

1

My earlier post has a method to rank between 0 and 1. Advice on classifier input correlation

However, the ranking I have used, Tmin/Tmax uses the sample min/max but you may find the population min/max more appropriate. Also look up z scores

edited Apr 13 '17 at 12:44

Community

1

answered Mar 14 '11 at 21:20

score 0 · Answer 9 · edited May 21 '21 at 15:17

0

there's a simple formula for normalization in $[0,1]$:

$$ x' = \begin{cases} \frac{1}{ 1 + 1 / x } & \text{if } x > 0\\ 0 & \text{else} \end{cases} $$

$$\lim_{x\rightarrow0} x' = 0$$

$$\lim_{x\rightarrow\infty} x' = 1$$

$$\lim_{x\rightarrow1} x' = 0.5$$

Is there a name for this function?

edited May 21 '21 at 15:17

chuse

762
6
12

answered May 21 '21 at 08:41

Michael Kaufmann

1

1

$x / (x + 1)$ is simpler and one-branch. The issue with this function is whether its curvature is what you want. – Nick Cox May 21 '21 at 08:55

score 0 · Answer 10 · answered Mar 16 '11 at 00:27

A very simple option is dividing each number in your data by the largest number in your data. If you have many small numbers and a few very large ones, this might not convey the information well. But it's relatively easy; if you think meaningful information is lost when you graph the data like this, you could try one of the more sophisticated techniques that others have suggested.

How to represent an unbounded variable as number between 0 and 1

10 Answers10

Linked