9

The logistic function has an output range 0 to 1, and asymptotic slope is zero on both sides.

What is an alternative to a logistic function that doesn't flatten out completely at its ends? Whose asymptotic slopes are approaching zero but not zero, and the range is infinite?

Neil G
  • 13,633
  • 3
  • 41
  • 84
Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • 2
    The title seems to disagree with how i read your question -- is this new function required to have asymptotes or not? – jld Mar 20 '19 at 16:17
  • Basically I want a function that looks like sigmoid but has a slope – Aksakal Mar 20 '19 at 16:24
  • Right, a sigmoid like shape that doesn’t completely flatten, e.g. log function doesn’t completely flatten – Aksakal Mar 20 '19 at 16:31
  • 6
    $\operatorname{sign}(x)\log(1 + |x|)$? – steveo'america Mar 20 '19 at 16:42
  • I like the ideas so far , will try them – Aksakal Mar 20 '19 at 16:43
  • $\operatorname{sign}(x)\sqrt{|x|+k}$ for some non-negative $k$? Note that your question cannot give a cumulative distribution function – Henry Mar 20 '19 at 18:24
  • 4
    Beginning of the decade called, it wants its neural network activation functions back. (Sorry bad joke, but realistically this is why people moved to ReLUs) (+1 though, relevant question) – usεr11852 Mar 20 '19 at 21:39
  • "Zero but not zero"? What is the shape of the function? Can you draw a picture (by hand as a graphic or take a picture of paper)? Your requirements are still unclear. I take it you don't want horizontal asymptotes, but surely some kind of sloped asymptote is OK (or maybe not?). Also, what is the purpose of having this new function? Is it like ReLU? – Mitch Mar 21 '19 at 12:35
  • @Mitch, I have a step like dependence of y on x, but suspect that there is a slope instead of flat step level change. so i want to fit the curve instead of a step – Aksakal Mar 22 '19 at 13:40
  • @Aksakal The words are not giving a good picture. Is it symmetric on the y-axis? Is it an [odd function](https://en.wikipedia.org/wiki/Even_and_odd_functions)? What is the action to the right? Is it asymptotic to linear or does it rise like the log function? So many questions that could be answered by a picture. Also after the picture, the more important question is 'Why?'? What does the rising to the right _get_ you? – Mitch Mar 22 '19 at 15:11

3 Answers3

12

You could just add a term to a logistic function:

$$ f(x; a, b, c, d, e)=\frac{a}{1+b\exp(-cx)} + dx + e $$

The asymptotes will have slopes $d$.

Here is an example with $a=10, b = 1, c = 2, d = \frac{1}{20}, e = -5$:

Sigmoid

COOLSerdash
  • 25,317
  • 8
  • 73
  • 123
  • 2
    I think this answer is the best because if you zoom out far enough it's just a straight line with a little wiggle in the middle. Gives the most intuitive behavior at large x but retains the sigmoid shape. – user1717828 Mar 21 '19 at 01:30
  • this seemed to work for my dataset, and I picked it, but the solution is not ideal since the asymptotic slope doesn't decrease – Aksakal Mar 22 '19 at 10:46
11

Initially I was thinking you did want the horizontal asymptotes at $0$ still; I moved my original answer to the end. If you instead want $\lim_{x\to\pm \infty} f(x) = \pm\infty$ then would something like the inverse hyperbolic sine work? $$ \text{asinh}(x) = \log\left(x + \sqrt{1 + x^2}\right) $$

This is unbounded but grows like $\log$ for large $|x|$ and looks like asinh

I like this function a lot as a data transformation when I've got heavy tails but possibly zeros or negative values.

Another nice thing about this function is that $\text{asinh}'(x) = \frac{1}{\sqrt{1+x^2}}$ so it has a nice simple derivative.

Original answer

$\newcommand{\e}{\varepsilon}$Let $f : \mathbb R\to\mathbb R$ be our function and we'll assume $$ \lim_{x\to\pm \infty} f(x) = 0. $$

Suppose $f$ is continuous. Fix $\e > 0$. From the asymptotes we have $$ \exists x_1 : x < x_1 \implies |f(x)| < \e $$ and analogously there's an $x_2$ such that $x > x_2 \implies |f(x)| < \e$. Therefore outside of $[x_1,x_2]$ $f$ is within $(-\e, \e)$. And $[x_1,x_2]$ is a compact interval so by continuity $f$ is bounded on it.

This means that any such function can't be continuous. Would something like $$ f(x) = \begin{cases} x^{-1} & x\neq 0 \\ 0 & x = 0\end{cases} $$ work?

Community
  • 1
jld
  • 18,405
  • 2
  • 52
  • 65
  • 2
    The "Related" threads include this unanswered question, in case anyone else has asked themselves the natural followup "what happens if you use asinh in a neural network?" https://stats.stackexchange.com/questions/359245/has-arcsinh-ever-been-considered-as-a-neural-network-activation-function?rq=1 – Sycorax Mar 20 '19 at 18:52
  • My ears did indeed prick up. I have in the past found asinh() useful when you want to 'do log stuff' to both positive and negative numbers. It also gets around the quandry you can get in, where you need to do a log transform on data with zeros and have to judge an appropriate value of $a$ for $log(x + a)$ – Ingolifs Mar 20 '19 at 23:41
  • how could you parameterize this function to change it's shape? in particular, to regulate the slope at the inflection point – Aksakal Mar 22 '19 at 10:48
  • @Aksakal if $a > 0$ then just doing $a\cdot\text{asinh}$ would keep the shape and asymptotics the same and the derivative is $\frac{a}{\sqrt{1+x^2}}$ so the slope at zero is just $a$ – jld Mar 22 '19 at 17:04
  • @Aksakal more generally we could consider the antiderivative of $\frac{a}{\sqrt{c^2 + (bx)^2}}$ which is $$\frac ab \log\left(b\left(bx + \sqrt{c^2 + (bx)^2}\right)\right)$$ and allows more ability to change the shape, or just something like $a\cdot\text{asinh}(bx)$ – jld Mar 22 '19 at 17:29
  • In a [tag:neural-networks] context, the shape change would be accomplished by an affine transformation, so you'd have $\text{asinh}(Wx+b)$ applied element-wise for matrix $W$ and vector $b$ and $x$. – Sycorax Mar 25 '19 at 15:44
7

I will go ahead and turn the comment into an answer. I suggest $$ f(x) = \operatorname{sign}(x)\log{\left(1 + |x|\right)}, $$ which has slope tending towards zero, but is unbounded.

edit by popular demand, a plot, for $|x|\le 30$: enter image description here

steveo'america
  • 503
  • 2
  • 11