29

I often read about a function being 'highly non linear'. In my understanding, there is "linear" and "non-linear", so what is this 'highly' about? Is there a formal difference from non linear? How is it defined?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Toby El Tejedor
  • 391
  • 3
  • 5
  • 8
    Informally: "Don't expect to be able to easily map change in input to change in output." – keshlam Apr 17 '14 at 21:58
  • 2
    Did you read this in a paper about Deep Learning? Highly non-linear function approximation is one of the motivations for deep learning because a shallow network has a hard time modeling the kinds of things Joe describes in his answer. – Neil G Apr 18 '14 at 05:24
  • 1
    I would say it depends on where you read it. If this is written by math-savvy people, then it could mean what answers here (so far) provide. If it was written by a practitioner, like a medical doctor or a biologist, it could mean the relationship is not straight, but highly curved. In my experience, most people think linear regression refers to fitting straight lines to data, which could be part of source of the confusion. – Roman Luštrik Apr 18 '14 at 06:45
  • No, I didn't @NeilG. – Toby El Tejedor Apr 22 '14 at 11:46
  • If one of these answers helped you understand this issue, consider accepting it by clicking the check mark to its left under the vote total. – gung - Reinstate Monica May 23 '14 at 02:03
  • 1
    It's not a singly defined term -- a physicist will tend to take a quite different meaning from the term than a cryptographer would. Without more context this question can't be properly answered -- we'd be guessing the context (or we'd have to account for every distinct one). – Glen_b Apr 17 '15 at 03:08

8 Answers8

48

I don't think there's a formal definition. It's my impression that it simply means that not only is it non-linear, but attempting to model it with a linear approximation won't yield reasonable results and may even cause instability in the fitting method. Someone may also use it to simply mean that small input changes can result in counter-intuitively large changes in output.

Wayne
  • 19,981
  • 4
  • 50
  • 99
16

In a formal sense, I believe one could say that the second derivative differs substantially from zero. If 0 were a "reasonable" approximation to the second derivative over the domain of interest, it is close to linear, but if it's not, the nonlinear effects become very important to capture.

I've rarely heard terms like this apply to relatively simple polynomials, often in practical use it seems to apply to divergent dynamical systems (chaos-theory sort of things), or very non-smooth functions (where much higher-order derivatives are nonzero).

Joe
  • 1,121
  • 7
  • 11
  • 4
    Btw, "smooth" really is a technical term, meaning that every derivative exists. `x -> e^x` is smooth even though its derivatives of all orders are everywhere non-zero :-) – Steve Jessop Apr 17 '14 at 21:03
10

The important aspect missing from the other excellent answers is the domain. E.g., $f(x)=x^2$ is

  • highly non-linear on $[-10;10]$ but
  • not on either half of the domain (i.e., on both $[-10;0]$ and $[0;10]$ one can possibly use a linear approximation of $f$ without an immediate disaster).

Another example is $f(x)=x^3-x$ which is

  • highly non-linear on $[-1;1]$ but
  • not on the larger domain $[-10;;10]$
sds
  • 2,016
  • 1
  • 22
  • 31
  • I disagree about $x^2$. It's always highly non-linear around 0. Consider $x =[0.1,0.2,0.3]$, where $f(x)=[0.01,0.04,0.09]$. How is this linear to you? It doesn't even have linear term (obviously), it can't be approximated linearly at all. – Aksakal Apr 18 '14 at 13:04
  • 3
    @Aksakal: the function is certainly not linear (anywhere), but, as I said, "one can possibly use a linear approximation of f without an immediate disaster" – sds Apr 18 '14 at 14:19
  • 1
    Any function can be approximated by a line, it's only a question of how bad the approximation is. And in x \in [0, 0.5], the error isn't that bad. – Joe Apr 18 '14 at 18:56
8

As other mentioned, I don't think there's a formal definition. I would define it as a function which can not be approximated linearly in the typical range of disturbances to the argument. For instance, you have $y=f(x)$, and $\sigma^2=var[x]$. Then if the approximation $f(x+\sigma)\approx f(x)+f'(x)\sigma$ breaks down, then it's highly non-linear. For instance, $f(x)=exp(x^2)$ would be highly non-linear for any $x$ around zero, because its Taylor series are $1+x^2+x^4/2+O(x^5)$.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
1

As professional mathematician I can confirm that "highly nonlinear" is not a mathematical precisely defined term. :)

And none of "highly anything" I can think of.

Nonlinear is precise and opposite of linear (obviously).

But linear occurs in two different meanings:

  • $f(x) = ax+b$ is called linear function
  • only function $f(x) = ax$ (with no constant term $b$) is called linear

To emphasize the difference and presence of the constant term, the first function $(ax + b)$ is also called affine

Community
  • 1
Dmitri Zaitsev
  • 111
  • 5
1

For smooth functions, we usually say that something is "highly" nonlinear if the magnitude of the second derivative (or perhaps the curvature) is high. A linear function has zero second derivative and zero curvature, so it represents the extreme of low curvature.

Ben
  • 91,027
  • 3
  • 150
  • 376
1

Informally ... "highly non linear" means "even a blind man can see its not a straight line!" ;) Personally I take it as a danger sign, that it will somehow "blow up in your face" when used with real world examples.

The Tower of Hanoi could be called an example of highly non linear ... the legend being when the monks finish a 64 disk stack, the world will end. If you count total time spent in training, feeding, housing, and motivating everyone to support a thankless boring pointless multi-generational task, I would expect the total cost in man hours to really blow out!

iheggie
  • 111
  • 2
0

As other answers said, there isn't a formal definition of the term from the mathematical point of view. But it has to do with the existence of the non zero derivative (the order) of the function.

A function that can be defined as being linear is some function $f(x)$ that can be described by a polynomial $P_n(x)$, even if the degree $n$ of the polynomial is big. That's because the total number of derivatives of the function that can be defined are finite (because the degree of the polynomial is finite).

But those functions that can only be described by other functions with infinite derivatives (infinite order) are said to be "highly nonlinear". For instance a function described by a exponential function who's derivatives of all orders are defined.

The word "highly" is prefixed to differentiate the function with regular polynomials of degree major or equal than 2 that are considered non linear functions (although polynomials of degree equal or major than 2 are considered "linear" from the linear algebra point of view given that they can be expressed in terms of a linear combination of the polynomial basis).

Related to all this has to do the domain of the function. As others said some functions are "more linear" in some intervals than in others. That's because the polynomial approximation theorem. Every real function can be exactly reproduced by a polynomial but that approximation at times involves a greater degree, some times the degree tends to infinity as with a power series. In that later case the function is said to be "highly nonlinear" because the existence of its infinite derivatives.

Carlos Davalillo
  • 1
  • 1