Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

See Wikipedia, which has a long list of various smoothing methods, and further references.

361 questions
73
votes
3 answers

How to use Pearson correlation correctly with time series

I have 2 time-series (both smooth) that I would like to cross-correlate to see how correlated they are. I intend to use the Pearson correlation coefficient. Is this appropriate? My second question is that I can choose to sample the 2 time-series as…
user1551817
  • 1,007
  • 1
  • 8
  • 11
29
votes
2 answers

Choosing a bandwidth for kernel density estimators

For univariate kernel density estimators (KDE), I use Silverman's rule for calculating $h$: \begin{equation} 0.9 \min(sd, IQR/1.34)\times n^{-0.2} \end{equation} What are the standard rules for multivariate KDE (assuming a Normal kernel).
csgillespie
  • 11,849
  • 9
  • 56
  • 85
20
votes
2 answers

How to smooth data and force monotonicity

I have some data which I would like to smooth so that the smoothed points are monotonically decreasing. My data sharply decreases and then begins to plateau. Here's an example using R df <- data.frame(x=1:10,…
Ben
  • 1,612
  • 3
  • 17
  • 30
19
votes
2 answers

If variable kernel widths are often good for kernel regression, why are they generally not good for kernel density estimation?

This question is prompted by discussion elsewhere. Variable kernels are often used in local regression. For example, loess is widely used and works well as a regression smoother, and is based on a kernel of variable width that adapts to data…
Rob Hyndman
  • 51,928
  • 23
  • 126
  • 178
19
votes
2 answers

Smoothing - when to use it and when not to?

There is quite an old post on William Briggs' blog which looks at the pitfalls of smoothing data and carrying that smoothed data through to analysis. The key argument is namely: If, in a moment of insanity, you do smooth time series data and you…
NickB2014
  • 743
  • 1
  • 9
  • 16
18
votes
2 answers

When will a Kalman filter give better results than a simple moving average?

I recently implemented a Kalman filter on the simple example of measuring a particles position with a random velocity and acceleration. I found that Kalman filter worked well, but I then asked myself what's the difference between this and just…
dvreed77
  • 737
  • 1
  • 6
  • 15
18
votes
2 answers

How to tune smoothing in mgcv GAM model

I am trying to figure out how to control the smoothing parameters in an mgcv:gam model. I have a binomial variable I am trying to model as primarily a function of x and y coordinates on a fixed grid, plus some other variables with more minor…
colonel triq
  • 181
  • 1
  • 1
  • 3
16
votes
6 answers

Finding inflection points in R from smoothed data

I have some data that I smooth using loess. I'd like to find the inflection points of the smoothed line. Is this possible? I'm sure someone has made a fancy method to solve this...I mean...after all, it's R! I'm fine with changing the smoothing…
user164846
  • 395
  • 2
  • 4
  • 9
16
votes
1 answer

The Pros and Cons of Smoothing spline

I have a general question. Recently I just learnt Basis Expansion and Regularization. There are several interesting techniques including: cubic spline, natural spline, b-spline and smoothing spline. The question is, what is the Pros and Cons(if…
penpen926
  • 169
  • 1
  • 4
16
votes
3 answers

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - D, 0\right\}}{\sum_{w'} C\left(w_{n-1}, w'\right)} +…
sunside
  • 311
  • 3
  • 10
15
votes
4 answers

Smoothing time series data

I am building an android application that records accelerometer data during sleep, so as to analyze sleep trends and optionally wake the user near a desired time during light sleep. I have already built the component that collects and stores data,…
Jon
  • 253
  • 2
  • 7
15
votes
1 answer

Adaptive GAM smooths in mgcv

Simon Wood's book on GAMs and his associated R package mgcv are both highly detailed and informative when it comes to GAM theory and model-fitting to real and simulated data. For 1D smooths, there is really not much to worry about, save for…
compbiostats
  • 1,017
  • 9
  • 18
12
votes
2 answers

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a symmetric Dirichlet distribution with parameter…
DanielX2010
  • 173
  • 1
  • 7
12
votes
4 answers

How to obtain the values used in plot.gam in mgcv?

I would like to find out the values (x, y) used in plotting plot(b, seWithMean=TRUE) in mgcv package. Does anyone know how I can extract or compute these values? Here is an example: library(mgcv) set.seed(0) dat <- gamSim(1, n=400, dist="normal",…
user2264
12
votes
2 answers

How to use a Kalman filter?

I have a trajectory of an object in a 2D space (a surface). The trajectory is given as a sequence of (x,y) coordinates. I know that my measurements are noisy and sometimes I have obvious outliers. So, I want to filter my observations. As far as I…
Roman
  • 1,013
  • 2
  • 23
  • 38
1
2 3
…
24 25