R Machine Learning: Linear Regression Smoothing

Introduction to Smoothing

• Smoothing is a very powerful technique used all across data analysis. It is designed to detect trends in the presence of noisy data in cases in which the shape of the trend is unknown.
• The concepts behind smoothing techniques are extremely useful in machine learning because conditional expectations/probabilities can be thought of as trends of unknown shapes that we need to estimate in the presence of uncertainty.

Code: Intro

data("polls_2008")
qplot(day, margin, data = polls_2008)

Bin Smoothing and Kernels

• The general idea of smoothing is to group data points into strata in which the value of f(x) can be assumed to be constant. We can make this assumption because we think f(x) changes slowly and, as a result, f(x) is almost constant in small windows of time.
• This assumption implies that a good estimate for f(x) is the average of the Yi values in the window. The estimate is:
  f^(x0)=1/N0 * ∑(i∈A0)Yi
• In smoothing, we call the size of the interval |x−x0| satisfying the particular condition the window size, bandwidth or span.

Code: Bin Smoothing & Kernels

# bin smoothers
span <- 7 
fit <- with(polls_2008,ksmooth(day, margin, x.points = day, kernel="box", bandwidth =span))
polls_2008 %>% mutate(smooth = fit$y) %>%
    ggplot(aes(day, margin)) +
    geom_point(size = 3, alpha = .5, color = "grey") + 
    geom_line(aes(day, smooth), color="red")

# kernel
span <- 7
fit <- with(polls_2008, ksmooth(day, margin,  x.points = day, kernel="normal", bandwidth = span))
polls_2008 %>% mutate(smooth = fit$y) %>%
  ggplot(aes(day, margin)) +
  geom_point(size = 3, alpha = .5, color = "grey") + 
  geom_line(aes(day, smooth), color="red")

Local Weighted Regression (loess)

• A limitation of the bin smoothing approach is that we need small windows for the approximately constant assumptions to hold which may lead to imprecise estimates of f(x). Local weighted regression (loess) permits us to consider larger window sizes.
• One important difference between loess and bin smoother is that we assume the smooth function is locally linear in a window instead of constant.
• The result of loess is a smoother fit than bin smoothing because we use larger sample sizes to estimate our local parameters.

Code: Loess Function

polls_2008 %>% ggplot(aes(day, margin)) +
  geom_point() + 
  geom_smooth(color="red", span = 0.15, method.args = list(degree=1))