Data Analysis Toolkit #10: Simple linear regression Page 1

Simple linear regression is the most commonly used technique for determining how one variable of interest (the response variable) is affected by changes in another variable (the explanatory variable). The terms "response " and "explanatory " mean the same thing as "dependent " and "independent", but the former terminology is preferred because the "independent " variable may actually be interdependent with many other variables as well. Simple linear regression is used for three main purposes: 1. To describe the linear dependence of one variable on another 2. To predict values of one variable from values of another, for which more data are available 3. To correct for the linear dependence of one variable on another, in order to clarify other features of its variability. Any line fitted through a cloud of data will deviate from each data point to greater or lesser degree. The vertical distance between a data point and the fitted line is termed a "residual". This distance is a measure of prediction error, in the sense that it is the discrepancy between the actual value of the response variable and the value predicted by the line. Linear regression determines the best-fit line through a scatterplot of data, such that the sum of squared residuals is minimized; equivalently, it minimizes the error variance. The fit is "best " in precisely that sense: the sum of squared errors is as small as possible. That is why it is also termed "Ordinary Least Squares " regression. Derivation of linear regression equations The mathematical problem is straightforward: given a set of n points (Xi,Yi) on a scatterplot, find the best-fit line, ii bXaY += such that the sum of squared errors in Y, ()2 ∑ − ii YY is minimized The derivation proceeds as follows: for convenience, name the sum of squares "Q"