Nonparametric smoothing techniques allow for the estimation of nonlinear relationships between continuous variables. In conjunction with standard statistical models, these smoothing techniques provide the means to test for, and estimate, nonlinear relationships in a wide variety of analyses. Until recently these methods have been little used within the social sciences. Semiparametric Regression for the Social Sciences sets out to address this situation by providing an accessible introduction to the subject, filled with examples drawn from the social and political sciences.

Readers are introduced to the principles of nonparametric smoothing and to a wide variety of smoothing methods. The author also explains how smoothing methods can be incorporated into parametric linear and generalized linear models. The use of smoothers with these standard statistical models allows the estimation of more flexible functional forms whilst retaining the interpretability of parametric models. The full potential of these techniques is highlighted via the use of detailed empirical examples drawn from the social and political sciences. Each chapter features exercises to aid in the understanding of the methods and applications.

Semiparametric Regression for the Social Sciences is supported by a supplementary website containing all the datasets used and computer code for implementing the methods in S-Plus and R. The book will prove essential reading for students and researchers using statistical models in areas such as sociology, economics, psychology, demography and marketing.

Luke J. Keele – Department of Political Science, Ohio State University, US Since acquiring his PhD, Dr Keele has published work in a number of international journals, including papers on this specific topic. He has also taught the material for the proposed book at Ohio State University and presented it at international meetings.Dr Keele is a political scientist by trade but has considerable experience in applying statistical techniques to social science applications.

Semiparametric Regression for the Social Sciences

By Luke John Keele

John Wiley & Sons

Copyright © 2008 John Wiley & Sons, Ltd
All right reserved.
ISBN: 978-0-470-31991-8

Chapter One

Introduction: Global versus Local Statistics

Statistical models are always simplifications, and even the most complicated model will be a pale imitation of reality. Given this fact, it might seem a futile effort to estimate statistical models, but George Box succinctly described the nature of statistical research: ‘All models are wrong, some are useful.’ Despite the fact that our models are always wrong, statistics provides us with considerable insight into the political, economic and sociological world that we inhabit. Statistical models become simplifications of reality because we must make assumptions about various aspects of reality. The practice of statistics is not shy about the need to make assumptions. A large part of statistical modeling is performing diagnostic tests to ensure that the assumptions of the model are satisfied. And in the social sciences, a great deal of time is devoted to checking assumptions about the nature of the error term: are the errors heteroskedastic or serially correlated? Social scientists are far more lax, however, when it comes to testing assumptions about the functional form of the model. In the social sciences, the linear functional (and usually additive) form reigns supreme, and researchers often do little to verify the linearity assumption. Much effort is devoted to specification and avoiding misspecification, but little is done to explore other functional forms when the incorrect functional form is in essence a specification error.

The reliance on linear functional forms is more widespread than many probably realize. For many analysts, the question of linearity is focused solely on the nature of the outcome variable in a statistical model. If the outcome variable is continuous, we can usually estimate a linear regression model using least squares. For discrete outcomes, analysts typically estimate generalized regression models such a logistic or Poisson regression. Researchers often believe that because they are estimating a logistic or Poisson regression model, they have abandoned the linear functional form. Often this is not the case, as the functional form for these models remains linear in an important way. The generalized linear model (GLM) notation developed by McCullagh and Nelder (1989) helps clarify the linearity assumption in models that many researchers think of as nonlinear.

In the GLM framework, the analyst makes three choices to specify the statistical model. First, the analyst chooses the stochastic component of the model by selecting a sampling distribution for the dependent variable. For example, we might choose the following sampling distribution for a continuous outcome:

[Y.sub.i] ~ N([.sub.i], [[sigma].sup.2]. (1.1)

Here, the outcome [Y.sub.i] follows a Normal sampling distribution with expected value [.sub.i] and a constant variance of [[sigma].sup.2]. This forms the stochastic component of the statistical model. Next, the analyst must define the systematic part of the model by choosing a set of predictor variables and a functional form. If we have data on k predictors, X is an n k matrix containing k predictors for n observations, and [eta] is an n 1 vector of linear predictions:

[eta] = X’ (1.2)

where is a vector of parameters whose values are unknown and must be estimated. Both X’ and [eta] are interchangeably referred to as the linear predictor. The linear predictor forms the systematic component of the model. The systematic component need not have a linear functional form, but linearity is typically assumed. Finally, the analyst must choose a link function. The link can be as simple as the identity function:

[.sub.i] = [eta]. (1.3)

The link function defines the connection between the stochastic and systematic parts of the model. To make the notation more general, we can write the link function in the following form:

[.sub.i] = g([[eta].sub.i]), (1.4)

where g() is a link function that must be monotonic and differentiable. When the stochastic component follows a Normal distribution and the identity function links the stochastic and systematic components, this notation describes a linear regression model. The GLM framework, however, generalizes beyond linear regression models. The stochastic component may come from any of the exponential family of distributions, and any link function that is monotonic and differentiable is acceptable.

As an example of this generality, consider the GLM notation for a model with a binary dependent variable. Let [Y.sub.i] be a binary variable without outcomes 0/1, where 1 represents a success for each [m.sub.i] trials and [[psi].sub.i] is the probability of a success for each trial. If this is true, we might assume that [Y.sub.i] follows a Bernoulli distribution, which would imply the following stochastic component for the model:

[Y.sub.i] | [[psi].sub.i] ~ B([m.sub.i], [[psi].sub.i]). (1.5)

The systematic component remains X’, the linear predictor. We must now select a link function that will ensure that the predictions from the systematic component lie between 0 and 1. The logistic link function is a common choice

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (1.6)

With the logit link function, no matter what value [[eta].sub.i] takes the predicted value for [Y.sub.i], [[psi].sub.i], will always be between zero and one.

What has been the purpose of recounting the GLM notation? The point of this exerciseistodemonstratethatwhilethelinkfunctionisanonlineartransformation of the linear predictor, the systematic component of the model remains linear. Across the two examples, both the link function and stochastic component of the model differed, but in both cases we used the linear predictor X’. The second model, a logistic regression, is thought of as a nonlinear model, but it has a linear functional form. Thus many of the models that analysts assume are nonlinear models retain a linearity assumption. There is nothing about the logistic regression model (or any other GLM) that precludes the possibility that the model is nonlinear in the variables. That is, instead of the effect of X on Y, which is summarized by the estimated coefficient, being constant, that effect varies across the values of X. For example, one is as likely to use a quadratic term in a Poisson regression model as a linear regression model.

Why, then, is the assumption of linearity so widespread? And why is model checking for deviations from nonlinearity so rare? This question cannot be answered definitively. Quite possibly, the answer lies in the nature of social science theory. As Beck and Jackman (1998) note: ‘few social scientific theories offer any guidance as to functional form whatsoever.’ When researchers stipulate the predicted relationship between X and Y, they do not go beyond ‘the relationship between X and Y is expected to be positive (or negative).’ Given that most theory is silent as to the exact functional form, linearity has become a default, while other options are rarely explored. We might ask what are the consequences of ignoring nonlinearity?

1.1 The Consequences of Ignoring Nonlinearity

What are the consequences for using the wrong functional form? Simulation provides some useful insights into how misspecification of functional form can affect our inferences. First, we need some basic notation. We start with the linear and additive functional form for Y

Y = [.sub.0] + [.sub.1][X.sub.1] + [.sub.2][X.sub.2]. (1.7)

The above equation is linear in the parameters in that we have specified that the mean of Y is a linear function of the variables [X.sub.1] and [X.sub.2]. It is possible that the effect of [X.sub.2] on Y is nonlinear. If so, the effect of [X.sub.2] on Y will vary across the values of [X.sub.2]. Such a model is said to be nonlinear in the variables but linear in the parameters. Estimation of models that are nonlinear in the variables presents few problems, which is less true of models that are nonlinear in the parameters. Consider a model of this type

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Here, the model is nonlinear in the parameters and estimation is less easy as nonlinear least squares is required. For a model that is nonlinear in the variables, we can still use least squares to estimate the model. What are the consequences of ignoring nonlinearity in the variables if it is present? For example, assume that the true data generating process is

Y = [.sub.0] + [.sub.1][X.sub.1] + [.sub.2][X.sup.2.sub.1]. (1.8)

Suppose an analyst omits the quadratic term when estimating the model. The effect of omitting the nonlinear term from the right hand side of a regression model is easily captured with simulated data. To capture the above data generating process, we simulate [X.sub.1] as 500 draws from a uniform distribution on the interval 1 to 50. Y is a function of this X variable with an error term drawn from a Normal distribution with zero mean and constant variance and the parameters are set to ones. What are the consequences of fitting a linear model without the quadratic term? Figure 1.1 contains a plot of the estimated regression line when only a linear term has been included on the right hand side of the estimated model and the true quadratic functional form.

Here, the model with the incorrect functional form would lead one to conclude that X is unrelated to Y as the regression line is virtually flat, when in fact the curved line reflects the strong but nonlinear relationship between the two variables. This simulated example illustrates the misspecification that results from assuming linearity when the relationship between X and Y is actually nonlinear. In this particular example, the consequences are particularly severe, given that an analyst might conclude that there is no relationship between X and Y, when the two are strongly related. Moreover, as the GLM framework implies, models such as logistic or Poisson regression are equally prone to this misspecification from an incorrect functional form. If [X.sup.2] is in the true data generating process, no link function will correct the specification error. In these models, the systematic component of the model often remains linear and the failure to include a nonlinear term in the model will have equally deleterious effects on the model estimates. In fact, given the serious distortions such a misspecification can cause, it would be prudent to test that the effect of any continuous covariate on the right hand side of a model does not have a nonlinear effect. If we admit that defaulting to linearity might be a problematic method of data analysis, what alternatives are available? While there are a number of possibilities, analysts in the social sciences usually rely on power transformations to address nonlinearity.

1.2 Power Transformations

Power transformations are a simple and flexible means of estimating a nonlinear functional form. While a variety of power transformations are possible, most researchers restrict themselves to only one or two transformations. A common notation exists for all power transformations that is useful to outline since we will often use power transformations in the data analytic examples that follow. For a strictly positive variable, X, we can define the following set of power transformations:

[X.sup.[lambda]]. (1.9)

Using different values for [lambda] produces a wide variety of transformations. If [lambda] is 2 or 3, the transformation is quadratic or cubic respectively, and if [lambda] is 1/2 or 1/3, the transformation is either the squared or cubic root. By convention, a value of 0 for [lambda] denotes a log transformation, and a value of 1 for [lambda] corresponds to no transformation (Weisberg 2005).

Power transformations are often a reasonable method for modeling nonlinear functional forms. For example, it is well understood that older people vote more often than younger people (Nagler 1991). So as age increases, we expect that the probability of a person voting increases. But we do not expect the effect of age on voter turnout to be linear since once people reach a more advanced age, it often prevents them from voting. To capture such nonlinearity, most analysts rely on a quadratic power transformation. In this model, the analyst would square the age variable and then include both the untransformed and squared variable on the right hand side of the model. If the quadratic term is statistically significant at conventional levels (p <0.05), the analyst concludes that the relationship between the two variables is nonlinear.

The process described above is often a reasonable way to proceed. Some care must be taken with the interpretation of the quadratic term as the standard error for the marginal effect must be calculated by the analyst, but the transformation method is easily to use. Moreover, the model is now a better representation of the theory, and power transformations avoid any misspecifiation due to an incorrect functional form. If the effect of age on voter turnout is truly quadratic, and we only include a linear term for age; we have misspecified the model and biased not only the estimate of age on voter turnout but all the other estimates in the model as well.

Power transformations, however, have several serious limitations. First and foremost, power transformations are global and not local fits. With a global fit, one assumes that the statistical relationship between X and Y does not vary over the range of X. When an analyst estimates a linear relationship or uses a power transformation, the assumption is that the relationship between X and Y is exactly the same for all possible values of X. The analyst must be willing to assume a global relationship when he or she estimates linear functional forms or uses power transformations. Quite often the relationship between X and Y is local: the statistical relationship between two variables is often specific to local regions of X. The assumption of a global relationship often undermines power transformation.

For example, quadratic power transformations assume that the relationship between X and Y for all values of X is strictly quadratic regardless of whether this is true or not. When it is not, the power transformation can overcorrect the nonlinearity between X and Y. While there are a variety of power transformations, each assumes a global form of nonlinearity, which may or may not be correct. A variety of more complex nonlinear forms cannot be easily modeled with power transformations. Therefore, while power transformations can model some forms of nonlinearity, the global nature of power transformation fits often undermines them as a data analytic technique. In several of the data examples that follow, we see how power transformations often cannot adequately capture the nonlinear relationship in the data.

More seriously, the choice about which power transformation to use is often arbitrary. At best our theory might indicate that an effect is nonlinear, such as it does with age and voter turnout, but to have a theory that actually indicates that the effect is quadratic is unlikely. For example, how might one choose between a logarithmic or quadratic power transformation? One can estimate a model where the value of [lambda] is estimated as the value that minimizes the sum of squared model errors. Unfortunately, this method often produces transformations that are uninterpretable (Berk 2006).

(Continues…)

Excerpted from Semiparametric Regression for the Social Sciencesby Luke John Keele Copyright © 2008 by John Wiley & Sons, Ltd. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.