THESIS
2018
xiv, 119 pages : illustrations ; 30 cm
Abstract
In this thesis, we propose three novel dimension reduction and coefficient estimation
methods for multivariate linear or nonlinear regression, respectively.
First, for simultaneous variable selection and parameter estimation under a linear
model, this thesis studies an iteratively reweighted L
2 penalized least squares
method, or broken adaptive ridge (BAR) regression. Specifically, the BAR estimator
is defined as the limit of an iterative algorithm that approximates an
L
0 penalized regression through a series of reweighted ridge regression. We establish
that the BAR estimator possesses the following oracle properties: when
the true model has a sparse representation, it correctly shrinks the estimate of
a zero component to zero and estimates the non-zero components as well as th...[
Read more ]
In this thesis, we propose three novel dimension reduction and coefficient estimation
methods for multivariate linear or nonlinear regression, respectively.
First, for simultaneous variable selection and parameter estimation under a linear
model, this thesis studies an iteratively reweighted L
2 penalized least squares
method, or broken adaptive ridge (BAR) regression. Specifically, the BAR estimator
is defined as the limit of an iterative algorithm that approximates an
L
0 penalized regression through a series of reweighted ridge regression. We establish
that the BAR estimator possesses the following oracle properties: when
the true model has a sparse representation, it correctly shrinks the estimate of
a zero component to zero and estimates the non-zero components as well as the
scenario when the true sub-model is known in advance. In addition, we show
that the BAR estimator enjoys a desirable grouping property that highly correlated
variables are naturally assigned similar coefficients, so that they are in or
out of a model together. We conduct extensive empirical studies to illustrate the
performance of the BAR method in both low and high dimensional settings.
Furthermore, we study the generalized broken adaptive ridge method to estimate
lower-dimensional patterns of coefficients in linear regression models. The extended method can simultaneously recover both the true sparsity and the inherent structures of the features. The resulting estimate is shown to enjoy the oracle property. The proposed method also contains a set of variable selection or pattern estimation methods. As a special case, the fused broken adaptive ridge, which penalizes the differences between adjacent coefficients, is thoroughly discussed with applications in signal approximation and image processing. The associated algorithms are numerically easy to implement. Various simulation studies and real data analyses illustrate its advantages over the fused lasso method.
Last but not least, for dimension reduction under the nonlinear regression, we propose the cross-validation metric learning approach to learn a distance metric for dimension reduction in the multiple-index model. We minimize a leave-one-out cross-validation-type loss function, where the unknown link function is approximated by a metric-based kernel-smoothing function. In contrast to existing methods, the new method has several distinctive features: (1) to the best of the authors' knowledge, we are the first to apply the cross-validation technique in learning a distance metric for regression problems; (2) the resultant metric contains crucial information on both dimension reduction subspace and the optimal kernel-smoothing bandwidth; (3) it is model-free in the sense that it requires nearly no assumption on the functional relationship between the response variable and predictors, which makes it theoretically and practically appealing. Under weak assumptions on the design of predictors, the coordinates of those redundant variables are shown to be consistently zero. The convergence rate and the optimal rate of bandwidth are also justified. It is relatively easy-to-implement numerically by employing fast gradient-based algorithms. Various simulation studies and real data analyses illustrate its advantages over other existing methods.
Post a Comment