In the field of statistical learning theory, matrix regularization generalizes notions of vector regularization to cases where the object to be learned is a matrix. The purpose of regularization is to enforce conditions, for example sparsity or smoothness, that can produce stable predictive functions. For example, in the more common vector framework, Tikhonov regularization optimizes over

to find a vector that is a stable solution to the regression problem. When the system is described by a matrix rather than a vector, this problem can be written as

Matrix regularization

where the vector norm enforcing a regularization penalty on has been extended to a matrix norm on .

Matrix regularization

Matrix regularization has applications in matrix completion, multivariate regression, and multi-task learning. Ideas of feature and group selection can also be extended to matrices, and these can be generalized to the nonparametric case of multiple kernel learning.

Matrix regularization

Basic definition

Consider a matrix to be learned from a set of examples, , where goes from to , and goes from to . Let each input matrix be , and let be of size . A general model for the output can be posed as

where the inner product is the Frobenius inner product. For different applications the matrices will have different forms, but for each of these the optimization problem to infer can be written as

where defines the empirical error for a given , and is a matrix regularization penalty. The function is typically chosen to be convex and is often selected to enforce sparsity (using -norms) and/or smoothness (using -norms). Finally, is in the space of matrices with Frobenius inner product .

General applications

Matrix completion

In the problem of matrix completion, the matrix takes the form

where and are the canonical basis in and . In this case the role of the Frobenius inner product is to select individual elements from the matrix . Thus, the output is a sampling of entries from the matrix .

The problem of reconstructing from a small set of sampled entries is possible only under certain restrictions on the matrix, and these restrictions can be enforced by a regularization function. For example, it might be assumed that is low-rank, in which case the regularization penalty can take the form of a nuclear norm.

where , with from to , are the singular values of .

Multivariate regression

Models used in multivariate regression are parameterized by a matrix of coefficients. In the Frobenius inner product above, each matrix is

such that the output of the inner product is the dot product of one row of the input with one column of the coefficient matrix. The familiar form of such models is

Many of the vector norms used in single variable regression can be extended to the multivariate case. One example is the squared Frobenius norm, which can be viewed as an -norm acting either entrywise, or on the singular values of the matrix:

In the multivariate case the effect of regularizing with the Frobenius norm is the same as the vector case; very complex models will have larger norms, and, thus, will be penalized more.

Multi-task learning

The setup for multi-task learning is almost the same as the setup for multivariate regression. The primary difference is that the input variables are also indexed by task (columns of ). The representation with the Frobenius inner product is then

The role of matrix regularization in this setting can be the same as in multivariate regression, but matrix norms can also be used to couple learning problems across tasks. In particular, note that for the optimization problem

the solutions corresponding to each column of are decoupled. That is, the same solution can be found by solving the joint problem, or by solving an isolated regression problem for each column. The problems can be coupled by adding an additional regularization penalty on the covariance of solutions

where models the relationship between tasks. This scheme can be used to both enforce similarity of solutions across tasks, and to learn the specific structure of task similarity by alternating between optimizations of and . When the relationship between tasks is known to lie on a graph, the Laplacian matrix of the graph can be used to couple the learning problems.