pythonsi.feature_selection
Feature selection methods with selective inference.
Classes
- class pythonsi.feature_selection.LassoFeatureSelection(lambda_: float = 10)[source]
LASSO feature selection with selective inference support.
This class implements LASSO feature selection with the capability to perform selective inference on the selected features. The LASSO optimization problem is:
\[\hat{\boldsymbol{\beta}} = \mathop{\arg \min}_{\boldsymbol{\beta}} \quad \frac{1}{2} \|\mathbf{y} - \mathbf{x}\boldsymbol{\beta}\|_2^2 + \lambda \|\boldsymbol{\beta}\|_1\]where \(\lambda\) is the regularization parameter that controls sparsity.
- Parameters:
lambda (float, optional) – Regularization parameter for LASSO, default 10
- lambda_
Regularization parameter
- Type:
float
- self.interval
Feasible interval from last inference call
- Type:
list or None
- self.active_set_data
Active set from last inference call
- Type:
array-like or None
- checkKKT_Lasso(x, Y, beta_hat, Lambda, tol=1e-10)[source]
Validate KKT conditions for LASSO solution.
Checks that the computed LASSO solution satisfies the Karush-Kuhn-Tucker optimality conditions:
Active features: \(\mathbf{x}_j^T(\mathbf{y} - \mathbf{x}\hat{\boldsymbol{\beta}}) = \lambda \cdot \text{sign}(\hat{\beta}_j)\) for \(\hat{\beta}_j \neq 0\)
Inactive features: \(|\mathbf{x}_j^T(\mathbf{y} - \mathbf{x}\hat{\boldsymbol{\beta}})| \leq \lambda\) for \(\hat{\beta}_j = 0\)
- Parameters:
x (array-like, shape (n, d)) – Design matrix
Y (array-like, shape (n, 1)) – Response vector
beta_hat (array-like, shape (d, 1)) – Estimated LASSO coefficients
Lambda (float) – Regularization parameter
tol (float, optional) – Numerical tolerance for checks, default 1e-10
- Raises:
AssertionError – If any KKT condition is violated
Notes
This is a helper function for debugging and validation. Prints detailed information about each constraint.
- forward(x: ndarray[tuple[Any, ...], dtype[floating]], y: ndarray[tuple[Any, ...], dtype[floating]]) Tuple[ndarray[tuple[Any, ...], dtype[floating]], ndarray[tuple[Any, ...], dtype[floating]], ndarray[tuple[Any, ...], dtype[floating]]][source]
Solve LASSO optimization and extract active set information.
Solves the LASSO problem and returns the active set (selected features), inactive set, and signs of active coefficients.
- Parameters:
x (array-like, shape (n, p)) – Feature matrix
y (array-like, shape (n, 1)) – Response vector
- Returns:
active_set (array-like, shape (k,)) – Indices of selected features
inactive_set (array-like, shape (p-k,)) – Indices of unselected features
sign_active (array-like, shape (k, 1)) – Signs of coefficients for active features
- inference(z: float) Tuple[list, ndarray[tuple[Any, ...], dtype[floating]]][source]
Find feasible interval of the Lasso Feature Selection for the parametrized data at z.
- Parameters:
z (float) – Inference parameter value
- Returns:
final_interval – Feasible interval [lower, upper] for z
- Return type:
list
- class pythonsi.feature_selection.SequentialFeatureSelection(n_features_to_select: int | None = None, direction: Literal['forward', 'backward'] = 'forward', criterion: Literal['aic', 'bic', 'adj_r2'] | None = None)[source]
Sequential feature selection with selective inference support.
This class performs sequential feature selection in a greedy manner, either by adding features (forward selection) or removing them (backward selection) to construct a subset of relevant features.
The number of selected features can be determined automatically using model selection criteria such as AIC, BIC, or adjusted R².
In addition, the class provides support for performing selective inference on the chosen feature set, allowing valid statistical inference after the selection process.
The selection problem for each step is to find the feature that maximizes the fit improvement (e.g., minimizes the residual sum of squares). For forward selection, at step $k$, the next feature $j$ is chosen to solve:
\[j = \mathop{\arg \min}_{j \notin \mathcal{A}_{k-1}} \quad \|\mathbf{y} - \mathbf{x}_{\mathcal{A}_{k-1} \cup \{j\}}\boldsymbol{\beta}_{k-1}\|_2^2\]where $mathcal{A}_{k-1}$ is the active set from the previous step.
- Parameters:
n_features_to_select (int, optional) – Number of features to select. If None, the number of features is determined automatically by the criterion. Default is None.
direction (Literal["forward", "backward"], optional) – The direction of the selection process. ‘forward’ adds features and ‘backward’ removes them. Default is ‘forward’.
criterion (Literal["aic", "bic", "adj_r2"], optional) – The model selection criterion to use. If None, the selection stops when n_features_to_select is reached. Default is None.
- n_features_to_select
Number of features to select, as specified by the user.
- Type:
int or None
- n_features_to_select_
The actual number of features selected. This may differ from n_features_to_select if a criterion is used.
- Type:
int or None
- direction
The direction of feature selection.
- Type:
Literal[“forward”, “backward”]
- criterion
The criterion used for automatic selection.
- Type:
Literal[“aic”, “bic”, “adj_r2”] or None
- active_set
Indices of the selected features.
- Type:
array-like or None
- interval
Feasible interval from the last inference call.
- Type:
list or None
- static RSS(Y, X)[source]
Calculates the Residual Sum of Squares (RSS).
- Returns:
rss (float) – The RSS value.
residual_vec (array-like) – The residual vector.
- fit(X, y, sigma=None)[source]
Fit the sequential feature selection model.
Performs the sequential selection process based on the configured direction and either the n_features_to_select or the criterion.
- Parameters:
X (array-like, shape (n, p)) – Feature matrix.
y (array-like, shape (n, 1)) – Response vector.
sigma (array-like, shape (n, n), optional) – Covariance matrix of the residuals.
- forward(x, y, sigma=None)[source]
Fit the sequential feature selection model.
Performs the sequential selection process based on the configured direction and either the n_features_to_select or the criterion.
- Parameters:
X (array-like, shape (n, p)) – Feature matrix.
y (array-like, shape (n, 1)) – Response vector.
sigma (array-like, shape (n, n), optional) – Covariance matrix of the residuals.
- inference(z: float) Tuple[list, ndarray[tuple[Any, ...], dtype[floating]]][source]
Find feasible interval of the Sequential Feature Selection for the parametrized data at z.
Calculates the feasible interval for the given parameter z, ensuring the selected feature set remains the same within this interval. The interval is determined by the stepwise selection process and, if applicable, the chosen model selection criterion.
- Parameters:
z (float) – Inference parameter value.
- Returns:
final_interval – Feasible interval [lower, upper] for z.
- Return type:
list
- interval_AIC(X, Portho, K, a, b, Sigma, z)[source]
Calculates the feasible interval for the AIC criterion.
- Returns:
intervals – The feasible interval.
- Return type:
list
- interval_BIC(X, Portho, K, a, b, Sigma, z)[source]
Calculates the feasible interval for the BIC criterion.
- Returns:
intervals – The feasible interval.
- Return type:
list
- interval_adjr2(X, Portho, K, a, b, z)[source]
Calculates the feasible interval for the adjusted R-squared criterion.
- Returns:
intervals – The feasible interval.
- Return type:
list
- list_residualvec(X, Y) list[source]
Generates a list of residual vectors during the selection process.
- Returns:
lst_SELEC_k (list) – A list of active sets at each step.
lst_Portho (list) – A list of orthogonal projection matrices at each step.
- run(x: Data, y: Data, covariance: Data | None = None) Data[source]
Configure sequential selection with input data and return active set node.
- Parameters:
x (array-like, shape (n, p)) – Feature matrix.
y (array-like, shape (n, 1)) – Response vector.
covariance (array-like, shape (n, n), optional) – Covariance matrix for the residuals.
- Returns:
active_set_node – Node containing selected feature indices.
- Return type: