Background: Best subset selection for sparse linear regression is a classical statistical problem that has received renewed attention following a new approach to directly solve using mixed integer quadratic optimization (MIQO) (Bertsimas et al., 2016). Prior to this approach, best subset selection was too computationally demanding to be practical for data sets with more than a few dozen variables. Accordingly, the best subset is often approximated with the heuristic algorithm of stepwise variable selection or replaced with relaxations such as the lasso. But approaching best subsets as MIQO problems has led to new developments in two complimentary areas of research: the efficient solving of MIQO problems representing best subset selection and the utility of best subset selection for prediction compared to the common alternatives. Our research adds to the former by applying reinforcement learning (RL) to accelerate the solving of these MIQO problems. Recent research in the latter area has shown that predictive models based on best subset selection can be improved by combining best subset selection with $l_2$ regularization, and we incorporate this into our research. Methods: We use RL to solve best subset selection problems with $l_2$ regularization, where the agent learns to solve these problems with increasing efficiency, as measured by the number of actions required to find an optimal solution while proving that solution's optimality. Specifically, our RL agent must choose to branch one of the active nodes of a Branch and Bound (B\&B) tree using one of the variables available in that node. This corresponds to dividing a best subset problem into two sub-problems: one in which the variable used for branching is included in the best subset and the other in which the variable is excluded. To the best of our knowledge, this is the first research to apply RL to simultaneously optimize both the choice of node and variable for branching, as well as the first to apply RL to B\&B with quadratic objective functions. This has led us to develop a novel set of statistics to summarize the state of a B\&B process and each of the available branching options, as well as two simple transformations of the optimality gap for use as RL rewards. After training our models, we compute the permutation importance of the statistics describing the state, in order to understand their relative importance in the agent's decisions. Results: We apply our technique to a synthetic data set with varying levels of exponential correlation, signal to noise ratios, number of variables, and regularization constants. We find that our method has performance comparable to that of the combination of the two common selection heuristics of breadth-first node selection and max fraction variable selection. And our research suggests several potential paths toward improving our method. Conclusions: We have confirmed the ability of RL to effectively address sequential decision making in highly non-intuitive settings. Our results suggest there is significant potential to our unified approach to the selection of nodes and variables for branching. And our research continues the process of making best subset selection a practical option for sparse variable selection at scale.
