In this paper, we study reinforcement learning (RL) with general function approximation, where either the value function or the model dynamics is approximated by a given abstract hypothesis space. We propose the generalized eluder coefficient (GEC), which measures the hardness of generalization from the historical in-sample error to the prediction error, and further serves to measure the hardness of learning an RL problem. In terms of the algorithmic design, we propose an optimization-based framework for RL with general function approximation, following the general principle of “Optimism in the Face of Uncertainty” (OFU). Compared to existing algorithms, the proposed framework does not explicitly maintain the confidence set, and neatly handles both model-free and model-based problems wi...[ Read more ]

In this paper, we study reinforcement learning (RL) with general function approximation, where either the value function or the model dynamics is approximated by a given abstract hypothesis space. We propose the generalized eluder coefficient (GEC), which measures the hardness of generalization from the historical in-sample error to the prediction error, and further serves to measure the hardness of learning an RL problem. In terms of the algorithmic design, we propose an optimization-based framework for RL with general function approximation, following the general principle of “Optimism in the Face of Uncertainty” (OFU). Compared to existing algorithms, the proposed framework does not explicitly maintain the confidence set, and neatly handles both model-free and model-based problems with a low GEC. Theoretical analysis shows that our regret results match those provided by existing frameworks.