Papers
Event:
-
2508.0001ViewThe Other Side of Foundation Models for Reinforcement Learning: Hacking Rewards with Vision-Language ModelsRecent studies have explored the integration of Vision Language Models (VLMs) and Reinforcement Learning (RL) to tackle complex decision-making tasks. By leveraging the zero-shot captioning capabilities of pre-trained VLMs, an agent can be trained to maximize rewards generated through text prompts. Despite the promise of these recent advances, we reveal a potentially significant limitation: generated rewards are susceptible to hacking. This means that an agent, when manipulated in-env, can inadvertently cause poor performance under true rewards. To illustrate this, we conduct experiments across six distinct environments that span both visual and state inputs, as well as manipulation and navigation tasks. Notably, our findings demonstrate that reward hacking is prevalent in all these setups. Given the lack of prior research on hacking in the context of rewards generated by VLMs for RL agents, we provide a comprehensive analysis of the root cause of this phenomenon and discuss potential mitigation strategies. Our findings underscore the need for increased vigilance when deploying such methods in real-world applications.
-
2507.0001ViewCode2Reward: Preference-Based Prompting for Reward DesignReward function design is a longstanding challenge in reinforcement learning (RL). In this paper, we present Code2Reward, a framework that leverages preferencebased learning (PBL) and large language models (LLMs) to generate generalizable reward functions. Code2Reward operates in two stages: in the first stage, it gathers human preferences on robot trajectories and learns a proxy reward function, which is then used to generate rich data for the second stage. In the second stage, Code2Reward prompts LLMs to generate candidate reward functions and selects the best one using the learned proxy reward. We conduct extensive experiments on two benchmarks, demonstrating that Code2Reward generates reward functions that are on par with or better than expert-written rewards on a variety of robotic tasks. You can find more information at https://code2reward.io/.
-
2505.0002ViewWorld GPT: An Auto-Regressive World Model for Reinforcement LearningReinforcement learning (RL) agents can significantly benefit from learning an internal world model to predict future observations, which can then be used to train a policy more efficiently. We introduce World GPT, an auto-regressive world model that combines a semantic prior with a quantized latent space to capture complex environments more accurately and efficiently. In contrast to prior approaches, World GPT does not require any re-configuration of the model to generate multiple future frames. Instead, it can fully benefit from the latent space of a pre-trained VQ-GAN model, which can be trained independently of the RL task. Our experiments in the Atari 100K benchmark show that World GPT outperforms prior model-based approaches in terms of data efficiency and planning abilities in complex environments while reducing computational costs. Finally, we demonstrate that World GPTβs generation capabilities open up exciting new possibilities for exploration and real-world applications such as training free-form interactive agents.
-
2505.0001ViewReversed Smoothed Quantile Regression for Distributed High-Dimensional DataHigh-dimensional distributed quantile regression (QR) is studied in this paper. To overcome the non-smooth issue of the check loss function, a popular approach is to smooth it. However, the smoothed QR estimator and its inferential procedures require a large minimum local sample size. To address the problem, we propose a new estimator by combining the reversed smoothed check loss and β1-penalization. Theoretically, in terms of estimation, we establish the minimax optimal convergence rate for the global estimator and the valid confidence interval for an individual coefficient. In terms of computation and communication, we show that the proposed iterative algorithm converges linearly for a fixed number of machines and requires only a logarithmic number of communication rounds. Additionally, our theoretical results hold under a weaker condition on the minimum local sample size. Numerical experiments corroborate our theoretical claims.