T3Mermaid to ASCII art (mermaid-ascii)T3Kimi K3, and what we can still learn from the pelican benchmarkT3Firefox in WebAssemblyT3Spot birds not golfT3[AINews] Kimi K3 2.8T-A50B: the largest open model ever released; Opus 4.8-class at Sonnet 5 pricingT1From physical surfaces to human-centric heat stress: LST and UTCI heat mapping reveals nonlinear effects of urban morphologyT1DualHNIE: Dual-Channel Hypergraph Learning for Node Importance Estimation in Heterogeneous Knowledge GraphsT1GenTL: A General Transfer Learning Model for Building Thermal DynamicsT1A short review on the maximum clique problem algorithms with classical, AI, and quantum methodsT1Man, Machine, and Masterpiece: Artistic Ownership in the AI EraT1HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective OptimizationT1How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay DraftsT3Mermaid to ASCII art (mermaid-ascii)T3Kimi K3, and what we can still learn from the pelican benchmarkT3Firefox in WebAssemblyT3Spot birds not golfT3[AINews] Kimi K3 2.8T-A50B: the largest open model ever released; Opus 4.8-class at Sonnet 5 pricingT1From physical surfaces to human-centric heat stress: LST and UTCI heat mapping reveals nonlinear effects of urban morphologyT1DualHNIE: Dual-Channel Hypergraph Learning for Node Importance Estimation in Heterogeneous Knowledge GraphsT1GenTL: A General Transfer Learning Model for Building Thermal DynamicsT1A short review on the maximum clique problem algorithms with classical, AI, and quantum methodsT1Man, Machine, and Masterpiece: Artistic Ownership in the AI EraT1HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective OptimizationT1How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts

RL without TD learning

Lab/vendor primary source

#216

T2digested

Canonical URL: http://bair.berkeley.edu/blog/2025/11/01/rl-without-td-learning/
Topic: Post-Training
First seen: 2026-07-16 19:08:00
Last seen: 2026-07-16 19:08:00

Source raw items (1)

Blog / Newsletter2026-07-16 19:07:20
RL without TD learning
 <p>In this post, I’ll introduce a reinforcement learning (RL) algorithm based on an “alternative” paradigm: <strong>divide and conquer</strong>. Unlike traditional methods, this algorithm is <em>not</em> based on temporal difference (TD) learning (which has <a href="https://seohong.me/blog/q-learning-is-not-yet-scalable/">scalability challenges</a>), and scales well to long-horizon tasks.</p> <p style="text-align: center;"> <img alt="" src="https://bair.berkeley.edu/static/blog/rl-without-td-learning/teaser_short.png" width="100%" /> <br /> <i style="font-size: 0.9em;">We can do Reinforcement Learning (RL) based on divide and conquer, instead of temporal difference (TD) learning.</i> </p>  <h2 id="problem-setting-off-policy-rl">Problem setting: off-policy RL</h2> <p>Our problem setting is <strong>off-policy RL</strong>. Let’s briefly review what this means.</p> <p>There are two classes of algorithms in RL: on-policy RL and off-policy RL. On-