T3Mermaid to ASCII art (mermaid-ascii)T3Kimi K3, and what we can still learn from the pelican benchmarkT3Firefox in WebAssemblyT3Spot birds not golfT3[AINews] Kimi K3 2.8T-A50B: the largest open model ever released; Opus 4.8-class at Sonnet 5 pricingT1From physical surfaces to human-centric heat stress: LST and UTCI heat mapping reveals nonlinear effects of urban morphologyT1DualHNIE: Dual-Channel Hypergraph Learning for Node Importance Estimation in Heterogeneous Knowledge GraphsT1GenTL: A General Transfer Learning Model for Building Thermal DynamicsT1A short review on the maximum clique problem algorithms with classical, AI, and quantum methodsT1Man, Machine, and Masterpiece: Artistic Ownership in the AI EraT1HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective OptimizationT1How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay DraftsT3Mermaid to ASCII art (mermaid-ascii)T3Kimi K3, and what we can still learn from the pelican benchmarkT3Firefox in WebAssemblyT3Spot birds not golfT3[AINews] Kimi K3 2.8T-A50B: the largest open model ever released; Opus 4.8-class at Sonnet 5 pricingT1From physical surfaces to human-centric heat stress: LST and UTCI heat mapping reveals nonlinear effects of urban morphologyT1DualHNIE: Dual-Channel Hypergraph Learning for Node Importance Estimation in Heterogeneous Knowledge GraphsT1GenTL: A General Transfer Learning Model for Building Thermal DynamicsT1A short review on the maximum clique problem algorithms with classical, AI, and quantum methodsT1Man, Machine, and Masterpiece: Artistic Ownership in the AI EraT1HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective OptimizationT1How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts

Thinking about High-Quality Human Data

Named practitioner synthesis

#236

T3digested

Canonical URL: https://lilianweng.github.io/posts/2024-02-05-human-data-quality/
Topic: Practitioner Notes
First seen: 2026-07-16 19:08:00
Last seen: 2026-07-16 19:08:00

Source raw items (1)

Blog / Newsletter2026-07-16 19:07:22
Thinking about High-Quality Human Data
<p><span class="update">[Special thank you to <a href="https://scholar.google.com/citations?user=FRBObOwAAAAJ&hl=en">Ian Kivlichan</a> for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ]</span><br /></p> <p>High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or <a href="https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/#rl-fine-tuning-with-human-preferences">RLHF</a> labeling (which can be constructed as classification format) for LLM alignment training. Lots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution. The community knows the value of high quality data, but somehow we have this subtle impression that “Everyone wants to do the model work, not the data work” (<a href="https://dl.