Over the past few days, I went from “What is JEPA?” to successfully installing and running LeWorldModel — one of the most promising recent approaches toward real world models from Yann LeCun’s new lab, Advanced Machine Intelligence (AMI).
Why This Matters
Current large language models are impressive but fundamentally limited. LeCun argues we need world models that understand physics, causality, and plan actions — not just predict the next token. JEPA (Joint Embedding Predictive Architecture) is his proposed solution: instead of generating pixels or tokens, it learns to predict in abstract latent space, making it more efficient and grounded in reality.
LeWorldModel (LeWM) is the latest practical implementation — stable, end-to-end trainable from raw pixels, and runnable even on a single GPU.
My Setup Journey (Debian + GT 1030)
- OS: Debian Linux
- GPU: NVIDIA GeForce GT 1030 (2GB VRAM)
- Environment: Clean Conda environment with Python 3.10
- Final working stack: PyTorch 2.6.0 + CUDA 12.4
Key Lessons Learned:
- Always match PyTorch’s CUDA version to your driver (this was the trickiest part).
- Use Conda environments for complex research code.
- Start tiny — big configs will instantly OOM on consumer GPUs.
What I Have Running Now
- ✅ Full LeWorldModel codebase installed
- ✅ Push-T robotics dataset ready
- ✅ Tiny JEPA model training pipeline working on my GT 1030
The model is currently learning basic physical intuitions (object motion, collisions, path prediction) in a simple robotic pushing environment.
What’s Next?
- Train longer on Push-T and visualize learned representations
- Try action-conditioned prediction
- Explore how the latent space encodes momentum and surprise (Violation-of-Expectation)
This is just the beginning of my journey into predictive world models — the path LeCun believes will take us beyond today’s LLMs toward truly intelligent systems.