How to Build a Personal AI Assistant That Understands Your Emails and Calendar
October 23, 2025Building Smarter AI Agents with Habit Memory
October 23, 2025Why does this matter?
Imagine if every time you cooked, you only ever used salt and pepper. Then one day, you discover that adding a little bit of garlic and paprika makes the dish taste so much better. That is what happened when researchers used evolutionary algorithms to discover a new way for AI models to pay attention to information. Instead of sticking with the standard method called softmax attention, they found that a different approach called sparsemax with output gating works significantly better. This discovery was not made by humans guessing, but by an AI system that tested over 384 different ways of doing attention, finally landing on one that works 4% better.
The researchers set up an experiment where they created many different “children” AI models, each with a slightly different way of handling attention. They let these models compete, and only the best ones were allowed to “reproduce” by creating new variants. Over 10 generations, with 12 models per generation, the system evolved until it found a winner. The winning model used sparsemax instead of softmax for attention, and it also used a technique called output gating. This combination performed better than the standard transformer model we use in most AI systems today.
- Evolved attention mechanism beats standard transformer by 4% on WikiText-2
- Uses sparsemax and output gating instead of softmax
- Ran on free Google Colab, no supercomputers needed
- Open source code and data available for anyone to use and improve
- Shows the power of evolutionary algorithms in AI research
Key results at a glance
In most AI models, attention is a core mechanism that decides which parts of the input are most important. The standard method uses a mathematical function called softmax, which works well but might not be optimal. The evolved alternative, sparsemax, is like a more decisive version. If softmax says “maybe this part is a little important,” sparsemax says “this part is important, and this other part is not.” This clarity helps the model learn more efficiently. Additionally, output gating acts like a final check, ensuring only the most relevant information is used. Together, they make the model not only smarter but also more efficient.
While this breakthrough was achieved on a small scale using models with only 2 layers and 128 dimensions, and tested only on WikiText-2, the results are promising. The team behind this project, who open-sourced all their code and data, hope that others will test these ideas at larger scales. They note that while their work might not immediately revolutionize giant models like GPT-4, it shows that our current best practices might not be the best possible. By exploring with AI, we can find better building blocks for AI. This is a reminder that sometimes the best way to solve a problem is not to think harder, but to let machines explore and discover for us.
