October 23, 2025Published by Phil at October 23, 2025Categories Artificial Intelligence Machine LearningHow an AI discovered a better attention mechanism than we use todayEvolutionary algorithms discover a novel attention mechanism that outperforms standard transformers by 4% on WikiText-2, using sparsemax and output gating instead of softmax.