How Deep Learning Building Blocks Shape Neural Network Behavior Through Symmetry

The Hidden Influence of Neural Network Primitives

Deep learning relies on fundamental building blocks that most researchers consider neutral choices, but emerging research suggests these components quietly shape how neural networks think and behave. Activation functions, normalizers, optimizers, and other primitives may be introducing unintended biases through symmetry properties that influence network representations. This discovery challenges the assumption that these building blocks are merely technical choices without broader implications for model behavior.

Three groundbreaking papers collectively demonstrate that current primitive formulations create absolute coordinate frames that force activations to align with specific axes. This alignment explains why neurons often appear to represent particular concepts and why interpretability phenomena emerge in predictable patterns. The research suggests that many observed behaviors in deep learning models might not be fundamental properties but rather side effects of our current component choices.

• Activation functions create hidden coordinate systems that influence neuron alignment
• Current primitives cause representations to cluster into discrete groups rather than smooth continua
• Symmetry-based alternatives can eliminate these biases and create more continuous representations
• Interpretability phenomena like grandmother neurons may be artifacts of primitive choices rather than fundamental network properties

Experimental Evidence and New Primitive Designs

The Quantised Representations paper introduced five novel activation functions designed with different symmetry properties to test whether removing the absolute frame would create smoother representation distributions. The experiment confirmed predictions from earlier work, demonstrating that symmetry-based alternatives maintain continuous activation patterns rather than forcing discrete clustering. This finding suggests that many interpretability observations are contingent on specific primitive formulations rather than inherent to deep learning architectures.

Our choices of activation functions and normalizers may be quietly biasing networks in ways we do not fully understand, with implications for interpretability, robustness, and model design

This research arc opens exciting new directions for neural network design and interpretation. By treating symmetry as a core design axis alongside architecture and data, researchers can potentially unlock new model behaviors and improve interpretability. The findings encourage reexamining fundamental assumptions about deep learning components and considering how symmetry properties might be leveraged for better performance, robustness, and understanding of neural networks.