Akshita shares intriguing findings from her research, highlighting the challenges faced with parametric layer norms and the decision to adopt non-parametric alternatives. She recounts a fascinating anecdote about unexpected training curve irregularities linked to the random number generator in PyTorch, emphasizing the importance of documenting such insights that often go unrecorded in formal papers. The discussion also touches on the architectural choices made to ensure model comparability with prior work.
