Double-negative activations occur in SwiGLU when the gate projection and up projection both produce negative values that multiply into a positive product
Agree
Disagree
Overview
Debates
Related claims
Topics
Activity
Debates
…
Claims
…
View all debates
Activity (0)
Join the conversation...
Loading comments...
Opening sign-in…