Double-negative activations occur in SwiGLU when the gate projection and up projection both produce negative values that multiply into a positive product

Activity

Activity (0)
Loading comments...
Opening sign-in…