Noam Shazeer proposed the SwiGLU gated multi-layer perceptron architecture in a 2020 paper

Opening sign-in…