For a given level of intelligence, making Mixture of Experts (MOE) models sparser can save a significant amount of flops

Opening sign-in…