American workers have largely not taken the jobs vacated by immigrant workers