Fix bug where outboud federation would stop talking to some servers when using workers