Fix bug where outbound federation would stop talking to some servers when using workers