Fix a long-standing bug where some to_device messages could be dropped when using workers.