Fix race when backfilling in rooms with worker mode.