Scaling synapse via workers
---------------------------

Synapse has experimental support for splitting out functionality into
multiple separate python processes, helping greatly with scalability.  These
processes are called 'workers', and are (eventually) intended to scale
horizontally independently.

All processes continue to share the same database instance, and as such, workers
only work with postgres based synapse deployments (sharing a single sqlite
across multiple processes is a recipe for disaster, plus you should be using
postgres anyway if you care about scalability).

The workers communicate with the master synapse process via a synapse-specific
TCP protocol called 'replication' - analogous to MySQL or Postgres style
database replication; feeding a stream of relevant data to the workers so they
can be kept in sync with the main synapse process and database state.

To enable workers, you need to add a replication listener to the master synapse, e.g.::

    listeners:
      - port: 9092
        bind_address: '127.0.0.1'
        type: replication

Under **no circumstances** should this replication API listener be exposed to the
public internet; it currently implements no authentication whatsoever and is
unencrypted.

You then create a set of configs for the various worker processes.  These should be
worker configuration files should be stored in a dedicated subdirectory, to allow
synctl to manipulate them.

The current available worker applications are:
 * synapse.app.pusher - handles sending push notifications to sygnal and email
 * synapse.app.synchrotron - handles /sync endpoints.  can scales horizontally through multiple instances.
 * synapse.app.appservice - handles output traffic to Application Services
 * synapse.app.federation_reader - handles receiving federation traffic (including public_rooms API)
 * synapse.app.media_repository - handles the media repository.
 * synapse.app.client_reader - handles client API endpoints like /publicRooms

Each worker configuration file inherits the configuration of the main homeserver
configuration file.  You can then override configuration specific to that worker,
e.g. the HTTP listener that it provides (if any); logging configuration; etc.
You should minimise the number of overrides though to maintain a usable config.

You must specify the type of worker application (worker_app) and the replication
endpoint that it's talking to on the main synapse process (worker_replication_host
and worker_replication_port).

For instance::

    worker_app: synapse.app.synchrotron

    # The replication listener on the synapse to talk to.
    worker_replication_host: 127.0.0.1
    worker_replication_port: 9092

    worker_listeners:
     - type: http
       port: 8083
       resources:
         - names:
           - client

    worker_daemonize: True
    worker_pid_file: /home/matrix/synapse/synchrotron.pid
    worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml

...is a full configuration for a synchrotron worker instance, which will expose a
plain HTTP /sync endpoint on port 8083 separately from the /sync endpoint provided
by the main synapse.

Obviously you should configure your loadbalancer to route the /sync endpoint to
the synchrotron instance(s) in this instance.

Finally, to actually run your worker-based synapse, you must pass synctl the -a
commandline option to tell it to operate on all the worker configurations found
in the given directory, e.g.::

    synctl -a $CONFIG/workers start

Currently one should always restart all workers when restarting or upgrading
synapse, unless you explicitly know it's safe not to.  For instance, restarting
synapse without restarting all the synchrotrons may result in broken typing
notifications.

To manipulate a specific worker, you pass the -w option to synctl::

    synctl -w $CONFIG/workers/synchrotron.yaml restart

All of the above is highly experimental and subject to change as Synapse evolves,
but documenting it here to help folks needing highly scalable Synapses similar
to the one running matrix.org!