Commit graph

18658 commits

Author SHA1 Message Date
Andrew Morgan 3186324260 Merge branch 'master' into develop 2021-04-22 11:23:56 +01:00
Andrew Morgan 0f2629ebc6 Synapse 1.32.2 (2021-04-22)
===========================
 
 This release includes a fix for a regression introduced in 1.32.0.
 
 Bugfixes
 --------
 
 - Fix a regression in Synapse 1.32.0 and 1.32.1 which caused `LoggingContext` errors in plugins. ([\#9857](https://github.com/matrix-org/synapse/issues/9857))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCBTGATHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9LB8EACvUiG5xkNjItcfv4udMKF9HbQCt0r2
 zlAmlffNrNqoCWpEi7cVY/48jeS6LEcN2pB4kQwKRrKl0RWdu9dL96m0DSUjwjiU
 gqJYEZLLgvnrAVoum91DvW3hiqUn8h1XEMd4VMo+4KeASoakrZOyo++Bj97gtN8H
 MiZbINYP51jDF8tSYNPhJW/kP7d67Zbf51Emi5JgMnUbonJc8ilg6uKsQg/fA1Gk
 iShEbOOqFM55VVbPq/bUnhWYzhiPq+pY1VJjMm0QmMnEupS6ANMYtXMt8ULpyZLO
 rEgpYlheADE7snrqLzPZPGjyi+jE35kUSOeYa3piB4OwZVhgron7dGWTdUSRkhx2
 cUu3JK3e2AO4eAH1sJKHs0k8T0q0jOQVcI0cTmmVskiqD2WueXWiM1YBq26IF953
 GTQaxHJ73ByOSCXb2eMlstljf5LPvuS+ywgN18jjX+adA0lBAKkcsce1btDxzlxF
 gY6f/DprV5X//H75gQt1tLml/G+58LbqPZRP4L8TcsyEviXk/zZ/K7w8e4+DqQ3/
 Zsu6rGxBEgX/ywVoPcYW6/P+ylsZ9rR2C9ujip+jdS/af4sDWh16DuGzj8eRoIeB
 M/e4bSopjVkWkh35cFgh/IMJlY3LbFJBNZyLQUG3sXiOXBRBrC+/7xTgYQXz82wW
 7M9fiwOmkLllOw==
 =KYWX
 -----END PGP SIGNATURE-----

Merge tag 'v1.32.2'

Synapse 1.32.2 (2021-04-22)
===========================

This release includes a fix for a regression introduced in 1.32.0.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 and 1.32.1 which caused `LoggingContext` errors in plugins. ([\#9857](https://github.com/matrix-org/synapse/issues/9857))
2021-04-22 11:23:34 +01:00
Andrew Morgan dac4445934 A regression can't be introduced twice 2021-04-22 11:09:31 +01:00
Andrew Morgan 79e6d9e4b1 Note regression was in 1.32.0 and 1.32.1 2021-04-22 11:04:51 +01:00
Andrew Morgan ca380881b1 Update dates in changelogs 2021-04-21 18:47:31 +01:00
Andrew Morgan 55159c48e3 1.32.2 2021-04-21 18:45:39 +01:00
Andrew Morgan ca6ecb8d67 Merge branch 'release-v1.32.1' of github.com:matrix-org/synapse into release-v1.32.2 2021-04-21 18:39:45 +01:00
Andrew Morgan 8798f2291c Merge branch 'master' of github.com:matrix-org/synapse into develop 2021-04-21 18:21:56 +01:00
Andrew Morgan 046175daba Merge branch 'release-v1.32.1' of github.com:matrix-org/synapse 2021-04-21 18:21:14 +01:00
Andrew Morgan 0c23aa393c
Note LoggingContext signature change incompatibility in 1.32.0 (#9859)
1.32.0 also introduced an incompatibility with Synapse modules that make use of `synapse.logging.context.LoggingContext`, such as [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider).

This PR adds a note to the 1.32.0 changelog and upgrade notes about it.
2021-04-21 18:16:58 +01:00
Richard van der Hoff d9bd62f9d1
Make LoggingContext's name optional (#9857)
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
2021-04-21 16:39:34 +01:00
Andrew Morgan 4b2217ace2 Merge branch 'master' into develop 2021-04-21 14:55:06 +01:00
Andrew Morgan a0972085ed Synapse 1.32.1 (2021-04-21)
===========================
 
 This release fixes [a
 regression](https://github.com/matrix-org/synapse/issues/9853)
 in Synapse 1.32.0 that caused connected Prometheus instances to become
 unstable. If you
 ran Synapse 1.32.0 with Prometheus metrics, first upgrade to Synapse
 1.32.1 and follow
 [these
 instructions](https://github.com/matrix-org/synapse/pull/9854#issuecomment-823472183)
 to clean up any excess writeahead logs.
 
 Bugfixes
 --------
 
 - Fix a regression in Synapse 1.32.0 which caused Synapse to report
 large numbers of Prometheus time series, potentially overwhelming
 Prometheus instances.
 ([\#9854](https://github.com/matrix-org/synapse/issues/9854))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCALNwTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9CLbEADBxSsZZkemuPTyRiyDOMCeElLraeAF
 xqTNG2II2u38kFPvZoyQDztw0Qblu5ErEvRDM7P3jCH+CKNS1A4GLeuroDx0AeGH
 ieDTBs9lkJ9/Omi8SC6TuDz1ywURXaU5uk3Uuo05WCnambSathmOkfkM9vO2UnsT
 KUpx9dgW48nkYv8/ynhUMVj7cypIRwvWDzQc9LkomZQaqqHmDFBry6wlytKNtOah
 KVBzZkFq4/R+QjEqhO5zU2TkdfExwNbNzQ9ZgyRGHjWkAWQEA9CO3WGCms3QyuDX
 EX2FooLOZ+Q1GZ1VlOcsDuOt5YAIlxpdxFkApKfN6sO4rDDxLdY7du0fMzrEOUvp
 SX3g1bITAGa6NYQm+ACZJwKizHu9oxo9aSh59DoLHMzuDFLkjLirW+dNwN9xFyDo
 J5imJUe6T3KGkZgH8UJDiWYuTzw8PzObRjLL0cjAAa8S0bf3VJ1t4UkhGlip9/LR
 xOvuo9pdUD6rquy9tVzIKxtg2MgyjyuRT2+C9GllvYHqqUV2UBDTGkOCT/uWCYfG
 IdxMf5IQEfATkRCzdFQ6Sh0v+GWkfzBNTy6sv8+JJmHAV9uoTC1jRXR9f1Kzd8H4
 RPnjQyKoNxVhA/f7pmv5XFNUM4UNZ0I7HX6I5mZTYjqr+xwqOthwnNnKU/DagxKq
 DypHBY8rq+dJRA==
 =olOn
 -----END PGP SIGNATURE-----

Merge tag 'v1.32.1'

Synapse 1.32.1 (2021-04-21)
===========================

This release fixes [a regression](https://github.com/matrix-org/synapse/issues/9853) in Synapse 1.32.0 that caused connected Prometheus instances to become unstable. If you ran Synapse 1.32.0 with Prometheus metrics, first upgrade to Synapse 1.32.1 and follow [these instructions](https://github.com/matrix-org/synapse/pull/9854#issuecomment-823472183) to clean up any excess writeahead logs.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 which caused Synapse to report large numbers of Prometheus time series, potentially overwhelming Prometheus instances. ([\#9854](https://github.com/matrix-org/synapse/issues/9854))
2021-04-21 14:54:03 +01:00
Andrew Morgan bdb4c20dc1 Clarify 1.32.0/1 changelog and upgrade notes 2021-04-21 14:44:04 +01:00
Andrew Morgan acb8c81041 Add regression notes to CHANGES.md; fix link in 1.32.0 changelog 2021-04-21 14:24:16 +01:00
Andrew Morgan 98a1b84631 Add link to fixing prometheus to 1.32.0 upgrade notes; 1.32.1 has a fix 2021-04-21 14:19:11 +01:00
Andrew Morgan 026a66f2b3 Fix typo in link to regression in 1.32.0 upgrade notes 2021-04-21 14:04:44 +01:00
Andrew Morgan a745531c10 1.32.1 2021-04-21 14:01:12 +01:00
Andrew Morgan 30c94862b4 Mention Prometheus metrics regression in v1.32.0 2021-04-21 14:00:31 +01:00
Richard van der Hoff 5d281c10dd
Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854)
This undoes part of b076bc276e.
2021-04-21 10:03:31 +01:00
Patrick Cloke 683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan eccacd72cb Merge branch 'master' into develop 2021-04-20 17:14:15 +01:00
Andrew Morgan b8c5f6fddb Mention Prometheus metrics regression in v1.32.0 2021-04-20 17:11:36 +01:00
Andrew Morgan 272402c4d7 Merge branch 'master' into develop 2021-04-20 16:07:53 +01:00
Andrew Morgan 05fa06834d Further tweaking on gpg signing key notice 2021-04-20 15:52:06 +01:00
Andrew Morgan 913f790bb2 Add note about expired Debian gpg signing keys to CHANGES.md 2021-04-20 15:33:56 +01:00
Andrew Morgan 6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Andrew Morgan 438a8594cb Update v1.32.0 changelog. It's m.login.application_service, not plural 2021-04-20 14:47:17 +01:00
Andrew Morgan e031c7e0cc 1.32.0 2021-04-20 14:31:27 +01:00
Andrew Morgan 0a88ec0a87
Add Application Service registration type requirement + py35, pg95 deprecation notices to v1.32.0 upgrade notes (#9849)
Fixes https://github.com/matrix-org/synapse/issues/9846.

Adds important removal information from the top of https://github.com/matrix-org/synapse/releases/tag/v1.32.0rc1 into UPGRADE.rst.
2021-04-20 14:19:35 +01:00
Patrick Cloke b076bc276e
Always use the name as the log ID. (#9829)
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Erik Johnston de0d088adc
Add presence federation stream (#9819) 2021-04-20 14:11:24 +01:00
Erik Johnston db70435de7
Fix bug where we sent remote presence states to remote servers (#9850) 2021-04-20 13:37:54 +01:00
Jonathan de Jong 495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Andrew Morgan 71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Denis Kasak e694a598f8
Sanity check identity server passed to bind/unbind. (#9802)
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-04-19 17:21:46 +01:00
Erik Johnston 2b7dd21655
Don't send normal presence updates over federation replication stream (#9828) 2021-04-19 10:50:49 +01:00
Andrew Morgan c571736c6c
User directory: use calculated room membership state instead (#9821)
Fixes: #9797.

Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
2021-04-16 18:17:18 +01:00
Erik Johnston 601b893352
Small speed up joining large remote rooms (#9825)
There are a couple of points in `persist_events` where we are doing a
query per event in series, which we can replace.
2021-04-16 14:44:55 +01:00
Richard van der Hoff 5a153772c1
remove HomeServer.get_config (#9815)
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
Patrick Cloke 936e69825a
Separate creating an event context from persisting it in the federation handler (#9800)
This refactoring allows adding logic that uses the event context
before persisting it.
2021-04-14 12:35:28 -04:00
Patrick Cloke e8816c6ace Revert "Check for space membership during a remote join of a restricted room. (#9763)"
This reverts commit cc51aaaa7a.

The PR was prematurely merged and not yet approved.
2021-04-14 12:33:37 -04:00
Patrick Cloke cc51aaaa7a
Check for space membership during a remote join of a restricted room. (#9763)
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
    
This only applies to an experimental room version, as defined in MSC3083.
2021-04-14 12:32:20 -04:00
Jonathan de Jong 05e8c70c05
Experimental Federation Speedup (#9702)
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-14 17:19:02 +01:00
Erik Johnston 00a6db9676
Move some replication processing out of generic_worker (#9796)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
rkfg c9a2b5d402
More robust handling of the Content-Type header for thumbnail generation (#9788)
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
2021-04-14 16:30:59 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Andrew Morgan 7e460ec2a5
Add a dockerfile for running a set of Synapse worker processes (#9162)
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:

* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.

Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.

`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.

Huge thanks to @MatMaul for helping get this all working 🎉 This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.

Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
2021-04-14 13:54:49 +01:00
Andrew Morgan f16c6cf59a
Add note to docker docs explaining platform support (#9801)
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894.

I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support.

The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
2021-04-14 12:06:19 +01:00
Andrew Morgan d9bd181a3f Update changelog for v1.32.0 2021-04-13 14:39:06 +01:00