Matrix/synapse
Matrix/synapse
1
0
Fork 0
mirror of https://github.com/element-hq/synapse synced 2024-07-07 03:13:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
21512 commits 883 branches 702 tags 481 MiB
Commit graph

6559 commits

Author SHA1 Message Date
Brendan Abolivier 8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) (#13799) 
Partial implementation of MSC3881
 2022-09-21 14:39:01 +00:00
Mathieu Velten 6bd8763804
Add cache invalidation across workers to module API (#13667) 
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
 2022-09-21 15:32:01 +02:00
Peter Scheu 16e1a9d9a7
Correct documentation for map_user_attributes of OpenID Mapping Providers (#13836) 
Co-authored-by: David Robertson <davidr@element.io>
 2022-09-21 13:08:16 +00:00
Quentin Gliech e0804ef898
Improve the synapse.api.auth.Auth mock used in unit tests. (#13809) 
To return the proper type (`Requester`) instead of a `dict`.
 2022-09-21 12:40:34 +00:00
reivilibre a35842caec
Fix the release script not publishing binary wheels. (#13850) 2022-09-21 09:43:08 +00:00
Olivier Wilkinson (reivilibre) 2b522cceb6 Merge branch 'release-v1.68' into develop 2022-09-20 14:54:05 +01:00
Quentin Gliech 85fc7ea1a1
Remove the complete_sso_login method from the Module API which was deprecated in Synapse 1.13.0. (#13843) 
Signed-off-by: Quentin Gliech <quenting@element.io>
 2022-09-20 15:18:07 +02:00
David Robertson fff9b955fa
Generate separate snapshots for logical databases (#13792) 
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
 2022-09-20 14:14:12 +01:00
Erik Johnston 42d261c32f
Port the push rule classes to Rust. (#13768) 2022-09-20 12:10:31 +01:00
Olivier Wilkinson (reivilibre) e124b24ad9 1.68.0rc1 2022-09-20 11:30:48 +01:00
Denis c802ef1411
Don't include redundant prev_state in new events (#13791) 2022-09-20 09:44:38 +01:00
Eric Eastwood 44be42338e
Add support to purge rows from MSC2716 and other tables when purging a room (#13825) 
`event_failed_pull_attempts` added in https://github.com/matrix-org/synapse/pull/13589

MSC2716 related tables added in:

 - https://github.com/matrix-org/synapse/pull/10245/files#diff-3d42dfb44d02f7de3aada105e0bdc1cc9dd7f953cbf0f36c5d0f50827bf0320aR1
    - Renamed in https://github.com/matrix-org/synapse/pull/10838/files#diff-2730bfbe9e688b55e46f9371aefe67dac2bd2b2b7d9d6b92774eea1fcfae156dR1
 - https://github.com/matrix-org/synapse/pull/10498/files#diff-c52bbfbb5921a3f6f023b24343668479d966fac164f13b7c39d2197ce3afa7a5R1
 2022-09-16 10:56:56 -05:00
Mathieu Velten d5292b8017
Fix Docker build when Rust .so has been build locally first (#13811) 
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
 2022-09-16 15:38:54 +00:00
David Robertson 642c4b253d
Compare ported to unported PG schemas in portdb test job (#13808) 2022-09-16 16:25:54 +01:00
David Robertson 5e84461653
Minor speedups to CI linting (#13827) 2022-09-16 16:18:32 +01:00
Sean Quah d64e85197a
Remove error spam when users query the keys of departed remote users (#13826) 
The error message introduced in #13749 has turned out to be very spammy.
Remove it for now.
 2022-09-16 16:16:05 +01:00
Mathieu Velten 384dca53d6
complement: init postgres DB directly inside the target image (#13819) 
Doing so in the base postgres image doesn't work with buildah because
changes in a declared VOLUME in the Dockerfile is supposed to be
discarded, cf https://docs.docker.com/engine/reference/builder/#volume

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
 2022-09-16 17:12:45 +02:00
Quentin Gliech 74f60cec92
Add an admin API endpoint to find a user based on its external ID in an auth provider. (#13810) 2022-09-16 12:29:03 +00:00
reivilibre f7a77ad717
Update request log format documentation to mention the format used when the authenticated user is controlling another user. (#13794) 2022-09-16 11:48:41 +00:00
Sean Quah b73cbb8215
Avoid putting rejected events in room state (#13723) 
Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-16 12:45:04 +01:00
Eric Eastwood 6986bcbf39
Document common fix of Poetry problems by removing egg-info (#13785) 
`matrix_synapse.egg-info/`

Mentioned at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$aKy_IjrKwb70aTVZWeW_6zt0k7OIZ1YkyZpkP9uiRaM?via=matrix.org&via=element.io&via=beeper.com and many other places.
 2022-09-15 16:28:03 -05:00
Eric Eastwood 5093cbf88d
Be able to correlate timeouts in reverse-proxy layer in front of Synapse (pull request ID from header) (#13801) 
Fix https://github.com/matrix-org/synapse/issues/13685

New config:

```diff
  listeners:
    - port: 8008
      tls: false
      type: http
      x_forwarded: true
+     request_id_header: "cf-ray"
      bind_addresses: ['::1', '127.0.0.1', '0.0.0.0']
```
 2022-09-15 15:32:25 -05:00
Eric Eastwood 140af0cdb6
Record any exception when processing a pulled event (#13814) 
Part of https://github.com/matrix-org/synapse/issues/13700 and https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13589
 2022-09-15 14:40:49 -05:00
Patrick Cloke b2b0c85279
Support providing an index predicate for upserts. (#13822) 
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
 2022-09-15 18:28:48 +00:00
David Robertson 742f9f9d78
A third batch of Pydantic validation for rest/client/account.py (#13736) 2022-09-15 18:36:02 +01:00
Andrew Morgan 918c74bfb5
Add a MXCUri class to make working with mxc uri's easier. (#13162) 2022-09-15 12:57:16 +00:00
Eric Eastwood 957e3d74fc
Keep track when we try and fail to process a pulled event (#13589) 
We can follow-up this PR with:

 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622
 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623
 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700

Fix https://github.com/matrix-org/synapse/issues/13621

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
 2022-09-14 13:57:50 -05:00
Patrick Cloke 666ae87729
Update event push action and receipt tables to support threads. (#13753) 
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.

The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.

`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".

See MSC3771 and MSC3773 for where this data will be useful.
 2022-09-14 17:11:16 +00:00
Patrick Cloke f2d12ccabe
Use partial indices on SQLIte. (#13802) 
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.

This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
 2022-09-14 12:01:42 -04:00
reivilibre 6302753012
Deduplicate is_server_notices_room. (#13780) 2022-09-14 15:53:18 +00:00
reivilibre cf65433de2
Fix a memory leak when running the unit tests. (#13798) 2022-09-14 15:29:05 +00:00
Quentin Gliech eaed4e6113
Remove unused method in synapse.api.auth.Auth. (#13795) 
Clean-up from b19060a29b (#13094)
and 73af10f419 (#13093) which removed
all callers.
 2022-09-14 10:33:54 -04:00
David Robertson 51a77e990b
Remove incorrect migration file from state logical DB (#13788) 
* Remove incorrect migration file from `state` logical DB

The table `ex_outlier_stream` is part of the `main` logical DB; it
should not have been created in the `state` logical DB. We remove this
migration now as a tidy-up.

Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new
migration, because some (most) instances of Synapse host both of these
logical DBs on the same DB cluster.

* Changelog
 2022-09-14 14:16:12 +01:00
Sean Quah c73774467e
Fix bug in device list caching when remote users leave rooms (#13749) 
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.

Fixes #13651.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-14 10:42:57 +01:00
reivilibre 21687ec189
Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of /get_missing_events federation requests. (#13789) 
* Don't accept a trailing slash on the end of /get_missing_events

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
 2022-09-14 09:28:12 +01:00
Mathieu Velten 12dacecabd
Make sequence cache_invalidation_stream_seq begin at 2 (#13766) 
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-09-13 16:14:28 +02:00
Erik Johnston 9772e362aa Merge branch 'master' into develop 2022-09-13 12:11:53 +01:00
David Robertson b60d47ab2c
Updates to the schema dump script (#13770) 2022-09-13 10:53:11 +01:00
David Robertson 540afb0bfc
Simplify CI tests DAG (#13784) 
* Simplify CI tests DAG

* Changelog
 2022-09-13 10:17:23 +01:00
Richard van der Hoff 41df25bbbd
installation.md: require libpq on M1 macs (#13480) 2022-09-13 09:01:21 +00:00
Nick Mills-Barrett cdbb641232
Add receipts event stream ordering (#13703) 2022-09-13 08:16:37 +01:00
Mathieu Velten fa2f3d8d0c
Fix GHA skippable syntax (#13778) 
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
 2022-09-12 17:31:23 +00:00
Brendan Abolivier 7571337445
Fix typo in ratelimiting documentation (#13727) 2022-09-12 14:11:18 +01:00
Erik Johnston dd7484b562
Fix CI on non-PR builds (#13769) 
Mark cargo-test as skippable since it only runs on Rust code change.
 2022-09-12 13:26:33 +01:00
Nick Mills-Barrett da41a7cd61
Remove check current state membership up to date (#13745) 
* Remove checks for membership column in current_state_events
* Add schema script to force through the
  `current_state_events_membership` background job

Contributed by Nick @ Beeper (@fizzadar).
 2022-09-12 12:58:33 +01:00
Erik Johnston ebfeac7c5d
Check if Rust lib needs rebuilding. (#13759) 
This protects against the common mistake of failing to remember to rebuild Rust code after making changes.
 2022-09-12 10:03:42 +00:00
Nick Mills-Barrett 4c4889cac0
Concurrently collect room unread counts for push badges (#13765) 
Most of the time this function is heavily cached, but when that isn't
the case fetching the counts room by room slows down push delivery on
users with many (thousands) of rooms.

Signed off by Nick @ Beeper.
 2022-09-09 19:00:21 +01:00
Eric Eastwood a911ffb42c
Tag trace with instance name (#13761) 
We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load.

As suggested by @clokep and @reivilibre in,

 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258
 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
 2022-09-09 11:31:37 -05:00
Eric Eastwood f694bb71b7
Strip number suffix from instance name to consolidate services that traces are spread over (#13729) 
The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985

Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through.

With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇

Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.
 2022-09-09 11:30:06 -05:00
Patrick Cloke 3d9f82efcb
Use an upsert for receipts_graph. (#13752) 
Instead of a delete, then insert.

This was previously done for `receipts_linearized` in
2dc430d36e (#7607).
 2022-09-09 07:08:41 -04:00
Erik Johnston c85c5ace52
Add rust to CI (#13763) 2022-09-09 11:29:04 +01:00
David Robertson f2d2481e56
Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
Sean Quah 69fa29700e
Re-type hint some collections in /sync code as read-only (#13754) 
Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-08 20:13:39 +01:00
reivilibre 5261d2e2e8
Remove unused Prometheus recording rules from synapse-v2.rules and add comments describing where the rest are used. (#13756) 2022-09-08 17:50:15 +00:00
Dirk Klimpel f799eac7ea
Add timestamp to user's consent (#13741) 
Co-authored-by: reivilibre <olivier@librepush.net>
 2022-09-08 15:41:48 +00:00
Sean Quah 906cead9ca
Update docstrings to explain the impact of partial state (#13750) 
Update the docstrings for `get_users_in_room` and
`get_current_hosts_in_room` to explain the impact of partial state.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-08 15:55:29 +01:00
Sean Quah 89e8b98b65
Avoid raising errors due to malformed IDs in get_current_hosts_in_room (#13748) 
Handle malformed user IDs with no colons in `get_current_hosts_in_room`.
It's not currently possible for a malformed user ID to join a room, so
this error would never be hit.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-08 15:55:03 +01:00
Sean Quah 8ef0c8ff14
Fix error in is_mine_id when encountering a malformed ID (#13746) 
Previously, `is_mine_id` would raise an exception when passed an ID with
no colons. Return `False` instead.

Fixes #13040.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-09-08 15:54:36 +01:00
reivilibre cf11919ddd
Fix cache metrics not being updated when not using the legacy exposition module. (#13717) 2022-09-08 15:30:48 +01:00
reivilibre 526f84bc2e
Fix Prometheus recording rules to not use legacy metric names. (#13718) 2022-09-08 15:01:42 +01:00
Erik Johnston 1cc729c177
Fix latest deps (#13743) 2022-09-08 13:58:31 +01:00
reivilibre b7e4bfd005
Fix a bug where Synapse fails to start if a signing key file contains an empty line. (#13738) 2022-09-08 11:18:03 +01:00
Eric Eastwood d4d3249ded
Instrument get_metadata_for_events for tracing (#13730) 
When backfilling, `_get_state_ids_after_missing_prev_event` calls [`get_metadata_for_events`](26bc26586b/synapse/handlers/federation_event.py (L1133)). For `#matrix:matrix.org`, it's called with 77k `state_events` which means 77 calls to the database and takes 28 seconds.
 2022-09-07 11:41:52 -05:00
Erik Johnston 8d7fcf9b76
Fix latest deps CI (#13734) 2022-09-07 14:07:06 +00:00
Erik Johnston dc0e896b68
Add some rust caching to CI (#13735) 2022-09-07 13:56:59 +00:00
David Robertson c46fecd1f2
Correct out-of-date doc for event_cache_size (#13726) 2022-09-07 14:46:11 +01:00
David Robertson 77f3986451
Define SQLite compat policy (#13728) 2022-09-07 12:07:42 +00:00
David Robertson b58386e37e
A second batch of Pydantic models for rest/client/account.py (#13687) 2022-09-07 12:16:10 +01:00
reivilibre d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
reivilibre c2fe48a6ff
Rename the EventFormatVersions enum values so that they line up with room version numbers. (#13706) 2022-09-07 11:08:20 +01:00
Connor Davis bb5b47b62a
Add Admin API to Fetch Messages Within a Particular Window (#13672) 
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
 2022-09-07 10:54:44 +01:00
reivilibre 26bc26586b
Remove the unspecced room_id field in the /hierarchy response. (#13506) 
This is a re-do of 57d334a13d (#13365),
which was backed out in 12abd72497 (#13501).

The `room_id` field represented the parent space for each room
and was made redundant by changes in the API shape where the
`children_state` is now nested underneath each `room`.

The room ID of each child is in the `state_key` field and is still
available.
 2022-09-06 15:28:44 -04:00
Erik Johnston c9b7e97355
Add a stub Rust crate (#12595) 2022-09-06 19:01:37 +01:00
Erik Johnston 3d20115115
Fix trial-olddeps (#13725) 2022-09-06 14:21:55 +00:00
David Robertson a4ecb8e353
Actually fix typechecking with latest types-jsonschema (#13724) 2022-09-06 14:29:16 +01:00
Erik Johnston b5effc7201
Update trial old deps CI to use poetry 1.2.0 (#13707) 2022-09-06 11:43:04 +00:00
reivilibre b455c2a5ec
Update Grafana dashboard to not use legacy metric names. (#13714) 2022-09-06 12:21:21 +01:00
Erik Johnston cdf7fb737b 1.67.0rc1 2022-09-06 09:01:26 +01:00
Patrick Cloke 32fc3b7ba4
Remove configuration options for direct TCP replication. (#13647) 
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
 2022-09-06 07:50:02 +00:00
Erik Johnston 8edf3f66d5
Reduce number of CI jobs run on PRs (#13713) 
* Reduce number of CI jobs run on PRs

* Newsfile

* Also limit sytest jobs

* Fix typo

* Fix up

* Fixup
 2022-09-05 13:31:42 +01:00
reivilibre c7b18d9d44
Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process. (#13483) 2022-09-05 12:16:59 +01:00
David Robertson 8cb9261598
Fix typechecking with latest types-jsonschema (#13712) 2022-09-05 11:13:13 +00:00
Brendan Abolivier 898fef2789
Share some metrics between the Prometheus exporter and the phone home stats (#13671) 2022-09-05 10:26:43 +00:00
Eric Eastwood 877bdfa889
Clarify (room_id, event_id) global uniqueness (#13701) 
Summarized from @richvdh's reply at https://github.com/matrix-org/synapse/pull/13589#discussion_r961116999
 2022-09-02 14:05:39 -05:00
Eric Eastwood 36b184b782
Update docs to make enabling metrics more clear (#13678) 
It was really easy to miss the `enable_metrics: True` step with the previous language.
 2022-09-02 11:29:51 -05:00
Erik Johnston 4fee4a339d
Update lock file for Poetry v1.2.0 (#13689) 2022-09-02 13:20:03 +00:00
Erik Johnston 0fdb685c2b
Only run trial CI on all python versions on non-PRs (#13698) 2022-09-02 13:12:47 +01:00
Dirk Klimpel 044900af6c
Fix two typos with colon in headlines (#13665) 2022-09-01 20:38:37 +00:00
Patrick Cloke 48a5c47a9f
Add a schema delta to drop unstable private read receipts. (#13692) 
Otherwise they'll be leaked due to the filtering code only respecting
the stable identifiers for private read receipts.
 2022-09-01 14:57:47 -04:00
Patrick Cloke 390b7ce946
Disable calculating unread counts unless the config flag is enabled. (#13694) 
This avoids doing work that will never be used (since the
resulting unread counts will never be sent in a /sync
response).

The negative of doing this is that unread counts will be
incorrect when the feature is initially enabled.
 2022-09-01 16:52:03 +00:00
reivilibre f48f4dd59e
Update the Grafana dashboard that is included with Synapse in the contrib directory. (#13697) 
* Add missing graph to contrib

* Update with minor but plausible changes, including positioning changes

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
 2022-09-01 16:27:06 +01:00
Erik Johnston 9d2823ab70
Cache is_partial_state_room (#13693) 
Fixes #13613.
 2022-09-01 16:07:01 +01:00
Will Hunt c913e440c0
Add monthly active users documentation (#13617) 
* Add monthly active users documentation

* changelog

* Tidy up notes

* more tidyup

* Rewrite #1

* link back to mau docs

* fix links

* s/appservice|AS/application service

* further review

* a newline

* Remove bit about shadow banned users.

I think talking about them is confusing, and the current text doesn't imply they get any special treatment.

* Update docs/usage/administration/monthly_active_users.md

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update docs/usage/administration/monthly_active_users.md

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2022-09-01 14:48:43 +00:00
Erik Johnston 2318603772
Add some logging to help track down #13444 (#13679) 2022-09-01 13:54:52 +01:00
Richard van der Hoff e8130f219b
Return keys for unwhitelisted servers from /_matrix/key/v2/query (#13683) 2022-09-01 13:54:02 +01:00
Erik Johnston 18e4092801
Bump docker image to use stable poetry version (#13688) 2022-09-01 13:46:47 +01:00
Šimon Brandner 0e99f07952
Remove support for unstable private read receipts (#13653) 
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
 2022-09-01 13:31:54 +01:00
Jacek Kuśnierz 84ddcd7bbf
Drop support for calling /_matrix/client/v3/rooms/{roomId}/invite without an id_access_token (#13241) 
Fixes #13206

Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
 2022-08-31 12:10:25 +00:00
Nick Mills-Barrett 42b11d5565
Remove cached wrap on _get_joined_users_from_context method (#13569) 
The method doesn't actually do any data fetching and the method that
does, `_get_joined_profile_from_event_id`, has its own cache.

Signed off by Nick @ Beeper (@Fizzadar).
 2022-08-31 12:19:39 +01:00
reivilibre 7bc110a19e
Generalise the @cancellable annotation so it can be used on functions other than just servlet methods. (#13662) 2022-08-31 11:16:05 +00:00
David Robertson a160406d24
Fix admin List Room API return type on sqlite (#13509) 2022-08-31 10:38:16 +00:00
Eric Eastwood 92c5817e34
Give the correct next event when the message timestamps are the same - MSC3030 (#13658) 
Discovered while working on https://github.com/matrix-org/synapse/pull/13589 and I had all the messages at the same timestamp in the tests.

Part of https://github.com/matrix-org/matrix-spec-proposals/pull/3030

Complement tests: https://github.com/matrix-org/complement/pull/457
 2022-08-30 14:50:06 -05:00
Shay 20c76cecb9
Drop unused column application_services_state.last_txn (#13627) 2022-08-30 10:29:16 -07:00
Richard van der Hoff 372136d3a8
Remove documentation of legacy frontend_proxy worker app (#13645) 
This has been the same as a generic_worker since #6964, so let's get rid of it.

Fixes #3717
 2022-08-30 18:01:51 +01:00
Patrick Cloke e761e8b475
Clarify documentation about replication traffic. (#13656) 
It can be authenticated with the worker_replication_secret setting,
but is always unencrypted.
 2022-08-30 12:21:19 +00:00
Erik Johnston 1c26acd815
Fix bug where we wedge media plugins if clients disconnect early (#13660) 
We incorrectly didn't use the returned `Responder` if the client had
disconnected, which meant that the resource used by the Responder
wasn't correctly released.

In particular, this exhausted the thread pools so that *all* requests
timed out.
 2022-08-30 12:17:48 +01:00
Patrick Cloke 303b40b988
Do not wait for background updates to complete do expire URL cache. (#13657) 
Media downloaded as part of a URL preview is normally deleted after two days.
However, while a background database migration is running, the process is
stopped. A long-running database migration can therefore cause the media
store to fill up with old preview files.

This logic was added in #2697 to make sure that we didn't try to run the expiry
without an index on `local_media_repository.created_ts`; the original logic that
needs that index was added in #2478 (in `get_url_cache_media_before`, as
amended by 93247a424a), and is still present.

Given that the background update was added before Synapse v1.0.0, just drop
this check and assume the index exists.
 2022-08-30 07:15:54 -04:00
Patrick Cloke 20df96a7a7
Speed up inserting event_push_actions_staging. (#13634) 
By using `execute_values` instead of `execute_batch`.
 2022-08-30 07:12:48 -04:00
Dirk Klimpel 682dfcfc0d
Fix that user cannot /forget rooms after the last member has left (#13546) 2022-08-30 09:58:38 +00:00
Eric Eastwood 51d732db3b
Optimize how we calculate likely_domains during backfill (#13575) 
Optimize how we calculate `likely_domains` during backfill because I've seen this take 17s in production just to `get_current_state` which is used to `get_domains_from_state` (see case [*2. Loading tons of events* in the `/messages` investigation issue](https://github.com/matrix-org/synapse/issues/13356)).

There are 3 ways we currently calculate hosts that are in the room:

 1. `get_current_state` -> `get_domains_from_state`
    - Used in `backfill` to calculate `likely_domains` and `/timestamp_to_event` because it was cargo-culted from `backfill`
    - This one is being eliminated in favor of `get_current_hosts_in_room` in this PR 🕳
 1. `get_current_hosts_in_room`
    - Used for other federation things like sending read receipts and typing indicators
 1. `get_hosts_in_room_at_events`
    - Used when pushing out events over federation to other servers in the `_process_event_queue_loop`

Fix https://github.com/matrix-org/synapse/issues/13626

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh)


### Query performance

#### Before

The query from `get_current_state` sucks just because we have to get all 80k events. And we see almost the exact same performance locally trying to get all of these events (16s vs 17s):
```
synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 16035.612 ms (00:16.036)

synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 4243.237 ms (00:04.243)
```

But what about `get_current_hosts_in_room`: When there is 8M rows in the `current_state_events` table, the previous query in `get_current_hosts_in_room` took 13s from complete freshness (when the events were first added). But takes 930ms after a Postgres restart or 390ms if running back to back to back.

```sh
$ psql synapse
synapse=# \timing on
synapse=# SELECT COUNT(DISTINCT substring(state_key FROM '@[^:]*:(.*)$'))
FROM current_state_events
WHERE
    type = 'm.room.member'
    AND membership = 'join'
    AND room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
 count
-------
  4130
(1 row)

Time: 13181.598 ms (00:13.182)

synapse=# SELECT COUNT(*) from current_state_events where room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
 count
-------
 80814

synapse=# SELECT COUNT(*) from current_state_events;
  count
---------
 8162847

synapse=# SELECT pg_size_pretty( pg_total_relation_size('current_state_events') );
 pg_size_pretty
----------------
 4702 MB
```

#### After

I'm not sure how long it takes from complete freshness as I only really get that opportunity once (maybe restarting computer but that's cumbersome) and it's not really relevant to normal operating times. Maybe you get closer to the fresh times the more access variability there is so that Postgres caches aren't as exact. Update: The longest I've seen this run for is 6.4s and 4.5s after a computer restart.

After a Postgres restart, it takes 330ms and running back to back takes 260ms.

```sh
$ psql synapse
synapse=# \timing on
Timing is on.
synapse=# SELECT
    substring(c.state_key FROM '@[^:]*:(.*)$') as host
FROM current_state_events c
/* Get the depth of the event from the events table */
INNER JOIN events AS e USING (event_id)
WHERE
    c.type = 'm.room.member'
    AND c.membership = 'join'
    AND c.room_id = '!OGEhHVWSdvArJzumhm:matrix.org'
GROUP BY host
ORDER BY min(e.depth) ASC;
Time: 333.800 ms
```

#### Going further

To improve things further we could add a `limit` parameter to `get_current_hosts_in_room`. Realistically, we don't need 4k domains to choose from because there is no way we're going to query that many before we a) probably get an answer or b) we give up. 

Another thing we can do is optimize the query to use a index skip scan:

 - https://wiki.postgresql.org/wiki/Loose_indexscan
 - Index Skip Scan, https://commitfest.postgresql.org/37/1741/
 - https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/
 2022-08-30 01:38:14 -05:00
Richard van der Hoff 4f6de33f41
Print complement failure results last (#13639) 
Since github always scrolls to the bottom of any test output, let's put the
failed tests last and hide any successful packages.
 2022-08-28 20:05:30 +00:00
Richard van der Hoff c4e29b6908
Improve documentation around user registration (#13640) 
Update a bunch of the documentation for user registration, add some cross
links, etc.
 2022-08-26 13:29:31 +00:00
Richard van der Hoff 5e5c8150d7
Generate missing configuration files at startup (#13615) 
If things like the signing key file are missing, let's just try to generate
them on startup.

Again, this is useful for k8s-like deployments where we just want to generate
keys on the first run.
 2022-08-26 11:26:06 +00:00
Brad Murray 967d7bad6c
Move the execution of the retention purge_jobs to the main worker (#13632) 
Fixes #9927

Signed-off-by: Brad Murray brad@beeper.com
 2022-08-26 08:38:10 +01:00
Richard van der Hoff d092e6f32a
Support registration_shared_secret in a file (#13614) 
A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
 2022-08-25 16:27:46 +00:00
Richard van der Hoff a2ce614447
register_new_matrix_user: read server url from config (#13616) 
Fixes https://github.com/matrix-org/synapse/issues/3672:
`https://localhost:8448` is virtually never right.
 2022-08-25 15:29:08 +01:00
Eric Eastwood 0bf180cbb4
Comment about a better future where we can get the state diff between two events (#13586) 
Split off from https://github.com/matrix-org/synapse/pull/13561

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh)
 2022-08-24 18:59:27 -05:00
David Robertson c406d50d2d
Rename event_map to unpersisted_events (#13603) 2022-08-24 21:06:31 +01:00
Eric Eastwood 1a209efdb2
Update get_users_in_room mis-use to get hosts with dedicated get_current_hosts_in_room (#13605) 
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
 2022-08-24 14:15:37 -05:00
Eric Eastwood d58615c82c
Directly lookup local membership instead of getting all members in a room first (get_users_in_room mis-use) (#13608) 
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
 2022-08-24 14:13:12 -05:00
Eric Eastwood b93bd95e8a
When loading current ids, sort by stream_id to avoid incorrect overwrite and avoid errors caused by sorting alphabetical instance name which can be null (#13585) 
When loading current ids, sort by stream ID so that we don't want to overwrite the `current_position` of an instance to a lower stream ID than we're actually at ([discussion](https://github.com/matrix-org/synapse/pull/13585#discussion_r951795379)). Previously, it sorted alphabetically by instance name which can be `null` and throw errors but more importantly, accomplishes nothing.

Fixes the following startup error which is why I started looking into this area:

```
$ poetry run synapse_homeserver --config-path homeserver.yaml
****************************************************************
 Error during initialisation:
    '<' not supported between instances of 'NoneType' and 'str'
 There may be more information in the logs.
****************************************************************
```

Somehow my database ended up looking like the following, notice the `instance_name` is `null` in the db, and we can't sort `NoneType` things. Another question is why do we see the `instance_name` as `null` sometimes instead of `master` in monolith mode?
```
$ psql synapse
synapse=# SELECT * FROM stream_positions;
   stream_name   | instance_name | stream_id
-----------------+---------------+-----------
 account_data    | master        |      1242
 events          | master        |      1787
 to_device       | master        |        58
 presence_stream | master        |    485638
 receipts        | master        |       341
 backfill        | master        |   -139106
(6 rows)
synapse=# SELECT instance_name, stream_id FROM receipts_linearized;
 instance_name | stream_id
---------------+-----------
               |       211
               |         3
               |         4
               |       212
               |       213
               |       224
               |       228
               |       164
               |       313
               |       253
               |        38
               |       321
               |       324
               |       189
               |       192
               |       193
               |       194
               |       195
               |       197
               |       198
               |       275
               |        79
               |       339
               |       340
               |        82
               |       341
               |        84
               |        85
               |        91
               |       119
```
 2022-08-24 12:53:46 -05:00
Eric Eastwood c807b814ae
Use dedicated get_local_users_in_room to find local users when calculating join_authorised_via_users_server of a /make_join request (#13606) 
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` ("the authorising user for joining a restricted room") of a `/make_join` request.

Found while working on https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755 but it's not related.
 2022-08-24 11:14:28 -05:00
reivilibre be4250c7a8
Add experimental configuration option to allow disabling legacy Prometheus metric names. (#13540) 
Co-authored-by: David Robertson <davidr@element.io>
 2022-08-24 11:35:54 +00:00
Nick Mills-Barrett b687010f89
Rewrite get push actions queries (#13597) 2022-08-24 10:12:51 +01:00
reivilibre ba882c0357
Faster Room Joins: fix /make_knock blocking indefinitely when the room in question is a partial-stated room. (#13583) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-08-24 09:09:59 +00:00
Eric Eastwood 7af07f9716
Instrument _check_sigs_and_hash_and_fetch to trace time spent in child concurrent calls (#13588) 
Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls because I've see `_check_sigs_and_hash_and_fetch` take [10.41s to process 100 events](https://github.com/matrix-org/synapse/issues/13587)

Fix https://github.com/matrix-org/synapse/issues/13587

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-23 21:53:37 -05:00
David Robertson a25a37002c
Write about the chain cover a little. (#13602) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2022-08-23 17:41:55 +00:00
Erik Johnston f7ddfe17a3
Speed up @cachedList (#13591) 
This speeds things up by ~2x.

The vast majority of the time is now spent in `LruCache` moving things around the linked lists.

We do this via two things:
1. Don't create a deferred per-key during bulk set operations in `DeferredCache`. Instead, only create them if a subsequent caller asks for the key.
2. Add a bulk lookup API to `DeferredCache` rather than use a loop.
 2022-08-23 14:53:27 +00:00
Erik Johnston 05c9c7363b
Fix regression caused by #13573 (#13600) 
Broke in #13573.
 2022-08-23 14:14:05 +00:00
David Robertson ca3d19b05f Synapse 1.66.0rc1 (2022-08-23) 
==============================
 
 This release removes the ability for homeservers to delegate email ownership
 verification and password reset confirmation to identity servers. This removal
 was originally planned for Synapse 1.64, but was later deferred until now.
 
 See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.
 
 Features
 --------
 
 - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](https://github.com/matrix-org/synapse/issues/13188), [\#13563](https://github.com/matrix-org/synapse/issues/13563))
 - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](https://github.com/matrix-org/synapse/issues/13503))
 - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852). ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
 - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716).  ([\#13551](https://github.com/matrix-org/synapse/issues/13551))
 - Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537))
 - Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522), [\#13547](https://github.com/matrix-org/synapse/issues/13547))
 
 Bugfixes
 --------
 
 - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514))
 - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525))
 - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566))
 - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574))
 
 Improved Documentation
 ----------------------
 
 - Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472))
 - Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491))
 - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492))
 - Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497))
 - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515))
 - Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536))
 - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538))
 
 Deprecations and Removals
 -------------------------
 
 - Remove the ability for homeservers to delegate email ownership verification
   and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.
 
 Internal Changes
 ----------------
 
 - Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459))
 - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477))
 - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531))
 
 - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453))
 - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489))
 - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499), [\#13554](https://github.com/matrix-org/synapse/issues/13554))
 - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533))
 - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534), [\#13541](https://github.com/matrix-org/synapse/issues/13541))
 - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535), [\#13584](https://github.com/matrix-org/synapse/issues/13584))
 - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544))
 - Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545))
 
 - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024))
 - Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471))
 - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474))
 - Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479), [\#13488](https://github.com/matrix-org/synapse/issues/13488))
 - Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485))
 - Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493))
 - Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502))
 - Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543))
 - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
 - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMExBwACgkQkD7OEIo5
 3t0sGA/9FLZLdTWeROLUenbInvz3POqi6DgqlE0f8IDdp3zHjjuDJIE4HZwnPhhj
 c3H5FcUnTPhQoVZ8Ybno3J3KN8jfpe4tFhnhlM1x9hOi1fujfqCnFd2qWvlzkYXX
 J4oN6Dz0BbhqdhqnS2tcNacZJOielJ47UtNNBZkzprpzjL3qqRUK3Hm/gGK8GtGN
 BLon7JK0pKG1GFMVaoIWsr+j7880QD8DGXTjoZrjo/yGdp+zJldwJTbN2ymNdMw1
 3Caq7eCk+m6jzCUJxI7tTJVioXvf7+TScIsozAXuQtBy9W1RctvVMMoR9cIlN+JJ
 dj5GKhKBv8uJpJU6dJnlYpjNC5+5hPHr1gfnkoYbcyFF6PybJrHx9jfhEIEXmI1+
 j0DlJJdFnX+eeq4deFd+LPU5hYaF8SkSVRK3G+/NmKI3YYTReD6ootZzV5O4Z9P0
 NUwLK/OSMZrsFUfuwysUttfHq0uHGGs7mytlRgeqb/1RKRC80nxVJu3qv/xF6kKj
 KF98Yr2IDgp/MFhBpGAO3ylNgK8aQ3YxED4qW+kfnDeHPkma5mTdAtJ1bpvKcec+
 3awpFGWcmYVQPflwp9J3HFJHqWkj32LMfuSgDffLs8mWuj87fP7eAdjzv6M4zeAM
 4qh+cmvNC7ArfQ+ngz3g+xVD1lHi84s+fMgo7L+ySGfgDqTLhT4=
 =1F3P
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMEz10ACgkQkD7OEIo5
 3t02GhAApioIpEedKaeSATI42N8rWhfuni3vaPFAXwmOD6BvBkZlP0gcg+vUoBjP
 I0VIkGWZ3MKIhCSJkjwPJh1vHcpE09KYqIYRKO5vSbqHZKHYUJ+ThjcCR4IkOBPD
 3Dc3IRgQaM2Nh8eMcyRY7eyxo+2y+VN2G+SXr1B+A5/4N58ZrxfV97MtLYJ1a6AH
 0iIknOxb9FPArYs4ORbk6x2rvRO2QPGh88uKwJ8uHXvHlUbJ3fmHLVvRa1m0RKAW
 ojkzj/Dysa+4gasMIvtzPePtIQnMyovaQLul+bGXzE0js4lqzSOIP12x1zNpLuuq
 LSiyD3gMgGeogumhCAU6b23xF6jBaGtgtKKpaYuK8pqkuBTRNN8kQ6LB+7g3WQ2L
 QWVos+/+wKUEHoVVD/q3u7o4CrKYqv4zE9Pj6kw5e1USOoy7803ub2ccjY1UgG3M
 mZ7c48WNQQ61o9oH2+9+Ct6bab3R/Lo1MbS3zIKN8uktB4CdrhVAz3kYgV5Bl8bo
 PsldxUccOTTeS5qmUGvm2/szVKjdqyCWRx9AY9eUH9Kc4u/GzmNq0UVOZF8yxM0q
 WEo7GFgjM3k9IIC3K/fSmSC1iBjPSHpYvqO+c8oSNnjq2iucZeIhiEB07e89+KSa
 Wiz9DDWP8Z7W6D9wRAeXyPJFB6ZdSSBgP1I90z6lQjSMzy46Kv4=
 =pJ2V
 -----END PGP SIGNATURE-----

Merge tag 'v1.66.0rc1' into develop

Synapse 1.66.0rc1 (2022-08-23)
==============================

This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.

See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.

Features
--------

- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](https://github.com/matrix-org/synapse/issues/13188), [\#13563](https://github.com/matrix-org/synapse/issues/13563))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](https://github.com/matrix-org/synapse/issues/13503))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852). ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716).  ([\#13551](https://github.com/matrix-org/synapse/issues/13551))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522), [\#13547](https://github.com/matrix-org/synapse/issues/13547))

Bugfixes
--------

- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574))

Improved Documentation
----------------------

- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538))

Deprecations and Removals
-------------------------

- Remove the ability for homeservers to delegate email ownership verification
  and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.

Internal Changes
----------------

- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531))

- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499), [\#13554](https://github.com/matrix-org/synapse/issues/13554))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534), [\#13541](https://github.com/matrix-org/synapse/issues/13541))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535), [\#13584](https://github.com/matrix-org/synapse/issues/13584))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545))

- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479), [\#13488](https://github.com/matrix-org/synapse/issues/13488))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558))
 2022-08-23 14:00:09 +01:00
Erik Johnston aec87a0f93
Speed up fetching large numbers of push rules (#13592) 2022-08-23 13:15:43 +01:00
David Robertson ea85a2bf6c
Remove manually-added changelog 2022-08-23 12:40:35 +01:00
David Robertson 956e015413
Drop support for delegating email validation, round 2 (#13596) 2022-08-23 11:40:00 +00:00
Nick Mills-Barrett 5e7847dc92
Cache user IDs instead of profile objects (#13573) 
The profile objects are never used and increase cache size significantly.
 2022-08-23 09:49:59 +00:00
David Robertson f0b23927fc
1.66.0rc1 2022-08-23 09:49:51 +01:00
Dirk Klimpel 37f329c9ad
Fix that sending server notices fail if avatar is None (#13566) 
Indroduced in #11846.
 2022-08-23 09:48:35 +01:00
Eric Eastwood 9385c41ba4
Fix Prometheus metrics being negative (mixed up start/end) (#13584) 
Fix:

 - https://github.com/matrix-org/synapse/pull/13535#discussion_r949582508
 - https://github.com/matrix-org/synapse/pull/13533#discussion_r949577244
 2022-08-23 08:47:30 +01:00
Quentin Gliech 3dd175b628
synapse.api.auth.Auth cleanup: make permission-related methods use Requester instead of the UserID (#13024) 
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
 2022-08-22 14:17:59 +01:00
Andrew Morgan 94375f7a91
Remove redundant opentracing spans for /sendToDevice and /keys/upload (#13574) 2022-08-22 10:03:11 +01:00
Eric Eastwood 06df5d4250
MSC2716v4 room version - remove namespace from MSC2716 event content fields (#13551) 
Complement PR: https://github.com/matrix-org/complement/pull/450

As suggested in
https://github.com/matrix-org/matrix-spec-proposals/pull/2716#discussion_r941444525
 2022-08-19 15:37:01 -05:00
Andrew Morgan f9f03426de
Implement MSC3852: Expose last_seen_user_agent to users for their own devices; also expose to Admin API (#13549) 2022-08-19 16:17:10 +00:00
Andrew Morgan 40e3e68cd7
Register homeserver modules when creating test homeserver (#13558) 2022-08-19 16:52:20 +01:00
Patrick Cloke f3fba4914d
Reduce the number of tests using TCP replication. (#13543) 
Uses Redis replication in additional test cases (instead of
TCP replication). A small step towards dropping TCP replication.
 2022-08-19 08:25:24 -04:00
reivilibre 3a245f6cfe
Fix validation problem that occurs when a user tries to deactivate their account or change their password. (#13563) 2022-08-19 11:03:29 +00:00
Eric Eastwood 2c42673a9b
Add metrics to track /messages response time by room size (#13545) 
Follow-up to https://github.com/matrix-org/synapse/pull/13533

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-18 14:15:53 -05:00
Sean Quah b251cff819
Fix incorrect juggling of logging contexts in _PerHostRatelimiter (#13554) 
Signed-off-by: Sean Quah <seanq@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2022-08-18 16:26:26 +01:00
Eric Eastwood d64653d062
Track number of hosts affected by the rate limiter (#13541) 
Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation.

Follow-up to https://github.com/matrix-org/synapse/pull/13534

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-18 10:05:07 -05:00
Ayush Anand 22ea51faf9
Add support for compression to federation responses (#13537) 
Closes #13415.

Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
 2022-08-18 15:14:47 +01:00
Sean Quah 84169a82dc
Avoid blocking lazy-loading /syncs during partial joins (#13477) 
Use a state filter or accept partial state in a few places where we
request state, to avoid blocking.

To make lazy-loading `/sync`s work, we need to provide the memberships
of event senders, which are not guaranteed to be in the room state.
Instead we dig through auth events for memberships to present to
clients. The auth events of an event are guaranteed to contain a
passable membership event, otherwise the event would have been rejected.

Note that this only covers the common code paths encountered during
testing. There has been no exhaustive checking of all sync code paths.

Fixes #13146.

Signed-off-by: Sean Quah <seanq@matrix.org>
 2022-08-18 11:53:02 +01:00
Eric Eastwood 49d04e43df
Add metrics to track how the rate limiter is affecting requests (sleep/reject) (#13534) 
Related to https://github.com/matrix-org/synapse/pull/13499

Part of https://github.com/matrix-org/synapse/issues/13356
 2022-08-17 16:10:07 -05:00