Commit graph

8146 commits

Author SHA1 Message Date
Erik Johnston
e5dfb6ecbf
Fix incorrect docker hub link in release script (#16910) 2024-02-20 12:20:31 +00:00
Rainer Zufall
1b7304c8b4
fix typo in admin_api/rooms.md (#16857)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-02-20 12:20:23 +00:00
Remi Rampin
0621e8eb0e
Add metric for emails sent (#16881)
This adds a counter `synapse_emails_sent_total` for emails sent. They
are broken down by `type`, which are `password_reset`, `registration`,
`add_threepid`, `notification` (matching the methods of `Mailer`).
2024-02-14 15:30:03 +00:00
Erik Johnston
bc1db16086 Merge branch 'master' into develop 2024-02-13 13:24:29 +00:00
Erik Johnston
7b4d7429f8
Don't invalidate the entire event cache when we purge history (#16905)
We do this by adding support to the LRU cache for "extra indices" based
on the cached value. This allows us to efficiently map from room ID to
the cached events and only invalidate those.
2024-02-13 13:24:11 +00:00
Erik Johnston
01910b981f
Add a config to not send out device list updates for specific users (#16909)
List of users not to send out device list updates for when they register
new devices. This is useful to handle bot accounts.

This is undocumented as its mostly a hack to test on matrix.org.

Note: This will still send out device list updates if the device is
later updated, e.g. end to end keys are added.
2024-02-13 13:23:03 +00:00
Erik Johnston
2252bae3df 1.101.0 2024-02-13 10:45:40 +00:00
Erik Johnston
ea1b30940e Merge remote-tracking branch 'origin/release-v1.101' into develop 2024-02-09 10:52:35 +00:00
Erik Johnston
bfa93d1d3b
Only do one concurrent fetch per server in keyring (#16894)
Otherwise if we've stacked a bunch of requests for the keys of a server,
we'll end up sending lots of concurrent requests for its keys,
needlessly.
2024-02-09 10:51:11 +00:00
Erik Johnston
02a147039c
Increase batching when fetching auth chains (#16893)
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.

The smaller batching can cause performance issues on busy servers and
databases.
2024-02-09 10:51:00 +00:00
Erik Johnston
7c805f00a7 1.101.0rc1 2024-02-06 16:33:19 +00:00
David Baker
71ca199165
Accept unprefixed form of MSC3981 recurse parameter (#16842)
Now that the MSC3981 has passed FCP
2024-02-06 09:48:39 +00:00
kegsay
93edd0932e
Update docs for MacOS installs (#16854)
ICU is an optional dependency and also a pain to install. Mention that
you can just not install it and still get a working installation.
2024-02-06 09:27:38 +00:00
kegsay
505cdd044b
Fix broken links on docs (#16853)
Some links seemed to be incorrect (vector-im/sygnal and vector-im/sytest
have never been A Thing iirc) so pointed them back to matrix-org/*).
2024-02-06 09:26:55 +00:00
Will Hunt
d24d115706
Update version picker for element-hq (#16880) 2024-02-01 14:30:16 +00:00
Olivier Wilkinson (reivilibre)
458c9f6a15 1.100.0rc3 2024-01-24 14:18:35 +00:00
Erik Johnston
adf15c4f6b
Run ANALYZE after fiddling with stats (#16849)
Introduced in #16833

Fixes #16844
2024-01-24 13:57:12 +00:00
Olivier Wilkinson (reivilibre)
cf7109fc35 1.100.0rc2 2024-01-24 12:01:20 +00:00
reivilibre
d0bdd41a75
Downgrade the download-artifact and upload-artifact actions to v3 due to breaking changes. (#16847)
Partially reverts #16796

This is causing errors of the form:

```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```

for the debs and wheels stages.


There were breaking changes that weren't included in the dependabot
changelog (:/):
https://github.com/actions/upload-artifact#breaking-changes


<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `release-v1.100` <!-- git-stack-base-branch:release-v1.100 -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Downgrade the `upload-artifact` and `download-artifact` actions to v3 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-24 11:59:15 +00:00
Olivier Wilkinson (reivilibre)
f366455682 1.100.0rc1 2024-01-23 14:26:45 +00:00
Erik Johnston
c925b45567
Speed up e2e device keys queries for bot accounts (#16841)
This helps with bot accounts with lots of non-e2e devices.

The change is basically to change the order of the join for the case of
using `INNER JOIN`
2024-01-23 11:37:16 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
14c725f73b
Preparatory work for tweaking performance of auth chain lookups (#16833) 2024-01-23 11:26:27 +00:00
reivilibre
fa2700f001
Add a --generate-only option to the Complement launcher. (#16828)
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Add a --generate-only option 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-22 16:00:04 +00:00
Arnold
8459ac9be2
listen http2 deprecated nginx (updating documentation) (#16831)
More info [here](https://www.nginx.com/blog/nginx-plus-r30-released/).

Nginx threw error's at me when I used all the options of the doc
2024-01-22 14:22:16 +00:00
Shay
a68b48a5dd
Allow room creation but not publishing to continue if room publication rules are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.

This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
2024-01-22 13:59:45 +00:00
Erik Johnston
3fefb15fef
Bump minimum Rust version to 1.65.0 (#16818)
The ecosystem e.g. regex crate, have bumped up their MSRV to 1.65.0,
which was released Nov 2022. In line with our policy, let's bump to
match.
2024-01-22 11:11:52 +00:00
Mo Balaa
b99f6db039
Handle wildcard type filters properly (#14984) 2024-01-22 10:46:30 +00:00
Erik Johnston
f95cfd898e
Fix broken links in issue template (#16810)
This happened during the migration
2024-01-17 13:40:00 +00:00
Hanadi
42e1aaea68
feat: add msc4028 to versions api (#16787)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-16 14:36:08 +00:00
Erik Johnston
c43f751013
Optimize query for fetching to-device messages in /sync (#16805)
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.

Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
2024-01-11 13:37:57 +00:00
Erik Johnston
b11f7b5122
Improve DB performance of calculating badge counts for push. (#16756)
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-11 11:52:13 +00:00
Erik Johnston
a986f86c82
Correctly handle OIDC config with no client_secret set (#16806)
In previous versions of authlib using `client_secret_basic` without a
`client_secret` would result in an invalid auth header. Since authlib
1.3 it throws an exception.

The configuration may be accepted in by very lax servers, so we don't
want to deny it outright. Instead, let's default the
`client_auth_method` to `none`, which does the right thing. If the
config specifies `client_auth_method` and no `client_secret` then that
is going to be bogus and we should reject it
2024-01-10 17:16:49 +00:00
Erik Johnston
cbe8a80d10
Faster load recents for sync (#16783)
This hopefully reduces the amount of state we need to keep in memory
2024-01-10 15:11:59 +00:00
Erik Johnston
0a96fa52a2
Pull less state out if we fail to backfill (#16788)
Sometimes we fail to fetch events during backfill due to missing state,
and we often end up querying the same bad events periodically (as people
backpaginate). In such cases its likely we will continue to fail to get
the state, and therefore we should try *before* loading the state that
we have from the DB (as otherwise it's wasted DB and memory).

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:42:13 +00:00
Erik Johnston
578c5c736e
Reduce amount of state pulled out when querying federation hierachy (#16785)
There are two changes here:

1. Only pull out the required state when handling the request.
2. Change the get filtered state return type to check that we're only
querying state that was requested

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:31:35 +00:00
Erik Johnston
4c67f0391b
Split up deleting devices into batches (#16766)
Otherwise for users with large numbers of devices this can cause a lot
of woe.
2024-01-10 13:55:16 +00:00
Erik Johnston
72e9b74bbf
Fix auto-merge CI to correctly wait for linting. (#16781)
Otherwise if you hit the `Enable auto-merge` button and the linting
fails the PR is still aut-merged.
2024-01-10 13:53:44 +00:00
Erik Johnston
8189942a1f
Remove CI check for sign off (#16776)
Since we don't require one anymore.
2024-01-10 13:53:20 +00:00
Andrew Morgan
13e3740f70
Add a link to the Request log format page from Logging Sample Config (#16778) 2024-01-10 13:34:55 +00:00
Erik Johnston
c3f2f0f063
Faster partial join to room with complex auth graph (#7)
Instead of persisting outliers in a bunch of batches, let's just do them
all at once.

This is fine because all `_auth_and_persist_outliers_inner` is doing is
checking the auth rules for each event, which requires the events to be
topologically sorted by the auth graph.
2024-01-10 12:29:42 +00:00
Erik Johnston
7109274c65 1.99.0rc1 2024-01-09 13:44:02 +00:00
reivilibre
a83a337c4d
Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. (#16759)
The idea here being that the directory server shouldn't advertise rooms
to a requesting server is the requesting server would not be allowed to
join or participate in the room.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Pass `from_federation_origin` down into room list retrieval code 

</li>
<li>

Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

fixup! Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

Cap the number of /publicRooms entries to 100 

</li>
<li>

Simplify code now that you can't request unlimited rooms 

</li>
<li>

Filter out rooms from federated requests that don't have the correct ACL

</li>
<li>

Request a handful more when filtering ACLs so that we can try to avoid
shortchanging the requester

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-08 17:24:20 +00:00
Erik Johnston
5d3850b038
Port EventInternalMetadata class to Rust (#16782)
There are a couple of things we need to be careful of here:

1. The current python code does no validation when loading from the DB,
so we need to be careful to ignore such errors (at least on jki.re there
are some old events with internal metadata fields of the wrong type).
2. We want to be memory efficient, as we often have many hundreds of
thousands of events in the cache at a time.

---------

Co-authored-by: Quentin Gliech <quenting@element.io>
2024-01-08 14:06:48 +00:00
Erik Johnston
81b1c56288
Fix linting (#16780)
Introduced in #16762
2024-01-05 13:29:00 +00:00
Erik Johnston
7469fa7585
Simplify internal metadata class. (#16762)
We remove these fields as they're just duplicating data the event
already stores, and (for reasons 🤫) I'd like to simplify
the class to only store simple types.

I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
2024-01-05 13:03:20 +00:00
Gaël Goinvic
9ee3db1de5
Implement cosign on docker image (#16774)
Signed-off-by: Gaël Goinvic <gaelg@element.io>
2024-01-04 11:49:33 +00:00
David Baker
25b3ba5328
Add recursion_depth to /relations if recursing (#16775)
This is an extra response parameter just added to MSC3981. In the
current impl, the recursion depth is always 3, so this just returns a
static 3 if the recurse parameter is supplied.
2024-01-04 09:57:21 +00:00
Adam Jędrzejewski
c7d0d02be7
Search non ASCII display names using Admin API (#16767)
Closes #16370

Signed-off-by: Adam Jedrzejewski <adamjedrzejewski@icloud.com>
2024-01-04 09:36:57 +00:00
FadhlanR
eabedd9520
Fix email verification redirection (#16761)
Previously, the response status of `HTMLResource` was hardcoded as
`200`. However, for proper redirection after the user verifies their
email, we require the status to be `302`. This PR addresses that issue
by using `code` as response status.
2024-01-02 16:25:26 +00:00
Andrew Morgan
1607ed5b2c
Delete the add version picker GHA workflow (#9453)
Added in https://github.com/matrix-org/synapse/pull/16533, this workflow
was intended to be run once to add the version picker to all historical
versions of the https://matrix-org.github.io/synapse documentation
website.

Note that the latest version of the docs built from this repo now exist
at https://element-hq.github.io/synapse/.

The workflow has been run successfully and the version picker was added
to the documentation. Thus we can now delete this workflow.

---

Note: Do not confuse this PR with
https://github.com/matrix-org/synapse/issues/9453. This PR was made
while we were populating this repo with "Dummy issues" after the
changeover from matrix-org/synapse to element-hq/synapse - therefore
referencing this PR may cause some confusion.
2024-01-02 16:10:19 +00:00
Christian Lölkes
35b6365317
Add another custom statistics collection server (#16769)
Signed-off-by: Christian Lölkes <christian.loelkes@gmail.com>
2024-01-02 11:54:12 +00:00
Dirk Klimpel
14ed84ac33
Enable user without password (#16770)
Closes:
- https://github.com/matrix-org/synapse/issues/10397
- #10397 

An administrator should know whether he wants to set a password or not.
There are many uses cases where a blank password is required.

- Use of only some users with SSO.
- Use of bots with password, users with SSO
2024-01-02 11:52:51 +00:00
Fredrik Lanker
c1fe945dd5
Remove config value from header (#16763)
Signed-off-by: Fredrik Lanker <fredrik@lanker.se>
2024-01-02 11:50:50 +00:00
Erik Johnston
8a50312099
Move the rust stubs inline for better IDE integration (#16757)
At least for vscode this allows click through / type checking / syntax
highlighting.
2023-12-21 13:31:36 +00:00
Erik Johnston
719014d9d5
Fix sample config doc CI (#16758)
I accidentally broke it during the move by removing a trailing new line.
2023-12-21 13:31:19 +00:00
Zeeshan Rafiq
e108cde669
Sentry Alert configuration based on production and development environment (#16738) 2023-12-12 16:04:41 +00:00
Mathieu Velten
e108c31fc0
Add avatar and topic settings for server notice room (#16679) 2023-12-12 15:22:19 +00:00
Mathieu Velten
9f6c644825
Add config to change the delay before sending a notification email (#16696) 2023-12-12 10:28:56 +00:00
Dmytro Kagirov
483d22afc3
Adding a version picker for Synapse docs (#16533) 2023-12-11 14:18:40 +00:00
elara-leitstellentechnik
10ada2ff6d
Write signing keys with file mode 0640 (#16740)
Co-authored-by: Fabian Klemp <fabian.klemp@frequentis.com>
2023-12-08 16:25:57 +00:00
Dirk Klimpel
aa983c7b0f
Clarify documentation for only_for_reauth (#16737) 2023-12-07 17:52:50 +00:00
David Robertson
fcf3c493c2
Expose OIDC discovery information under the CSAPI (#16726)
Co-authored-by: Quentin Gliech <quenting@element.io>
2023-12-06 13:48:09 +00:00
David Robertson
3c83d8f0af
1.98.0rc1 2023-12-05 13:14:36 +00:00
David Robertson
0a00c99823
Fix upgrading a room without events field in power levels (#16725) 2023-12-05 12:06:21 +00:00
Amanda H. L. de Andrade Katz
e87499b3f4
Add how to validate configuration file with synapse.config script (#16714) 2023-12-05 11:42:56 +00:00
Will Hunt
ea783550bb
Set response values to zero if None for /_synapse/admin/v1/federation/destinations (#16729) 2023-12-05 11:40:27 +00:00
David Robertson
0bb8e418a4
Fix postgres schema after dropping old tables (#16730) 2023-12-05 11:08:40 +00:00
reivilibre
51e4e35653
Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (take 2, now with no added deadlocks!) (#16658)
* Add `ALTER TABLE ... REPLICA IDENTITY ...` for individual tables

We can't combine them into one file as it makes it likely to hit a deadlock

if Synapse is running, as it only takes one other transaction to access two

tables in a different order to the schema delta.

* Add notes

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Re-introduce REPLICA IDENTITY test

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-12-04 14:57:28 +00:00
villepeh
0aa4d3b6f7
Switch UNIX socket paths to /run, and add a UNIX socket example for HAProxy (#16700) 2023-12-04 12:38:46 +00:00
Mathieu Velten
9e7f80037d
Server notices: add an autojoin setting for the notices room (#16699)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-12-04 12:31:42 +00:00
Travis Ralston
d6e194b2bc
Implement MSC4069: Inhibit profile propagation (#16636)
MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
2023-12-04 11:36:12 +00:00
Andrew Yasinishyn
63d96bfc61
ModuleAPI SSO auth callbacks (#15207)
Signed-off-by: Andrii Yasynyshyn yasinishyn.a.n@gmail.com
2023-12-01 14:31:50 +00:00
Patrick Cloke
579c6be5f6
Drop unused tables & unneeded access token ID for events. (#16522) 2023-12-01 10:12:00 +00:00
Mo Balaa
3a092699e5
Upgrade poetry-core range to fix issue with .so file (#16702)
poetry-core 1.8.x includes a fix which properly moves the generate
synapse_rust.abi3.so file to the synapse directory when using an
editable install.

Without this change developers are left with a confusing experience
of the synapse.synapse_rust module not being found after installation.
2023-11-29 15:46:43 -05:00
Patrick Cloke
dcf949cd87
Declare support for Matrix v1.7, v1.8, and v1.9. (#16707) 2023-11-29 15:02:09 -05:00
Patrick Cloke
d6c3b7584f
Request & follow redirects for /media/v3/download (#16701)
Implement MSC3860 to follow redirects for federated media downloads.

Note that the Client-Server API doesn't support this (yet) since the media
repository in Synapse doesn't have a way of supporting redirects.
2023-11-29 19:03:42 +00:00
Erik Johnston
a14678492e
Reduce DB load when forget on leave setting is disabled (#16668)
* Reduce DB load when forget on leave setting is disabled

* Newsfile
2023-11-29 18:21:30 +00:00
Erik Johnston
19dac97480
Add a workflow to try and automatically fixup a PR (#16704)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-11-29 14:07:32 +00:00
Erik Johnston
df366966b4
Speed up pruning of user_ips table (#16667)
Silly query planner
2023-11-29 11:54:42 +00:00
Patrick Cloke
77882b6a7d
Document which versions of Synapse have compatible schema versions. (#16661) 2023-11-28 11:01:24 -05:00
Mathieu Velten
b0ed14d815
Ignore encryption_enabled_by_default_for_room_type for notices room (#16677) 2023-11-28 13:15:26 +00:00
Patrick Cloke
d199b84006
Remove old full schema dumps. (#16697)
These are not useful and make it difficult to search for
table definitions, etc.
2023-11-28 07:28:07 -05:00
David Robertson
8751f0ef32
Fix poetry version typo in contributors' guide (#16695) 2023-11-27 15:16:20 +00:00
David Robertson
0619c2bbd2
Move media retention tests out of rest tests (#16684)
* Move media retention tests out of rest tests

AFAICS this doesn't make any HTTP requests and so it ought not to belong
in `tests.rest`.

* Changelog
2023-11-27 01:29:46 +00:00
David Robertson
c3627d0f99
Correctly read to-device stream pos on SQLite (#16682) 2023-11-24 13:42:38 +00:00
David Robertson
32a59a6495
Keep track of user_ips and monthly_active_users when delegating auth (#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests

sick of copypasting

* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
2023-11-23 12:35:37 +00:00
Charles Wright
1a5f9bb651
Enable refreshable tokens on the admin registration endpoint (#16642)
Signed-off-by: Charles Wright <cvwright@futo.org>
2023-11-22 15:01:09 +00:00
V02460
f2430b16d1
Bump pyo3 (0.20), pythonize (0.20), pyo3-log (0.9) (#16673)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2023-11-22 14:55:43 +00:00
Mathieu Velten
c432d8f18f
Admin API for server notice: consistently bypass rate limits (#16670)
* Admin API for server notice: disable rate limit for all calls

* Add changelog

* Update changelog.d/16670.bugfix
2023-11-22 13:47:29 +00:00
Jason Little
460743da16
Filter out auth chain queries that don't exist (#16552) 2023-11-22 10:59:16 +00:00
David Robertson
bb86eb9814
1.97.0rc1 2023-11-21 12:38:46 +00:00
Erik Johnston
6088303efb
Speed up how quickly we launch new tasks (#16660)
Now that we're reducing concurrency (#16656), this is more important.
2023-11-17 16:36:02 +00:00
Patrick Cloke
d9dcfe2a35
Bump requests-toolbelt from 0.10.1 to 1.0.0. (#16659) 2023-11-17 10:23:07 -05:00
Erik Johnston
9c02ef21e0
Speed up purge room by adding index (#16657)
What it says on the tin
2023-11-17 14:15:44 +00:00
Erik Johnston
6fec2d035f
Also discard 'caches' and 'backfill' stream POSITIONS (#16655)
Follow on from #16640
2023-11-17 14:14:29 +00:00
Michael Weimann
518e4de758
Update admin user API return types in docs. (#16654) 2023-11-17 13:38:25 +00:00
Erik Johnston
700c8a0de5
Reduce task concurrency (#16656) 2023-11-17 13:14:26 +00:00
Erik Johnston
4d6b800385
Revert "Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)" (#16652)
This reverts commit 830988ae72.
2023-11-16 16:57:26 +00:00
Erik Johnston
ef5329a9f9
Revert "Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651)
This reverts commit 69afe3f7a0.
2023-11-16 16:48:48 +00:00
Erik Johnston
3e8531d3ba
Speed up deleting device messages (#16643)
Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
2023-11-16 15:19:35 +00:00
Erik Johnston
1b238e8837
Speed up persisting large number of outliers (#16649)
Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
2023-11-16 14:25:35 +00:00
Erik Johnston
fef08cbee8
Fix sending out of order POSITION over replication (#16639)
If a worker reconnects to Redis we send out the current positions of all our streams. However, if we're also trying to send out a backlog of RDATA at the same time then we can end up sending a `POSITION` with the current token *before* we've sent all the RDATA before the current token.

This doesn't cause actual bugs as the receiving servers see the POSITION, fetch the relevant rows from the DB, and then ignore the old RDATA as they come in. However, this is inefficient so it'd be better if we didn't  send out-of-order positions
2023-11-16 13:05:09 +00:00
Erik Johnston
898655fd12
More efficiently handle no-op POSITION (#16640)
We may receive `POSITION` commands where we already know that worker has
advanced past that position, so there is no point in handling it.
2023-11-16 12:32:17 +00:00
reivilibre
830988ae72
Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)
* Fix the CI query that did not detect all cases of missing primary keys

* Add more missing REPLICA IDENTITY entries

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-16 12:26:27 +00:00
David Robertson
43d1aa75e8
Add an Admin API to temporarily grant the ability to update an existing cross-signing key without UIA (#16634) 2023-11-15 17:28:10 +00:00
Sumner Evans
999bd77d3a
Asynchronous Uploads (#15503)
Support asynchronous uploads as defined in MSC2246.
2023-11-15 09:19:24 -05:00
Patrick Cloke
80922dc46e
Add links to pre-1.0 changelog issue/PR references. (#16638) 2023-11-15 13:31:24 +00:00
Patrick Cloke
f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Will Hunt
4dd18bdc2e
Improve documentation for /_synapse/admin/v1/rooms/<room_id>/timestamp_to_event (#16631) 2023-11-14 11:43:44 -05:00
Nick Mills-Barrett
0e36a57b60
Remove whole table locks on push rule add/delete (#16051)
The statements are already executed within a transaction thus a table
level lock is unnecessary.
2023-11-13 16:57:44 +00:00
reivilibre
69afe3f7a0
Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)
* Add Postgres replica identities to tables that don't have an implicit one

Fixes #16224

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Move the delta to version 83 as we missed the boat for 82

* Add a test that all tables have a REPLICA IDENTITY

* Extend the test to include when indices are deleted

* isort

* black

* Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11

* Update tests/storage/test_database.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add missed tables

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-11-13 16:03:22 +00:00
David Robertson
fb2554b11f
Fix outbound_federation_restricted_to docs & note when added (#16628) 2023-11-13 14:26:49 +00:00
Patrick Cloke
2c6a7dfcbf
Use attempt_to_set_autocommit everywhere. (#16615)
To avoid asserting the type of the database connection.
2023-11-09 16:19:42 -05:00
reivilibre
dc7f068d9c
Fix a long-standing bug where Synapse would not unbind third-party identifiers for Application Service users when deactivated and would not emit a compliant response. (#16617)
* Don't skip unbinding 3PIDs and returning success status when deactivating AS user

Fixes #16608

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-09 20:18:25 +00:00
Patrick Cloke
bc4372ad81
Use dbname instead of database for Postgres config. (#16618) 2023-11-09 14:40:45 -05:00
Patrick Cloke
9f514dd0fb
Use _invalidate_cache_and_stream_bulk in more places. (#16616)
This takes advantage of the new bulk method in more places to
invalidate caches for many keys at once (and then to stream that
over replication).
2023-11-09 14:40:30 -05:00
Patrick Cloke
ab3f1b3b53
Convert simple_select_one_txn and simple_select_one to return tuples. (#16612) 2023-11-09 11:13:31 -05:00
Patrick Cloke
ff716b483b
Return attrs for more media repo APIs. (#16611) 2023-11-09 11:00:30 -05:00
David Robertson
91587d4cf9
Bulk-invalidate e2e cached queries after claiming keys (#16613)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-11-09 15:57:09 +00:00
Patrick Cloke
455ef04187
Avoid updating the same rows multiple times with simple_update_many_txn. (#16609)
simple_update_many_txn had a bug in it which would cause each
update to be applied twice.
2023-11-07 14:02:09 -05:00
Patrick Cloke
9738b1c497
Avoid executing no-op queries. (#16583)
If simple_{insert,upsert,update}_many_txn is called without any data
to modify then return instead of executing the query.

This matches the behavior of simple_{select,delete}_many_txn.
2023-11-07 14:00:25 -05:00
Patrick Cloke
ec9ff389f4
More tests for the simple_* methods. (#16596)
Expand tests for the simple_* database methods, additionally
test against both PostgreSQL and SQLite variants.
2023-11-07 09:34:23 -05:00
Patrick Cloke
7e5d3b06fa
Collect information for PushRuleEvaluator in parallel. (#16590)
Fetch information needed for push rule evaluation in parallel.
Ideally this would use query pipelining, but this is not
available in psycopg2.

Due to the database thread pool this may result in little
to no parallelization.
2023-11-06 15:41:57 -05:00
Patrick Cloke
1dd3074629
Bump setuptools_rust to match pinned version. (#16605) 2023-11-06 09:13:53 -05:00
Patrick Cloke
cc4fe68adf
Support reactor timing metric on more reactors. (#16532)
Previously only Twisted's EPollReactor was compatible with the
reactor timing metric, notably not working when asyncio was used.

After this change, the following configurations support the reactor
timing metric:

* poll, epoll, or select reactors
* asyncio reactor with a poll, epoll, select, /dev/poll, or kqueue event loop.
2023-11-06 08:31:22 -05:00
Patrick Cloke
92828a7f95
Simplify event persistence code (#16584)
The event persistence code used to handle multiple rooms
at a time, but was simplified to only ever be called with a
single room at a time (different rooms are now handled in
parallel). The code is still generic to multiple rooms causing
a lot of work that is unnecessary (e.g. unnecessary loops, and
partitioning data by room).

This strips out the ability to handle multiple rooms at once, greatly
simplifying the code.
2023-11-03 07:30:31 -04:00
Patrick Cloke
0afbef30cf
Use simple_select_many_txn in event persistance code. (#16585)
Just to standardize on the normal helpers, it might also have
a slight perf improvement on PostgreSQL which will now use
`ANY (?)` instead of `IN (?, ?, ...)`.
2023-11-02 09:41:00 -04:00
dependabot[bot]
c812f43bd7
Bump twisted from 23.8.0 to 23.10.0 (#16588) 2023-11-01 10:23:13 +00:00
Patrick Cloke
ed1b879576
Do not call getfullargspec on every call. (#16589)
getfullargspec is relatively expensive and the results will
not change between calls, so precalculate it outside the
wrapper.
2023-10-31 20:16:17 +00:00
Patrick Cloke
cfb6d38c47
Remove remaining usage of cursor_to_dict. (#16564) 2023-10-31 13:13:28 -04:00
Erik Johnston
4bb2b4aa9a 1.96.0rc1 2023-10-31 14:09:13 +00:00
Erik Johnston
60c5e8d79b Revert "1.96.0rc1"
This reverts commit 4724a6ded1.
2023-10-31 14:08:45 +00:00
Erik Johnston
4724a6ded1 1.96.0rc1 2023-10-31 13:47:08 +00:00
David Robertson
de981ae567
Claim local one-time-keys in bulk (#16565)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-10-30 21:25:21 +00:00
Niranjan Kurhade
91aa52c911
Clients link fixed in README (#16569) 2023-10-30 16:05:34 +00:00
Erik Johnston
408c13801a
Add fast path for replication events stream fetch (#16580)
We can bail early if the from token is greater than or equal to the
current token.
2023-10-30 14:47:57 +00:00
David Robertson
fdce83ee60
Claim fallback keys in bulk (#16570) 2023-10-30 14:34:37 +00:00
dependabot[bot]
a3f6200d65
Bump setuptools-rust from 1.7.0 to 1.8.0 (#16574)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Robertson <davidr@element.io>
2023-10-30 13:40:17 +00:00
David Robertson
4e1a19d375
Run actions/setup-go after checking out complement (#16567) 2023-10-30 13:07:08 +00:00
Erik Johnston
8c63e93286
Fix HTTP repl response to use minimum token (#16578) 2023-10-30 12:27:14 +00:00
David Robertson
747416e94c
Portdb: don't copy a table that gets rebuilt (#16563) 2023-10-27 20:14:02 +01:00
kegsay
11a8ae0632
complement: enable dirty runs (#16520)
* complement: enable dirty runs

* Add changelog

* Set a low connpool limit when running in Complement

Dirty runs can cause many containers to be running concurrently,
which seems to easily exhaust resources on the host. The increased
speedup from dirty runs also seems to use more db connections on
workers, which are misconfigured currently to have
`SUM(workers * cp_max) > max_connections`, causing
```
FATAL:  sorry, too many clients already
```
which results in tests failing.

* Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake

* Debug logging

* Only run flakey tests

* Only adjust connection pool limits in worker mode

* Move cp vars to somewhere where they get executed in CI

* Move cp values back to where they actually work

* Debug logging

* Try p=1 to see if this makes worker mode happier

* Remove debug logging
2023-10-27 18:29:20 +01:00
Patrick Cloke
2bf9341406
Ensure local invited & knocking users leave before purge. (#16559)
This is mostly useful for federated rooms where some users
would get stuck in the invite or knock state when the room
was purged from their homeserver.
2023-10-27 12:50:50 -04:00
Erik Johnston
5413cefe32
Reduce amount of caches POSITIONS we send (#16561)
Follow on from / actually correctly does #16557
2023-10-27 16:07:11 +01:00
Erik Johnston
89dbbd68e1
Reduce spurious replication catchup (#16555) 2023-10-27 13:27:20 +00:00
Erik Johnston
928e964857
Fix cross-worker ratelimiting (#16558)
c.f. #16481
2023-10-27 12:52:40 +01:00
Erik Johnston
0680d76659
Reduce replication traffic due to reflected cache stream POSITION (#16557) 2023-10-27 12:51:08 +01:00
Erik Johnston
c02406ac71
Add new module API for adding custom fields to events unsigned section (#16549) 2023-10-27 09:04:08 +00:00
Patrick Cloke
679c691f6f
Remove more usages of cursor_to_dict. (#16551)
Mostly to improve type safety.
2023-10-26 15:12:28 -04:00