Commit graph

8063 commits

Author SHA1 Message Date
V02460
2bdf6280f6
Raise poetry-core version cap to 1.9.0 (#16986)
A new poetry-core version was released. See if CI is happy. Required for
the latest Fedora Synapse package.
2024-03-13 16:40:08 +00:00
Erik Johnston
5c0b87ff95 Update changelog 2024-03-12 15:12:19 +00:00
Erik Johnston
0d44f64c4e Merge remote-tracking branch 'origin/develop' into release-v1.103 2024-03-12 15:11:03 +00:00
Gerrit Gogel
1f88790764
Prevent locking up while processing batched_auth_events (#16968)
This PR aims to fix #16895, caused by a regression in #7 and not fixed
by #16903. The PR #16903 only fixes a starvation issue, where the CPU
isn't released. There is a second issue, where the execution is blocked.
This theory is supported by the flame graphs provided in #16895 and the
fact that I see the CPU usage reducing and far below the limit.

Since the changes in #7, the method `check_state_independent_auth_rules`
is called with the additional parameter `batched_auth_events`:


6fa13b4f92/synapse/handlers/federation_event.py (L1741-L1743)


It makes the execution enter this if clause, introduced with #15195


6fa13b4f92/synapse/event_auth.py (L178-L189)

There are two issues in the above code snippet.

First, there is the blocking issue. I'm not entirely sure if this is a
deadlock, starvation, or something different. In the beginning, I
thought the copy operation was responsible. It wasn't. Then I
investigated the nested `store.get_events` inside the function `update`.
This was also not causing the blocking issue. Only when I replaced the
set difference operation (`-` ) with a list comprehension, the blocking
was resolved. Creating and comparing sets with a very large amount of
events seems to be problematic.

This is how the flamegraph looks now while persisting outliers. As you
can see, the execution no longer locks up in the above function.

![output_2024-02-28_13-59-40](https://github.com/element-hq/synapse/assets/13143850/6db9c9ac-484f-47d0-bdde-70abfbd773ec)

Second, the copying here doesn't serve any purpose, because only a
shallow copy is created. This means the same objects from the original
dict are referenced. This fails the intention of protecting these
objects from mutation. The review of the original PR
https://github.com/matrix-org/synapse/pull/15195 had an extensive
discussion about this matter.

Various approaches to copying the auth_events were attempted:
1) Implementing a deepcopy caused issues due to
builtins.EventInternalMetadata not being pickleable.
2) Creating a dict with new objects akin to a deepcopy.
3) Creating a dict with new objects containing only necessary
attributes.

Concluding, there is no easy way to create an actual copy of the
objects. Opting for a deepcopy can significantly strain memory and CPU
resources, making it an inefficient choice. I don't see why the copy is
necessary in the first place. Therefore I'm proposing to remove it
altogether.

After these changes, I was able to successfully join these rooms,
without the main worker locking up:
- #synapse:matrix.org
- #element-android:matrix.org
- #element-web:matrix.org
- #ecips:matrix.org
- #ipfs-chatter:ipfs.io
- #python:matrix.org
- #matrix:matrix.org
2024-03-12 15:07:36 +00:00
Erik Johnston
9d7880c0c6 1.103.0rc1 2024-03-12 15:03:45 +00:00
Alexander Fechler
48f59d3806
deactivated flag refactored to filter deactivated users. (#16874)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-03-11 16:08:04 +00:00
Patrick Cloke
696cc9e802
Stabilize support for Retry-After header (MSC4014) (#16947) 2024-03-08 09:33:46 +00:00
Quentin Gliech
4af33015af
Fix joining remote rooms when a on_new_event callback is registered (#16973)
Since Synapse 1.76.0, any module which registers a `on_new_event`
callback would brick the ability to join remote rooms.
This is because this callback tried to get the full state of the room,
which would end up in a deadlock.

Related:
https://github.com/matrix-org/synapse-auto-accept-invite/issues/18

The following module would brick the ability to join remote rooms:

```python
from typing import Any, Dict, Literal, Union
import logging

from synapse.module_api import ModuleApi, EventBase

logger = logging.getLogger(__name__)

class MyModule:
    def __init__(self, config: None, api: ModuleApi):
        self._api = api
        self._config = config

        self._api.register_third_party_rules_callbacks(
            on_new_event=self.on_new_event,
        )

    async def on_new_event(self, event: EventBase, _state_map: Any) -> None:
        logger.info(f"Received new event: {event}")

    @staticmethod
    def parse_config(_config: Dict[str, Any]) -> None:
        return None
```

This is technically a breaking change, as we are now passing partial
state on the `on_new_event` callback.
However, this callback was broken for federated rooms since 1.76.0, and
local rooms have full state anyway, so it's unlikely that it would
change anything.
2024-03-06 16:00:20 +01:00
Andrew Morgan
ab80b3412e
Revert "Ignore notification counts from rooms you've left" (#16981) 2024-03-05 16:02:54 +00:00
Andrew Morgan
274f289a52
Ignore notification counts from rooms you've left (#16954)
Co-authored-by: reivilibre <oliverw@matrix.org>
2024-02-23 14:12:10 +00:00
Twilight Sparkle
8de3283ebe
Add docs on upgrading from a very old version (#16951)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-22 17:36:41 +00:00
kegsay
0c55c76da8
Better complement docs (#16946) 2024-02-20 17:14:50 +00:00
Andrew Morgan
3eb0a3b468 Merge branch 'release-v1.102' into develop 2024-02-20 16:57:37 +00:00
Georg
7c1c011942
Add HAProxy example for single port operation (#16768) 2024-02-20 16:15:58 +00:00
Andrew Morgan
7856ec96ef 1.102.0rc1 2024-02-20 15:51:17 +00:00
Erik Johnston
cdbbf3653d
Don't lock up when joining large rooms (#16903)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-02-20 14:29:18 +00:00
kegsay
c51a2240d1
bugfix: always prefer unthreaded receipt when >1 exist (MSC4102) (#16927)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-20 14:12:06 +00:00
Erik Johnston
e5dfb6ecbf
Fix incorrect docker hub link in release script (#16910) 2024-02-20 12:20:31 +00:00
Rainer Zufall
1b7304c8b4
fix typo in admin_api/rooms.md (#16857)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-02-20 12:20:23 +00:00
Remi Rampin
0621e8eb0e
Add metric for emails sent (#16881)
This adds a counter `synapse_emails_sent_total` for emails sent. They
are broken down by `type`, which are `password_reset`, `registration`,
`add_threepid`, `notification` (matching the methods of `Mailer`).
2024-02-14 15:30:03 +00:00
Erik Johnston
bc1db16086 Merge branch 'master' into develop 2024-02-13 13:24:29 +00:00
Erik Johnston
7b4d7429f8
Don't invalidate the entire event cache when we purge history (#16905)
We do this by adding support to the LRU cache for "extra indices" based
on the cached value. This allows us to efficiently map from room ID to
the cached events and only invalidate those.
2024-02-13 13:24:11 +00:00
Erik Johnston
01910b981f
Add a config to not send out device list updates for specific users (#16909)
List of users not to send out device list updates for when they register
new devices. This is useful to handle bot accounts.

This is undocumented as its mostly a hack to test on matrix.org.

Note: This will still send out device list updates if the device is
later updated, e.g. end to end keys are added.
2024-02-13 13:23:03 +00:00
Erik Johnston
2252bae3df 1.101.0 2024-02-13 10:45:40 +00:00
Erik Johnston
ea1b30940e Merge remote-tracking branch 'origin/release-v1.101' into develop 2024-02-09 10:52:35 +00:00
Erik Johnston
bfa93d1d3b
Only do one concurrent fetch per server in keyring (#16894)
Otherwise if we've stacked a bunch of requests for the keys of a server,
we'll end up sending lots of concurrent requests for its keys,
needlessly.
2024-02-09 10:51:11 +00:00
Erik Johnston
02a147039c
Increase batching when fetching auth chains (#16893)
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.

The smaller batching can cause performance issues on busy servers and
databases.
2024-02-09 10:51:00 +00:00
Erik Johnston
7c805f00a7 1.101.0rc1 2024-02-06 16:33:19 +00:00
David Baker
71ca199165
Accept unprefixed form of MSC3981 recurse parameter (#16842)
Now that the MSC3981 has passed FCP
2024-02-06 09:48:39 +00:00
kegsay
93edd0932e
Update docs for MacOS installs (#16854)
ICU is an optional dependency and also a pain to install. Mention that
you can just not install it and still get a working installation.
2024-02-06 09:27:38 +00:00
kegsay
505cdd044b
Fix broken links on docs (#16853)
Some links seemed to be incorrect (vector-im/sygnal and vector-im/sytest
have never been A Thing iirc) so pointed them back to matrix-org/*).
2024-02-06 09:26:55 +00:00
Will Hunt
d24d115706
Update version picker for element-hq (#16880) 2024-02-01 14:30:16 +00:00
Olivier Wilkinson (reivilibre)
458c9f6a15 1.100.0rc3 2024-01-24 14:18:35 +00:00
Erik Johnston
adf15c4f6b
Run ANALYZE after fiddling with stats (#16849)
Introduced in #16833

Fixes #16844
2024-01-24 13:57:12 +00:00
Olivier Wilkinson (reivilibre)
cf7109fc35 1.100.0rc2 2024-01-24 12:01:20 +00:00
reivilibre
d0bdd41a75
Downgrade the download-artifact and upload-artifact actions to v3 due to breaking changes. (#16847)
Partially reverts #16796

This is causing errors of the form:

```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```

for the debs and wheels stages.


There were breaking changes that weren't included in the dependabot
changelog (:/):
https://github.com/actions/upload-artifact#breaking-changes


<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `release-v1.100` <!-- git-stack-base-branch:release-v1.100 -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Downgrade the `upload-artifact` and `download-artifact` actions to v3 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-24 11:59:15 +00:00
Olivier Wilkinson (reivilibre)
f366455682 1.100.0rc1 2024-01-23 14:26:45 +00:00
Erik Johnston
c925b45567
Speed up e2e device keys queries for bot accounts (#16841)
This helps with bot accounts with lots of non-e2e devices.

The change is basically to change the order of the join for the case of
using `INNER JOIN`
2024-01-23 11:37:16 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
14c725f73b
Preparatory work for tweaking performance of auth chain lookups (#16833) 2024-01-23 11:26:27 +00:00
reivilibre
fa2700f001
Add a --generate-only option to the Complement launcher. (#16828)
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Add a --generate-only option 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-22 16:00:04 +00:00
Arnold
8459ac9be2
listen http2 deprecated nginx (updating documentation) (#16831)
More info [here](https://www.nginx.com/blog/nginx-plus-r30-released/).

Nginx threw error's at me when I used all the options of the doc
2024-01-22 14:22:16 +00:00
Shay
a68b48a5dd
Allow room creation but not publishing to continue if room publication rules are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.

This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
2024-01-22 13:59:45 +00:00
Erik Johnston
3fefb15fef
Bump minimum Rust version to 1.65.0 (#16818)
The ecosystem e.g. regex crate, have bumped up their MSRV to 1.65.0,
which was released Nov 2022. In line with our policy, let's bump to
match.
2024-01-22 11:11:52 +00:00
Mo Balaa
b99f6db039
Handle wildcard type filters properly (#14984) 2024-01-22 10:46:30 +00:00
Erik Johnston
f95cfd898e
Fix broken links in issue template (#16810)
This happened during the migration
2024-01-17 13:40:00 +00:00
Hanadi
42e1aaea68
feat: add msc4028 to versions api (#16787)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-16 14:36:08 +00:00
Erik Johnston
c43f751013
Optimize query for fetching to-device messages in /sync (#16805)
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.

Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
2024-01-11 13:37:57 +00:00
Erik Johnston
b11f7b5122
Improve DB performance of calculating badge counts for push. (#16756)
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-11 11:52:13 +00:00
Erik Johnston
a986f86c82
Correctly handle OIDC config with no client_secret set (#16806)
In previous versions of authlib using `client_secret_basic` without a
`client_secret` would result in an invalid auth header. Since authlib
1.3 it throws an exception.

The configuration may be accepted in by very lax servers, so we don't
want to deny it outright. Instead, let's default the
`client_auth_method` to `none`, which does the right thing. If the
config specifies `client_auth_method` and no `client_secret` then that
is going to be bogus and we should reject it
2024-01-10 17:16:49 +00:00