Commit graph

23875 commits

Author SHA1 Message Date
Mathieu Velten
74ab329eaa
Pass module API to OIDC mapping provider (#16974)
As done for SAML mapping provider, let's pass the module API to the OIDC
one so the mapper can do more logic in its code.
2024-03-19 17:20:10 +00:00
V02460
05489d89c6
Specify IP subnet literals in canonical form (#16953)
This is needed, because the netaddr package removed support for the
implicit prefix form in version 1.0.0:
https://github.com/netaddr/netaddr/pull/360
2024-03-19 17:19:12 +00:00
Richard van der Hoff
9635822cc1
Clarify docs for some room state functions (#16950)
State *before* an event is different to state *after* that event, and
people tend to assume the wrong one.
2024-03-19 17:16:37 +00:00
Olivier Wilkinson (reivilibre)
42fa47a2a4 Merge branch 'master' into develop 2024-03-19 14:19:00 +00:00
Olivier Wilkinson (reivilibre)
0b4dc4de7c 1.103.0 2024-03-19 12:24:41 +00:00
Richard van der Hoff
52f456a822
/sync: Fix edge-case in calculating the "device_lists" response (#16949)
Fixes https://github.com/element-hq/synapse/issues/16948. If the `join`
and the `leave` are in the same sync response, we need to count them as
a "left" user.
2024-03-14 17:34:19 +00:00
Richard van der Hoff
6d5bafb2c8
Split up SyncHandler.compute_state_delta (#16929)
This is a huge method, which melts my brain.

This is a non-functional change which lays some groundwork for future
work in this area.
2024-03-14 17:18:48 +00:00
Will Hunt
1198f649ea
Sort versions in the documentation version picker appropriately. (#16966)
Fixes #16964 

This adds a proper sorter for versions which takes into account semantic
versions, rather than just relying on localeCompare.
2024-03-14 15:18:51 +00:00
Richard van der Hoff
acc2f00eca
upgrade.md: fix grammatical errors (#16965)
comma splice
"rollback" is a noun
2024-03-14 13:54:01 +00:00
Andrew Morgan
1c1b0bfa77
Add query to update local cache of a remote user's device list to docs (#16892) 2024-03-14 13:53:25 +00:00
Mathieu Velten
cb562d73aa
Improve lock performance when a lot of locks are waiting (#16840)
When a lot of locks are waiting for a single lock, notifying all locks
independently with `call_later` on each release is really costly and
incurs some kind of async contention, where the CPU is spinning a lot
for not much.

The included test is taking around 30s before the change, and 0.5s
after.

It was found following failing tests with
https://github.com/element-hq/synapse/pull/16827.
2024-03-14 13:49:54 +00:00
dependabot[bot]
a111ba0207
Bump types-psycopg2 from 2.9.21.16 to 2.9.21.20240311 (#16995) 2024-03-14 10:36:21 +00:00
dependabot[bot]
1cc1d6b655
Bump pyo3 from 0.20.2 to 0.20.3 (#16962) 2024-03-14 10:36:13 +00:00
Richard van der Hoff
92f2069627
Multi-worker-docker-container: disable log buffering (#16919)
Background: we have a `matrixdotorg/synapse-workers` docker image, which
is intended for running multiple workers within the same container. That
image includes a `prefix-log` script which, for each line printed to
stdout or stderr by one of the processes, prepends the name of the
process.

This commit disables buffering in that script, so that lines are logged
quickly after they are printed. This makes it much easier to understand
the output, since they then come out in a natural order.
2024-03-13 17:21:37 +00:00
dependabot[bot]
9b5eef95ad
Bump ruff from 0.1.14 to 0.3.2 (#16994) 2024-03-13 17:06:23 +00:00
dependabot[bot]
e161103b46
Bump mypy from 1.5.1 to 1.8.0 (#16901) 2024-03-13 17:05:57 +00:00
dependabot[bot]
f4e12ceb1f
Bump dawidd6/action-download-artifact from 3.1.1 to 3.1.2 (#16960) 2024-03-13 16:50:47 +00:00
dependabot[bot]
10e56b162f
Bump cryptography from 41.0.7 to 42.0.5 (#16958) 2024-03-13 16:50:11 +00:00
dependabot[bot]
74fb3e1996
Bump serde_json from 1.0.113 to 1.0.114 (#16961) 2024-03-13 16:49:54 +00:00
dependabot[bot]
a91fb6cc06
Bump serde from 1.0.196 to 1.0.197 (#16963) 2024-03-13 16:49:19 +00:00
dependabot[bot]
6cb8839f67
Bump log from 0.4.20 to 0.4.21 (#16977) 2024-03-13 16:49:06 +00:00
dependabot[bot]
1e68b56a62
Bump black from 23.10.1 to 24.2.0 (#16936) 2024-03-13 16:46:44 +00:00
V02460
2bdf6280f6
Raise poetry-core version cap to 1.9.0 (#16986)
A new poetry-core version was released. See if CI is happy. Required for
the latest Fedora Synapse package.
2024-03-13 16:40:08 +00:00
Erik Johnston
5c0b87ff95 Update changelog 2024-03-12 15:12:19 +00:00
Erik Johnston
0d44f64c4e Merge remote-tracking branch 'origin/develop' into release-v1.103 2024-03-12 15:11:03 +00:00
Gerrit Gogel
1f88790764
Prevent locking up while processing batched_auth_events (#16968)
This PR aims to fix #16895, caused by a regression in #7 and not fixed
by #16903. The PR #16903 only fixes a starvation issue, where the CPU
isn't released. There is a second issue, where the execution is blocked.
This theory is supported by the flame graphs provided in #16895 and the
fact that I see the CPU usage reducing and far below the limit.

Since the changes in #7, the method `check_state_independent_auth_rules`
is called with the additional parameter `batched_auth_events`:


6fa13b4f92/synapse/handlers/federation_event.py (L1741-L1743)


It makes the execution enter this if clause, introduced with #15195


6fa13b4f92/synapse/event_auth.py (L178-L189)

There are two issues in the above code snippet.

First, there is the blocking issue. I'm not entirely sure if this is a
deadlock, starvation, or something different. In the beginning, I
thought the copy operation was responsible. It wasn't. Then I
investigated the nested `store.get_events` inside the function `update`.
This was also not causing the blocking issue. Only when I replaced the
set difference operation (`-` ) with a list comprehension, the blocking
was resolved. Creating and comparing sets with a very large amount of
events seems to be problematic.

This is how the flamegraph looks now while persisting outliers. As you
can see, the execution no longer locks up in the above function.

![output_2024-02-28_13-59-40](https://github.com/element-hq/synapse/assets/13143850/6db9c9ac-484f-47d0-bdde-70abfbd773ec)

Second, the copying here doesn't serve any purpose, because only a
shallow copy is created. This means the same objects from the original
dict are referenced. This fails the intention of protecting these
objects from mutation. The review of the original PR
https://github.com/matrix-org/synapse/pull/15195 had an extensive
discussion about this matter.

Various approaches to copying the auth_events were attempted:
1) Implementing a deepcopy caused issues due to
builtins.EventInternalMetadata not being pickleable.
2) Creating a dict with new objects akin to a deepcopy.
3) Creating a dict with new objects containing only necessary
attributes.

Concluding, there is no easy way to create an actual copy of the
objects. Opting for a deepcopy can significantly strain memory and CPU
resources, making it an inefficient choice. I don't see why the copy is
necessary in the first place. Therefore I'm proposing to remove it
altogether.

After these changes, I was able to successfully join these rooms,
without the main worker locking up:
- #synapse:matrix.org
- #element-android:matrix.org
- #element-web:matrix.org
- #ecips:matrix.org
- #ipfs-chatter:ipfs.io
- #python:matrix.org
- #matrix:matrix.org
2024-03-12 15:07:36 +00:00
Erik Johnston
9d7880c0c6 1.103.0rc1 2024-03-12 15:03:45 +00:00
Alexander Fechler
48f59d3806
deactivated flag refactored to filter deactivated users. (#16874)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-03-11 16:08:04 +00:00
Patrick Cloke
696cc9e802
Stabilize support for Retry-After header (MSC4014) (#16947) 2024-03-08 09:33:46 +00:00
Quentin Gliech
4af33015af
Fix joining remote rooms when a on_new_event callback is registered (#16973)
Since Synapse 1.76.0, any module which registers a `on_new_event`
callback would brick the ability to join remote rooms.
This is because this callback tried to get the full state of the room,
which would end up in a deadlock.

Related:
https://github.com/matrix-org/synapse-auto-accept-invite/issues/18

The following module would brick the ability to join remote rooms:

```python
from typing import Any, Dict, Literal, Union
import logging

from synapse.module_api import ModuleApi, EventBase

logger = logging.getLogger(__name__)

class MyModule:
    def __init__(self, config: None, api: ModuleApi):
        self._api = api
        self._config = config

        self._api.register_third_party_rules_callbacks(
            on_new_event=self.on_new_event,
        )

    async def on_new_event(self, event: EventBase, _state_map: Any) -> None:
        logger.info(f"Received new event: {event}")

    @staticmethod
    def parse_config(_config: Dict[str, Any]) -> None:
        return None
```

This is technically a breaking change, as we are now passing partial
state on the `on_new_event` callback.
However, this callback was broken for federated rooms since 1.76.0, and
local rooms have full state anyway, so it's unlikely that it would
change anything.
2024-03-06 16:00:20 +01:00
Andrew Morgan
2d1bb0b06b Merge remote-tracking branch 'origin/release-v1.102' into develop 2024-03-05 16:03:24 +00:00
Andrew Morgan
ab80b3412e
Revert "Ignore notification counts from rooms you've left" (#16981) 2024-03-05 16:02:54 +00:00
Andrew Morgan
1dee1b72ec Switch #16979 changelog type from internal change to bugfix 2024-03-05 15:13:32 +00:00
Andrew Morgan
571ca0c004 1.102.0 2024-03-05 14:47:35 +00:00
Andrew Morgan
8a05304222
Revert "Improve DB performance of calculating badge counts for push. (#16756)" (#16979) 2024-03-05 12:27:27 +00:00
Andrew Morgan
274f289a52
Ignore notification counts from rooms you've left (#16954)
Co-authored-by: reivilibre <oliverw@matrix.org>
2024-02-23 14:12:10 +00:00
Twilight Sparkle
8de3283ebe
Add docs on upgrading from a very old version (#16951)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-22 17:36:41 +00:00
dependabot[bot]
4ad70f115b
Bump anyhow from 1.0.79 to 1.0.80 (#16935)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:40:34 +00:00
dependabot[bot]
3778cea776
Bump pyopenssl from 23.3.0 to 24.0.0 (#16937)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:38:38 +00:00
dependabot[bot]
5ce9498047
Bump JasonEtco/create-an-issue from 2.9.1 to 2.9.2 (#16934)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:37:32 +00:00
dependabot[bot]
f2c5f1564e
Bump types-netaddr from 0.10.0.20240106 to 1.2.0.20240219 (#16938)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:36:35 +00:00
dependabot[bot]
91694907da
Bump furo from 2023.9.10 to 2024.1.29 (#16939)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:35:05 +00:00
dependabot[bot]
e0b19a4777
Bump dawidd6/action-download-artifact from 3.0.0 to 3.1.1 (#16933)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:34:03 +00:00
kegsay
0c55c76da8
Better complement docs (#16946) 2024-02-20 17:14:50 +00:00
Andrew Morgan
3eb0a3b468 Merge branch 'release-v1.102' into develop 2024-02-20 16:57:37 +00:00
Georg
7c1c011942
Add HAProxy example for single port operation (#16768) 2024-02-20 16:15:58 +00:00
Andrew Morgan
7856ec96ef 1.102.0rc1 2024-02-20 15:51:17 +00:00
Erik Johnston
cdbbf3653d
Don't lock up when joining large rooms (#16903)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-02-20 14:29:18 +00:00
kegsay
c51a2240d1
bugfix: always prefer unthreaded receipt when >1 exist (MSC4102) (#16927)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-20 14:12:06 +00:00
Erik Johnston
e5dfb6ecbf
Fix incorrect docker hub link in release script (#16910) 2024-02-20 12:20:31 +00:00