Commit graph

8097 commits

Author SHA1 Message Date
Erik Johnston
5360baeb64
Pull out fewer receipts from DB when doing push (#17049)
Before we were pulling out *all* read receipts for a user for every
event we pushed. Instead let's only pull out the relevant receipts.

This also pulled out the event rows for each receipt, causing load on
the events table.
2024-04-05 12:46:34 +01:00
Richard van der Hoff
0e68e9b7f4
Fix bug in calculating state for non-gappy syncs (#16942)
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.

Fixes https://github.com/element-hq/synapse/issues/16941.

~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
2024-04-04 16:15:35 +00:00
Richard van der Hoff
230b709d9d
/sync: fix bug in calculating state response (#16930)
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.

Fixes: https://github.com/element-hq/synapse/issues/16928
2024-04-04 12:14:24 +00:00
Richard van der Hoff
05957ac70f
Fix bug in /sync response for archived rooms (#16932)
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.

The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.

The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
2024-04-04 12:47:59 +01:00
Erik Johnston
31122b71bc
Add missing index to access_tokens table (#17045)
This was causing sequential scans when using refresh tokens.
2024-04-04 11:05:40 +01:00
Erik Johnston
51776745b9 Merge branch 'master' into develop 2024-04-02 18:44:47 +01:00
Erik Johnston
ca27b51665 1.104.0 2024-04-02 17:17:02 +01:00
Erik Johnston
ec174d0470
Refactor chain fetching (#17044)
Since these queries are duplicated in two places.
2024-04-02 15:33:56 +01:00
Erik Johnston
fd48fc4585
Fixups to new push stream (#17038)
Follow on from #17037
2024-03-28 16:29:23 +00:00
Erik Johnston
ea6bfae0fc
Add support for moving /push_rules off of main process (#17037) 2024-03-28 15:44:07 +00:00
Erik Johnston
34878b6bc9 Merge remote-tracking branch 'origin/develop' into release-v1.104 2024-03-26 13:42:09 +00:00
Erik Johnston
c900d18647
Fix OIDC login regression (#17031)
Requests may require a User-Agent header, and the change in #16972
accidentally removed it, resulting in requests getting rejected causing
login to fail.
2024-03-26 13:26:46 +00:00
Erik Johnston
03f0d746c3 1.104.0rc1 2024-03-26 11:49:11 +00:00
Richard van der Hoff
b5322b4daf
Ensure that pending to-device events are sent over federation at startup (#16925)
Fixes https://github.com/element-hq/synapse/issues/16680, as well as a
related bug, where servers which we had *never* successfully sent an
event to would not be retried.

In order to fix the case of pending to-device messages, we hook into the
existing `wake_destinations_needing_catchup` process, by extending it to
look for destinations that have pending to-device messages. The
federation transmission loop then attempts to send the pending to-device
messages as normal.
2024-03-22 13:24:11 +00:00
Mathieu Velten
b7af076ab5
Add OIDC config to add extra parameters to the authorize URL (#16971) 2024-03-22 10:35:11 +00:00
SpiritCroc
9ad49e7ecf
Do not refuse to set read_marker if previous event_id is in wrong room (#16990) 2024-03-21 18:43:07 +00:00
Hanadi
f7a3ebe44d
Fix reject knocks on deactivating account (#17010) 2024-03-21 18:05:54 +00:00
Sam Wedgwood
bef765b262
generate configuration with correct user in start.py for docker (#16978) 2024-03-21 17:55:44 +00:00
Mathieu Velten
3ab9e6d524
OIDC: try to JWT decode userinfo response if JSON parsing failed (#16972) 2024-03-21 17:49:44 +00:00
Richard van der Hoff
db95b75515
Patch the db conn pool sooner in tests (#17017)
When running unit tests, we patch the database connection pool so that
it runs queries "synchronously". This is ok, except that if any queries
are launched before we do the patching, those queries get left in limbo
and never complete.

To fix this, let's change the way we do the switcheroo, by patching out
the method which creates the connection pool in the first place.
2024-03-21 17:48:16 +00:00
Tadeusz Sośnierz
5a59c68b3d
Remove the hardcoded poetry version from contributing guide (#17002) 2024-03-21 17:12:02 +00:00
Andrew Morgan
21daa56ee1
Prevent start_for_complement.sh from setting START_POSTGRES to false when it's already set (#16985)
I have a use case where I'd like the Synapse image to start up a
postgres instance that I can use, but don't want to force Synapse to use
postgres as well.

This commit prevents postgres from being started when it has already
been explicitly enabled elsewhere.
2024-03-21 13:50:51 +00:00
Shay
cf5adc80e1
Update power level default for public rooms (#16907) 2024-03-19 17:55:31 +00:00
Shay
8fb5b0f335
Improve event validation (#16908)
As the title states.
2024-03-19 17:52:53 +00:00
Mathieu Velten
74ab329eaa
Pass module API to OIDC mapping provider (#16974)
As done for SAML mapping provider, let's pass the module API to the OIDC
one so the mapper can do more logic in its code.
2024-03-19 17:20:10 +00:00
V02460
05489d89c6
Specify IP subnet literals in canonical form (#16953)
This is needed, because the netaddr package removed support for the
implicit prefix form in version 1.0.0:
https://github.com/netaddr/netaddr/pull/360
2024-03-19 17:19:12 +00:00
Richard van der Hoff
9635822cc1
Clarify docs for some room state functions (#16950)
State *before* an event is different to state *after* that event, and
people tend to assume the wrong one.
2024-03-19 17:16:37 +00:00
Richard van der Hoff
52f456a822
/sync: Fix edge-case in calculating the "device_lists" response (#16949)
Fixes https://github.com/element-hq/synapse/issues/16948. If the `join`
and the `leave` are in the same sync response, we need to count them as
a "left" user.
2024-03-14 17:34:19 +00:00
Richard van der Hoff
6d5bafb2c8
Split up SyncHandler.compute_state_delta (#16929)
This is a huge method, which melts my brain.

This is a non-functional change which lays some groundwork for future
work in this area.
2024-03-14 17:18:48 +00:00
Will Hunt
1198f649ea
Sort versions in the documentation version picker appropriately. (#16966)
Fixes #16964 

This adds a proper sorter for versions which takes into account semantic
versions, rather than just relying on localeCompare.
2024-03-14 15:18:51 +00:00
Richard van der Hoff
acc2f00eca
upgrade.md: fix grammatical errors (#16965)
comma splice
"rollback" is a noun
2024-03-14 13:54:01 +00:00
Andrew Morgan
1c1b0bfa77
Add query to update local cache of a remote user's device list to docs (#16892) 2024-03-14 13:53:25 +00:00
Mathieu Velten
cb562d73aa
Improve lock performance when a lot of locks are waiting (#16840)
When a lot of locks are waiting for a single lock, notifying all locks
independently with `call_later` on each release is really costly and
incurs some kind of async contention, where the CPU is spinning a lot
for not much.

The included test is taking around 30s before the change, and 0.5s
after.

It was found following failing tests with
https://github.com/element-hq/synapse/pull/16827.
2024-03-14 13:49:54 +00:00
Richard van der Hoff
92f2069627
Multi-worker-docker-container: disable log buffering (#16919)
Background: we have a `matrixdotorg/synapse-workers` docker image, which
is intended for running multiple workers within the same container. That
image includes a `prefix-log` script which, for each line printed to
stdout or stderr by one of the processes, prepends the name of the
process.

This commit disables buffering in that script, so that lines are logged
quickly after they are printed. This makes it much easier to understand
the output, since they then come out in a natural order.
2024-03-13 17:21:37 +00:00
V02460
2bdf6280f6
Raise poetry-core version cap to 1.9.0 (#16986)
A new poetry-core version was released. See if CI is happy. Required for
the latest Fedora Synapse package.
2024-03-13 16:40:08 +00:00
Erik Johnston
5c0b87ff95 Update changelog 2024-03-12 15:12:19 +00:00
Erik Johnston
0d44f64c4e Merge remote-tracking branch 'origin/develop' into release-v1.103 2024-03-12 15:11:03 +00:00
Gerrit Gogel
1f88790764
Prevent locking up while processing batched_auth_events (#16968)
This PR aims to fix #16895, caused by a regression in #7 and not fixed
by #16903. The PR #16903 only fixes a starvation issue, where the CPU
isn't released. There is a second issue, where the execution is blocked.
This theory is supported by the flame graphs provided in #16895 and the
fact that I see the CPU usage reducing and far below the limit.

Since the changes in #7, the method `check_state_independent_auth_rules`
is called with the additional parameter `batched_auth_events`:


6fa13b4f92/synapse/handlers/federation_event.py (L1741-L1743)


It makes the execution enter this if clause, introduced with #15195


6fa13b4f92/synapse/event_auth.py (L178-L189)

There are two issues in the above code snippet.

First, there is the blocking issue. I'm not entirely sure if this is a
deadlock, starvation, or something different. In the beginning, I
thought the copy operation was responsible. It wasn't. Then I
investigated the nested `store.get_events` inside the function `update`.
This was also not causing the blocking issue. Only when I replaced the
set difference operation (`-` ) with a list comprehension, the blocking
was resolved. Creating and comparing sets with a very large amount of
events seems to be problematic.

This is how the flamegraph looks now while persisting outliers. As you
can see, the execution no longer locks up in the above function.

![output_2024-02-28_13-59-40](https://github.com/element-hq/synapse/assets/13143850/6db9c9ac-484f-47d0-bdde-70abfbd773ec)

Second, the copying here doesn't serve any purpose, because only a
shallow copy is created. This means the same objects from the original
dict are referenced. This fails the intention of protecting these
objects from mutation. The review of the original PR
https://github.com/matrix-org/synapse/pull/15195 had an extensive
discussion about this matter.

Various approaches to copying the auth_events were attempted:
1) Implementing a deepcopy caused issues due to
builtins.EventInternalMetadata not being pickleable.
2) Creating a dict with new objects akin to a deepcopy.
3) Creating a dict with new objects containing only necessary
attributes.

Concluding, there is no easy way to create an actual copy of the
objects. Opting for a deepcopy can significantly strain memory and CPU
resources, making it an inefficient choice. I don't see why the copy is
necessary in the first place. Therefore I'm proposing to remove it
altogether.

After these changes, I was able to successfully join these rooms,
without the main worker locking up:
- #synapse:matrix.org
- #element-android:matrix.org
- #element-web:matrix.org
- #ecips:matrix.org
- #ipfs-chatter:ipfs.io
- #python:matrix.org
- #matrix:matrix.org
2024-03-12 15:07:36 +00:00
Erik Johnston
9d7880c0c6 1.103.0rc1 2024-03-12 15:03:45 +00:00
Alexander Fechler
48f59d3806
deactivated flag refactored to filter deactivated users. (#16874)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-03-11 16:08:04 +00:00
Patrick Cloke
696cc9e802
Stabilize support for Retry-After header (MSC4014) (#16947) 2024-03-08 09:33:46 +00:00
Quentin Gliech
4af33015af
Fix joining remote rooms when a on_new_event callback is registered (#16973)
Since Synapse 1.76.0, any module which registers a `on_new_event`
callback would brick the ability to join remote rooms.
This is because this callback tried to get the full state of the room,
which would end up in a deadlock.

Related:
https://github.com/matrix-org/synapse-auto-accept-invite/issues/18

The following module would brick the ability to join remote rooms:

```python
from typing import Any, Dict, Literal, Union
import logging

from synapse.module_api import ModuleApi, EventBase

logger = logging.getLogger(__name__)

class MyModule:
    def __init__(self, config: None, api: ModuleApi):
        self._api = api
        self._config = config

        self._api.register_third_party_rules_callbacks(
            on_new_event=self.on_new_event,
        )

    async def on_new_event(self, event: EventBase, _state_map: Any) -> None:
        logger.info(f"Received new event: {event}")

    @staticmethod
    def parse_config(_config: Dict[str, Any]) -> None:
        return None
```

This is technically a breaking change, as we are now passing partial
state on the `on_new_event` callback.
However, this callback was broken for federated rooms since 1.76.0, and
local rooms have full state anyway, so it's unlikely that it would
change anything.
2024-03-06 16:00:20 +01:00
Andrew Morgan
ab80b3412e
Revert "Ignore notification counts from rooms you've left" (#16981) 2024-03-05 16:02:54 +00:00
Andrew Morgan
274f289a52
Ignore notification counts from rooms you've left (#16954)
Co-authored-by: reivilibre <oliverw@matrix.org>
2024-02-23 14:12:10 +00:00
Twilight Sparkle
8de3283ebe
Add docs on upgrading from a very old version (#16951)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-22 17:36:41 +00:00
kegsay
0c55c76da8
Better complement docs (#16946) 2024-02-20 17:14:50 +00:00
Andrew Morgan
3eb0a3b468 Merge branch 'release-v1.102' into develop 2024-02-20 16:57:37 +00:00
Georg
7c1c011942
Add HAProxy example for single port operation (#16768) 2024-02-20 16:15:58 +00:00
Andrew Morgan
7856ec96ef 1.102.0rc1 2024-02-20 15:51:17 +00:00
Erik Johnston
cdbbf3653d
Don't lock up when joining large rooms (#16903)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-02-20 14:29:18 +00:00