Commit graph

23869 commits

Author SHA1 Message Date
Eric Eastwood
afb6627b6f Add rust changes for event.internal_metadata.instance_name 2024-06-12 15:11:29 -05:00
Eric Eastwood
c94550d542 Add event.internal_metadata.instance_name and event position to get_last_event_in_room_before_stream_ordering(...) 2024-06-12 14:53:36 -05:00
Eric Eastwood
03547b09e3 Merge branch 'madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering' into madlittlemods/msc3575-sliding-sync-sort
Conflicts:
	synapse/storage/databases/main/stream.py
	tests/storage/test_stream.py
2024-06-11 22:40:08 -05:00
Eric Eastwood
42f24de8fd Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-sort 2024-06-11 22:39:04 -05:00
Eric Eastwood
4d585b6886 Merge branch 'develop' into madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering 2024-06-11 22:37:24 -05:00
Eric Eastwood
54bdc0ca92 Fix invalid syntax 2024-06-11 22:36:15 -05:00
Eric Eastwood
3f317a9929 We're actually using sub-query syntax so we can ORDER each query 2024-06-11 22:05:27 -05:00
Eric Eastwood
a8056ae67d Add changelog 2024-06-11 22:04:48 -05:00
Eric Eastwood
d7f40aedf7 Try to better explain why
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
2024-06-11 20:50:31 -05:00
Eric Eastwood
431b31e0f2 Add actual guranteed order for UNION
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.

See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:49:20 -05:00
Eric Eastwood
87ad458d77 Fix get_last_event_in_room_before_stream_ordering(...) not finding the last event
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.

`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:49:14 -05:00
Eric Eastwood
901ce62d07 Try to better explain why
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
2024-06-11 20:48:02 -05:00
Eric Eastwood
b1af992113 Some clean-up 2024-06-11 20:30:32 -05:00
Eric Eastwood
2e1d142892 Add actual guranteed order for UNION
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.

See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:29:40 -05:00
Eric Eastwood
2a82ac0cbe Fix get_last_event_in_room_before_stream_ordering(...) not finding the last event
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.

`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:14:01 -05:00
Eric Eastwood
c8a240f59d Prefer ? < a AND a <= ?
See https://github.com/element-hq/synapse/pull/17293#discussion_r1634863810

> FWIW I mildly prefer the styling `? < a AND a <= ?` as then my brain can more easily convert that to `a in (?...?]`
2024-06-11 16:00:02 -05:00
Quentin Gliech
0248ed70a9
Merge branch 'release-v1.109' into develop 2024-06-11 16:25:26 +02:00
Quentin Gliech
e6816babf6
CHANGES.md: s/OTKs/one-time-keys/ 2024-06-11 15:39:30 +02:00
Quentin Gliech
a8069e9739
1.109.0rc2 2024-06-11 15:22:21 +02:00
dependabot[bot]
863578bfcf
Bump regex from 1.10.4 to 1.10.5 (#17290) 2024-06-11 10:50:13 +01:00
dependabot[bot]
9e59d18022
Bump dawidd6/action-download-artifact from 3.1.4 to 5 (#17289) 2024-06-11 10:50:03 +01:00
dependabot[bot]
491365f199
Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520 (#17285) 2024-06-11 10:47:03 +01:00
Eric Eastwood
75b701f0ee Add changelog 2024-06-10 17:21:09 -05:00
Eric Eastwood
ce45cc1d44 Sliding sync sort stub 2024-06-10 17:18:48 -05:00
Eric Eastwood
dad1559721
Reorganize Pydantic models and types used in handlers (#17279)
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.

 - Moved Pydantic parsing models to `synapse/types/rest`
 - Moved handler types to `synapse/types/handlers`
2024-06-10 15:03:50 -05:00
Erik Johnston
8c4937b216
Fix bug where device lists would break sync (#17292)
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.

The fix is to make sure we add the unconverted table to the list of
tables we check at start up.

Broke in https://github.com/element-hq/synapse/pull/17229
2024-06-10 15:56:57 +01:00
Erik Johnston
b84e31375b Update changelog 2024-06-10 15:55:42 +01:00
Erik Johnston
06953bc193 Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-10 15:53:45 +01:00
Eric Eastwood
265ee88f34
Wrong retention policy being used when filtering events (lint ControlVarUsedAfterBlockViolation WPS441) (#17272)
Fix loop var being used outside block.

Before this change, we were always using the last room_id's retention policy for all events being filtered.

I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning! 

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-06-10 09:48:38 -05:00
Shay
ab94bce02c
Support MSC3916 by adding a federation /download endpoint (#17172) 2024-06-07 13:54:28 +01:00
reivilibre
17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266)
Fixes: #17013

Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.

The existing opentracing logging is not enough because it is not enabled
for all users.
2024-06-07 12:01:21 +01:00
Eric Eastwood
4a7c58642c
Add Sliding Sync /sync endpoint (initial implementation) (#17187)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston
ce9385819b
Handle OTK uploads off master (#17271)
And fallback keys uploads. Only device keys need handling on master
2024-06-06 17:47:02 +01:00
Erik Johnston
a963f579de
Don't try and resync devices for down hosts (#17273)
It's just a waste of time if we won't even query the remote host as its
marked as down.
2024-06-06 17:46:52 +01:00
Erik Johnston
3f06bbc0ac
Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-06 17:10:58 +01:00
Shay
fcbc79bb87
Ratelimiting of remote media downloads (#17256) 2024-06-05 13:43:36 +01:00
Erik Johnston
aabf577166
Handle hyphens in user dir search porperly (#17254)
c.f. #16675
2024-06-05 10:40:34 +01:00
Eric Eastwood
7d8f0ef351
Use fully-qualified PersistedEventPosition when returning RoomsForUser (#17265)
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.

Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
2024-06-04 12:58:03 -05:00
Andrew Morgan
eab0b548e4 Merge branch 'release-v1.109' into develop 2024-06-04 12:37:07 +01:00
dependabot[bot]
81cef38d4b
Bump sentry-sdk from 2.1.1 to 2.3.1 (#17263) 2024-06-04 11:58:27 +01:00
Andrew Morgan
e2f8476044 Fix typo in CHANGES.md 2024-06-04 09:47:28 +01:00
Andrew Morgan
18c1196893 1.109.0rc1 2024-06-04 09:46:09 +01:00
dependabot[bot]
8a3270075b
Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425 (#17260)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:26:41 +01:00
dependabot[bot]
f458dff16d
Bump mypy-zope from 1.0.3 to 1.0.4 (#17262)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:23:29 +01:00
dependabot[bot]
6b709c512d
Bump lxml from 5.2.1 to 5.2.2 (#17261)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:21:20 +01:00
Erik Johnston
5c2a837e3c
Fix bug where typing replication breaks (#17252)
This can happen on restarts of the service, due to old rooms being
pruned.
2024-05-31 16:07:05 +01:00
Erik Johnston
64f5a4a353
Fix logging errors when receiving invalid User ID for key querys (#17250) 2024-05-31 11:27:56 +01:00
Erik Johnston
7dd14fadb1
Fix sentry default tags (#17251)
This was broken by the sentry 2.0 upgrade

Broke in v1.108.0
2024-05-31 11:27:47 +01:00
Erik Johnston
5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Erik Johnston
4e3868dc46
Fix deduplicating of membership events to not create unused state groups. (#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes #3791
2024-05-30 11:33:48 +00:00