Matrix/synapse
Matrix/synapse
1
0
Fork 0
mirror of https://github.com/element-hq/synapse synced 2024-06-30 16:03:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
23871 commits 881 branches 700 tags 466 MiB
Commit graph

23871 commits

This branch
This branch
All branches
Author SHA1 Message Date
Eric Eastwood bd49c3415b Add some tests 2024-06-12 17:53:24 -05:00
Eric Eastwood af60f7b508 First pass on sort_rooms and refactor to include room membership alongside the sync rooms 2024-06-12 15:46:37 -05:00
Eric Eastwood afb6627b6f Add rust changes for event.internal_metadata.instance_name 2024-06-12 15:11:29 -05:00
Eric Eastwood c94550d542 Add event.internal_metadata.instance_name and event position to get_last_event_in_room_before_stream_ordering(...) 2024-06-12 14:53:36 -05:00
Eric Eastwood 03547b09e3 Merge branch 'madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering' into madlittlemods/msc3575-sliding-sync-sort 
Conflicts:
	synapse/storage/databases/main/stream.py
	tests/storage/test_stream.py
 2024-06-11 22:40:08 -05:00
Eric Eastwood 42f24de8fd Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-sort 2024-06-11 22:39:04 -05:00
Eric Eastwood 4d585b6886 Merge branch 'develop' into madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering 2024-06-11 22:37:24 -05:00
Eric Eastwood 54bdc0ca92 Fix invalid syntax 2024-06-11 22:36:15 -05:00
Eric Eastwood 3f317a9929 We're actually using sub-query syntax so we can ORDER each query 2024-06-11 22:05:27 -05:00
Eric Eastwood a8056ae67d Add changelog 2024-06-11 22:04:48 -05:00
Eric Eastwood d7f40aedf7 Try to better explain why 
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
 2024-06-11 20:50:31 -05:00
Eric Eastwood 431b31e0f2 Add actual guranteed order for UNION 
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.

See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
 2024-06-11 20:49:20 -05:00
Eric Eastwood 87ad458d77 Fix get_last_event_in_room_before_stream_ordering(...) not finding the last event 
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.

`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
 2024-06-11 20:49:14 -05:00
Eric Eastwood 901ce62d07 Try to better explain why 
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
 2024-06-11 20:48:02 -05:00
Eric Eastwood b1af992113 Some clean-up 2024-06-11 20:30:32 -05:00
Eric Eastwood 2e1d142892 Add actual guranteed order for UNION 
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.

See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
 2024-06-11 20:29:40 -05:00
Eric Eastwood 2a82ac0cbe Fix get_last_event_in_room_before_stream_ordering(...) not finding the last event 
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.

`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
 2024-06-11 20:14:01 -05:00
Eric Eastwood c8a240f59d Prefer ? < a AND a <= ? 
See https://github.com/element-hq/synapse/pull/17293#discussion_r1634863810

> FWIW I mildly prefer the styling `? < a AND a <= ?` as then my brain can more easily convert that to `a in (?...?]`
 2024-06-11 16:00:02 -05:00
Quentin Gliech 0248ed70a9
Merge branch 'release-v1.109' into develop 2024-06-11 16:25:26 +02:00
Quentin Gliech e6816babf6
CHANGES.md: s/OTKs/one-time-keys/ 2024-06-11 15:39:30 +02:00
Quentin Gliech a8069e9739
1.109.0rc2 2024-06-11 15:22:21 +02:00
dependabot[bot] 863578bfcf
Bump regex from 1.10.4 to 1.10.5 (#17290) 2024-06-11 10:50:13 +01:00
dependabot[bot] 9e59d18022
Bump dawidd6/action-download-artifact from 3.1.4 to 5 (#17289) 2024-06-11 10:50:03 +01:00
dependabot[bot] 491365f199
Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520 (#17285) 2024-06-11 10:47:03 +01:00
Eric Eastwood 75b701f0ee Add changelog 2024-06-10 17:21:09 -05:00
Eric Eastwood ce45cc1d44 Sliding sync sort stub 2024-06-10 17:18:48 -05:00
Eric Eastwood dad1559721
Reorganize Pydantic models and types used in handlers (#17279) 
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.

 - Moved Pydantic parsing models to `synapse/types/rest`
 - Moved handler types to `synapse/types/handlers`
 2024-06-10 15:03:50 -05:00
Erik Johnston 8c4937b216
Fix bug where device lists would break sync (#17292) 
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.

The fix is to make sure we add the unconverted table to the list of
tables we check at start up.

Broke in https://github.com/element-hq/synapse/pull/17229
 2024-06-10 15:56:57 +01:00
Erik Johnston b84e31375b Update changelog 2024-06-10 15:55:42 +01:00
Erik Johnston 06953bc193 Always return OTK counts (#17275) 
Broke in https://github.com/element-hq/synapse/pull/17215
 2024-06-10 15:53:45 +01:00
Eric Eastwood 265ee88f34
Wrong retention policy being used when filtering events (lint ControlVarUsedAfterBlockViolation WPS441) (#17272) 
Fix loop var being used outside block.

Before this change, we were always using the last room_id's retention policy for all events being filtered.

I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning! 

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
 2024-06-10 09:48:38 -05:00
Shay ab94bce02c
Support MSC3916 by adding a federation /download endpoint (#17172) 2024-06-07 13:54:28 +01:00
reivilibre 17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266) 
Fixes: #17013

Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.

The existing opentracing logging is not enough because it is not enabled
for all users.
 2024-06-07 12:01:21 +01:00
Eric Eastwood 4a7c58642c
Add Sliding Sync /sync endpoint (initial implementation) (#17187) 
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
 2024-06-06 14:44:32 -05:00
Erik Johnston ce9385819b
Handle OTK uploads off master (#17271) 
And fallback keys uploads. Only device keys need handling on master
 2024-06-06 17:47:02 +01:00
Erik Johnston a963f579de
Don't try and resync devices for down hosts (#17273) 
It's just a waste of time if we won't even query the remote host as its
marked as down.
 2024-06-06 17:46:52 +01:00
Erik Johnston 3f06bbc0ac
Always return OTK counts (#17275) 
Broke in https://github.com/element-hq/synapse/pull/17215
 2024-06-06 17:10:58 +01:00
Shay fcbc79bb87
Ratelimiting of remote media downloads (#17256) 2024-06-05 13:43:36 +01:00
Erik Johnston aabf577166
Handle hyphens in user dir search porperly (#17254) 
c.f. #16675
 2024-06-05 10:40:34 +01:00
Eric Eastwood 7d8f0ef351
Use fully-qualified PersistedEventPosition when returning RoomsForUser (#17265) 
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.

Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
 2024-06-04 12:58:03 -05:00
Andrew Morgan eab0b548e4 Merge branch 'release-v1.109' into develop 2024-06-04 12:37:07 +01:00
dependabot[bot] 81cef38d4b
Bump sentry-sdk from 2.1.1 to 2.3.1 (#17263) 2024-06-04 11:58:27 +01:00
Andrew Morgan e2f8476044 Fix typo in CHANGES.md 2024-06-04 09:47:28 +01:00
Andrew Morgan 18c1196893 1.109.0rc1 2024-06-04 09:46:09 +01:00
dependabot[bot] 8a3270075b
Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425 (#17260) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-03 14:26:41 +01:00
dependabot[bot] f458dff16d
Bump mypy-zope from 1.0.3 to 1.0.4 (#17262) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-03 14:23:29 +01:00
dependabot[bot] 6b709c512d
Bump lxml from 5.2.1 to 5.2.2 (#17261) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-03 14:21:20 +01:00
Erik Johnston 5c2a837e3c
Fix bug where typing replication breaks (#17252) 
This can happen on restarts of the service, due to old rooms being
pruned.
 2024-05-31 16:07:05 +01:00
Erik Johnston 64f5a4a353
Fix logging errors when receiving invalid User ID for key querys (#17250) 2024-05-31 11:27:56 +01:00
Erik Johnston 7dd14fadb1
Fix sentry default tags (#17251) 
This was broken by the sentry 2.0 upgrade

Broke in v1.108.0
 2024-05-31 11:27:47 +01:00