Eric Eastwood
d5929f15af
Adjust wording
2024-06-12 18:25:20 -05:00
Eric Eastwood
5243a30e00
Fix ban test case
2024-06-12 18:20:47 -05:00
Eric Eastwood
5060588012
Fix newly_left not being added back if we returned early (when membership_snapshot_token.is_before_or_eq(to_token.room_key)
)
...
Wasn't caught in the tests because the test was wrong
2024-06-12 18:18:20 -05:00
Eric Eastwood
bd49c3415b
Add some tests
2024-06-12 17:53:24 -05:00
Eric Eastwood
af60f7b508
First pass on sort_rooms
and refactor to include room membership alongside the sync rooms
2024-06-12 15:46:37 -05:00
Eric Eastwood
afb6627b6f
Add rust changes for event.internal_metadata.instance_name
2024-06-12 15:11:29 -05:00
Eric Eastwood
c94550d542
Add event.internal_metadata.instance_name
and event position to get_last_event_in_room_before_stream_ordering(...)
2024-06-12 14:53:36 -05:00
Eric Eastwood
03547b09e3
Merge branch 'madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering' into madlittlemods/msc3575-sliding-sync-sort
...
Conflicts:
synapse/storage/databases/main/stream.py
tests/storage/test_stream.py
2024-06-11 22:40:08 -05:00
Eric Eastwood
42f24de8fd
Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-sort
2024-06-11 22:39:04 -05:00
Eric Eastwood
4d585b6886
Merge branch 'develop' into madlittlemods/fix-and-tests-for-get_last_event_in_room_before_stream_ordering
2024-06-11 22:37:24 -05:00
Eric Eastwood
54bdc0ca92
Fix invalid syntax
2024-06-11 22:36:15 -05:00
Eric Eastwood
3f317a9929
We're actually using sub-query syntax so we can ORDER each query
2024-06-11 22:05:27 -05:00
Eric Eastwood
a8056ae67d
Add changelog
2024-06-11 22:04:48 -05:00
Eric Eastwood
d7f40aedf7
Try to better explain why
...
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
2024-06-11 20:50:31 -05:00
Eric Eastwood
431b31e0f2
Add actual guranteed order for UNION
...
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.
See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:49:20 -05:00
Eric Eastwood
87ad458d77
Fix get_last_event_in_room_before_stream_ordering(...)
not finding the last event
...
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.
`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:49:14 -05:00
Eric Eastwood
901ce62d07
Try to better explain why
...
See https://github.com/element-hq/synapse/pull/17293#discussion_r1633904606
2024-06-11 20:48:02 -05:00
Eric Eastwood
b1af992113
Some clean-up
2024-06-11 20:30:32 -05:00
Eric Eastwood
2e1d142892
Add actual guranteed order for UNION
...
We use `union all` because we don't need any of the deduplication logic
(`union` is really a union + distinct). `UNION ALL`` does preserve the
ordering of the operand queries but there is no actual gurantee that it
has this behavior in all scenarios so we need the extra `ORDER BY` at the
bottom.
See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:29:40 -05:00
Eric Eastwood
2a82ac0cbe
Fix get_last_event_in_room_before_stream_ordering(...)
not finding the last event
...
Previously, it would use the event with the lowest `stream_ordering` that was
fetched (we want the highest `stream_ordering. `union` does not work how you
might think at first and does not preserve the ordering of the operand queries.
`union all` also doesn't necessarily have that gurantee but it does behave
that way. See https://dba.stackexchange.com/questions/316818/are-results-from-union-all-clauses-always-appended-in-order/316835#316835
2024-06-11 20:14:01 -05:00
Eric Eastwood
c8a240f59d
Prefer ? < a AND a <= ?
...
See https://github.com/element-hq/synapse/pull/17293#discussion_r1634863810
> FWIW I mildly prefer the styling `? < a AND a <= ?` as then my brain can more easily convert that to `a in (?...?]`
2024-06-11 16:00:02 -05:00
Quentin Gliech
0248ed70a9
Merge branch 'release-v1.109' into develop
2024-06-11 16:25:26 +02:00
Quentin Gliech
e6816babf6
CHANGES.md: s/OTKs/one-time-keys/
2024-06-11 15:39:30 +02:00
Quentin Gliech
a8069e9739
1.109.0rc2
2024-06-11 15:22:21 +02:00
dependabot[bot]
863578bfcf
Bump regex from 1.10.4 to 1.10.5 ( #17290 )
2024-06-11 10:50:13 +01:00
dependabot[bot]
9e59d18022
Bump dawidd6/action-download-artifact from 3.1.4 to 5 ( #17289 )
2024-06-11 10:50:03 +01:00
dependabot[bot]
491365f199
Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520 ( #17285 )
2024-06-11 10:47:03 +01:00
Eric Eastwood
75b701f0ee
Add changelog
2024-06-10 17:21:09 -05:00
Eric Eastwood
ce45cc1d44
Sliding sync sort stub
2024-06-10 17:18:48 -05:00
Eric Eastwood
dad1559721
Reorganize Pydantic models and types used in handlers ( #17279 )
...
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.
- Moved Pydantic parsing models to `synapse/types/rest`
- Moved handler types to `synapse/types/handlers`
2024-06-10 15:03:50 -05:00
Erik Johnston
8c4937b216
Fix bug where device lists would break sync ( #17292 )
...
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.
The fix is to make sure we add the unconverted table to the list of
tables we check at start up.
Broke in https://github.com/element-hq/synapse/pull/17229
2024-06-10 15:56:57 +01:00
Erik Johnston
b84e31375b
Update changelog
2024-06-10 15:55:42 +01:00
Erik Johnston
06953bc193
Always return OTK counts ( #17275 )
...
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-10 15:53:45 +01:00
Eric Eastwood
265ee88f34
Wrong retention policy being used when filtering events (lint ControlVarUsedAfterBlockViolation
WPS441
) ( #17272 )
...
Fix loop var being used outside block.
Before this change, we were always using the last room_id's retention policy for all events being filtered.
I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769 ), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning!
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog ).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html ) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters ))
2024-06-10 09:48:38 -05:00
Shay
ab94bce02c
Support MSC3916 by adding a federation /download
endpoint ( #17172 )
2024-06-07 13:54:28 +01:00
reivilibre
17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. ( #17266 )
...
Fixes : #17013
Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.
The existing opentracing logging is not enough because it is not enabled
for all users.
2024-06-07 12:01:21 +01:00
Eric Eastwood
4a7c58642c
Add Sliding Sync /sync
endpoint (initial implementation) ( #17187 )
...
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575 ): Sliding Sync
This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).
Rooms appear in the Sliding sync response based on:
- `invite`, `join`, `knock`, `ban` membership events
- Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
- `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
- In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.
### Example request
`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync `
```json
{
"lists": {
"foo-list": {
"ranges": [ [0, 99] ],
"sort": [ "by_notification_level", "by_recency", "by_name" ],
"required_state": [
["m.room.join_rules", ""],
["m.room.history_visibility", ""],
["m.space.child", "*"]
],
"timeline_limit": 100
}
}
}
```
Response:
```json
{
"next_pos": "s58_224_0_13_10_1_1_16_0_1",
"lists": {
"foo-list": {
"count": 1,
"ops": [
{
"op": "SYNC",
"range": [0, 99],
"room_ids": [
"!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
]
}
]
}
},
"rooms": {},
"extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston
ce9385819b
Handle OTK uploads off master ( #17271 )
...
And fallback keys uploads. Only device keys need handling on master
2024-06-06 17:47:02 +01:00
Erik Johnston
a963f579de
Don't try and resync devices for down hosts ( #17273 )
...
It's just a waste of time if we won't even query the remote host as its
marked as down.
2024-06-06 17:46:52 +01:00
Erik Johnston
3f06bbc0ac
Always return OTK counts ( #17275 )
...
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-06 17:10:58 +01:00
Shay
fcbc79bb87
Ratelimiting of remote media downloads ( #17256 )
2024-06-05 13:43:36 +01:00
Erik Johnston
aabf577166
Handle hyphens in user dir search porperly ( #17254 )
...
c.f. #16675
2024-06-05 10:40:34 +01:00
Eric Eastwood
7d8f0ef351
Use fully-qualified PersistedEventPosition
when returning RoomsForUser
( #17265 )
...
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.
Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
2024-06-04 12:58:03 -05:00
Andrew Morgan
eab0b548e4
Merge branch 'release-v1.109' into develop
2024-06-04 12:37:07 +01:00
dependabot[bot]
81cef38d4b
Bump sentry-sdk from 2.1.1 to 2.3.1 ( #17263 )
2024-06-04 11:58:27 +01:00
Andrew Morgan
e2f8476044
Fix typo in CHANGES.md
2024-06-04 09:47:28 +01:00
Andrew Morgan
18c1196893
1.109.0rc1
2024-06-04 09:46:09 +01:00
dependabot[bot]
8a3270075b
Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425 ( #17260 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:26:41 +01:00
dependabot[bot]
f458dff16d
Bump mypy-zope from 1.0.3 to 1.0.4 ( #17262 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:23:29 +01:00
dependabot[bot]
6b709c512d
Bump lxml from 5.2.1 to 5.2.2 ( #17261 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:21:20 +01:00