Commit graph

20511 commits

Author SHA1 Message Date
David Robertson
535a689cfc
Reintroduce the lint targets in the linter script (#12455) 2022-04-14 11:33:06 +01:00
David Robertson
6b3e0ea6bd
Use poetry to manage the virtualenv in debian packages (#12449)
* Use `poetry` to build venv in debian packages

Co-authored-by: Dan Callahan <danc@element.io>
Co-authored-by: Shay <hillerys@element.io>

* Changelog

* Only pull in from requirements.txt

Addresses the same problem as #12439.

* Include `test` and `all` extras

`poetry export` helpfully silently ignores an unknown extra

Haven't seen this before because it's the only place we export `all` and
`test`. I could have __sworm__ that the syntax `--extra "all test"`
worked for `poetry install`...

* Clean up requirements file on subsequence builds

* Fix shell syntax

Co-authored-by: Dan Callahan <danc@element.io>
Co-authored-by: Shay <hillerys@element.io>
2022-04-14 11:03:24 +01:00
Richard van der Hoff
8af8a9bce5
Dockerfile-workers: give the master its own log config (#12466)
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:

```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```

- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.

To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
2022-04-13 20:50:08 +01:00
Shay
8e2759f2d8
Limit device_id size to 512B (#12454)
*
2022-04-13 10:04:01 -07:00
Andrew Morgan
0922462fc7
docs: Don't render the table of contents on the print page (#12340) 2022-04-13 08:27:51 -07:00
David Baker
73d8ded0b0
Prevent a sync request from removing a user's busy presence status (#12213)
In trying to use the MSC3026 busy presence status, the user's status
would be set back to 'online' next time they synced. This change makes
it so that syncing does not affect a user's presence status if it
is currently set to 'busy': it must be removed through the presence
API.

The MSC defers to implementations on the behaviour of busy presence,
so this ought to remain compatible with the MSC.
2022-04-13 16:21:07 +01:00
Nick Mills-Barrett
e3a49f4784
Fix missing sync events during historical batch imports (#12319)
Discovered after much in-depth investigation in #12281.

Closes: #12281
Closes: #3305

Signed off by: Nick Mills-Barrett nick@beeper.com
2022-04-13 11:38:35 +01:00
David Robertson
d24cd17820
Use poetry lockfile in twisted trunk CI job (#12425)
Fixes #12458

Co-authored-by: Sean Quah <seanq@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-13 11:26:53 +01:00
Jonathan de Jong
36d8b83888
Rename Mutual Rooms unstable_features flag to match MSC (#12445)
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2022-04-13 10:32:44 +01:00
Shay
32545d2e26
Bump twisted version to the latest in lockfile (#12441) 2022-04-12 13:46:55 -07:00
David Robertson
5a275a2377
Run "main" trial tests under poetry (#12438)
* Run "main" trial tests under poetry

Olddeps and twisted trunk tests are handled in separate PRs.

The PyPy config is a best-effort only; it's completely untested.

Pulled out from #12337.

* Changelog
2022-04-12 17:41:21 +01:00
David Robertson
58c657322a
Run lints under poetry in CI; remove lint tox jobs (#12434)
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-12 17:35:48 +01:00
Erik Johnston
aa28110264
Process device list updates asynchronously (#12365) 2022-04-12 16:50:40 +01:00
Patrick Cloke
4bdbebccb9
Remove the unstable event field for /send_join per MSC3083. (#12395)
This was missed when initially stabilising room version 8 and was
left in as a compatibility shim. Most homeservers have upgraded
to a version which expects the proper field name, and the failure
mode is reasonable (a user on an older server may have to attempt
joining the room twice with an obscure error message the first time).
2022-04-12 11:27:45 -04:00
Erik Johnston
ba1588461b
Fix typos in release script docs (#12450) 2022-04-12 15:12:57 +00:00
Erik Johnston
a468768104 Merge branch 'release-v1.57' into develop 2022-04-12 15:28:41 +01:00
Patrick Cloke
9535fd0f9c
Disable groups/communities by default. (#12344)
This disables the endpoints (and sync response fields) for
groups/communities by default.
2022-04-12 10:20:46 -04:00
Erik Johnston
9b1f360091 Update changelog 2022-04-12 15:07:08 +01:00
Richard van der Hoff
643c0c50c1
Update UPGRADE notes to include more details about v1.57 upgrade failure mode (#12448) 2022-04-12 14:56:05 +01:00
Richard van der Hoff
320186319a
Resync state after partial-state join (#12394)
We work through all the events with partial state, updating the state at each
of them. Once it's done, we recalculate the state for the whole room, and then
mark the room as having complete state.
2022-04-12 13:23:43 +00:00
Erik Johnston
e31d06f6f0 Split changelogs by year 2022-04-12 13:54:36 +01:00
Patrick Cloke
86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Erik Johnston
3810730ba5 Don't line wrap 2022-04-12 13:41:58 +01:00
Erik Johnston
641f43ba81 1.57.0rc1 2022-04-12 13:39:35 +01:00
Dirk Klimpel
1783156dbc
Add some type hints to datastore (#12423)
* Add some type hints to datastore

* newsfile

* change `Collection` to `List`

* refactor return type of `select_users_txn`

* correct type hint in `stream.py`

* Remove `Optional` in `select_users_txn`

* remove not needed return type in `__init__`

* Revert change in `get_stream_id_for_event_txn`

* Remove import from `Literal`
2022-04-12 11:54:00 +01:00
David Robertson
4e13743738
Poetry: select olddeps using poetry (#12407)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-12 10:50:11 +00:00
Erik Johnston
3ad74b63e5
Specify tls extra for Twisted dependency. (#12444)
* Specify `tls` extra for Twisted dependency.

It was already pulled in for us by `treq`, but we should be explicit
that we do use the `tls` functionality of Twisted directly.

* Mark `idna` as dev-dependency

This doesn't actually change anything, as `Twisted[tls]` will put it in
as a main dependency anyway.
2022-04-12 10:49:02 +00:00
David Robertson
5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes (#12439)
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205     Twisted[tls]>=18.7.0 from 38622ff95b/Twisted-22.4.0-py3-none-any.whl (sha256)=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
Patrick Cloke
ab3165efb7
Remove experimental configuration flag for MSC3666. (#12436) 2022-04-11 15:37:46 -04:00
Patrick Cloke
4586119f0b
Add missing type hints to config classes. (#12402) 2022-04-11 12:07:23 -04:00
Richard van der Hoff
214f3b7d21
Enable certificate checking during complement tests (#12435) 2022-04-11 16:35:41 +01:00
Patrick Cloke
772bad2562
Do not consider events by ignored users for bundled aggregations (#12235)
Consider the requester's ignored users when calculating the
bundled aggregations.

See #12285 / 4df10d3214
for corresponding changes for the `/relations` endpoint.
2022-04-11 10:09:57 -04:00
Richard van der Hoff
3cdf5a1386
Fix up healthcheck generation for workers docker image (#12405)
This wasn't quite generating the right thing.
2022-04-11 13:38:58 +00:00
David Robertson
961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
Richard van der Hoff
5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
reivilibre
85ca963c1a
Add Module API for reading and writing global account data. (#12391) 2022-04-11 10:05:43 +01:00
David Robertson
98ec375b26
CI: Fix the export-data job to run under poetry (#12418)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 19:18:54 +00:00
reivilibre
e630722f11
Optimise _update_client_ips_batch_txn to batch together database operations. (#12252)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 15:29:13 +01:00
David Robertson
0cd182f296
Make synapse._scripts pass typechecks (#12421) 2022-04-08 14:00:12 +00:00
David Robertson
dd5cc37aa4
Stop maintaining a list of lint targets in lint.sh (#12420) 2022-04-08 14:36:09 +01:00
David Robertson
95a038c106
Unify HTTP query parameter type hints (#12415)
* Pull out query param types to `synapse.http.types`
* Use QueryParams everywhere
* Simplify `encode_query_args`
* Add annotation which would have caught #12410
2022-04-08 13:06:51 +01:00
Jorge Florian
2e2d8cc2f9
Update the server notices user profile in room if changed. (#12115) 2022-04-08 07:51:27 -04:00
David Robertson
7851a2c62f
CI: Get the portdb job to run under poetry (#12412) 2022-04-08 12:00:05 +01:00
Jorge Florian
78e4d96a4d
Add missing type definitions for scripts in docker folder (#12280)
Signed-off-by: Jorge Florian <jafn28@gmail.com>
2022-04-08 10:10:58 +00:00
Erik Johnston
7732c4902c
Fix rejecting invites over federation (#12409)
Currently causes future incremental syncs to fail.

Broke by #12191
2022-04-07 15:54:16 +01:00
Erik Johnston
36af768c13
Fix fetching public rooms over federation (#12410)
Broke by #12364
2022-04-07 14:18:02 +00:00
Patrick Cloke
1a90c1e3af
Do not add groups to sync results if disabled. (#12408) 2022-04-07 09:29:56 -04:00
Erik Johnston
d1cd96ce29
Add opentracing spans to calls to external cache (#12380) 2022-04-07 13:18:29 +01:00
David Robertson
3a7e97c7ad
Poetry: use locked environment in Docker images (#12385) 2022-04-07 11:43:31 +00:00
Patrick Cloke
0bcb651b3f
Support the v1 endpoint for /relations. (#12403)
Now that MSC2675 has passed FCP and the implementation is
compliant with the final version.
2022-04-07 07:08:23 -04:00