Commit graph

7507 commits

Author SHA1 Message Date
Eric Eastwood
1903c7e5ed
Remove duplicate timestamp from test logs (_trial_temp/test.log) (#15636)
Fix https://github.com/matrix-org/synapse/issues/15618

### Before

```
2023-05-17 22:51:36-0500 [-] 2023-05-17 22:51:36,889 - synapse.server - 338 - INFO - sentinel - Finished setting up.
```

### After

```
2023-05-19 18:16:20-0500 [-] synapse.server - 338 - INFO - sentinel - Finished setting up.
```


### Dev notes

The `Twisted.Logger` controls the `2023-05-19 18:16:20-0500 [-]` prefix, see : [`twisted/twisted` -> `src/twisted/logger/_format.py#L362-L374`](34b161e66b/src/twisted/logger/_format.py (L362-L374))

And we delegate our logs to the Twisted Logger for the tests which puts it in `_trial_temp/test.log`
2023-05-22 13:49:01 -05:00
Andrew Morgan
737f7ddf58
Remove outdated comment in log config (#15648) 2023-05-22 17:58:58 +00:00
Patrick Cloke
c5d1e6d414
Properly parse event_fields in filters (#15607)
The event_fields property in filters should use the proper
escape rules, namely backslashes can be escaped with
an additional backslash.

This adds tests (adapted from matrix-js-sdk) and implements
the logic to properly split the event_fields strings.
2023-05-22 11:31:22 -04:00
dependabot[bot]
201597fc86
Bump pygithub from 1.58.1 to 1.58.2 (#15643)
* Bump pygithub from 1.58.1 to 1.58.2

Bumps [pygithub](https://github.com/pygithub/pygithub) from 1.58.1 to 1.58.2.
- [Release notes](https://github.com/pygithub/pygithub/releases)
- [Changelog](https://github.com/PyGithub/PyGithub/blob/v1.58.2/doc/changes.rst)
- [Commits](https://github.com/pygithub/pygithub/compare/v1.58.1...v1.58.2)

---
updated-dependencies:
- dependency-name: pygithub
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-05-22 15:39:19 +01:00
Sean Quah
cc53c96bf8
Limit the size of the HomeServerConfig cache in trial test runs (#15646)
...to try to control memory usage. `HomeServerConfig`s hold on to
many Jinja2 objects, which come out to over 0.5 MiB per config.

Over the course of a full test run, the cache grows to ~360 entries.
Limit it to 8 entries.

Part of #15622.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-22 13:25:39 +01:00
dependabot[bot]
a47b2065f0
Bump furo from 2023.3.27 to 2023.5.20 (#15642)
* Bump furo from 2023.3.27 to 2023.5.20

Bumps [furo](https://github.com/pradyunsg/furo) from 2023.3.27 to 2023.5.20.
- [Release notes](https://github.com/pradyunsg/furo/releases)
- [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md)
- [Commits](https://github.com/pradyunsg/furo/compare/2023.03.27...2023.05.20)

---
updated-dependencies:
- dependency-name: furo
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-05-22 12:12:59 +01:00
dependabot[bot]
875015d512
Bump sphinx from 6.1.3 to 6.2.1 (#15641)
* Bump sphinx from 6.1.3 to 6.2.1

Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 6.2.1.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v6.2.1)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-05-22 10:38:08 +01:00
dependabot[bot]
8516001566
Bump types-pillow from 9.5.0.2 to 9.5.0.4 (#15640)
* Bump types-pillow from 9.5.0.2 to 9.5.0.4

Bumps [types-pillow](https://github.com/python/typeshed) from 9.5.0.2 to 9.5.0.4.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-05-22 10:38:01 +01:00
dependabot[bot]
adae1cfc8c
Bump types-setuptools from 67.7.0.2 to 67.8.0.0 (#15639)
* Bump types-setuptools from 67.7.0.2 to 67.8.0.0

Bumps [types-setuptools](https://github.com/python/typeshed) from 67.7.0.2 to 67.8.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2023-05-22 10:37:50 +01:00
Eric Eastwood
703a8f9c67
Instrument state and state_group storage related things (tracing) (#15610)
Instrument `state` and `state_group` storage related things (tracing) so it's a little more clear where these database transactions are coming from as there is a lot of wires crossing in these functions.

Part of `/messages` performance investigation: https://github.com/matrix-org/synapse/issues/13356
2023-05-19 12:26:58 -05:00
Eric Eastwood
ca3c07e833
Trace how many new events from the backfill response we need to process (#15633)
You can kinda derive this information from how many `_process_pulled_event` spans there are but it would be nice to quickly glance.
2023-05-19 11:18:45 -05:00
reivilibre
736199b763
Remove old R30 because R30v2 supercedes it (#10428)
R30v2 has been out since 2021-07-19 (https://github.com/matrix-org/synapse/pull/10332)
and we started collecting stats on 2021-08-16. Since it's been over a year now
(almost 2 years), this is enough grace period for us to now rip it out.
2023-05-19 11:13:44 -05:00
Patrick Cloke
1e89976b26
Rename blacklist/whitelist internally. (#15620)
Avoid renaming configuration settings for now and rename internal code
to use blocklist and allowlist instead.
2023-05-19 12:25:25 +00:00
Patrick Cloke
89a23c9406
Do not allow deactivated users to login with JWT. (#15624)
To improve the organization of this code it moves the JWT login
checks to a separate handler and then fixes the bug (and a
deprecation warning).
2023-05-19 08:06:54 -04:00
Patrick Cloke
07771fa487
Remove experimental configuration flags & unstable values for faster joins (#15625)
Synapse will no longer send (or respond to) the unstable flags
for faster joins. These were only available behind a configuration
flag and handled in parallel with the stable flags.
2023-05-19 07:23:09 -04:00
Sean Quah
d0de452d12
Fix HomeServers leaking during trial test runs (#15630)
This change fixes two memory leaks during `trial` test runs.

Garbage collection is disabled during each test case and a gen-0 GC is
run at the end of each test. However, when the gen-0 GC is run, the
`TestCase` object usually still holds references to the `HomeServer`
used during the test. As a result, the `HomeServer` gets promoted to
gen-1 and then never garbage collected.

Fix this by periodically running full GCs.

Additionally, fix `HomeServer`s leaking after tests that touch inbound
federation due to `FederationRateLimiter`s adding themselves to a global
set, by turning the set into a `WeakSet`.

Resolves #15622.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-19 11:17:12 +01:00
Nick Mills-Barrett
ad50510a06
Handle missing previous read marker event. (#15464)
If the previous read marker is pointing to an event that no longer exists
(e.g. due to retention) then assume that the newly given read marker
is newer.
2023-05-18 14:37:31 -04:00
Jonathan de Jong
e5b4d93770
Update Mutual Rooms (MSC2666) implementation (#15621)
To track changes in MSC2666:

- The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`.
- The addition of `next_batch_token` (and logic).
- Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`.
- The error code when your own user is requested.
2023-05-18 12:49:12 -04:00
Patrick Cloke
5dc1f25c53
Fix olddeps build (#15626)
Do an `apt update` before install packages.
2023-05-18 10:53:57 -04:00
axel simon
4ec40b16ac
flake.nix: start synapse automatically, add space usage warning (#15613)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2023-05-18 15:44:28 +01:00
Sean Quah
68dcd2cbcb
Re-type config paths in ConfigErrors to be StrSequences (#15615)
Part of #14809.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-18 11:11:30 +01:00
Sean Quah
e15aa00bc0
Fix error message when app_service_config_files validation fails (#15614)
The second argument of `ConfigError` is a path, passed as an optional
`Iterable[str]` and not a `str`. If a string is passed directly,
Synapse unhelpfully emits "Error in configuration at
a.p.p._.s.e.r.v.i.c.e._.c.o.n.f.i.g._.f.i.l.e.s'" when the config
option has the wrong data type.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-18 10:58:13 +01:00
Quentin Gliech
41b9def9f2
Add a new admin API to create a new device for a user. (#15611)
This allows an external service (e.g. the matrix-authentication-service)
to create devices for users.
2023-05-17 14:39:06 +00:00
Patrick Cloke
4ee82c0576
Apply url_preview_url_blacklist to oEmbed and pre-cached images (#15601)
There are two situations which were previously not properly checked:

1. If the requested URL was replaced with an oEmbed URL, then the
   oEmbed URL was not checked against url_preview_url_blacklist.
2. Follow-up URLs (either via autodiscovery of oEmbed or to pre-cache
   images) were not checked against url_preview_url_blacklist.
2023-05-16 16:25:01 -04:00
Patrick Cloke
375b0a8a11
Update code to refer to "workers". (#15606)
A bunch of comments and variables are out of date and use
obsolete terms.
2023-05-16 15:56:38 -04:00
Eric Eastwood
7148c2a0d6
Run mypy type checking with the minimum supported Python version (#15602)
We use the oldest Python version because later Python versions can include some overloads which don't work in the older versions which we still support.

We're using Python 3.8 instead of 3.7 which is our actual minimum support version because it's EOL is in a matter of weeks so can avoid the extra effort. And in any case, minimum Python 3.8 support is better than winging it on Python 3.11.
2023-05-16 13:27:47 -05:00
Shay
9f6ff6a0eb
Add not null constraint to column full_user_id of tables profiles and user_filters (#15537) 2023-05-16 10:57:39 -07:00
Eric Eastwood
77cda342be traceback.format_exception(...) usage that is compatible with Python 3.7 and 3.11 (#15599)
* Usage that is compatible with Python 3.8 and 3.11

> Since Python 3.10, instead of passing value and tb, an exception object can
  be passed as the first argument. If value and tb are provided, the first
  argument is ignored in order to provide backwards compatibility.
>
> -- https://docs.python.org/3/library/traceback.html

* Add changelog
2023-05-16 12:33:18 -05:00
Eric Eastwood
c51d2e6199
Fix subscriptable type usage in Python <3.9 (#15604)
Fix the following `mypy` errors when running `mypy` with Python 3.7:
```
synapse/storage/controllers/stats.py:58: error: "Counter" is not subscriptable, use "typing.Counter" instead  [misc]

tests/test_state.py:267: error: "dict" is not subscriptable, use "typing.Dict" instead  [misc]
```

Part of https://github.com/matrix-org/synapse/issues/15603

In Python 3.9, `typing` is deprecated and the types are subscriptable (generics) by default, https://peps.python.org/pep-0585/#implementation
2023-05-16 12:19:46 -05:00
Olivier Wilkinson (reivilibre)
3ec9f3b0cc 1.84.0rc1 2023-05-16 11:23:05 +01:00
Shay
ba572647b2
Export run_as_background_process from the module API (#15577) 2023-05-15 13:11:21 -07:00
Patrick Cloke
f2905d827f
Implement MSC3821 to update redaction rules (third_party_invite.signed) (#15563)
Updates the redaction rules to protect enough information that the
event can still be properly verified.
2023-05-15 15:02:24 -04:00
Patrick Cloke
eb3c1823d8
Reject instead of erroring on invalid membership events. (#15564)
Instead of resulting in an internal server error for invalid events,
return that the event is invalid.
2023-05-15 15:01:29 -04:00
Patrick Cloke
ba6b21c81e
Implement MSC3389 to protect relations from redaction. (#15565)
MSC3389 proposes protecting the relation type & parent event ID
from redaction. This keeps the relation information intact after
redaction which helps with some UX flaws (e.g. deleting an
event causes it to no longer be in a thread, which is confusing).
2023-05-15 12:58:09 +00:00
Mathieu Velten
8583346335 Revert "Bump pillow from 9.4.0 to 9.5.0 (#15593)"
This reverts commit 34ab801379.
2023-05-15 14:22:07 +02:00
icp
b3ada9bfb4
Allow poetry-core 1.6.0 (#15588) 2023-05-15 11:19:11 +02:00
villepeh
aa5c0592e7
Update Mastodon SSO instructions (#15587) 2023-05-15 11:17:24 +02:00
Michael Weimann
3690d5bd89
Add an unstable feature flag for MSC3981 to the /versions endpoint (#15558)
Signed-off-by: Michael Weimann <michaelw@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-05-15 10:54:49 +02:00
dependabot[bot]
7b6c9f4c04
Bump phonenumbers from 8.13.7 to 8.13.11 (#15590) 2023-05-15 10:45:34 +02:00
dependabot[bot]
2e8a2bda52
Bump types-psycopg2 from 2.9.21.9 to 2.9.21.10 (#15591) 2023-05-15 10:45:15 +02:00
dependabot[bot]
3fd8eb81de
Bump types-commonmark from 0.9.2.2 to 0.9.2.3 (#15592) 2023-05-15 10:44:47 +02:00
dependabot[bot]
1b4782a37d
Bump types-setuptools from 67.7.0.1 to 67.7.0.2 (#15594) 2023-05-15 10:44:31 +02:00
dependabot[bot]
34ab801379
Bump pillow from 9.4.0 to 9.5.0 (#15593) 2023-05-15 10:44:06 +02:00
dependabot[bot]
bcd2495469
Bump serde from 1.0.162 to 1.0.163 (#15589) 2023-05-15 10:42:51 +02:00
Patrick Cloke
def480442d
Declare support for Matrix 1.6 (#15559)
Adds logging for key server requests which include a key ID.
This is technically in violation of the 1.6 spec, but is the only
way to remain backwards compatibly with earlier versions of
Synapse (and possibly other homeservers) which *did* include
the key ID.
2023-05-12 07:31:50 -04:00
Erik Johnston
808105bd31
Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)" (#15580)
This reverts commit a7b3e9ce65.
2023-05-12 11:38:16 +01:00
David Robertson
c96a1d2a27
Relax poetry-core lower bound to 1.1.0 (#15571)
See https://github.com/matrix-org/synapse/pull/15566#issuecomment-1543844104

Also check you can `pip install` in the old-deps CI job
2023-05-12 11:21:11 +01:00
helix-loop
08297f2f18
Add pkg-config package to Stage 0 (#15567) 2023-05-12 11:32:09 +02:00
David Robertson
7c76514f1e
Deal with more GHA deprecations (#15576)
* Bump netlify PR

* Manually cache mypy cache dir

cache cache cache cache cache cache cache cache cache cache

* Changelog
2023-05-11 18:24:32 +00:00
Eric Eastwood
d19d1edbcf
Print full startup/initialization error (#15569)
I found the error in the **Before** really vague and obtuse and didn't realize port `5432` corresponded to the Postgres port until searching the codebase. It says to check the logs but that wasn't my first instinct. It's just more obvious if we just print the full thing which gives context of the error type and the traceback to the relevant area of code.

#### Before

```
$ poetry run python -m synapse.app.homeserver -c homeserver.yaml
**********************************************************************************
 Error during initialisation:
    connection to server at "localhost" (::1), port 5432 failed: Connection refused
 	Is the server running on that host and accepting TCP/IP connections?
 connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
 	Is the server running on that host and accepting TCP/IP connections?
 
 There may be more information in the logs.
**********************************************************************************
```

#### After

```sh
$ poetry run python -m synapse.app.homeserver -c homeserver.yaml
**********************************************************************************
 Error during initialisation:
     Traceback (most recent call last):
       File "/home/eric/Documents/github/element/synapse/synapse/app/homeserver.py", line 352, in setup
         hs.setup()
       File "/home/eric/Documents/github/element/synapse/synapse/server.py", line 337, in setup
         self.datastores = Databases(self.DATASTORE_CLASS, self)
       File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/__init__.py", line 65, in __init__
         with make_conn(database_config, engine, "startup") as db_conn:
       File "/home/eric/Documents/github/element/synapse/synapse/storage/database.py", line 161, in make_conn
         native_db_conn = engine.module.connect(**db_params)
       File "/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.10/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
         conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
     psycopg2.OperationalError: connection to server at "localhost" (::1), port 5432 failed: Connection refused
     	Is the server running on that host and accepting TCP/IP connections?
     connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
     	Is the server running on that host and accepting TCP/IP connections?
 
 
 There may be more information in the logs.
**********************************************************************************
```
2023-05-11 11:50:46 -05:00