Commit graph

18625 commits

Author SHA1 Message Date
Erik Johnston
e550ab17ad
Increase default join burst ratelimiting (#9674)
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23 14:52:20 +00:00
Jonathan de Jong
0caf2a338e
Fix federation stall on concurrent access errors (#9639) 2021-03-23 13:52:30 +00:00
Richard van der Hoff
4ecba9bd5c
Federation API for Space summary (#9652)
Builds on the work done in #9643 to add a federation API for space summaries.

There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
2021-03-23 11:51:12 +00:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Andrew Morgan
5b268997bd
Allow providing credentials to HTTPS_PROXY (#9657)
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
2021-03-22 17:20:47 +00:00
Johannes Wienke
4612302399
Include opencontainers labels in Docker image (#9612)
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys

Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-03-22 15:31:00 +00:00
Ankit Dobhal
d66f9070cd
Fixed code misc. quality issues (#9649)
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
2021-03-22 11:18:13 -04:00
Erik Johnston
d600d4506b Merge branch 'master' into develop 2021-03-22 13:36:36 +00:00
Brendan Abolivier
e09838c78f
Merge pull request #9644 from matrix-org/babolivier/msc3026
Implement MSC3026: busy presence state
2021-03-22 14:28:19 +01:00
Erik Johnston
e2904f720d 1.30.0 2021-03-22 13:15:55 +00:00
Brendan Abolivier
b6ed4f55ac
Incorporate review 2021-03-19 18:19:50 +01:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026 2021-03-19 16:12:40 +01:00
Brendan Abolivier
0b56481caa
Fix lint 2021-03-19 16:11:08 +01:00
Richard van der Hoff
066068f034 fix mypy 2021-03-19 12:20:11 +00:00
Richard van der Hoff
0e35584734
federation_client: handle inline signing_keys in hs.yaml (#9647) 2021-03-18 21:12:07 +00:00
Richard van der Hoff
201178db1a
federation_client: stop adding URL prefix (#9645) 2021-03-18 20:31:47 +00:00
Patrick Cloke
9b0e3009fa Fix type-hints from bad merge. 2021-03-18 14:40:56 -04:00
Richard van der Hoff
004234f03a
Initial spaces summary API (#9643)
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
2021-03-18 18:24:16 +00:00
Brendan Abolivier
066c703729
Move support for MSC3026 behind an experimental flag 2021-03-18 18:37:19 +01:00
Dirk Klimpel
8dd2ea65a9
Consistently check whether a password may be set for a user. (#9636) 2021-03-18 12:54:08 -04:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. (#9640)
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17 16:51:55 +00:00
Patrick Cloke
cc324d53fe
Fix up types for the typing handler. (#9638)
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
2021-03-17 11:30:21 -04:00
Hubert Chathi
73dbce5523
only save remote cross-signing keys if they're different from the current ones (#9634)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-17 11:04:57 -04:00
Erik Johnston
ad721fc559
Fix bad naming of storage function (#9637)
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
2021-03-17 13:20:08 +00:00
Richard van der Hoff
567f88f835
Prep work for removing outlier from internal_metadata (#9411)
* Populate `internal_metadata.outlier` based on `events` table

Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.

* Move `outlier` out of InternalMetadata._dict

Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
2021-03-17 12:33:18 +00:00
Patrick Cloke
b449af0379
Add type hints to the room member handler. (#9631) 2021-03-17 07:14:39 -04:00
Jonathan de Jong
27d2820c33
Enable flake8-bugbear, but disable most checks. (#9499)
* Adds B00 to ignored checks.
* Fixes remaining issues.
2021-03-16 14:19:27 -04:00
Hubbe
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers (#9609)
Allows limiting who can login using OIDC via the claims
made from the IdP.
2021-03-16 11:46:07 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588)
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
Andrew Morgan
45ef73fd4f Fix jemalloc changelog entry wording 2021-03-16 14:46:40 +00:00
Andrew Morgan
e3bc0e6f7c Changelog typo 2021-03-16 14:33:23 +00:00
Andrew Morgan
ad5d2e7ec0 Pull up appservice login deprecation notice 2021-03-16 13:51:24 +00:00
Andrew Morgan
d315e96443 1.30.0rc1 2021-03-16 13:45:46 +00:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626)
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
Mathieu Velten
ccf1dc51d7
Install jemalloc in docker image (#8553)
Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-03-16 11:32:18 +00:00
Patrick Cloke
1383508f29
Handle an empty cookie as an invalid macaroon. (#9620)
* Handle an empty cookie as an invalid macaroon.

* Newsfragment
2021-03-16 11:29:35 +00:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617)
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Richard van der Hoff
5b5bc188cf
Clean up config settings for stats (#9604)
... and complain if people try to turn it off.
2021-03-16 10:57:54 +00:00
Andrew Morgan
1b0eaed21f
Prevent bundling aggregations for state events (#9619)
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
2021-03-16 10:27:51 +00:00
Richard van der Hoff
1c8a2541da
Fix Internal Server Error on GET /saml2/authn_response (#9623)
* Fix Internal Server Error on `GET /saml2/authn_response`

Seems to have been introduced in #8765 (Synapse 1.24.0)

* Fix newsfile
2021-03-16 10:20:20 +00:00
Patrick Cloke
f87dfb9403
Revert requiring a specific version of Twisted for mypy checks. (#9618) 2021-03-15 12:18:35 -04:00
Patrick Cloke
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily (#9561)
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00
Richard van der Hoff
af2248f8bf
Optimise missing prev_event handling (#9601)
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from 
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.

This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)

The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).

Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
2021-03-15 13:51:02 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Richard van der Hoff
1e67bff833
Reject concurrent transactions (#9597)
If more transactions arrive from an origin while we're still processing the
first one, reject them.

Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
2021-03-12 15:14:55 +00:00
Richard van der Hoff
2b328d7e02
Improve logging when processing incoming transactions (#9596)
Put the room id in the logcontext, to make it easier to understand what's going on.
2021-03-12 15:08:03 +00:00
Richard van der Hoff
464e5da7b2
Add logging for redis connection setup (#9590) 2021-03-11 18:35:09 +00:00