Matrix/synapse
Matrix/synapse
1
0
Fork 0
mirror of https://github.com/element-hq/synapse synced 2024-10-08 06:13:32 +00:00
Code Issues Projects Releases Packages Wiki Activity
19861 commits 932 branches 720 tags 480 MiB
Commit graph

5193 commits

Author SHA1 Message Date
Patrick Cloke
6b3ac3b8cd
Convert device handler to async/await (#7871) 2020-07-17 07:09:25 -04:00
Patrick Cloke
00e57b755c
Convert synapse.app to async/await. (#7868) 2020-07-17 07:08:56 -04:00
Patrick Cloke
6fca1b3506
Convert _base, profile, and _receipts handlers to async/await (#7860) 2020-07-17 07:08:30 -04:00
Michael Albert
fff483ea96
Add admin endpoint to get members in a room. (#7842) 2020-07-16 16:43:23 -04:00
Patrick Cloke
f460da6031
Consistently use db_to_json to convert from database values to JSON objects. (#7849) 2020-07-16 11:32:19 -04:00
Richard van der Hoff
e5300063ed
Optimise queueing of inbound replication commands (#7861) 
When we get behind on replication, we tend to stack up background processes
behind a linearizer. Bg processes are heavy (particularly with respect to
prometheus metrics) and linearizers aren't terribly efficient once the queue
gets long either.

A better approach is to maintain a queue of requests to be processed, and
nominate a single process to work its way through the queue.

Fixes: #7444
 2020-07-16 15:49:37 +01:00
Richard van der Hoff
346476df21
Reject attempts to join empty rooms over federation (#7859) 
We shouldn't allow others to make_join through us if we've left the room;
reject such attempts with a 404.

Fixes #7835. Fixes #6958.
 2020-07-16 15:17:31 +01:00
Erik Johnston
f2e38ca867
Allow moving typing off master (#7869) 2020-07-16 15:12:54 +01:00
Erik Johnston
649a7ead5c
Add ability to run multiple pusher instances (#7855) 
This reuses the same scheme as federation sender sharding
 2020-07-16 14:06:28 +01:00
Richard van der Hoff
a827838706
Merge pull request #7866 from matrix-org/rav/fix_guest_user_id 
Fix guest user registration with lots of client readers
 2020-07-16 13:54:45 +01:00
Richard van der Hoff
a973bcb8a4
Add some tiny type annotations (#7870) 
I found these made pycharm have more of a clue as to what was going on in other places.
 2020-07-16 13:52:29 +01:00
Richard van der Hoff
16368c8a34 changelog 2020-07-16 13:01:11 +01:00
Patrick Cloke
35450519de
Ensure that calls to json.dumps are compatible with the standard library json. (#7836) 2020-07-15 13:40:54 -04:00
Richard van der Hoff
a57df9b827
Avoid brand new rooms in delete_old_current_state_events (#7854) 
When considering rooms to clean up in `delete_old_current_state_events`, skip
rooms which we are creating, which otherwise look a bit like rooms we have
left.

Fixes #7834.
 2020-07-15 18:33:03 +01:00
Erik Johnston
97e1159ac1 Merge branch 'erikj/faster_typing' of github.com:matrix-org/synapse into develop 2020-07-15 16:54:30 +01:00
Patrick Cloke
8c7d0f163d
Allow accounts to be re-activated from the admin APIs. (#7847) 2020-07-15 11:00:21 -04:00
Erik Johnston
3032b54ac9 Newsfile 2020-07-15 15:45:19 +01:00
Erik Johnston
f13061d515
Fix client reader sharding tests (#7853) 
* Fix client reader sharding tests

* Newsfile

* Fix typing

* Update changelog.d/7853.misc

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Move mocking of http_client to tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2020-07-15 15:27:35 +01:00
Patrick Cloke
b11450dedc
Convert E2E key and room key handlers to async/await. (#7851) 2020-07-15 08:48:58 -04:00
Patrick Cloke
111e70d75c
Return the proper 403 Forbidden error during errors with JWT logins. (#7844) 2020-07-15 07:10:21 -04:00
Richard van der Hoff
1d9dca02f9
remove retry_on_integrity_error wrapper for persist_events (#7848) 
As far as I can tell from the sentry logs, the only time this has actually done
anything in the last two years is when we had two master workers running at
once, and even then, it made a bit of a mess of it (see
https://github.com/matrix-org/synapse/issues/7845#issuecomment-658238739).

Generally I feel like this code is doing more harm than good.
 2020-07-15 10:34:53 +01:00
Patrick Cloke
8d0097bef1
Fix bug in per-room message retention policies. (#7850) 2020-07-14 15:51:13 -04:00
Brendan Abolivier
85223106f3
Allow email subjects to be customised through Synapse's configuration (#7846) 2020-07-14 19:10:42 +01:00
Dirk Klimpel
491f0dab1b
Add delete room admin endpoint (#7613) 
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2020-07-14 12:36:23 +01:00
Patrick Cloke
77d2c05410
Add the option to validate the iss and aud claims for JWT logins. (#7827) 2020-07-14 07:16:43 -04:00
Patrick Cloke
4db1509516
Improve the type hints of synapse.api.errors. (#7820) 2020-07-14 07:03:58 -04:00
Erik Johnston
f886a69916
Correctly pass app_name to all email templates. (#7829) 
We didn't do this for e.g. registration emails.
 2020-07-14 10:00:53 +01:00
Patrick Cloke
457096e6df
Support handling registration requests across multiple client readers. (#7830) 2020-07-13 13:31:46 -04:00
Brendan Abolivier
504c8f3483
Fix handling of "off" in encryption_enabled_by_default_for_room_type (#7822) 
Fixes https://github.com/matrix-org/synapse/issues/7821, introduced in https://github.com/matrix-org/synapse/pull/7639

Turns out PyYAML translates `off` into a `False` boolean if it's
unquoted (see https://stackoverflow.com/questions/36463531/pyyaml-automatically-converting-certain-keys-to-boolean-values),
which seems to be a liberal interpretation of this bit of the YAML spec: https://yaml.org/spec/1.1/current.html#id864510

An alternative fix would be to implement the solution mentioned in the
SO post linked above, but I'm aware it might break existing setups
(which might use these values in the configuration file) so it's
probably better just to add an extra check for this one. We should be
aware that this is a thing for the next times we do that though.

I didn't find any other occurrence of this bug elsewhere in the
codebase.
 2020-07-13 17:14:42 +01:00
Patrick Cloke
66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802) 2020-07-10 14:30:08 -04:00
Patrick Cloke
d9e47af617
Add types to the server code and remove unused parameter (#7813) 2020-07-10 14:28:42 -04:00
Sorunome
1bca21e1da
Include room states on invite events sent to ASes (#6455) 2020-07-10 18:44:56 +01:00
Erik Johnston
f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Erik Johnston
f1245dc3c0
Fix resync remote devices on receive PDU in worker mode. (#7815) 
The replication client requires that arguments are given as keyword
arguments, which was not done in this case. We also pull out the logic
so that we can catch and handle any exceptions raised, rather than
leaving them unhandled.
 2020-07-10 18:23:17 +01:00
Erik Johnston
e29c44340b
Fix recursion error when fetching auth chain over federation (#7817) 
When fetching the state of a room over federation we receive the event
IDs of the state and auth chain. We then fetch those events that we
don't already have.

However, we used a function that recursively fetched any missing auth
events for the fetched events, which can lead to a lot of recursion if
the server is missing most of the auth chain. This work is entirely
pointless because would have queued up the missing events in the auth
chain to be fetched already.

Let's just diable the recursion, since it only gets called from one
place anyway.
 2020-07-10 18:15:35 +01:00
Richard van der Hoff
43726783e4 1.17.0rc1 2020-07-09 16:53:19 +01:00
Patrick Cloke
38e1fac886
Fix some spelling mistakes / typos. (#7811) 2020-07-09 09:52:58 -04:00
Richard van der Hoff
53ee214f2f
update_membership declaration: now always returns an event id. (#7809) 2020-07-09 13:01:42 +01:00
Richard van der Hoff
8ca39bd2c3
Improve stacktraces from exceptions in background processes (#7808) 
use `Failure()` to fish out the real exception.
 2020-07-09 13:01:33 +01:00
Richard van der Hoff
08c5181a8d
Fix can only concatenate list (not "tuple") to list exception (#7810) 
It seems auth_events can be either a list or a tuple, depending on Things.
 2020-07-09 12:48:15 +01:00
Patrick Cloke
8fa7fdd4cb
Pass original request headers from workers to the main process. (#7797) 2020-07-09 07:34:46 -04:00
Richard van der Hoff
2ab0b021f1
Generate real events when we reject invites (#7804) 
Fixes #2181. 

The basic premise is that, when we
fail to reject an invite via the remote server, we can generate our own
out-of-band leave event and persist it as an outlier, so that we have something
to send to the client.
 2020-07-09 10:40:19 +01:00
Richard van der Hoff
67593b1728
Add HomeServer.signing_key property (#7805) 
... instead of duplicating `config.signing_key[0]` everywhere
 2020-07-08 17:51:56 +01:00
Richard van der Hoff
ef5ed5292b Revert "Update the installation docs on apt-transport-https (#7801)" 
This reverts commit e0c0129693.

As discussed at
https://github.com/matrix-org/synapse/pull/7801#pullrequestreview-444652786, I
don't think this is an improvement.
 2020-07-08 16:57:10 +01:00
Patrick Cloke
e7efd8f827
Do not use simplejson in Synapse. (#7800) 2020-07-08 07:15:08 -04:00
Patrick Cloke
ff0680f69d
Stop passing bytes when dumping JSON (#7799) 2020-07-08 07:14:56 -04:00
Dirk Heinrichs
e0c0129693
Update the installation docs on apt-transport-https (#7801) 
* Starting with apt 1.6, https support has moved into the main package and apt-transport-https has become a transitional dummy package.

Signed-off-by: Dirk Heinrichs <dirk.heinrichs@altum.de>
 2020-07-08 11:34:13 +01:00
Nicolai Søborg
96bb01d8ec
Change Caddy links (old is deprecated) (#7789) 
* Change Caddy links

Current links points to Caddy v1 which is deprecated.

Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
 2020-07-08 10:09:16 +01:00
Richard van der Hoff
76dbd7b8d6
Stop populating unused table local_invites. (#7793) 
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
 2020-07-07 14:20:40 +01:00
Erik Johnston
67d7756fcf
Refactor getting replication updates from database v2. (#7740) 2020-07-07 12:11:35 +01:00
Juho Vanhanen
d378c3da78
Add libwebp dependency to Dockerfile (#7791) 
* Add libwebp dependency to Dockerfile

Signed-off-by: Juho Vanhanen <juho@vanhanen.io>
 2020-07-06 13:37:39 +01:00
Patrick Cloke
2a266f4511
Add documentation for JWT login type and improve sample config. (#7776) 2020-07-06 08:31:51 -04:00
Patrick Cloke
6d687ebba1
Convert the appservice handler to async/await. (#7775) 2020-07-06 07:40:35 -04:00
reivilibre
57feeab364
Don't ignore set_tweak actions with no explicit value. (#7766) 
* Fix spec compliance; tweaks without values are valid

(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)

* Changelog for 7766.

* Add documentation to `tweaks_for_actions`

May as well tidy up when I'm here.

* Add a test for `tweaks_for_actions`
 2020-07-06 11:43:41 +01:00
Oliver Kurz
4e118742ca
Allow to use higher versions of prometheus_client (#7780) 
Fixes https://github.com/matrix-org/synapse/issues/7641

The package was pinned to <0.8.0 without an obvious reasoning with
7ad1d7635
in https://github.com/matrix-org/synapse/pull/5636
while the version selection looks to just try to exclude an arbitrary
next minor version number that might introduce API breaking changes.
Selecting the next minor number might be a good conservative selection.

Downstream distributions already reported success patching out the version
requirements.

This also fixes the integration of upgraded packages into openSUSE packages,
e.g. for openSUSE Tumbleweed which already ships prometheus_client >= 0.8 .

Signed-off-by: Oliver Kurz <okurz@suse.de>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-07-06 10:21:41 +01:00
Will Hunt
62b1ce8539
isort 5 compatibility (#7786) 
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
 2020-07-05 16:32:02 +01:00
Erik Johnston
5cdca53aa0
Merge different Resource implementation classes (#7732) 2020-07-03 19:02:19 +01:00
Dirk Klimpel
21a212f8e5
Fix inconsistent handling of upper and lower cases of email addresses. (#7021) 
fixes #7016
 2020-07-03 14:03:13 +01:00
Alex Kotov
8097659f6e
Allow YAML config file to contain None (#7779) 
Useful when config file is fully commented

Signed-off-by: Alex Kotov <kotovalexarian@gmail.com>
 2020-07-03 13:19:03 +01:00
Patrick Cloke
f3e0f16240 Synapse 1.16.0rc2 (2020-07-02) 
==============================
 
 Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
 Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.
 
 Improved Documentation
 ----------------------
 
 - Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))
 
 Internal Changes
 ----------------
 
 - Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl79+qgACgkQM/xY9qcR
 MEhcaRAAjWLW3ojN1F0DUfE85jziZK2VdnMQC3g+uEOLX6QRbfqFNaNNMjLdK+vl
 K/+2ZoHkRsg6g8noSPhPmI1z1+hb5xDJaxjltzHxonIipW8XSU8o2PQMkf8O/BAy
 VS58y3GyLkhEgzWC+/hcII+LBgcqXpLuNM0xrKTHmxclIjdewlwe1v+hxkP+6wsX
 9Whhn1f4sNHrCtyFVK9uzMFcVyzcQaiWZRjEDMj2uR7rWT6UbCUifN/G4fWmtGbY
 xWoNoC4Qv8xiqXOG4U7juPp9T3bRyWMKyjBFM5PWO6Ec2zfafDyFzhBxJhlQhODG
 g21tS4PowX/dM/pBpJFEOPh1BVrPZzzTD+YMmTcd3NO79HeaQGqEX/+tzFCFUyPp
 0daJK3Y85+l5w/M09WU8DDN8CiR3PFJyGDIZp+nweMsiJZkbEbLOkh1tx6TL+5/6
 zwewU6cq8nTVGrn53Tn58l8C7Sj4w+Qk+1XDzymAoidyoWqAKW9Y/fw53PaViUSx
 voDu0rpsEUXR1OzCBG8SAPQCFy9gdEWV04OvIpzHuq2uojkz66f7NAXy+Wz+Occ9
 AYb/s6Ei80bGCLgRd5jg+myqavwRbzCyv+LIC6dxpopxZJ3AzrFuD11eXKtrIxOC
 FZYf3U4KeBk4Q9TV5IFV1xcGFrq5aK36LdmP6WOsEl3PXVT9p/Q=
 =YaJn
 -----END PGP SIGNATURE-----

Merge tag 'v1.16.0rc2' into develop

Synapse 1.16.0rc2 (2020-07-02)
==============================

Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
Please see [below](https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1152-2020-07-02) for more details.

Improved Documentation
----------------------

- Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))

Internal Changes
----------------

- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
 2020-07-02 11:25:56 -04:00
Patrick Cloke
1319e53251 1.16.0rc2 2020-07-02 11:06:35 -04:00
Patrick Cloke
f2bcc6ecbf Merge branch 'master' into release-v1.16.0 2020-07-02 11:02:42 -04:00
Patrick Cloke
4d978d7db4 Merge branch 'master' into develop 2020-07-02 10:55:41 -04:00
reivilibre
e5808c4cfb
Hack to add push priority to push notifications (#7765) 
* Remove obsolete comment about ancient temporary code

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Implement hack to set push priority

based on whether the tweaks indicate the event might cause
effects.

* Changelog for 7765

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Antilint

* Add tests for push priority

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Update synapse/push/httppusher.py

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Antilint

* Remove needless invites from tests.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2020-07-01 17:02:31 +01:00
Richard van der Hoff
e866512367
Add early returns to _check_for_soft_fail (#7769) 
my editor was complaining about unset variables, so let's add some early
returns to fix that and reduce indentation/cognitive load.
 2020-07-01 16:41:19 +01:00
Richard van der Hoff
f01e2ca039
Use symbolic names for replication stream names (#7768) 
This makes it much easier to find where streams are referenced.
 2020-07-01 16:35:40 +01:00
Richard van der Hoff
a6eae69ffe
Type checking for FederationHandler (#7770) 
fix a few things to make this pass mypy.
 2020-07-01 16:21:02 +01:00
Erik Johnston
1e03513f9a
Fix new metric where we used ms instead of seconds (#7771) 
Introduced in #7755, not yet released.
 2020-07-01 15:23:58 +01:00
Richard van der Hoff
244dbb04f7
Fix incorrect error message when database CTYPE was set incorrectly. (#7760) 2020-07-01 13:56:16 +01:00
Andrew Morgan
dc80a0762d 1.16.0rc1 2020-07-01 11:26:58 +01:00
Brendan Abolivier
74d3e177f0
Back out MSC2625 implementation (#7761) 2020-07-01 11:08:25 +01:00
Patrick Cloke
71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston
a99658074d
Add some metrics for inbound and outbound federation processing times (#7755) 2020-06-30 16:58:06 +01:00
Andrew Morgan
2f6afdd8b4
Explain the purpose of the "tests" conditional dependency requirement (#7751) 2020-06-30 10:11:36 +01:00
Erik Johnston
831b31e563
Add another yield point to state res v2 (#7746) 2020-06-26 10:44:52 +01:00
Andrew Morgan
177b2d0c19
Move flake8 to end. Don't exit script on failure (#7738) 2020-06-25 17:58:55 +01:00
Dagfinn Ilmari Mannsåker
b099ef07d6
Make tox actions work on Debian 10 (#7703) 
- Remove the requirement for a specific version of Python
- Move dep comment to a separate line, Tox 3.7.0 like trailing ones

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
 2020-06-25 17:45:35 +01:00
Erik Johnston
0e0a2817a2
Yield during large v2 state res. (#7735) 
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
 2020-06-24 18:48:18 +01:00
Sorunome
6920e58136
add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675) 2020-06-24 10:23:55 +01:00
Christian Svensson
8bbe87f42d
Set Content-Length for Metrics requests (#7730) 
HTTP requires the response to contain a Content-Length header unless chunked encoding is being used.
Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics.

Signed-off-by: Christian Svensson <blue@cmd.nu>
 2020-06-23 18:06:01 +01:00
Patrick Cloke
24110255cd
Sync ignored table names in synapse_port_db to current database schema (#7717) 2020-06-23 07:33:25 -04:00
Patrick Cloke
95e41f368b
Allow local media to be marked as safe from being quarantined. (#7718) 2020-06-22 08:04:14 -04:00
Patrick Cloke
e060bf4462
Convert directory handler to async/await (#7727) 2020-06-22 07:18:00 -04:00
Erik Johnston
91e886d615
Speed up state res v2 across large state differences. (#7725) 2020-06-19 13:56:35 +01:00
Jesse Riddle
1b1489ff18
Fixed typo by adding a 'g' to PostgreSQL (#7724) 2020-06-19 07:19:21 -04:00
Brendan Abolivier
e35d44c01d
Merge pull request #7716 from matrix-org/babolivier/unread_fix 
Fix unread counts in sync
 2020-06-17 15:44:15 +01:00
Patrick Cloke
3630825612
Convert the typing handler to async/await. (#7679) 2020-06-17 10:37:59 -04:00
Oleg Girko
96bc110a68
Require parameterized package version to be at least 0.7.0. (#7680) 
Older versions of `parameterized` package have no `parameterized_class` decorator. This decorator is used in tests.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
 2020-06-17 15:31:40 +01:00
Brendan Abolivier
5a5cf6460e
Fix unread counts in sync 
* Always return an unread_count in get_unread_event_push_actions_by_room_for_user
* Don't always expect unread_count to be there so we don't take out sync entirely if something goes wrong
 2020-06-17 15:10:44 +01:00
Patrick Cloke
6418b0379f
Ignore the UI Auth sessions when porting from sqlite to postgresql (#7711) 2020-06-17 10:01:18 -04:00
Erik Johnston
e07a8caf58
Add support for using rust-python-jaeger-reporter (#7697) 2020-06-17 14:13:41 +01:00
Erik Johnston
b44bdd7f7b
Support running multiple media repos. (#7706) 
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
 2020-06-17 14:13:30 +01:00
Patrick Cloke
434716e1d3
Fetch from the r0 media path instead of the unspecced v1. (#7714) 2020-06-17 08:36:46 -04:00
lub
890c0c041d
Update postgres in the Docker compose example to 12-alpine. (#7696) 2020-06-17 08:29:08 -04:00
Brendan Abolivier
46613aaf79
Implement unread counter (MSC2625) (#7673) 
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
 2020-06-17 10:58:32 +01:00
Richard van der Hoff
e452973fd2
fix broken link in sample config (#7712) 2020-06-16 19:50:16 +01:00
Erik Johnston
f6f7511a4c
Refactor getting replication updates from database. (#7636) 
The aim here is to make it easier to reason about when streams are limited and when they're not, by moving the logic into the database functions themselves. This should mean we can kill of `db_query_to_update_function` function.
 2020-06-16 17:10:28 +01:00
Patrick Cloke
231252516c
Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708) 2020-06-16 12:01:18 -04:00
hungrymonkey
5c5516f80e
Add instructions for authing with Keycloak via OpenID (#7659) 2020-06-16 11:28:21 -04:00
Patrick Cloke
ac51bd581a
Include a user agent in federation requests. (#7677) 2020-06-16 10:43:29 -04:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Patrick Cloke
98c4e35e3c
Convert the device message and pagination handlers to async/await. (#7678) 2020-06-16 08:06:17 -04:00
Richard van der Hoff
03619324fc
Create a ListenerConfig object (#7681) 
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
 2020-06-16 12:44:07 +01:00
Brendan Abolivier
65eb078498
1.15.1 2020-06-16 10:28:58 +01:00
Andrew Morgan
3e6b5bba71
Wrap register_device coroutine in an ensureDeferred (#7684) 
Fixes https://github.com/matrix-org/synapse/issues/7683

Broke in: #7649

We had a `yield` acting on a coroutine. To be fair this one is a bit difficult to notice as there's a function in the middle that just passes the coroutine along.
 2020-06-16 10:13:59 +01:00
Patrick Cloke
cc32fa7358
Ensure the body is a string before comparing push rules. (#7701) 2020-06-15 16:20:34 -04:00
Hubert Chathi
2b2344652b
Ensure etag is a string for GET room_keys/version response (#7691) 2020-06-15 13:42:44 -04:00
Andrew Morgan
b8ee03caff
Update m.id.phone to use 'phone' instead of 'number' (#7687) 
The spec [states](https://matrix.org/docs/spec/client_server/r0.6.1#phone-number) that `m.id.phone` requires the field `country` and `phone`.

In Synapse, we've been enforcing `country` and `number`.

I am not currently sure whether this affects any client implementations.

This issue was introduced in #1994.
 2020-06-15 17:46:23 +01:00
Erik Johnston
4241a10673
Fix "There was no active span when trying to log." error (#7698) 2020-06-15 16:42:21 +01:00
Brendan Abolivier
6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:52 +01:00
Brendan Abolivier
c2b4621630 Merge branch 'develop' into babolivier/mark_unread 2020-06-15 16:37:08 +01:00
Brendan Abolivier
6d5985e1f2
Enable 3PID add/bind/unbind endpoints on r0 routes 2020-06-15 16:14:13 +01:00
Patrick Cloke
7d2532be36
Discard RDATA from already seen positions. (#7648) 2020-06-15 08:44:54 -04:00
Patrick Cloke
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Patrick Cloke
2d11ea385c
Fix warnings about losing log context during UI auth. (#7688) 2020-06-12 15:01:00 -04:00
Patrick Cloke
d0a43d431e
Fix a typo when comparing the URI & method during UI Auth. (#7689) 2020-06-12 14:12:04 -04:00
Brendan Abolivier
e47e5a2dcd
Incorporate review bits 2020-06-12 15:13:12 +01:00
Will Hunt
a3fbc23c39
Remove "user_id" from GET /presence. (#7606) 2020-06-11 14:13:53 -04:00
Patrick Cloke
b9df7f70bb
Increase the default SAML session expirary time to 15 minutes. (#7664) 2020-06-11 07:55:45 -04:00
wondratsch
c746889bb0
fix typo in sample_config.yaml (#7652) 
Just a simple typo fix.

Signed-off-by: wondratsch 28294257+wondratsch@users.noreply.github.com
 2020-06-11 11:51:10 +01:00
Richard van der Hoff
0df618f813
Take out a lock before modifying _CACHES (#7663) 
This should fix #7610.
 2020-06-10 18:27:49 +01:00
Brendan Abolivier
aad40e38e1
Changelog 2020-06-10 17:56:33 +01:00
Andrew Morgan
fcd6961441
Add option to enable encryption by default for new rooms (#7639) 
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
 2020-06-10 17:44:34 +01:00
Patrick Cloke
191dc98f80
Clean-up the fallback login code. (#7657) 2020-06-10 09:50:39 -04:00
Brendan Abolivier
ec0a7b9034 Merge branch 'develop' into babolivier/mark_unread 2020-06-10 11:42:30 +01:00
Brendan Abolivier
236d2d699d 1.15.0rc1 2020-06-09 16:37:14 +01:00
Brendan Abolivier
2dc9468c27 Revert "1.15.0rc1" 
This reverts commit 8587b0426f.
 2020-06-09 16:34:37 +01:00
Brendan Abolivier
8587b0426f 1.15.0rc1 2020-06-09 16:33:36 +01:00
Erik Johnston
664409b169
Fix bug in account data replication stream. (#7656) 
* Ensure account data stream IDs are unique.

The account data stream is shared between three tables, and the maximum
allocated ID was tracked in a dedicated table. Updating the max ID
happened outside the transaction that allocated the ID, leading to a
race where if the server was restarted then the same ID could be
allocated but the max ID failed to be updated, leading it to be reused.

The ID generators have support for tracking across multiple tables, so
we may as well use that instead of a dedicated table.

* Fix bug in account data replication stream.

If the same stream ID was used in both global and room account data then
the getting updates for the replication stream would fail due to
`heapq.merge(..)` trying to compare a `str` with a `None`. (This is
because you'd have two rows like `(534, '!room')` and `(534, None)` from
the room and global account data tables).

Fix is just to order by stream ID, since we don't rely on the ordering
beyond that. The bug where stream IDs can be reused should be fixed now,
so this case shouldn't happen going forward.

Fixes #7617
 2020-06-09 16:28:57 +01:00
Patrick Cloke
3c45a78090
Convert the registration handler to async/await. (#7649) 2020-06-08 11:15:02 -04:00
Patrick Cloke
375ca0cceb
Accept device information at the login fallback endpoint. (#7629) 2020-06-08 10:13:24 -04:00
Patrick Cloke
737b4a936e
Convert user directory handler and related classes to async/await. (#7640) 2020-06-05 14:42:55 -04:00
Travis Ralston
09099313e6
Add an option to disable autojoin for guest accounts (#6637) 
Fixes https://github.com/matrix-org/synapse/issues/3177
 2020-06-05 18:18:15 +01:00
Richard van der Hoff
1bc00fd76d
Clarifications to the admin api documentation (#7647) 
* Clarify how to authenticate
* path params are not the same thing as query params
* Fix documentation for `/_synapse/admin/v2/users/<user_id>`
 2020-06-05 17:31:05 +01:00
Patrick Cloke
a0d2d81cf9
Update to the stable SSO prefix for UI Auth. (#7630) 2020-06-05 10:50:08 -04:00
Richard van der Hoff
eea124370b
Fix type information on assert_*_is_admin methods (#7645) 
These things don't return Deferreds.
 2020-06-05 14:33:49 +01:00
Richard van der Hoff
b4f8dcb4bd
Remove some unused constants. (#7644) 2020-06-05 14:33:35 +01:00
Dirk Klimpel
908f9e2d24
Allow new users to be registered via the admin API even if the monthly active user limit has been reached (#7263) 2020-06-05 13:08:49 +01:00
Dirk Klimpel
2970ce8367
Add device management to admin API (#7481) 
- Admin is able to
  - change displaynames
  - delete devices
  - list devices
  - get device informations

Fixes #7330
 2020-06-05 13:07:22 +01:00
Patrick Cloke
02f345d053
Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. (#7634) 2020-06-05 07:36:47 -04:00
Andrew Morgan
139bc86f3d
Support CS API v0.6.0 (#6585) 2020-06-05 12:27:37 +01:00
WGH
e55ee7c32f
Add support for webp thumbnailing (#7586) 
Closes #4382

Signed-off-by: Maxim Plotnikov <wgh@torlan.ru>
 2020-06-05 11:54:27 +01:00
Andrew Morgan
f4e6495b5d
Performance improvements and refactor of Ratelimiter (#7595) 
While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both:

* Rather undocumented, and
* causing a *lot* of config checks

This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. 

Best to be reviewed commit-by-commit.
 2020-06-05 10:47:20 +01:00
Andrew Morgan
c389bfb6ea
Fix encryption algorithm typos in tests/comments (#7637) 
@uhoreg has confirmed these were both typos. They are only in comments and tests though, rather than anything critical.

Introduced in:

* https://github.com/matrix-org/synapse/pull/7157
* https://github.com/matrix-org/synapse/pull/5726
 2020-06-04 20:03:40 +01:00
Patrick Cloke
f8b9ead3ee
Advertise the token login type when OpenID Connect is enabled. (#7631) 2020-06-04 06:49:51 -04:00
Richard van der Hoff
11de843626
Cleanups to the OpenID Connect integration (#7628) 
docs, default configs, comments. Nothing very significant.
 2020-06-03 21:13:17 +01:00
Andrew Morgan
e91abfd291
async/await get_user_id_by_threepid (#7620) 
Based on #7619 

async's `get_user_id_by_threepid` and its call stack.
 2020-06-03 17:15:57 +01:00
Richard van der Hoff
86d814cdde
Check the changelog number in check-newsfragment (#7623) 2020-06-03 17:01:43 +01:00
Andrew Morgan
0188daf32c
Replace instances of reactor pumping with get_success. (#7619) 
Calls `self.get_success` on all deferred methods instead of abusing `self.pump()`. This has the benefit of working with coroutines, as well as checking that method execution completed successfully.

There are also a few small cleanups that I made in the process.
 2020-06-03 16:39:30 +01:00
Brendan Abolivier
c9507be989
Check if the localpart is reserved for guests earlier in the registration flow (#7625) 
This is so the user is warned about the username not being valid as soon as possible, rather than only once they've finished UIA.
 2020-06-03 16:55:02 +02:00
Erik Johnston
11dc2b4698
Fix exceptions when fetching events from a down host. (#7622) 
We already caught some exceptions, but not all.
 2020-06-03 14:12:13 +01:00
Richard van der Hoff
38d4ebbac7
synctl restart should start synapse if it wasn't running (#7624) 2020-06-03 13:16:15 +01:00
Richard van der Hoff
2a8ed93bd4
Switch back to upstream dh-virtualenv (#7621) 
Upstream have merged our changes
(https://github.com/spotify/dh-virtualenv/pull/300), so let's switch back to it
instead of using our fork.
 2020-06-03 12:21:58 +01:00
Richard van der Hoff
1bbc9e2df6
Clean up exception handling in SAML2ResponseResource (#7614) 
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
 2020-06-03 10:41:12 +01:00
Andrew Morgan
3e557447cb
Mention #synapse:matrix.org in README troubleshooting (#7603) 
Just in case people head straight to the troubleshooting section and find themselves at a dead end.
 2020-06-01 19:45:39 +01:00
Andrew Morgan
25e2d193e3
Advertise Python 3.8 support in setup.py (#7602) 
Synapse supports Python 3.8. We've been using it in CI for a while now.
 2020-06-01 19:45:01 +01:00
Olof Johansson
fe434cd3c9
Fix a bug in automatic user creation with m.login.jwt. (#7585) 2020-06-01 12:55:07 -04:00
Brendan Abolivier
33c39ab93c
Process cross-signing keys when resyncing device lists (#7594) 
It looks like `user_device_resync` was ignoring cross-signing keys from the results received from the remote server. This patch fixes this, by processing these keys using the same process `_handle_signing_key_updates` does (and effectively factor that part out of that function).
 2020-06-01 17:47:30 +02:00
Dirk Klimpel
901b1fa561
Email notifications for new users when creating via the Admin API. (#7267) 2020-06-01 15:34:33 +01:00
Dagfinn Ilmari Mannsåker
df8a3cef6b
Improve performance of _get_state_groups_from_groups_txn (#7567) 
The query keeps showing up in my slow query log.

This changes the plan under the top-level Sort node from

```
    WindowAgg  (cost=280335.88..292963.15 rows=561212 width=80) (actual time=138.651..160.562 rows=27112 loops=1)
      ->  Sort  (cost=280335.88..281738.91 rows=561212 width=84) (actual time=138.597..140.622 rows=27112 loops=1)
            Sort Key: state_groups_state.type, state_groups_state.state_key, state_groups_state.state_group
            Sort Method: quicksort  Memory: 4581kB
            ->  Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=21.548..47.657 rows=27112 loops=1)
                  ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=21.526..21.535 rows=20 loops=1)
                        Group Key: state.state_group
                        ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=21.280..21.493 rows=20 loops=1)
                  ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.005..0.991 rows=1356 loops=20)
                        Index Cond: (state_group = state.state_group)
```

to

```
    Nested Loop  (cost=2.83..226745.22 rows=561212 width=84) (actual time=24.194..52.834 rows=27112 loops=1)
      ->  HashAggregate  (cost=2.27..3.28 rows=101 width=8) (actual time=24.130..24.138 rows=20 loops=1)
            Group Key: state.state_group
            ->  CTE Scan on state  (cost=0.00..2.02 rows=101 width=8) (actual time=23.887..24.113 rows=20 loops=1)
      ->  Index Scan using state_groups_state_type_idx on state_groups_state  (cost=0.56..2189.40 rows=5557 width=84) (actual time=0.016..1.159 rows=1356 loops=20)
            Index Cond: (state_group = state.state_group)
```

This cuts the execution time from ~190ms to ~130ms, i.e. a reduction
of ~30%.

The full plans are visualised at https://explain.depesz.com/s/WpbT and
https://explain.depesz.com/s/KlEk

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
 2020-06-01 15:23:43 +01:00
Patrick Cloke
6af9cdca24
Convert groups local and server to async/await. (#7600) 2020-06-01 07:28:43 -04:00
Brendan Abolivier
c1bdd4fac7
Don't fail all of an iteration of the device list retry loop on error (#7609) 
Without this patch, if an error happens which isn't caught by `user_device_resync`, then `_maybe_retry_device_resync` would fail, without retrying the next users in the iteration. This patch fixes this so that it now only logs an error in this case.
 2020-06-01 12:55:14 +02:00
Dagfinn Ilmari Mannsåker
2dc430d36e
Use upsert when inserting read receipts (#7607) 
Fixes #7469

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
 2020-06-01 10:53:06 +01:00
hashashini
91a7c5ff6d
Update OpenBSD installation instructions (#7587) 
Synapse was added to the ports tree in Nov, 2019 by Renaud Allard (https://marc.info/?l=openbsd-ports&m=157417848805329).
With the release of OpenBSD 6.7 on May 22, 2020 a pre-compiled binary is available as well.
 2020-05-30 17:08:07 +01:00
Erik Johnston
cb495f526d
Fix 'FederationGroupsRoomsServlet' API when group has room server is not in. (#7599) 2020-05-29 17:49:47 +01:00
Erik Johnston
f5353eff21
Make inflight background metrics more efficient. (#7597) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2020-05-29 13:25:32 +01:00
David Rio Deiros
47db2c3673
Add entry to set dependency against psql service (#7591) 2020-05-28 16:02:41 +01:00
Erik Johnston
8c5f88fa4d
Merge pull request #7584 from matrix-org/erikj/save_and_send_fed_token_in_bg 
Speed up processing of federation stream RDATA rows.
 2020-05-27 20:06:29 +01:00
Erik Johnston
a6a40a1519 Newsfile 2020-05-27 19:35:03 +01:00
Christopher Cooper
c4a820b32a
allow emails to be passed through SAML (#7385) 
Signed-off-by: Christopher Cooper <cooperc@ocf.berkeley.edu>
 2020-05-27 17:40:08 +01:00
Andrew Morgan
0a6e837aaa
Fix incorrect placeholder syntax in database prepartion code (#7575) 
We were using `logger` syntax which isn't supported by `Exception`s.
 2020-05-27 16:26:59 +01:00
Brendan Abolivier
b4109499b4 1.14.0rc2 2020-05-27 17:22:28 +02:00
Jason Robinson
4be968d05d
Fix sample config docs error (#7581) 
'client_auth_method' commented out value was erronously 'client_auth_basic',
when code and docstring says it should be 'client_secret_basic'.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2020-05-27 13:52:18 +01:00
Erik Johnston
4ba55559ac
Fix specifying cache factors via env vars with * in name. (#7580) 
This mostly applise to `*stateGroupCache*` and co.

Broke in #6391.
 2020-05-27 13:17:01 +01:00
Erik Johnston
eefc6b3a0d
Don't apply cache factor to event cache. (#7578) 
This is already correctly done when we instansiate the cache, but wasn't
when it got reloaded (which always happens at least once on startup).
 2020-05-27 12:04:37 +01:00
Erik Johnston
9bac5d62b3
Ensure ReplicationStreamer is always started when replication enabled. (#7579) 
Fixes #7566.
 2020-05-27 11:44:19 +01:00
Brendan Abolivier
98483890ee Merge branch 'develop' of github.com:matrix-org/synapse into develop 2020-05-26 20:30:41 +02:00
Patrick Cloke
ef884f6d04
Convert identity handler to async/await. (#7561) 2020-05-26 13:46:22 -04:00
Brendan Abolivier
3b19c17247 1.14.0 2020-05-26 16:45:37 +02:00
Richard van der Hoff
edd9a7214c
Replace device_27_unique_idx bg update with a fg one (#7562) 
The bg update never managed to complete, because it kept being interrupted by
transactions which want to take a lock.

Just doing it in the foreground isn't that bad, and is a good deal simpler.
 2020-05-26 11:43:17 +01:00
Richard van der Hoff
04729b86f8
Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563) 
Introduced in #7556
 2020-05-26 11:42:22 +01:00
Richard van der Hoff
00db90f409
Fix recording of federation stream token (#7564) 
A couple of changes of significance:

 * remove the `_last_ack < federation_position` condition, so that
   updates will still be correctly processed after restart

 * Correctly wire up send_federation_ack to the right class.
 2020-05-26 11:41:38 +01:00
Richard van der Hoff
d14c4d6b6d
Simplify reap_monthly_active_users (#7558) 
we can use `make_in_list_sql_clause` rather than doing our own half-baked
equivalent, which has the benefit of working just fine with empty lists.

(This has quite a lot of tests, so I think it's pretty safe)
 2020-05-23 01:20:10 +01:00
Richard van der Hoff
f4269694ce
Optimise some references to hs.config (#7546) 
These are surprisingly expensive, and we only really need to do them at startup.
 2020-05-22 21:47:07 +01:00
Erik Johnston
2901f54359
Fix missing CORS headers on OPTION responses (#7560) 
Broke in #7534.
 2020-05-22 17:42:39 +01:00
Erik Johnston
e5c67d04db
Add option to move event persistence off master (#7517) 2020-05-22 16:11:35 +01:00
Patrick Cloke
4429764c9f
Return 200 OK for all OPTIONS requests (#7534) 2020-05-22 09:30:07 -04:00
Erik Johnston
1531b214fc
Add ability to wait for replication streams (#7542) 
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).

Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.

People probably want to look at this commit by commit.
 2020-05-22 14:21:54 +01:00
Erik Johnston
06a02bc1ce
Convert sending mail to async/await. (#7557) 
Mainly because sometimes the email push code raises exceptions where the
stack traces have gotten lost, which is hopefully fixed by this.
 2020-05-22 13:41:11 +01:00
Patrick Cloke
66f2ebc22f
Use a non-empty RelayState for user interactive auth with SAML. (#7552) 2020-05-22 07:17:30 -04:00
Erik Johnston
710d958c64
On upgrade room only send canonical alias once. (#7547) 
Instead of doing a complicated dance of deleting and moving aliases one
by one, which sends a canonical alias update into the old room for each
one, lets do it all in one go.

This also changes the function to move *all* local alias events to the new
room, however that happens later on anyway.
 2020-05-22 11:41:41 +01:00
Erik Johnston
547e4dd83e
Fix exception reporting due to HTTP request errors. (#7556) 
These are business as usual errors, rather than stuff we want to log at
error.
 2020-05-22 11:39:20 +01:00
Ivan Shapovalov
ac481a738e
synapse.metrics: implement detailed memory usage reporting on PyPy (#7536) 
PyPy's gc.get_stats() returns an object containing detailed allocator statistics
which could be beneficial to collect as metrics.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
 2020-05-22 11:08:41 +01:00
Richard van der Hoff
8c75da916c
Refresh apt cache when building dh_virtualenv docker image (#7555) 
When we tried to build debs for 1.13.0, the build failed because docker used a
base docker image which had a stale apt cache.

Fixes: #7540
 2020-05-22 10:17:47 +01:00
Richard van der Hoff
a0f99f81b3
Fix stacktrace mangling in patch_inline_callbacks (#7554) 
`Failure()` is more cunning than `Failure(e)`.
 2020-05-22 10:17:36 +01:00
Richard van der Hoff
d84bdfe599
mypy for synapse.http.site (#7553) 2020-05-22 10:12:17 +01:00
Richard van der Hoff
66a564c859
Fix some DETECTED VIOLATIONS in the config file (#7550) 
consistency ftw
 2020-05-22 10:11:50 +01:00
Brendan Abolivier
d1ae1015ec
Retry to sync out of sync device lists (#7453) 
When a call to `user_device_resync` fails, we don't currently mark the remote user's device list as out of sync, nor do we retry to sync it.

https://github.com/matrix-org/synapse/pull/6776 introduced some code infrastructure to mark device lists as stale/out of sync.

This commit uses that code infrastructure to mark device lists as out of sync if processing an incoming device list update makes the device handler realise that the device list is out of sync, but we can't resync right now.

It also adds a looping call to retry all failed resync every 30s. This shouldn't cause too much spam in the logs as this commit also removes the "Failed to handle device list update for..." warning logs when catching `NotRetryingDestination`.

Fixes #7418
 2020-05-21 17:41:12 +02:00
Richard van der Hoff
0bbbd10513
Stub out GET presence requests in the frontend proxy (#7545) 
We don't really make any promises about returning accurate presence data when
presence is disabled, so we may as well just return a static response, rather
than making the master handle a request.
 2020-05-21 14:36:46 +01:00
David Vo
d74cdc1a42
Ensure worker config exists in systemd service (#7528) 2020-05-21 13:47:23 +01:00
Erik Johnston
f6f92845f8
Fix bug in persist events when dealing with non member types. (#7548) 
`_is_server_still_joined` will throw if it is given state updates with non-user ID state keys with local user leaves. This is actually rarely a problem since local leaves almost always get persisted by themselves.

(I discovered this on a branch that was otherwise broken, so I haven't seen this in the wild)
 2020-05-21 13:20:10 +01:00
Patrick Cloke
b2b8699070
Remove Ubuntu Cosmic and Disco which are both EOL. (#7539) 2020-05-20 10:08:46 -04:00
Patrick Cloke
9dc6f3075a
Hash passwords earlier in the password reset process (#7538) 
This now matches the logic of the registration process as modified in
56db0b1365 / #7523.
 2020-05-20 09:48:03 -04:00
Richard van der Hoff
4fa74c7606
Minor clarifications to the TURN docs (#7533) 2020-05-20 11:04:34 +01:00
Romain Bouyé
a57863d2b4
synctl warns when no process is stopped and avoids start (#6598) 
* If an error occurs when stopping a process synctl now logs a warning.
* During a restart, synctl will avoid attempting to start Synapse if an error
  occurs during stopping Synapse.
 2020-05-19 08:47:45 -04:00
Aaron Raimist
250f3eb991
Omit displayname or avatar_url if they aren't set instead of returning null (#7497) 
Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec.

Fixes https://github.com/matrix-org/synapse/issues/7333

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2020-05-19 10:31:25 +01:00
Erik Johnston
51055c8c44
Allow ReplicationRestResource to be added to workers (#7515) 
This allows workers to talk to each other over HTTP replication.
 2020-05-18 12:24:48 +01:00
Richard van der Hoff
4d1afb1dfe
Merge pull request #7519 from matrix-org/rav/kill_py2_code 
Kill off some old python 2 code
 2020-05-18 10:45:30 +01:00
Richard van der Hoff
164f50f5f2
fix mypy for tests/replication (#7518) 2020-05-18 10:43:05 +01:00
Patrick Cloke
c29915bd05
Add type hints to room member handlers (#7513) 2020-05-15 15:05:25 -04:00
Richard van der Hoff
ab57353de3 changelog 2020-05-15 19:37:41 +01:00
Richard van der Hoff
6c1f7c722f
Fix limit logic for AccountDataStream (#7384) 
Make sure that the AccountDataStream presents complete updates, in the right
order.

This is much the same fix as #7337 and #7358, but applied to a different stream.
 2020-05-15 19:03:25 +01:00
Patrick Cloke
a3cf36f76e
Support UI Authentication for OpenID Connect accounts (#7457) 2020-05-15 12:26:02 -04:00
Erik Johnston
03aff4c75e
Add a worker store for search insertion. (#7516) 
This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
 2020-05-15 17:22:47 +01:00
Andrew Morgan
16090a077f
Prevent 0-member/null room_version rooms from appearing in group room queries (#7465) 2020-05-15 17:17:42 +01:00
Erik Johnston
1f36ff69e8
Move event stream handling out of slave store. (#7491) 
This allows us to have the logic on both master and workers, which is necessary to move event persistence off master.

We also combine the instantiation of ID generators from DataStore and slave stores to the base worker stores. This allows us to select which process writes events independently of the master/worker splits.
 2020-05-15 16:43:59 +01:00
Patrick Cloke
5355421295
Add type hints to event_auth code. (#7505) 2020-05-15 11:19:43 -04:00
Andrew Morgan
86614e251f
Fix a small typo in the arguments of simple_update in update_remote_profile_cache (#7511) 2020-05-15 16:17:12 +01:00
Richard van der Hoff
24d9151a08
Formatting for reverse-proxy docs (#7514) 
also a small clarification to nginx
 2020-05-15 15:13:39 +01:00
Jeff Peeler
572b444dab
Add Caddy 2 example (#7463) 
The specific headers that are passed using this new configuration format
are Host and X-Forwarded-For, which should be all that's required.

Note that for production another matcher should be added in the first
section to properly handle the base_url lookup:
reverse_proxy /.well-known/matrix/* http://localhost:8008

Signed-off-by: Jeff Peeler <jpeeler@gmail.com>
 2020-05-15 14:36:01 +01:00
Patrick Cloke
e9f3de0bab
Update the room member handler to use async/await. (#7507) 2020-05-15 09:32:13 -04:00
Patrick Cloke
08bc80ef09
Implement room version 6 (MSC2240). (#7506) 2020-05-15 09:30:10 -04:00
Andrew Morgan
02d97fc3ba
Ignore incoming presence updates when presence is disabled (#7508) 2020-05-15 11:44:00 +01:00
Patrick Cloke
56b66db78a
Strictly enforce canonicaljson requirements in a new room version (#7381) 2020-05-14 13:24:01 -04:00
Patrick Cloke
fef3ff5cc4
Enforce MSC2209: auth rules for notifications in power level event (#7502) 
In a new room version, the "notifications" key of power level events are
subject to restricted auth rules.
 2020-05-14 12:38:17 -04:00
Andrew Morgan
5611644519
Workaround for failure to wrap reason in Failure (#7473) 2020-05-14 17:07:24 +01:00
Richard van der Hoff
eafd103fc7
Fix b'GET' in prometheus metrics (#7503) 2020-05-14 17:01:34 +01:00
Andrew Morgan
225c165087
Allow expired accounts to logout (#7443) 2020-05-14 16:32:49 +01:00
Erik Johnston
4734a7bbe4
Move EventStream handling into default ReplicationDataHandler (#7493) 
This is so that the logic can happen on both master and workers when we move event persistence out.
 2020-05-14 14:01:39 +01:00
Erik Johnston
1de36407d1
Add instance_map config and route replication calls (#7495) 2020-05-14 14:00:58 +01:00
Erik Johnston
1124111a12
Allow censoring of events to happen on workers. (#7492) 
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
 2020-05-13 17:15:40 +01:00
Paul Tötterman
46cb2550bb
Fix copypasted comment (#7477) 
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
 2020-05-13 16:55:43 +01:00
Erik Johnston
18c1e52d82
Clean up replication unit tests. (#7490) 2020-05-13 16:01:47 +01:00
Erik Johnston
782e4e64df
Shuffle persist event data store functions. (#7440) 
The aim here is to get to a stage where we have a `PersistEventStore` that holds all the write methods used during event persistence, so that we can take that class out of the `DataStore` mixin and instansiate it separately. This will allow us to instansiate it on processes other than master, while also ensuring it is only available on processes that are configured to write to events stream.

This is a bit of an architectural change, where we end up with multiple classes per data store (rather than one per data store we have now). We end up having:

1. Storage classes that provide high level APIs that can talk to multiple data stores.
2. Data store modules that consist of classes that must point at the same database instance.
3. Classes in a data store that can be instantiated on processes depending on config.
 2020-05-13 13:38:22 +01:00
Erik Johnston
7ee24c5674
Have all instances correctly respond to REPLICATE command. (#7475) 
Before all streams were only written to from master, so only master needed to respond to `REPLICATE` commands.

Before all instances wrote to the cache invalidation stream, but didn't respond to `REPLICATE`. This was a bug, which could lead to missed rows from cache invalidation stream if an instance is restarted, however all the caches would be empty in that case so it wasn't a problem.
 2020-05-13 10:27:02 +01:00
Erik Johnston
8ca79613e6
Fix Redis reconnection logic (#7482) 
Proactively send out `POSITION` commands (as if we had just received a `REPLICATE`) when we connect to Redis. This is important as other instances won't notice we've connected to issue a `REPLICATE` command (unlike for direct TCP connections). This is only currently an issue if master process reconnects without restarting (if it restarts then it won't have written anything and so other instances probably won't have missed anything).
 2020-05-13 09:57:15 +01:00
Patrick Cloke
51fb0fc2e5
Update documentation about SSO mapping providers (#7458) 2020-05-12 10:51:07 -04:00
Erik Johnston
1a1da60ad2
Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Patrick Cloke
8c8858e124
Convert federation handler to async/await. (#7459) 2020-05-11 15:12:46 -04:00
Patrick Cloke
be309d99cf
Convert search code to async/await. (#7460) 2020-05-11 15:12:39 -04:00
Amber Brown
7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Andrew Morgan
a8580c5f19
Remove unused store method get_hosts_in_room (#7448) 2020-05-11 16:55:57 +01:00
Andrew Morgan
5cf758cdd6 Merge branch 'release-v1.13.0' into develop 
* release-v1.13.0:
  Don't UPGRADE database rows
  RST indenting
  Put rollback instructions in upgrade notes
  Fix changelog typo
  Oh yeah, RST
  Absolute URL it is then
  Fix upgrade notes link
  Provide summary of upgrade issues in changelog. Fix )
  Move next version notes from changelog to upgrade notes
  Changelog fixes
  1.13.0rc1
  Documentation on setting up redis (#7446)
  Rework UI Auth session validation for registration (#7455)
  Fix errors from malformed log line (#7454)
  Drop support for redis.dbid (#7450)
 2020-05-11 16:46:33 +01:00
Andrew Morgan
20ffaa7209 1.13.0rc1 2020-05-11 14:54:38 +01:00
Neil Johnson
85155654c5
Documentation on setting up redis (#7446) 2020-05-11 13:21:15 +01:00
Patrick Cloke
0ad6d28b0d
Rework UI Auth session validation for registration (#7455) 
Be less strict about validation of UI authentication sessions during
registration to match client expecations.
 2020-05-08 16:08:58 -04:00
Andrew Morgan
67feea8044
Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Quentin Gliech
616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Manuel Stahl
a4a5ec4096
Add room details admin endpoint (#7317) 2020-05-07 15:33:07 -04:00
Richard van der Hoff
aa5aa6f96a
Fix errors from malformed log line (#7454) 2020-05-07 19:51:38 +01:00
Richard van der Hoff
da9b2db3af
Drop support for redis.dbid (#7450) 
Since we only use pubsub, the dbid is irrelevant.
 2020-05-07 16:46:15 +01:00
Brendan Abolivier
5bb26b7c4f Merge branch 'release-v1.13.0' into develop 2020-05-07 17:31:19 +02:00
Patrick Cloke
9e0384dd3f
Fixes typo (bellow -> below) (#7449) 2020-05-07 09:31:06 -04:00
Patrick Cloke
22246919e3
Add more type hints to SAML handler. (#7445) 2020-05-07 09:30:45 -04:00
Erik Johnston
d7983b63a6
Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Brendan Abolivier
2929ce29d6
Merge pull request #7398 from Starbix/alpine-3.11 
Update docker runtime image to Alpine v3.11
 2020-05-07 11:56:56 +02:00
Brendan Abolivier
d9b8d27494
Add a configuration setting for the dummy event threshold (#7422) 
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
 2020-05-07 10:35:23 +01:00
Patrick Cloke
d7c2df2fa3
Improve per-block CPU and DB usage metrics (#7426) 2020-05-06 16:43:39 -04:00
Andrew Morgan
4162c39dcf
Port group attestation renewal slow down from matrix-org-hotfixes (#7442) 2020-05-06 20:21:38 +01:00
Richard van der Hoff
e053c86a96
Make redis go faster with hiredis (#7439) 
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
 2020-05-06 17:36:46 +01:00
Richard van der Hoff
62ee862119 Merge branch 'release-v1.13.0' into develop 2020-05-06 15:56:03 +01:00
Andrew Morgan
aee9130a83
Stop Auth methods from polling the config on every req. (#7420) 2020-05-06 15:54:58 +01:00
Richard van der Hoff
fa0b2bd28d
Merge pull request #7428 from matrix-org/rav/cross_signing_keys_cache 
Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
 2020-05-06 12:00:01 +01:00
Richard van der Hoff
16b67c404d Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk 
... mostly because the latter has a cache.
 2020-05-06 11:59:19 +01:00
Richard van der Hoff
2e0c46ca07 Merge branch 'release-v1.13.0' into develop 2020-05-06 11:58:31 +01:00
Richard van der Hoff
30a19daa02 Merge branch 'develop' into rav/upsert_for_device_list 2020-05-06 11:43:11 +01:00
Richard van der Hoff
e48361545d use an upsert to update device_lists_outbound_last_success 2020-05-06 11:41:23 +01:00
Erik Johnston
b26f3e582c
Merge pull request #7423 from matrix-org/erikj/faster_device_lists_fetch 
Speed up fetching device lists changes in sync.
 2020-05-06 11:14:13 +01:00
Richard van der Hoff
a8c17da245 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 23:01:12 +01:00
Richard van der Hoff
1242267316 Merge branch 'release-v1.13.0' into rav/fix_dropped_messages 2020-05-05 22:38:44 +01:00
Richard van der Hoff
7bf788ac73 changelog 2020-05-05 22:38:16 +01:00
Brendan Abolivier
5b8023dc7f
Move logs about discarded RDATA to debug (#7421) 2020-05-05 21:07:33 +02:00
Richard van der Hoff
13dd458b8d Merge branch 'release-v1.13.0' into erikj/faster_device_lists_fetch 2020-05-05 18:14:00 +01:00
Richard van der Hoff
714560e325
Update changelog.d/7423.misc 2020-05-05 18:03:59 +01:00
Erik Johnston
79fe3e068b Newsfile 2020-05-05 17:40:29 +01:00
Richard van der Hoff
16b1a34e80
Fix typing annotations in synapse/federation (#7382) 
We're pretty close to having mypy working for `synapse.federation`, so let's
finish the job.
 2020-05-05 14:27:13 +01:00
Patrick Cloke
fe69fb6263
Add backwards compatibility codepath to LoggingContext. (#7408) 2020-05-05 09:21:34 -04:00
Erik Johnston
7941a70fa8
Fix bug in EventContext.deserialize. (#7393) 
This caused `prev_state_ids` to be incorrect if the state event was not
replacing an existing state entry.
 2020-05-05 14:17:27 +01:00
Richard van der Hoff
d5aa7d93ed
Fix catchup-on-reconnect for the Federation Stream (#7374) 
looks like we managed to break this during the refactorathon.
 2020-05-05 14:15:57 +01:00
Erik Johnston
8123b2f909
Add MultiWriterIdGenerator. (#7281) 
This will be used to coordinate stream IDs across multiple writers.

Functions as the equivalent of both `StreamIdGenerator` and
`SlavedIdTracker`.
 2020-05-04 17:17:45 +01:00
Brendan Abolivier
15aa09bbe6 Merge branch 'release-v1.13.0' into develop 2020-05-04 16:33:56 +02:00
Brendan Abolivier
ad088716bc
Merge pull request #7404 from matrix-org/babolivier/fix_manifest 
Fix MANIFEST.in
 2020-05-04 16:24:15 +02:00
Brendan Abolivier
068da604c2
Fix MANIFEST.in 
An update of check-manifest shone some light on some issues with MANIFEST.in, specifically that we didn't ignore/prune the contrib directory, and that we were using prune instead of exclude for files. This fixes both issues.

Fixes #7403
 2020-05-04 15:18:06 +02:00
Erik Johnston
350421e058
Fix redis password support. (#7401) 
We forgot to set the password on the subscriber connection, as well as
not calling super methods for overridden connectionMade/connectionLost
functions.
 2020-05-04 14:04:09 +01:00
Patrick Cloke
eab59d758d
Convert the room handler to async/await. (#7396) 2020-05-04 07:43:52 -04:00
Cédric Laubacher
a251e0f4ba
Update runtime docker image to Alpine v3.11 2020-05-03 16:07:24 +02:00
Patrick Cloke
032e5a2aca
Convert synapse.server_notices to async/await. (#7394) 2020-05-01 15:28:59 -04:00
Patrick Cloke
b0cbc57375
Convert the synapse.notifier module to async/await. (#7395) 2020-05-01 15:14:49 -04:00
Richard van der Hoff
97ef1471c6
Wait for current_state_events_membership before populate_stats_process_rooms (#7387) 
populate_stats_process_rooms was added in #5971 / v1.4.0; current_state_events_membership was added in #5706 / v1.3.0.

Fixes #7380.
 2020-05-01 18:05:58 +01:00
Erik Johnston
0e719f2398
Thread through instance name to replication client. (#7369) 
For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
 2020-05-01 17:19:56 +01:00
Erik Johnston
3085cde577
Use stream.current_token() and remove stream_positions() (#7172) 
We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
 2020-05-01 15:21:35 +01:00
Andrew Morgan
6b22921b19
async/await is_server_admin (#7363) 2020-05-01 15:15:36 +01:00
Andrew Morgan
2e8955f4a6
Further improvements to requesting the public rooms list on a homeserver which has it set to private (#7368) 2020-05-01 15:15:08 +01:00
Richard van der Hoff
b2dba06079
Workaround for assertion errors from db_query_to_update_function (#7378) 
Hopefully this is no worse than what we have on master...
 2020-05-01 09:25:16 +01:00
Patrick Cloke
627b0f5f27
Persist user interactive authentication sessions (#7302) 
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
 2020-04-30 13:47:49 -04:00
Andrew Morgan
9d8ecc9e6c
Apply federation check for /publicRooms with filter list (#7367) 2020-04-30 11:38:07 +01:00
Erik Johnston
37f6823f5b
Add instance name to RDATA/POSITION commands (#7364) 
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
 2020-04-29 16:23:08 +01:00
Erik Johnston
3eab76ad43
Don't relay REMOTE_SERVER_UP cmds to same conn. (#7352) 
For direct TCP connections we need the master to relay REMOTE_SERVER_UP
commands to the other connections so that all instances get notified
about it. The old implementation just relayed to all connections,
assuming that sending back to the original sender of the command was
safe. This is not true for redis, where commands sent get echoed back to
the sender, which was causing master to effectively infinite loop
sending and then re-receiving REMOTE_SERVER_UP commands that it sent.

The fix is to ensure that we only relay to *other* connections and not
to the connection we received the notification from.

Fixes #7334.
 2020-04-29 14:10:59 +01:00
Richard van der Hoff
c2e1a2110f
Fix limit logic for EventsStream (#7358) 
* Factor out functions for injecting events into database

I want to add some more flexibility to the tools for injecting events into the
database, and I don't want to clutter up HomeserverTestCase with them, so let's
factor them out to a new file.

* Rework TestReplicationDataHandler

This wasn't very easy to work with: the mock wrapping was largely superfluous,
and it's useful to be able to inspect the received rows, and clear out the
received list.

* Fix AssertionErrors being thrown by EventsStream

Part of the problem was that there was an off-by-one error in the assertion,
but also the limit logic was too simple. Fix it all up and add some tests.
 2020-04-29 12:30:36 +01:00
Andrew Morgan
eeef9633af
Fix fallback value for account_threepid_delegates.email (#7316) 2020-04-29 11:30:06 +01:00
Andrew Morgan
c58ae367d8
Clean up admin api docs (#7361) 2020-04-28 20:06:03 +01:00
Manuel Stahl
04dd7d182d
Return total number of users and profile attributes in admin users endpoint (#6881) 
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2020-04-28 18:19:36 +01:00
Erik Johnston
fce663889b
Add some replication tests (#7278) 
Specifically some tests for the typing stream, which means we test streams that fetch missing updates via HTTP (rather than via the DB).

We also shuffle things around a bit so that we create two separate `HomeServer` objects, rather than trying to insert a slaved store into places.

Note: `test_typing.py` is heavily inspired by `test_receipts.py`
 2020-04-28 17:42:03 +01:00
Richard van der Hoff
fb8ff79efd
Fix collation for postgres for unit tests (#7359) 
When running the UTs against a postgres deatbase, we need to set the collation
correctly.
 2020-04-28 14:21:48 +01:00
Erik Johnston
38919b521e
Run replication streamers on workers (#7146) 
Currently we never write to streams from workers, but that will change soon
 2020-04-28 13:34:12 +01:00
Richard van der Hoff
07337fe30b
Fix incorrect metrics reporting for renew_attestations (#7344) 
We need to wait for the renewals to finish, so that the metrics are correctly
reported.
 2020-04-27 22:20:10 +01:00
Brendan Abolivier
036fab5d8a
Document monitoring workers (#7357) 
It doesn't seem to be documented anywhere and means that you suddenly start losing metrics without any obvious reason when you go from monolith to workers (e.g. #7312).
 2020-04-27 21:36:47 +02:00
lub
aa2492907f
Add some explanation to application_services.md (#7091) 
Signed-off-by: Simon Körner <git@lubiland.de>
 2020-04-27 15:03:09 +01:00
Brendan Abolivier
cc9eceb00d
Don't crash when one of the configuration files is empty (#7341) 
If the admin adds a `.yaml` file that's either empty or doesn't parse into a dict to a config directory (e.g. `conf.d` for debs installs), stuff like https://github.com/matrix-org/synapse/issues/7322 would happen. This PR checks that the file is correctly parsed into a dict, or ignores it with a warning if it parses into any other type (including `None` for empty files).

Fixes https://github.com/matrix-org/synapse/issues/7322
 2020-04-27 15:01:03 +02:00
Patrick Cloke
7bfe0902ce
Add documentation to the sample config about the templates for SSO. (#7343) 2020-04-24 15:03:49 -04:00
Patrick Cloke
33bceb7f70
Convert some of the federation handler methods to async/await. (#7338) 2020-04-24 14:36:38 -04:00
Richard van der Hoff
3655eafe85 changelog 2020-04-24 13:59:21 +01:00
Brendan Abolivier
2e3b9a0fcb
Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token"" 
This reverts commit 1adf6a5587.
 2020-04-23 11:23:53 +02:00
Richard van der Hoff
71a1abb8a1
Stop the master relaying USER_SYNC for other workers (#7318) 
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.

In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.

Fixes (I hope) #7257.
 2020-04-22 22:39:04 +01:00
Erik Johnston
841c581c40
Fix replication metrics when using redis (#7325) 2020-04-22 16:26:19 +01:00
Richard van der Hoff
f16beaa969 Synapse 1.12.4rc1 (2020-04-22) 
==============================
 
 Features
 --------
 
 - Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
 - Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))
 
 Bugfixes
 --------
 
 - Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
 - Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
 - Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl6gS+IACgkQOSor00I9
 eP8CTwf+KLehHQMcrz0yoHN+/O86mpNv3Uz83BJ0qDIgezVbtqT7+ZwnK2Jb8dwM
 zTcGRq5a6V+JLdfSTnXfrPzvAPt0WLaEENSAEGwZ6unqqxcG2PgATkRb0XtY8Rzh
 Feu8WgJTnORbiiM3sBLI8toZMxal3X3ZcftPEAan1SaoyqLPQ2s14edDwgx64S5e
 rBGzPDNSxZU1McNBq7Q11QTg3zgBUF+TYwyhnLbC5X1RdDwQpKhNZhwhYBS1yOD6
 TsPjtlixC+HXb21Rmob7XY8VFxTBSBdUdtVQKAT4h3+l9bNFQLpBfDSVc7xkPiLt
 /Gj+PL+YgY9xQDvHO2fVw8m4ololZw==
 =p53e
 -----END PGP SIGNATURE-----

Merge tag 'v1.12.4rc1' into develop

Synapse 1.12.4rc1 (2020-04-22)
==============================

Features
--------

- Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
- Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))

Bugfixes
--------

- Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
- Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
- Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
 2020-04-22 14:54:08 +01:00
Richard van der Hoff
82d8b1dd1f
Another go at fixing one-word commands (#7326) 
I messed this up last time I tried (#7239 / e13c6c7).
 2020-04-22 14:34:31 +01:00
Richard van der Hoff
dc8003f921 1.12.4rc1 2020-04-22 14:29:39 +01:00
Richard van der Hoff
7c7618c7e3 fix changelog name 2020-04-22 14:01:11 +01:00
Richard van der Hoff
13683a3a22
Extend StreamChangeCache to support multiple entities per stream ID (#7303) 
First some background: StreamChangeCache is used to keep track of what "entities" have 
changed since a given stream ID. So for example, we might use it to keep track of when the last
to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2].

Now, it turns out that StreamChangeCache didn't support more than one thing being changed at
a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send
to-device messages to more than one user at a time.

As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped
ok with having multiple things changing on the same stream ID, and it seems we never actually
use the methods which don't work on the stream change caches where we allow multiple
changes at the same stream ID. But that feels horribly fragile, hence: let's update
StreamChangeCache to properly support this, and add some typing and some more tests while
we're at it.

[1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301
[2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
 2020-04-22 13:45:40 +01:00
Dirk Klimpel
6b6685db9f
Extend room admin api with additional attributes (#7225) 2020-04-22 13:38:41 +01:00
Richard van der Hoff
2aa5bf13c8 Merge branch 'release-v1.12.4' into develop 2020-04-22 13:09:23 +01:00
Erik Johnston
51f7eaf908
Add ability to run replication protocol over redis. (#7040) 
This is configured via the `redis` config options.
 2020-04-22 13:07:41 +01:00
Patrick Cloke
51f358e2fe Do not treat display names as globs for push rules. (#7271) 2020-04-22 13:07:12 +01:00
Michael Kaye
5308239d5d
Reduce logging verbosity of URL cache cleanup. (#7295) 2020-04-22 07:45:16 -04:00
Andrew Morgan
f89ad3b6df
Query missing cross-signing keys on local sig upload (#7289) 2020-04-22 12:29:36 +01:00
Andrew Morgan
ff5604e7f1
import urllib.parse when using urllib.parse.quote (#7319) 2020-04-22 12:18:51 +01:00
Michael Kaye
336989a57f
Reduce federation logging on success (#7321) 
Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
 2020-04-22 11:18:18 +01:00
Richard van der Hoff
556566f0b8 Fix changelog file 
I updated the PR and forgot to update the changelog.
 2020-04-21 13:20:16 +01:00
Richard van der Hoff
974c0d726a
Support GET account_data requests on a worker (#7311) 2020-04-21 10:46:30 +01:00
Richard van der Hoff
461f01ad43 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:55:39 +01:00
Richard van der Hoff
d41c8f6d4d Revert "Query missing cross-signing keys on local sig upload" 
This was incorrectly merged to the release branch before it was ready.

This reverts commit 72fe2affb6.
 2020-04-20 17:54:35 +01:00
Richard van der Hoff
a46ff43319 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:42:27 +01:00
David Baker
40f79f58bf Always send the user updates to their own device list (#7160) 2020-04-20 17:20:38 +01:00
Lars Franke
13917232d5
Fix indention in generated config file (#7300) 
Also adjust sample_config.yaml

Signed-off-by: Lars Franke <frcl@mailbox.org>
 2020-04-20 16:51:27 +01:00
Patrick Cloke
f5ea8b48bd
Reject unknown UI auth sessions (instead of silently generating a new one) (#7268) 2020-04-20 08:54:42 -04:00
Richard van der Hoff
0f8f02bc39
On catchup, process each row with its own stream id (#7286) 
Other parts of the code (such as the StreamChangeCache) assume that there will
not be multiple changes with the same stream id.

This code was introduced in #7024, and I hope this fixes #7206.
 2020-04-20 11:43:29 +01:00
Patrick Cloke
054c231e58
Use a template for the SSO success page to allow for customization. (#7279) 2020-04-17 13:34:55 -04:00
James
701788a227
Added explicit Python build tools to snap requirements (#7213) 
Signed-off-by: James Hebden <james@ec0.io>
 2020-04-17 17:28:00 +01:00
Andrew Morgan
72fe2affb6 Query missing cross-signing keys on local sig upload 
Add changelog

Save retrieved keys to the db

lint

Fix and de-brittle remote result dict processing

Use query_user_devices instead, assume only master, self_signing key types

Make changelog more useful

Remove very specific exception handling

Wrap get_verify_key_from_cross_signing_key in a try/except

Note that _get_e2e_cross_signing_verify_key can raise a SynapseError

lint

Add comment explaining why this is useful

Only fetch master and self_signing key types

Fix log statements, docstrings

Remove extraneous items from remote query try/except

lint

Factor key retrieval out into a separate function

Send device updates, modeled after SigningKeyEduUpdater._handle_signing_key_updates

Update method docstring
 2020-04-17 15:47:49 +01:00
Richard van der Hoff
67ff7b8ba0
Improve type checking in replication.tcp.Stream (#7291) 
The general idea here is to get rid of the type: ignore annotations on all of the current_token and update_function assignments, which would have caught #7290.

After a bit of experimentation, it seems like the least-awful way to do this is to pass the offending functions in as parameters to the Stream constructor. Unfortunately that means that the concrete implementations no longer have the same constructor signature as Stream itself, which means that it gets hard to correctly annotate STREAMS_MAP.

I've also introduced a couple of new types, to take out some duplication.
 2020-04-17 14:49:55 +01:00
Tristan Lins
c07fca9e2f
Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Patrick Cloke
01294e6b3a
Do not treat display names as globs for push rules. (#7271) 2020-04-16 10:52:55 -04:00
Richard van der Hoff
d7d42387f5
Fix 'generator object is not subscriptable' error (#7290) 
Some of the query functions return generators rather than lists, so we can't
index into the result. Happily we already have a copy of the results.

(think this was introduced in #7024)
 2020-04-16 14:37:06 +01:00
Patrick Cloke
eed7c5b89e
Convert auth handler to async/await (#7261) 2020-04-15 12:40:18 -04:00
Patrick Cloke
17a2433b0d
Add notes to the changelog about an additional SSO template. (#7259) 2020-04-15 10:36:33 -04:00
Andrew Morgan
a48138784e
Allow specifying the value of Accept-Language header for URL previews (#7265) 2020-04-15 13:35:29 +01:00
Richard van der Hoff
f2049a8d21
Fix a potentially-huge sql query (#7274) 
We could end up looking up tens of thousands of events, which could cause large
amounts of data to be logged to the postgres log.
 2020-04-15 10:16:35 +01:00
Zay11Zay
f1097e7720
Fix the parameters of a test fixture (#7243) 2020-04-14 15:37:28 -04:00
Andrew Morgan
ac6a84818f Only register devices edu handler on the master process (#7255) 2020-04-14 11:36:24 +01:00
Andrew Morgan
fef82f4e22
Only register devices edu handler on the master process (#7255) 2020-04-14 10:09:58 +01:00
Ryan Hovland
4a0dadafbe
Add setting to nginx configuration to allow larger file uploads (#7251) 2020-04-13 17:23:36 +01:00
PeerD
f41b742161
Check on room creation if the user is allowed to publish the room to the room directory. (#7260) 2020-04-13 07:42:32 -04:00
Andrew Morgan
ac978ab3da
Default PL100 to enable encryption in a room (#7230) 2020-04-09 18:45:38 +01:00
Patrick Cloke
b85d7652ff
Do not allow a deactivated user to login via SSO. (#7240) 2020-04-09 13:28:13 -04:00
Andrew Morgan
967f99b9f8 Unblacklist /upgrade creates a new room (#7228) 
This commit was originally merged in #7228 but reverted in #7254 as the
associated sytest was not ready yet. Now that it is, we can merge this
again.
 2020-04-09 13:53:35 +01:00
Andrew Morgan
a026bdaab7
Add matrix-synapse-shared-secret-auth as an example password provider (#7248) 2020-04-09 12:49:05 +01:00
Andrew Morgan
e3cd28d8a4
Revert "Unblacklist /upgrade creates a new room (#7228)" (#7254) 
This reverts commit 59f0ca8b87.
 2020-04-09 12:47:16 +01:00
Andrew Morgan
59f0ca8b87
Unblacklist /upgrade creates a new room (#7228) 2020-04-09 12:44:48 +01:00
Richard van der Hoff
5a709630bf
Fix --help commandline argument (#7249) 
I don't really remember why this was so complicated; I think it dates
back to the time when we had to instantiate the Config classes before
we could call `add_arguments` - ie before #5597. In any case, I don't
think there's a good reason for it any more, and the impact of it
being complicated is that `--help` doesn't work correctly.
 2020-04-09 12:44:37 +01:00
Andrew Morgan
7f7f9968b3
Make it clearer to the user that they haven't overridden onLogin/Register (#7237) 2020-04-09 12:38:38 +01:00
Andrew Morgan
55d46da59a
Upgrade jQuery to 3.x on fallback login/registration screens (#7236) 2020-04-09 12:23:30 +01:00
Richard van der Hoff
23f8d285eb
Remove redundant checks on daemonize from synctl (#7233) 
We pass --daemonize on the commandline, which (since at least #4853) overrides
whatever the config file, so there is no need for it to be set in the config
file.
 2020-04-08 11:59:47 +01:00
Richard van der Hoff
cae4121484
Make systemd-with-workers doc official (#7234) 
Simplify and update this documentation, and make it part of the core dist.
 2020-04-08 11:59:26 +01:00
Brendan Abolivier
c11d24d48c
Fix changelog for #7235 2020-04-08 11:59:51 +02:00
Andrew Morgan
29b7e22b93
Add documentation to password_providers config option (#7238) 2020-04-08 00:46:50 +01:00
Richard van der Hoff
f31e65a749
bg update to clear out duplicate outbound_device_list_pokes (#7193) 
We seem to have some duplicates, which could do with being cleared out.
 2020-04-07 23:06:39 +01:00
Richard van der Hoff
aedeedc206
Merge pull request #7239 from matrix-org/rav/replication_cleanup 
Miscellaneous cleanups to replication code
 2020-04-07 22:00:55 +01:00
Patrick Cloke
1722b8a527
Convert delete_url_cache_media to async/await. (#7241) 2020-04-07 21:56:34 +01:00
Patrick Cloke
d78cb31588
Add typing information to federation_server. (#7219) 2020-04-07 15:03:23 -04:00
Richard van der Hoff
bd2ea3432b changelog 2020-04-07 17:44:51 +01:00
Brendan Abolivier
ec5ac8e2b1
Fix typo in the login fallback javascript (#7235) 
* Fix typo in the login fallback javascript

* Changelog
 2020-04-07 18:31:50 +02:00
Richard van der Hoff
2e105c156b
Remove sent outbound device list pokes from the database (#7192) 
They just get in the way.
 2020-04-07 15:19:19 +01:00
Erik Johnston
ce72355d7f
Fix race in replication (#7226) 
Fixes a race between handling `POSITION` and `RDATA` commands. We do this by simply linearizing handling of them.
 2020-04-07 11:01:04 +01:00
Erik Johnston
82498ee901
Move server command handling out of TCP protocol (#7187) 
This completes the merging of server and client command processing.
 2020-04-07 10:51:07 +01:00
Andrew Morgan
4b0f00ad0c
Remove stream before/after debug log lines (#7207) 2020-04-06 12:40:34 +01:00
Andrew Morgan
b21000a44f
Improve error responses when a remote server doesn't allow you to access its public rooms list (#6899) 2020-04-06 12:35:30 +01:00
Erik Johnston
5016b162fc
Move client command handling out of TCP protocol (#7185) 
The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
 2020-04-06 09:58:42 +01:00
Brendan Abolivier
d73bf18d13
Server notices: Dissociate room creation/lookup from invite (#7199) 
Fixes #6815

Before figuring out whether we should alert a user on MAU, we call get_notice_room_for_user to get some info on the existing server notices room for this user. This function, if the room doesn't exist, creates it and invites the user in it. This means that, if we decide later that no server notice is needed, the user gets invited in a room with no message in it. This happens at every restart of the server, since the room ID returned by get_notice_room_for_user is cached.

This PR fixes that by moving the inviting bit to a dedicated function, that's only called when the server actually needs to send a notice to the user. A potential issue with this approach is that the room that's created by get_notice_room_for_user doesn't match how that same function looks for an existing room (i.e. it creates a room that doesn't have an invite or a join for the current user in it, so it could lead to a new room being created each time a user syncs), but I'm not sure this is a problem given it's cached until the server restarts, so that function won't run very often.

It also renames get_notice_room_for_user into get_or_create_notice_room_for_user to make what it does clearer.
 2020-04-04 17:27:45 +02:00
Patrick Cloke
694d8bed0e
Support CAS in UI Auth flows. (#7186) 2020-04-03 15:35:05 -04:00
Martin Milata
b0db928c63
Extend web_client_location to handle absolute URLs (#7006) 
Log warning when filesystem path is used.

Signed-off-by: Martin Milata <martin@martinmilata.cz>
 2020-04-03 11:57:34 -04:00
Amber Brown
334bfdbc90
Add some benchmarks for LruCache (#6446) 2020-04-03 16:31:52 +01:00
Andrew Morgan
07b88c546d
Convert http.HTTPStatus objects to their int equivalent (#7188) 2020-04-03 14:26:07 +01:00
Richard van der Hoff
0f05fd1530
Reduce the number of calls to resource.getrusage (#7183) 
Let's just call `getrusage` once on each logcontext change, rather than twice.
 2020-04-03 13:21:30 +01:00
Richard van der Hoff
fd4c975b5b
Merge pull request #7190 from matrix-org/rav/one_bg_update_at_a_time 
Only run one background update at a time
 2020-04-03 13:17:30 +01:00
Richard van der Hoff
bae32740da
Remove some run_in_background calls in replication code (#7203) 
By running this stuff with `run_in_background`, it won't be correctly reported
against the relevant CPU usage stats.

Fixes #7202
 2020-04-03 12:29:30 +01:00
Richard van der Hoff
29ce90358c 1.12.3 2020-04-03 10:57:07 +01:00
Richard van der Hoff
daa1ac89a0
Fix device list update stream ids going backward (#7158) 
Occasionally we could get a federation device list update transaction which
looked like:

```
[
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D2', 'prev_id': [], 'stream_id': 12, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D1', 'prev_id': [12], 'stream_id': 11, 'deleted': True}},
    {'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D3', 'prev_id': [11], 'stream_id': 13, 'deleted': True}}
]
```

Having `stream_ids` which are lower than `prev_ids` looks odd. It might work
(I'm not actually sure), but in any case it doesn't seem like a reasonable
thing to expect other implementations to support.
 2020-04-03 10:40:22 +01:00
Richard van der Hoff
6d7cec7a57
Fix the debian build in a better way. (#7212) 2020-04-03 10:23:36 +01:00
Patrick Cloke
b9930d24a0
Support SAML in the user interactive authentication workflow. (#7102) 2020-04-01 08:48:00 -04:00
Patrick Cloke
468dcc767b
Allow admins to create aliases when they are not in the room (#7191) 2020-04-01 08:27:05 -04:00
siroccal
250f87d0de
Update postgres.md (#7119) 2020-04-01 12:44:51 +01:00
Erik Johnston
dfa0782254
Remove connections per replication stream metric. (#7195) 
This broke in a recent PR (#7024) and is no longer useful due to all
replication clients implicitly subscribing to all streams, so let's
just remove it.
 2020-04-01 10:40:46 +01:00
Richard van der Hoff
b413ab8aa6 changelog 2020-03-31 17:44:36 +01:00
Jostein Kjønigsen
2e826cd80c
Improve TURN documentation. (#7167) 2020-03-31 15:50:48 +01:00
Richard van der Hoff
60adcbed91
Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181) 
Fixes #7179.
 2020-03-31 15:18:41 +01:00
Karlinde
fe1580bfd9
Fill in the 'default' field for user-defined push rules (#6639) 
Signed-off-by: Karl Linderhed <git@karlinde.se>
 2020-03-31 15:08:56 +01:00
Andrew Morgan
b994e86e35
Only setdefault for signatures if device has key_json (#7177) 2020-03-31 14:51:22 +01:00
Patrick Cloke
0a7b0882c1
Fix use of async/await in media code (#7184) 2020-03-31 09:33:02 -04:00
Richard van der Hoff
62a7289133
Fix a bug which could cause incorrect 'cyclic dependency' error. (#7178) 
If there was an exception setting up one of the attributes of the Homeserver
god object, then future attempts to fetch that attribute would raise a
confusing "Cyclic dependency" error. Let's make sure that we clear the
`building` flag so that we just get the original exception.

Ref: #7169
 2020-03-31 13:09:16 +01:00
Andrew Morgan
3fb9fc40f5 1.12.1rc1 2020-03-31 11:49:43 +01:00
Erik Johnston
5d99bde788 Newsfile 2020-03-31 11:30:34 +01:00
Andrew Morgan
2cf115f0ea Rewrite changelog 2020-03-31 11:30:16 +01:00
Andrew Morgan
2cb38ca871 Add changelog 2020-03-31 11:30:05 +01:00
David Vo
5bd2b27525 Only import sqlite3 when type checking 
Fixes: #7127
Signed-off-by: David Vo <david@vovo.id.au>
 2020-03-31 11:27:17 +01:00
Richard van der Hoff
7966a1cde9
Rewrite prune_old_outbound_device_pokes for efficiency (#7159) 
make sure we clear out all but one update for the user
 2020-03-30 19:06:52 +01:00
Andrew Morgan
7042840b32
Transfer alias mappings when joining an upgraded room (#6946) 2020-03-30 17:53:25 +01:00
Andrew Morgan
d9f29f8dae
Fix a small typo in the metrics_flags config option. (#7171) 2020-03-30 17:38:21 +01:00
Erik Johnston
4f21c33be3
Remove usage of "conn_id" for presence. (#7128) 
* Remove `conn_id` usage for UserSyncCommand.

Each tcp replication connection is assigned a "conn_id", which is used
to give an ID to a remotely connected worker. In a redis world, there
will no longer be a one to one mapping between connection and instance,
so instead we need to replace such usages with an ID generated by the
remote instances and included in the replicaiton commands.

This really only effects UserSyncCommand.

* Add CLEAR_USER_SYNCS command that is sent on shutdown.

This should help with the case where a synchrotron gets restarted
gracefully, rather than rely on 5 minute timeout.
 2020-03-30 16:37:24 +01:00
David Baker
07569f25d1
Merge pull request #7160 from matrix-org/dbkr/always_send_own_device_list_updates 
Always send the user updates to their own device list
 2020-03-30 14:34:28 +01:00
Richard van der Hoff
6486c96b65
Merge pull request #7157 from matrix-org/rev.outbound_device_pokes_tests 
Add tests for outbound device pokes
 2020-03-30 13:59:07 +01:00
Patrick Cloke
c5f89fba55
Add developer documentation for running a local CAS server (#7147) 2020-03-30 07:28:42 -04:00
Richard van der Hoff
b7da598a61 Always whitelist the login fallback for SSO (#7153) 
That fallback sets the redirect URL to itself (so it can process the login
token then return gracefully to the client). This would make it pointless to
ask the user for confirmation, since the URL the confirmation page would be
showing wouldn't be the client's.
 2020-03-27 20:24:52 +00:00
Brendan Abolivier
84f7eaed16 Improve the UX of the login fallback when using SSO (#7152) 
* Don't show the login forms if we're currently logging in with a
  password or a token.
* Submit directly the SSO login form, showing only a spinner to the
  user, in order to eliminate from the clunkiness of SSO through this
  fallback.
 2020-03-27 20:19:54 +00:00
Dirk Klimpel
fb69690761
Admin API to join users to a room. (#7051) 2020-03-27 19:16:43 +00:00
Dirk Klimpel
8327eb9280
Add options to prevent users from changing their profile. (#7096) 2020-03-27 19:15:23 +00:00
txt-file
ae219fb411
update debian installation instructions to recommend installing virtualenv instead of python3-virtualenv (#6892) 
* change debian package from python3-virtualenv to virtualenv

The virtualenv package is needed for the virtualenv command. The
virtualenv package depends on python3-virtualenv (at least since
debian jessie) so there is no need to specify python3-virtualenv
additionally.

Signed-off-by: Vieno Hakkerinen <vieno@hakkerinen.eu>

* Add changelog

Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
 2020-03-27 15:02:00 +00:00
Andrew Morgan
12aa5a7fa7
Ensure is_verified on /_matrix/client/r0/room_keys/keys is a boolean (#7150) 2020-03-27 13:30:22 +00:00
David Vo
fbf0782c63
Only import sqlite3 when type checking (#7155) 
Fixes: #7127
Signed-off-by: David Vo <david@vovo.id.au>
 2020-03-27 13:20:00 +00:00
David Baker
d9965fb8d6 changelog 2020-03-27 12:30:59 +00:00
Richard van der Hoff
665630fcaa Add tests for outbound device pokes 2020-03-27 12:01:37 +00:00
Jason Robinson
7496d3d2f6
Merge pull request #7151 from matrix-org/jaywink/saml-redirect-fix 
Allow RedirectResponse in SAML response handler
 2020-03-26 22:10:31 +02:00
Patrick Cloke
fa4f12102d
Refactor the CAS code (move the logic out of the REST layer to a handler) (#7136) 2020-03-26 15:05:26 -04:00
Jason Robinson
55ca6cf88c
Update changelog.d/7151.bugfix 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-03-26 20:35:50 +02:00
Nektarios Katakis
825fb5d0a5
Don't default to an invalid sqlite config if no database configuration is provided (#6573) 2020-03-26 17:13:14 +00:00
Jason Robinson
060e7dce09 Allow RedirectResponse in SAML response handler 
Allow custom SAML handlers to redirect after processing an auth response.

Fixes #7149

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2020-03-26 19:02:35 +02:00
Dirk Klimpel
e8e2ddb60a
Allow server admins to define and enforce a password policy (MSC2000). (#7118) 2020-03-26 16:51:13 +00:00
Patrick Cloke
1c1242acba
Validate that the session is not modified during UI-Auth (#7068) 2020-03-26 07:39:34 -04:00
Aaron Raimist
6ca5e56fd1
Remove unused captcha_bypass_secret option (#7137) 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2020-03-25 17:49:34 +00:00
Erik Johnston
4cff617df1
Move catchup of replication streams to worker. (#7024) 
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
 2020-03-25 14:54:01 +00:00
Andrew Morgan
7bab642707
Various cleanups to INSTALL.md (#7141) 2020-03-25 13:56:40 +00:00
Erik Johnston
b1cfaf08af
Merge pull request #7133 from matrix-org/erikj/fix_worker_startup 
Fix starting workers when federation sending not split out.
 2020-03-25 09:42:39 +00:00
Richard van der Hoff
39230d2171
Clean up some LoggingContext stuff (#7120) 
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
 2020-03-24 14:45:33 +00:00
Naugrimm
1fcf9c6f95
Fix CAS redirect url (#6634) 
Build the same service URL when requesting the CAS ticket and when calling the proxyValidate URL.
 2020-03-24 11:59:04 +00:00
Erik Johnston
d6828c129f Newsfile 2020-03-24 10:36:44 +00:00
Patrick Cloke
190ab593b7
Use the proper error code when a canonical alias that does not exist is used. (#7109) 2020-03-23 15:21:54 -04:00
Kartikaya Gupta (kats)
e341518f92
Update pre-built package name for FreeBSD (#7107). (#7107) 
Signed-off-by: Kartikaya Gupta <kats@trevize.staktrace.com>
 2020-03-23 15:31:02 +00:00
Richard van der Hoff
a564b92d37
Convert *StreamRow classes to inner classes (#7116) 
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
 2020-03-23 13:59:11 +00:00
Richard van der Hoff
b3cee0ce67
Fix processing of groups stream, and use symbolic names for streams (#7117) 
`groups` != `receipts`

Introduced in #6964
 2020-03-23 11:39:36 +00:00
Dionysis Grigoropoulos
96071eea8f
Set Referrer-Policy to no-referrer for media (#7009) 2020-03-23 09:48:28 +00:00
Patrick Cloke
477c4f5b1c
Clean-up some auth/login REST code (#7115) 2020-03-20 16:22:47 -04:00
Richard van der Hoff
c165c1233b
Improve database configuration docs (#6988) 
Attempts to clarify the sample config for databases, and add some stuff about
tcp keepalives to `postgres.md`.
 2020-03-20 15:24:22 +00:00
Erik Johnston
fdb1344716
Remove concept of a non-limited stream. (#7011) 2020-03-20 14:40:47 +00:00
Patrick Cloke
caec7d4fa0
Convert some of the media REST code to async/await (#7110) 2020-03-20 07:20:02 -04:00
Patrick Cloke
c2db6599c8
Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors (#7089). 2020-03-19 08:22:56 -04:00
Erik Johnston
a319cb1dd1
Change device list streams to have one row per ID (#7010) 
* Add 'device_lists_outbound_pokes' as extra table.

This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.

* Change device lists stream to have one row per id.

This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.

* Change device list replication to match new semantics.

Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).

* Newsfile

* Remove handling of multiple rows per ID

* Fix worker handling

* Comments from review
 2020-03-19 11:36:53 +00:00
Richard van der Hoff
e43e78b985 1.12.0rc1 2020-03-19 11:07:16 +00:00
Richard van der Hoff
e913823a22
Fix concurrent modification errors in pusher metrics (#7106) 
add a lock to try to make this metric actually work
 2020-03-19 10:28:49 +00:00
Richard van der Hoff
8c75667ad7
Add prometheus metrics for the number of active pushers (#7103) 2020-03-19 10:00:24 +00:00
Richard van der Hoff
443162e577
Move pusherpool startup into _base.setup (#7104) 
This should be safe to do on all workers/masters because it is guarded by
a config option which will ensure it is only actually done on the worker
assigned as a pusher.
 2020-03-19 09:48:45 +00:00
Erik Johnston
4a17a647a9
Improve get auth chain difference algorithm. (#7095) 
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
 2020-03-18 16:46:41 +00:00
Patrick Cloke
88b41986db
Add an option to the set password API to choose whether to logout other devices. (#7085) 2020-03-18 07:50:00 -04:00
Richard von Kellner
6d110ddea4
Update INSTALL.md updated CentOS8 install instructions (#6925) 2020-03-17 21:48:23 +00:00
Richard van der Hoff
c37db0211e
Share SSL contexts for non-federation requests (#7094) 
Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests.

Fixes #7092.
 2020-03-17 21:32:25 +00:00
The Stranjer
5e477c1deb
Set charset to utf-8 when adding headers for certain text content types (#7044) 
Fixes #7043
 2020-03-17 13:29:09 +00:00
Patrick Cloke
7581d30e9f
Remove unused federation endpoint (query_auth) (#7026) 2020-03-17 08:04:49 -04:00
Patrick Cloke
60724c46b7
Remove special casing of m.room.aliases events (#7034) 2020-03-17 07:37:04 -04:00
Richard van der Hoff
6a35046363 Revert "Add options to disable setting profile info for prevent changes. (#7053)" 
This reverts commit 54dd28621b, reversing
changes made to 6640460d05.
 2020-03-17 11:25:01 +00:00
Brendan Abolivier
7df04ca0e6
Populate the room version from state events (#7070) 
Fixes #7065 

This is basically the same as https://github.com/matrix-org/synapse/pull/6847 except it tries to populate events from `state_events` rather than `current_state_events`, since the latter might have been cleared from the state of some rooms too early, leaving them with a `NULL` room version.
 2020-03-16 22:31:47 +00:00
Brendan Abolivier
beb19cf61a
Fix buggy condition in account validity handler (#7074) 2020-03-16 12:16:30 +00:00
Brendan Abolivier
d8d91983bc
Merge pull request #7067 from matrix-org/babolivier/saml_error_moar 
Move the default SAML2 error HTML to a dedicated file
 2020-03-13 19:53:19 +00:00
Patrick Cloke
77d0a4507b
Add type annotations and comments to auth handler (#7063) 2020-03-12 11:36:27 -04:00
Brendan Abolivier
e55a240681
Changelog 2020-03-11 19:37:04 +00:00
Brendan Abolivier
1cde4cf3f1
Changelog 2020-03-11 18:03:56 +00:00
Brendan Abolivier
54dd28621b
Add options to disable setting profile info for prevent changes. (#7053) 2020-03-10 22:23:01 +00:00
Brendan Abolivier
156f271867
Changelog 2020-03-10 14:01:24 +00:00
dklimpel
885134529f updates after review 2020-03-09 22:09:29 +01:00
Brendan Abolivier
14b2ebe767
Merge pull request #7055 from matrix-org/babolivier/get_time_of_last_push_action_before 
Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore
 2020-03-09 14:53:50 +00:00
Brendan Abolivier
f9e3a3f4d0
Changelog 
It's the same as in #6964 since it's the most likely cause of the bug
and that change hasn't been released yet.
 2020-03-09 14:21:01 +00:00
Patrick Cloke
06eb5cae08
Remove special auth and redaction rules for aliases events in experimental room ver. (#7037) 2020-03-09 08:58:25 -04:00
Patrick Cloke
66315d862f
Update routing of fallback auth in the worker docs. (#7048) 2020-03-09 07:19:24 -04:00
Brendan Abolivier
bbf725e7da
Merge pull request #7045 from matrix-org/babolivier/room_keys_check 
Make sure that is_verified is a boolean when processing room keys
 2020-03-09 09:54:48 +00:00
dklimpel
fb078f921b changelog 2020-03-08 15:19:07 +01:00
Neil Pilgrim
2bff4457d9
Add type hints to logging/context.py (#6309) 
* Add type hints to logging/context.py

Signed-off-by: neiljp (Neil Pilgrim) <github@kepier.clara.net>
 2020-03-07 17:57:26 +00:00
Neil Johnson
1d66dce83e
Break down monthly active users by appservice_id (#7030) 
* Break down monthly active users by appservice_id and emit via prometheus.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2020-03-06 18:14:19 +00:00
Brendan Abolivier
a27056d539
Changelog 2020-03-06 11:06:47 +00:00
Patrick Cloke
87972f07e5
Convert remote key resource REST layer to async/await. (#7020) 2020-03-05 11:29:56 -05:00
Richard van der Hoff
78a15b1f9d
Store room_versions in EventBase objects (#6875) 
This is a bit fiddly because it all has to be done on one fell swoop:

* Wherever we create a new event, pass in the room version (and check it matches the format version)
* When we prune an event, use the room version of the unpruned event to create the pruned version.
* When we pass an event over the replication protocol, pass the room version over alongside it, and use it when deserialising the event again.
 2020-03-05 15:46:44 +00:00
Brendan Abolivier
83b6c69d3d
Changelog 2020-03-04 17:29:09 +00:00
Patrick Cloke
13892776ef
Allow deleting an alias if the user has sufficient power level (#6986) 2020-03-04 11:30:46 -05:00
Richard van der Hoff
8ef8fb2c1c
Read the room version from database when fetching events (#6874) 
This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
 2020-03-04 13:11:04 +00:00
Brendan Abolivier
43f874055d
Merge branch 'master' into develop 2020-03-03 15:20:49 +00:00
Brendan Abolivier
fd983fad96 v1.11.1 2020-03-03 14:58:37 +00:00
Patrick Cloke
7dcbc33a1b
Validate the alt_aliases property of canonical alias events (#6971) 2020-03-03 07:12:45 -05:00
Erik Johnston
65a941d1f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_devices_stream 2020-03-02 16:55:55 +00:00
Erik Johnston
b29474e0aa
Always return a deferred from get_current_state_deltas. (#7019) 
This currently causes presence notify code to log exceptions when there
is no state changes to process. This doesn't actually cause any problems
as we'd simply do nothing anyway.
 2020-03-02 16:52:15 +00:00
Erik Johnston
3ab8e9c293
Fix py35-old CI by using native tox. (#7018) 
I'm not really sure how this was going wrong, but this seems like the
right approach anyway.
 2020-03-02 16:17:11 +00:00
Richard van der Hoff
174aaa1d62 remove spurious changelog 2020-03-02 14:53:56 +00:00
Dirk Klimpel
bbeee33d63 Fixed set a user as an admin with the new API (#6928) 
Fix #6910
 2020-03-02 13:28:50 +00:00
Uday Bansal
e4ffb14d57
Fix last date for ACMEv1 install (#7015) 
Support for getting TLS certificates through ACMEv1 ended on November 2019.

Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
 2020-02-29 23:37:23 +00:00
Patrick Cloke
12d4259000
Add some type annotations to the federation base & client classes (#6995) 2020-02-28 07:31:07 -05:00
Erik Johnston
59ad93d2a4 Newsfile 2020-02-28 11:27:37 +00:00
Dirk Klimpel
9b06d8f8a6
Fixed set a user as an admin with the new API (#6928) 
Fix #6910
 2020-02-28 09:58:05 +00:00
Patrick Cloke
ab0073a6c0 Merge remote-tracking branch 'origin/release-v1.11.1' into develop 2020-02-27 13:47:44 -05:00
Erik Johnston
2201bc9795
Don't refuse to start worker if media listener configured. (#7002) 
Instead lets just warn if the worker has a media listener configured but
has the media repository disabled.

Previously non media repository workers would just ignore the media
listener.
 2020-02-27 16:33:21 +00:00
Richard van der Hoff
cab4a52535
set worker_app for frontend proxy test (#7003) 
to stop the federationhandler trying to do master stuff
 2020-02-27 13:08:43 +00:00
James
b32ac60c22
Expose common commands via snap run interface to allow easier invocation (#6315) 
Signed-off-by: James Hebden <james@ec0.io>
 2020-02-27 12:47:40 +00:00
Richard van der Hoff
132b673dbe
Add some type annotations in synapse.storage (#6987) 
I cracked, and added some type definitions in synapse.storage.
 2020-02-27 11:53:40 +00:00
Richard van der Hoff
3e99528f2b
Store room version on invite (#6983) 
When we get an invite over federation, store the room version in the rooms table.

The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
 2020-02-26 16:58:33 +00:00
Patrick Cloke
380122866f
Cast a coroutine into a Deferred in the federation base (#6996) 
Properly convert a coroutine into a Deferred in federation_base to fix an error when joining a room.
 2020-02-26 11:32:13 -05:00
Erik Johnston
1f773eec91
Port PresenceHandler to async/await (#6991) 2020-02-26 15:33:26 +00:00
Uday Bansal
7728d87fd7
Updated warning for incorrect database collation/ctype (#6985) 
Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
 2020-02-26 15:17:03 +00:00
Andrew Morgan
8c75b621bf
Ensure 'deactivated' parameter is a boolean on user admin API, Fix error handling of call to deactivate user (#6990) 2020-02-26 12:22:55 +00:00
Richard van der Hoff
e66f099ca9
Sanity-check database before running upgrades (#6982) 
Some of the database deltas rely on `config.server_name` being set correctly,
so we should check that it is before running the deltas.

Fixes #6870.
 2020-02-25 17:46:00 +00:00
Erik Johnston
bbf8886a05
Merge worker apps into one. (#6964) 2020-02-25 16:56:55 +00:00
Fridtjof Mund
4aea0bd292
contrib/docker: remove quotes for POSTGRES_INITDB_ARGS (#6984) 
I made a mistake in https://github.com/matrix-org/synapse/pull/6921 - the quotes break the postgres container's startup script (or docker-compose), which makes initdb fail: https://github.com/matrix-org/synapse/pull/6921#issuecomment-590657154

Signed-off-by: Fridtjof Mund <fridtjof@das-labor.org>
 2020-02-25 10:48:13 +00:00
Richard van der Hoff
691659568f
Remove redundant store_room call (#6979) 
`_process_received_pdu` is only called by `on_receive_pdu`, which ignores any
events for unknown rooms, so this is redundant.
 2020-02-24 17:20:44 +00:00
Richard van der Hoff
a301934f46
Upsert room version when we join over federation (#6968) 
This is intended as a precursor to storing room versions when we receive an
invite over federation, but has the happy side-effect of fixing #3374 at last.

In short: change the store_room with try/except to a proper upsert which
updates the right columns.
 2020-02-24 15:46:41 +00:00
Richard van der Hoff
4c2ed3f20e
Fix minor issues with email config (#6962) 
* Give `notif_template_html`, `notif_template_text` default values (fixes #6960)
 * Don't complain if `smtp_host` and `smtp_port` are unset, since they have sensible defaults (fixes #6961)
 * Set the example for `enable_notifs` to `True`, for consistency and because it's more useful
 * Raise errors as ConfigError rather than RuntimeError for nicer formatting
 2020-02-24 15:18:38 +00:00
Patrick Cloke
af6c389501
No longer use room alias events to calculate room names for push notifications. (#6966) 2020-02-21 12:50:48 -05:00
Dirk Klimpel
7b0e2d961c
Change displayname of user as admin in rooms (#6876) 2020-02-21 17:44:03 +00:00
Patrick Cloke
fcf4599488
Stop returning aliases as part of the room list. (#6970) 2020-02-21 12:40:23 -05:00
Patrick Cloke
7936d2a96e
Publishing/removing from the directory requires a power level greater than canonical aliases. 2020-02-21 07:18:33 -05:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) 
Ensure good comprehension hygiene using flake8-comprehensions.
 2020-02-21 07:15:07 -05:00
Richard van der Hoff
1fcb9a1a7a changelog 2020-02-21 09:06:18 +00:00
Richard van der Hoff
9c1b83b007 1.11.0 2020-02-21 08:56:04 +00:00
Andrew Morgan
8f6d9c4cf0
Small grammar fixes to the ACME v1 deprecation notice (#6944) 
Some small fixes to the copy in #6907.
 2020-02-21 08:53:01 +00:00
Patrick Cloke
99eed85a77
Do not send alias events when creating / upgrading a room (#6941) 
Stop emitting room alias update events during room creation/upgrade.
 2020-02-20 16:24:04 -05:00
Hubert Chathi
a90d0dc5c2
don't insert into the device table for remote cross-signing keys (#6956) 2020-02-20 09:59:00 -05:00
Ruben Barkow-Kuder
4fb5f4d0ce
Add some clarifications to README.md in the database schema directory. (#6615) 
Signed-off-by: Ruben Barkow-Kuder <github@r.z11.de>
 2020-02-20 10:37:57 +00:00
Erik Johnston
7b7c3cedf2 Minor perf fixes to get_auth_chain_ids. 2020-02-19 15:47:11 +00:00
Erik Johnston
fc87d2ffb3
Freeze allocated objects on startup. (#6953) 
This may make gc go a bit faster as the gc will know things like
caches/data stores etc. are frozen without having to check.
 2020-02-19 15:09:00 +00:00
Erik Johnston
2b37eabca1
Reduce auth chains fetched during v2 state res. (#6952) 
The state res v2 algorithm only cares about the difference between auth
chains, so we can pass in the known common state to the `get_auth_chain`
storage function so that it can ignore those events.
 2020-02-19 15:04:47 +00:00
Richard van der Hoff
197b08de35 1.11.0rc1 2020-02-19 13:48:32 +00:00
Erik Johnston
099c96b89b
Revert get_auth_chain_ids changes (#6951) 2020-02-19 11:37:35 +00:00
Richard van der Hoff
2fb7794e60
Merge pull request #6949 from matrix-org/rav/list_room_aliases_peekable 
Make room alias lists peekable
 2020-02-19 11:19:11 +00:00
Brendan Abolivier
bbe39f808c
Merge pull request #6940 from matrix-org/babolivier/federate.md 
Clean up and update federation docs
 2020-02-19 10:58:59 +00:00
Richard van der Hoff
880aaac1d8
Move MSC2432 stuff onto unstable prefix (#6948) 
it's not in the spec yet, so needs to be unstable. Also add a feature flag for it. Also add a test for admin users.
 2020-02-19 10:40:27 +00:00
Richard van der Hoff
abf1e5c526
Tiny optimisation for _get_handler_for_request (#6950) 
we have hundreds of path_regexes (see #5118), so let's not convert the same
bytes to str for each of them.
 2020-02-19 10:38:20 +00:00
Erik Johnston
0d0bc35792
Increase DB/CPU perf of _is_server_still_joined check. (#6936) 
* Increase DB/CPU perf of `_is_server_still_joined` check.

For rooms with large amount of state a single user leaving could cause
us to go and load a lot of membership events and then pull out
membership state in a large number of batches.

* Newsfile

* Update synapse/storage/persist_events.py

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Fix adding if too soon

* Update docstring

* Review comments

* Woops typo

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-19 10:15:49 +00:00
Brendan Abolivier
5e4a438556
Merge pull request #6945 from matrix-org/babolivier/fix-retention-debug-log 
Fix log in message retention purge jobs
 2020-02-19 10:12:55 +00:00
Brendan Abolivier
fa64f836ec
Update changelog.d/6945.bugfix 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-19 09:54:13 +00:00
Erik Johnston
5a5abd55e8
Limit size of get_auth_chain_ids query (#6947) 2020-02-19 09:39:26 +00:00
Richard van der Hoff
603618c002 changelog 2020-02-19 08:53:32 +00:00
Brendan Abolivier
771d70e89c
Changelog 2020-02-18 17:31:02 +00:00
Richard van der Hoff
adfaea8c69
Implement GET /_matrix/client/r0/rooms/{roomId}/aliases (#6939) 
per matrix-org/matrix-doc#2432
 2020-02-18 16:23:25 +00:00
Richard van der Hoff
3f1cd14791
Merge pull request #6872 from matrix-org/rav/dictproperty 
Rewrite _EventInternalMetadata to back it with a dict
 2020-02-18 16:21:02 +00:00
Brendan Abolivier
d484126bf7
Merge pull request #6907 from matrix-org/babolivier/acme-config 
Add mention and warning about ACME v1 deprecation to the TLS config
 2020-02-18 16:11:31 +00:00
Erik Johnston
8a380d0fe2
Increase perf of get_auth_chain_ids used in state res v2. (#6937) 
We do this by moving the recursive query to be fully in the DB.
 2020-02-18 15:39:09 +00:00
Erik Johnston
818def8248
Fix worker docs to point /publicised_groups API correctly. (#6938) 2020-02-18 15:27:45 +00:00
Brendan Abolivier
7a782c32a2
Merge pull request #6909 from matrix-org/babolivier/acme-install 
Update INSTALL.md to recommend reverse proxying and warn about ACMEv1 deprecation
 2020-02-18 15:06:06 +00:00
Brendan Abolivier
b1255077f5 Changelog 2020-02-18 14:35:51 +00:00
Patrick Cloke
fe3941f6e3
Stop sending events when creating or deleting aliases (#6904) 
Stop sending events when creating or deleting associations (room aliases). Send an updated canonical alias event if one of the alt_aliases is deleted.
 2020-02-18 07:29:44 -05:00
Richard van der Hoff
3be2abd0a9
Kill off deprecated "config-on-the-fly" docker mode (#6918) 
Lots of people seem to get confused by this mode, and it's been deprecated
since Synapse 1.1.0. It's time for it to go.
 2020-02-18 11:41:53 +00:00
Richard van der Hoff
bc831d1d9a #6924 has been released in 1.10.1 2020-02-17 16:34:13 +00:00
Andrew Morgan
3404ad289b
Raise the default power levels for invites, tombstones and server acls (#6834) 2020-02-17 13:23:37 +00:00
Richard van der Hoff
46fa66bbfd
wait for current_state_events_membership before delete_old_current_state_events (#6924) 2020-02-17 11:30:50 +00:00
Patrick Cloke
10027c80b0
Add type hints to the spam check module (#6915) 
Add typing information to the spam checker modules.
 2020-02-14 12:49:40 -05:00
Richard van der Hoff
5a78f47f6e changelog 2020-02-14 16:42:40 +00:00
Fridtjof Mund
32873efa87
contrib/docker: Ensure correct encoding and locale settings on DB creation (#6921) 
Signed-off-by: Fridtjof Mund <fridtjof@das-labor.org>
 2020-02-14 16:27:29 +00:00
Richard van der Hoff
97a42bbc3a
Add a warning about indentation to generated config (#6920) 
Fixes #6916.
 2020-02-14 16:22:30 +00:00
Patrick Cloke
02e89021f5
Convert the directory handler tests to use HomeserverTestCase (#6919) 
Convert directory handler tests to use HomeserverTestCase.
 2020-02-14 09:05:43 -05:00
Patrick Cloke
49f877d32e
Filter the results of user directory searching via the spam checker (#6888) 
Add a method to the spam checker to filter the user directory results.
 2020-02-14 07:17:54 -05:00
Brendan Abolivier
79460ce9c9
Changelog 2020-02-13 17:24:14 +00:00
Brendan Abolivier
df1c98c22a
Update changelog for #6905 to group it with upcoming PRs 2020-02-13 16:12:20 +00:00
Brendan Abolivier
f3f142259e
Changelog 2020-02-13 16:10:16 +00:00
Patrick Cloke
361de49c90
Add documentation for the spam checker module (#6906) 
Add documentation for the spam checker.
 2020-02-13 07:40:57 -05:00
Brendan Abolivier
f48bf4febd
Merge pull request #6905 from matrix-org/babolivier/acme.md 
Update ACME.md to mention ACME v1 deprecation
 2020-02-13 12:13:18 +00:00
Aaron Raimist
dc3f998706
Remove m.lazy_load_members from unstable features since it is in CS r0.5.0 (#6877) 
Fixes #5528
 2020-02-13 12:02:32 +00:00
Brendan Abolivier
e88a5dd108
Changelog 2020-02-12 20:15:41 +00:00
Brendan Abolivier
6cd34da8b1
Merge pull request #6891 from matrix-org/babolivier/retention-doc-amend 
Spell out that the last event sent to a room won't be deleted by a purge
 2020-02-12 20:12:20 +00:00
Andrew Morgan
d8994942f2
Return a 404 for admin api user lookup if user not found (#6901) 2020-02-12 18:14:10 +00:00
Patrick Cloke
ba547ec3a9
Use BSD-compatible in-place editing for sed. (#6887) 2020-02-12 07:02:19 -05:00
Brendan Abolivier
a0c4769f1a
Update the changelog file 2020-02-11 17:56:42 +00:00
Brendan Abolivier
705c978366
Changelog 2020-02-11 17:38:27 +00:00
Patrick Cloke
a92e703ab9
Reject device display names that are too long (#6882) 
* Reject device display names that are too long.

Too long is currently defined as 100 characters in length.

* Add a regression test for rejecting a too long device display name.
 2020-02-10 16:35:26 -05:00
Patrick Cloke
3a3118f4ec
Add an additional test to the SyTest blacklist for worker mode. (#6883) 2020-02-10 11:47:18 -05:00
Richard van der Hoff
db0fee738d Synapse 1.10.0rc3 (2020-02-10) 
==============================
 
 Features
 --------
 
 - Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))
 
 Internal Changes
 ----------------
 
 - Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl5BLBwACgkQOSor00I9
 eP/9Ogf9HZ+XD5tt1uYkGqDSvUBS4sHdNcvxpNf1UMo6VaY4K5b9kF28MjY07ur2
 nGq/nnLyLZZzwVXtV3q8upIdIK1hcMp2HCU0qDhEljepOkGmyQTqZooxrdrLTjPS
 232bbifB6VFdK8P4gPIGopCRQI9a+TII4jODBobiJR43e0gEPfSREJKAnJs7DYaD
 Quf3Svzu+jB+X8ymWLVbCFXuk4A4UqPzrRPb8TQBDN3w35G4JBDAgIpdxhQdJpdR
 RIS1HgSm95P03doS23RLUjXuIpBfYveyStuGfPx4P9Lf30M2o6bxDeTPo2+IGbST
 yhl1jB0l8Ve5/6veuTieA6vuXWU55g==
 =2rgX
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0rc3' into develop

Synapse 1.10.0rc3 (2020-02-10)
==============================

Features
--------

- Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))

Internal Changes
----------------

- Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))
 2020-02-10 10:15:32 +00:00
Richard van der Hoff
3de57e7062 1.10.0rc3 2020-02-10 09:56:42 +00:00
Matthew Hodgson
8e64c5a24c filter out m.room.aliases from the CS API until a better solution is specced (#6878) 
We're in the middle of properly mitigating spam caused by malicious aliases being added to a room. However, until this work fully lands, we temporarily filter out all m.room.aliases events from /sync and /messages on the CS API, to remove abusive aliases. This is considered acceptable as m.room.aliases events were never a reliable record of the given alias->id mapping and were purely informational, and in their current state do more harm than good.
 2020-02-10 09:36:23 +00:00
Richard van der Hoff
cc0800ebfc Merge remote-tracking branch 'origin/release-v1.10.0' into develop 2020-02-10 00:41:49 +00:00
Richard van der Hoff
fe73f0d533
Update setuptools for python 3.5 tests (#6880) 
Workaround for jaraco/zipp#40
 2020-02-10 00:41:20 +00:00
Erik Johnston
21db35f77e
Add support for putting fed user query API on workers (#6873) 2020-02-07 15:45:39 +00:00
Richard van der Hoff
e1d858984d
Remove unused get_room_stats_state method. (#6869) 2020-02-07 15:30:26 +00:00
Richard van der Hoff
799001f2c0
Add a make_event_from_dict method (#6858) 
... and use it in places where it's trivial to do so.

This will make it easier to pass room versions into the FrozenEvent
constructors.
 2020-02-07 15:30:04 +00:00
Erik Johnston
b08b0a22d5
Add typing to synapse.federation.sender (#6871) 2020-02-07 13:56:38 +00:00
Erik Johnston
de2d267375
Allow moving group read APIs to workers (#6866) 2020-02-07 11:14:19 +00:00
Dirk Klimpel
56ca93ef59
Admin api to add an email address (#6789) 2020-02-07 10:29:36 +00:00
Richard van der Hoff
f4884444c3
remove unused room_version_to_event_format (#6857) 2020-02-07 09:26:57 +00:00
Richard van der Hoff
e1b240329e
Merge pull request #6856 from matrix-org/rav/redact_changes/6 
Pass room_version into `event_from_pdu_json`
 2020-02-07 09:22:15 +00:00
Patrick Cloke
7765bf3989
Limit the number of events that can be requested when backfilling events (#6864) 
Limit the maximum number of events requested when backfilling events.
 2020-02-06 13:25:24 -05:00
Richard van der Hoff
928edef979 Pass room_version into event_from_pdu_json 
It's called from all over the shop, so this one's a bit messy.
 2020-02-06 16:08:27 +00:00
Richard van der Hoff
b0c8bdd49d
pass room version into FederationClient.send_join (#6854) 
... which allows us to sanity-check the create event.
 2020-02-06 15:50:39 +00:00
timfi
bce557175b
Allow empty federation_certificate_verification_whitelist (#6849) 2020-02-06 14:45:01 +00:00
PeerD
99fcc96289
Third party event rules Update (#6781) 2020-02-06 14:15:29 +00:00
Erik Johnston
ed630ea17c
Reduce amount of logging at INFO level. (#6862) 
A lot of the things we log at INFO are now a bit superfluous, so lets
make them DEBUG logs to reduce the amount we log by default.

Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
 2020-02-06 13:31:05 +00:00
Richard van der Hoff
9bcd37146e
Merge pull request #6823 from matrix-org/rav/redact_changes/5 
pass room versions around
 2020-02-06 11:32:33 +00:00
Erik Johnston
6a7e90ad78 1.10.0rc2 2020-02-06 10:40:08 +00:00
Patrick Cloke
5e019069ab
Merge pull request #6855 from matrix-org/clokep/readme-pip-install 
Add quotes around the pip install target to avoid my shell complaining
 2020-02-05 13:29:52 -05:00
Patrick Cloke
39c2d26e0b Add quotes around pip install target (my shell complained without them). 2020-02-05 12:53:18 -05:00
Richard van der Hoff
ff70ec0a00 Newsfile 2020-02-05 17:43:57 +00:00
Richard van der Hoff
577f460369
Merge pull request #6840 from matrix-org/rav/federation_client_async 
Port much of `synapse.federation.federation_client` to async/await
 2020-02-05 16:56:39 +00:00
Erik Johnston
a58860e480
Check sender_key matches on inbound encrypted events. (#6850) 
If they don't then the device lists are probably out of sync.
 2020-02-05 14:02:39 +00:00
Hubert Chathi
60d0672426 Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-05 10:54:49 +00:00
Michael Kaye
a831d2e4e3
Reduce performance logging to DEBUG (#6833) 
* Reduce tnx performance logging to DEBUG
* Changelog.d
 2020-02-05 08:57:37 +00:00
Richard van der Hoff
d88e0ec080
Database updates to populate rooms.room_version (#6847) 
We're going to need this so that we can figure out how to handle redactions when fetching events from the database.
 2020-02-04 21:31:08 +00:00
Erik Johnston
6475382d80
Fix detecting unknown devices from remote encrypted events. (#6848) 
We were looking at the wrong event type (`m.room.encryption` vs
`m.room.encrypted`).

Also fixup the duplicate `EvenTypes` entries.

Introduced in #6776.
 2020-02-04 17:25:54 +00:00
Hubert Chathi
74bf3fdbb9
Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fed 
add device signatures to device key query results
 2020-02-04 12:03:54 -05:00
Michael Kaye
c87572d6e4
Update CONTRIBUTING.md about merging PRs. (#6846) 2020-02-04 16:21:09 +00:00
Richard van der Hoff
5ef91b96f1 Merge remote-tracking branch 'origin/develop' into rav/federation_client_async 2020-02-04 12:07:05 +00:00
Richard van der Hoff
c7d6d5c69e
Merge pull request #6837 from matrix-org/rav/federation_async 
Port much of `synapse.handlers.federation` to async/await.
 2020-02-04 12:06:18 +00:00
Hubert Chathi
245ee14220 add changelog 2020-02-04 00:21:07 -05:00
Richard van der Hoff
4b4536dd02 newsfile 2020-02-03 22:28:45 +00:00
Erik Johnston
b9391c9575
Add typing to SyncHandler (#6821) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-03 18:05:44 +00:00
Erik Johnston
ae5b3104f0
Fix stacktraces when using ObservableDeferred and async/await (#6836) 2020-02-03 17:10:54 +00:00
Richard van der Hoff
e49eb1a886 changelog 2020-02-03 16:30:21 +00:00
Andrew Morgan
370080531e
Allow URL-encoded user IDs on user admin api paths (#6825) 2020-02-03 13:18:42 +00:00
Richard van der Hoff
b0d112e78b
Fix room_version in on_invite_request flow (#6827) 
I messed this up a bit in #6805, but fortunately we weren't actually doing
anything with the room_version so it didn't matter that it was a str not a RoomVersion.
 2020-02-03 13:15:23 +00:00
Erik Johnston
ac0d45b78b 1.10.0rc1 2020-01-31 15:35:37 +00:00
Erik Johnston
83b0ea047b
Fix deleting of stale marker for device lists (#6819) 
We were in fact only deleting stale marker when we got an incremental
update, rather than when we did a full resync.
 2020-01-31 14:04:15 +00:00
Richard van der Hoff
7f93eb1903
pass room_version into compute_event_signature (#6807) 2020-01-31 13:47:43 +00:00
Richard van der Hoff
a5afdd15e5
Merge pull request #6806 from matrix-org/rav/redact_changes/3 
Pass room_version into add_hashes_and_signatures
 2020-01-31 10:57:03 +00:00
Richard van der Hoff
160522e32c
Merge pull request #6820 from matrix-org/rav/get_room_version_id 
Make `get_room_version` return a RoomVersion object
 2020-01-31 10:56:42 +00:00
Richard van der Hoff
f6fa2c0b31 newsfile 2020-01-31 10:30:29 +00:00
Erik Johnston
7d846e8704
Fix bug with getting missing auth event during join 500'ed (#6810) 2020-01-31 09:49:13 +00:00
Richard van der Hoff
540c5e168b changelog 2020-01-30 22:15:50 +00:00
Richard van der Hoff
46a446828d
pass room version into FederationHandler.on_invite_request (#6805) 2020-01-30 22:13:02 +00:00
Erik Johnston
e0992fcc5b
Log when we delete room in bg update (#6816) 2020-01-30 17:55:34 +00:00
Richard van der Hoff
184303b865
MSC2260: Block direct sends of m.room.aliases events (#6794) 
as per MSC2260
 2020-01-30 17:20:55 +00:00
Erik Johnston
57ad702af0
Backgroud update to clean out rooms from current state (#6802) 2020-01-30 17:17:44 +00:00
Erik Johnston
b660327056
Resync remote device list when detected as stale. (#6786) 2020-01-30 17:06:38 +00:00
Erik Johnston
c3d4ad8afd
Fix sending server up commands from workers (#6811) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2020-01-30 16:42:11 +00:00
Erik Johnston
a5bab2d058
When server leaves room check for stale device lists. (#6801) 
When a server leaves a room it may stop sharing a room with remote
users, and thus not get any updates to their device lists. So we need to
check for this case and delete those device lists from the cache.

We don't need to do this if we stop sharing a room because the remote
user leaves the room, because we track that case via looking at
membership changes.
 2020-01-30 16:10:30 +00:00
Erik Johnston
c80a9fe13d
When a client asks for remote keys check if should resync. (#6797) 
If we detect that the remote users' keys may have changed then we should
attempt to resync against the remote server rather than using the
(potentially) stale local cache.
 2020-01-30 15:06:58 +00:00
Richard van der Hoff
5a246611e3
Type defintions for use in refactoring for redaction changes (#6803) 
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
 2020-01-30 11:25:59 +00:00
Erik Johnston
a855b7c3a8
Remove unused DeviceRow class (#6800) 2020-01-29 12:06:31 +00:00
Richard van der Hoff
750d4d7599 changelog 2020-01-29 11:52:52 +00:00
Erik Johnston
6b9e1014cf
Fix race in federation sender that delayed device updates. (#6799) 
We were sending device updates down both the federation stream and
device streams. This mean there was a race if the federation sender
worker processed the federation stream first, as when the sender checked
if there were new device updates the slaved ID generator hadn't been
updated with the new stream IDs and so returned nothing.

This situation is correctly handled by events/receipts/etc by not
sending updates down the federation stream and instead having the
federation sender worker listen on the other streams and poke the
transaction queues as appropriate.
 2020-01-29 11:23:01 +00:00
Erik Johnston
611215a49c
Delete current state when server leaves a room (#6792) 
Otherwise its just stale data, which may get deleted later anyway so
can't be relied on. It's also a bit of a shotgun if we're trying to get
the current state of a room we're not in.
 2020-01-29 11:01:32 +00:00
Erik Johnston
2cad8baa70
Fix bug when querying remote user keys that require a resync. (#6796) 
We ended up only returning a single device, rather than all of them.
 2020-01-29 09:56:41 +00:00
Erik Johnston
fcfb591b31
Fix outbound federation request metrics (#6795) 2020-01-28 18:59:48 +00:00
Richard van der Hoff
cc109b79dd
Merge pull request #6787 from matrix-org/rav/msc2260 
Implement updated auth rules from MSC2260
 2020-01-28 15:11:22 +00:00
Erik Johnston
e17a110661
Detect unknown remote devices and mark cache as stale (#6776) 
We just mark the fact that the cache may be stale in the database for
now.
 2020-01-28 14:43:21 +00:00
Richard van der Hoff
fbe0a82c0d update changelog 2020-01-28 14:20:10 +00:00
Richard van der Hoff
99e205fc21 changelog 2020-01-28 14:20:10 +00:00
Richard van der Hoff
a8ce7aeb43
Pass room version object into event_auth.check and check_redaction (#6788) 
These are easier to work with than the strings and we normally have one around.

This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
 2020-01-28 14:18:29 +00:00
Erik Johnston
02b44db922
Warn if postgres database has non-C locale. (#6734) 
As using non-C locale can cause issues on upgrading OS.
 2020-01-28 13:44:21 +00:00
Erik Johnston
8df862e45d
Add rooms.room_version column (#6729) 
This is so that we don't have to rely on pulling it out from `current_state_events` table.
 2020-01-27 14:30:57 +00:00
Erik Johnston
d5275fc55f
Propagate cache invalidates from workers to other workers. (#6748) 
Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
 2020-01-27 13:47:50 +00:00
Brendan Abolivier
f74d178b17
Merge pull request #6775 from matrix-org/jaywink/worker-docs-tweaks 
Clarifications to the workers documentation
 2020-01-27 12:30:54 +00:00
Jason Robinson
cf9d56e5cf
Formatting of changelog 
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
 2020-01-27 14:09:59 +02:00
Andrew Morgan
9f7aaf90b5
Validate client_secret parameter (#6767) 2020-01-24 14:28:40 +00:00
Jason Robinson
aa6ad288f1 Clarifications to the workers documentation 
* Add note that user_dir requires disabling user dir
  updates from the main synapse process.
* Add note that federation_reader should have
  the federation listener resource.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2020-01-24 11:08:50 +02:00
Erik Johnston
fa4d609e20
Make 'event.redacts' never raise. (#6771) 
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
 2020-01-23 15:19:03 +00:00
Brendan Abolivier
51fc3f693e Merge branch 'master' into develop 2020-01-23 13:45:23 +00:00
Brendan Abolivier
1dc5a791cf Fixup changelog 2020-01-23 12:59:29 +00:00
Richard van der Hoff
6b7462a13f
a bit of debugging for media storage providers (#6757) 
* a bit of debugging for media storage providers

* changelog
 2020-01-23 12:11:44 +00:00
Richard van der Hoff
5bd3cb7260
Minor fixes to user admin api (#6761) 
* don't insist on a password (this is valid if you have an SSO login)
* fix reference to undefined `requester`
 2020-01-23 12:03:58 +00:00
Andrew Morgan
04345338e1 Merge branch 'release-v1.9.0' into develop 2020-01-23 11:38:29 +00:00
Andrew Morgan
d31f5f4d89
Update admin room docs with correct endpoints (#6770) 2020-01-23 11:37:26 +00:00
Andrew Morgan
ce84dd9e20
Remove unnecessary abstractions in admin handler (#6751) 2020-01-22 15:09:57 +00:00
Brendan Abolivier
0d0f32bc53 1.9.0rc1 2020-01-22 14:03:46 +00:00
Andrew Morgan
90a28fb475
Admin API to list, filter and sort rooms (#6720) 2020-01-22 13:36:43 +00:00
Brendan Abolivier
6ae0c8db33
Lint + changelog 2020-01-22 12:38:18 +00:00
Andrew Morgan
aa9b00fb2f
Fix and add test to deprecated quarantine media admin api (#6756) 2020-01-22 11:05:50 +00:00
Neil Johnson
5e52d8563b Allow monthly active user limiting support for worker mode, fixes #4639. (#6742) 2020-01-22 11:05:14 +00:00
Erik Johnston
5d7a6ad223
Allow streaming cache invalidate all to workers. (#6749) 2020-01-22 10:37:00 +00:00
Erik Johnston
2093f83ea0
Remove unused CI docker compose files (#6754) 
These now exist in the pipelines repo.
 2020-01-22 10:36:48 +00:00
Ivan Vilata-i-Balaguer
837f62266b Avoid attribute error when password_config present but empty (#6753) 
The old statement returned `None` for such a `password_config` (like the one
created on first run), thus retrieval of the `pepper` key failed with
`AttributeError`.

Fixes #5315

Signed-off-by: Ivan Vilata i Balaguer <ivan@selidor.net>
 2020-01-22 07:32:52 +00:00
Brendan Abolivier
07124d028d
Port synapse_port_db to async/await (#6718) 
* Raise an exception if there are pending background updates

So we return with a non-0 code

* Changelog

* Port synapse_port_db to async/await

* Port update_database to async/await

* Add version string to mocked homeservers

* Remove unused imports

* Convert overseen bits to async/await

* Fixup logging contexts

* Fix imports

* Add a way to print an error without raising an exception

* Incorporate review
 2020-01-21 19:04:58 +00:00
Erik Johnston
0e68760078
Add a DeltaState to track changes to be made to current state (#6716) 2020-01-20 18:07:20 +00:00
Erik Johnston
b0a66ab83c
Fixup synapse.rest to pass mypy (#6732) 2020-01-20 17:38:21 +00:00
Erik Johnston
74b74462f1
Fix /events/:event_id deprecated API. (#6731) 2020-01-20 17:38:09 +00:00
Erik Johnston
0f6e525be3
Fixup synapse.api to pass mypy (#6733) 2020-01-20 17:34:13 +00:00
Erik Johnston
ceecedc68b
Fix changing password via user admin API. (#6730) 2020-01-20 17:23:59 +00:00
Andrew Morgan
351fdfede6
Update changelog.d/6747.bugfix 
Co-Authored-By: Erik Johnston <erik@matrix.org>
 2020-01-20 15:58:44 +00:00
Andrew Morgan
026f4bdf3c Add changelog 2020-01-20 14:12:21 +00:00
Brendan Abolivier
a17f64361c
Add more logging around message retention policies support (#6717) 
So we can debug issues like #6683 more easily
 2020-01-17 20:51:44 +00:00
Erik Johnston
5909751936 Fix up changelog 2020-01-17 15:13:27 +00:00
Satsuki Yanagi
722b4f302d Fix syntax error in run_upgrade for schema 57 (#6728) 
Fix #6727
Related #6655

Co-authored-by: Erik Johnston <erikj@jki.re>
 2020-01-17 14:30:35 +00:00
Brendan Abolivier
3b72bb780a
Merge pull request #6714 from matrix-org/babolivier/retention_select_event 
Fix instantiation of message retention purge jobs
 2020-01-17 14:23:51 +00:00
Richard van der Hoff
59dc87c618
Merge pull request #6724 from matrix-org/rav/log_saml_attributes 
Log saml assertions rather than the whole response
 2020-01-17 10:33:24 +00:00
Richard van der Hoff
2b6a77fcde
Delegate remote_user_id mapping to the saml mapping provider (#6723) 
Turns out that figuring out a remote user id for the SAML user isn't quite as obvious as it seems. Factor it out to the SamlMappingProvider so that it's easy to control.
 2020-01-17 10:32:47 +00:00
Erik Johnston
a8a50f5b57
Wake up transaction queue when remote server comes back online (#6706) 
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
 2020-01-17 10:27:19 +00:00
Richard van der Hoff
5ce0b17e38
Clarify the account_validity and email sections of the sample configuration. (#6685) 
Generally try to make this more comprehensible, and make it match the
conventions.

I've removed the documentation for all the settings which allow you to change
the names of the template files, because I can't really see why they are
useful.
 2020-01-17 10:04:15 +00:00
Richard van der Hoff
95c5b9bfb3 changelog 2020-01-16 22:29:06 +00:00
Brendan Abolivier
4fb3cb208a
Precise changelog 2020-01-16 20:27:07 +00:00
Erik Johnston
d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Andrew Morgan
7b14c4a018
Add tips for the changelog to the pull request template (#6663) 2020-01-16 09:46:36 +00:00
Neil Johnson
38e0e59f42
Add org.matrix.e2e_cross_signing to unstable_features in /versions as per MSC1756 (#6712) 2020-01-16 09:46:14 +00:00
Erik Johnston
48c3a96886
Port synapse.replication.tcp to async/await (#6666) 
* Port synapse.replication.tcp to async/await

* Newsfile

* Correctly document type of on_<FOO> functions as async

* Don't be overenthusiastic with the asyncing....
 2020-01-16 09:16:12 +00:00
Brendan Abolivier
48e57a6452
Rename changelog 2020-01-15 19:40:52 +00:00
Brendan Abolivier
914e73cdd9
Changelog 2020-01-15 19:36:19 +00:00
Erik Johnston
19a1aac48c
Fix purge_room admin API (#6711) 2020-01-15 18:13:47 +00:00
Andrew Morgan
edc244eec4
Remove duplicate session check in web fallback servlet (#6702) 2020-01-15 18:05:18 +00:00
Richard van der Hoff
107f256cd8 Merge branch 'develop' into rav/module_api_extensions 2020-01-15 16:00:24 +00:00
Richard van der Hoff
8f5d7302ac
Implement RedirectException (#6687) 
Allow REST endpoint implemnentations to raise a RedirectException, which will
redirect the user's browser to a given location.
 2020-01-15 15:58:55 +00:00
Erik Johnston
28c98e51ff
Add local_current_membership table (#6655) 
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
 2020-01-15 14:59:33 +00:00
Erik Johnston
b5ce7f5874
Process EDUs in parallel with PDUs. (#6697) 
This means that things like to device messages don't get blocked behind
processing PDUs, which can potentially take *ages*.
 2020-01-14 14:08:35 +00:00
Erik Johnston
e8b68a4e4b
Fixup synapse.replication to pass mypy checks (#6667) 2020-01-14 14:08:06 +00:00
Andrew Morgan
1177d3f3a3
Quarantine media by ID or user ID (#6681) 2020-01-13 18:10:43 +00:00
Richard van der Hoff
47f4f493f0
Document more supported endpoints for workers (#6698) 2020-01-13 15:32:02 +00:00
Richard van der Hoff
326c893d24
Kill off RegistrationError (#6691) 
This is pretty pointless. Let's just use SynapseError.
 2020-01-13 12:48:22 +00:00
Richard van der Hoff
2d07c73777
Don't assign numeric IDs for empty usernames (#6690) 
Fix a bug where we would assign a numeric userid if somebody tried registering
with an empty username
 2020-01-13 12:47:30 +00:00
Richard van der Hoff
3cfac9593c
Merge pull request #6689 from matrix-org/rav/saml_mapping_provider_updates 
Updates to the SAML mapping provider API
 2020-01-13 12:44:55 +00:00
Richard van der Hoff
8039685051
Allow additional_resources to implement Resource directly (#6686) 
AdditionalResource really doesn't add any value, and it gets in the way for
resources which want to support child resources or the like. So, if the
resource object already implements the IResource interface, don't bother
wrapping it.
 2020-01-13 12:42:44 +00:00
Richard van der Hoff
feee819973
Fix exceptions on requests for non-ascii urls (#6682) 
Fixes #6402
 2020-01-13 12:41:51 +00:00
Richard van der Hoff
d56e95ea8b changelog 2020-01-12 21:42:15 +00:00
Richard van der Hoff
96ed33739a changelog 2020-01-12 21:36:10 +00:00
Richard van der Hoff
c3843fd075 changelog 2020-01-09 18:11:04 +00:00
Manuel Stahl
d2906fe666 Allow admin users to create or modify users without a shared secret (#6495) 
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2020-01-09 13:31:00 +00:00
Erik Johnston
d773290cb1 Merge branch 'master' into develop 2020-01-09 13:25:48 +00:00
Erik Johnston
24b2c940fb 1.8.0 2020-01-09 11:39:29 +00:00
Erik Johnston
7c232bd98b
Merge pull request #6664 from matrix-org/erikj/media_admin_apis 
Fix media repo admin APIs when using a media worker.
 2020-01-08 15:50:06 +00:00
Brendan Abolivier
e16521faab
Merge pull request #6665 from matrix-org/babolivier/retention_doc_typo 
Fix typo in message retention policies doc
 2020-01-08 13:57:02 +00:00
Erik Johnston
4e2a072a05 Newsfile 2020-01-08 13:28:19 +00:00
Brendan Abolivier
32ad2a3349
Changelog 2020-01-08 13:28:12 +00:00
Richard van der Hoff
b064a41291 Merge remote-tracking branch 'origin/release-v1.8.0' into develop 2020-01-08 13:27:17 +00:00
Manuel Stahl
7caaa29daa Fix GET request on /_synapse/admin/v2/users endpoint (#6563) 
Fixes #6552
 2020-01-08 13:26:40 +00:00
Richard van der Hoff
573fee759c
Back out ill-advised notary server hackery (#6657) 
This was ill-advised. We can't modify verify_keys here, because the response
object has already been signed by the requested key.

Furthermore, it's somewhat unnecessary because existing versions of Synapse
(which get upset that the notary key isn't present in verify_keys) will fall
back to a direct fetch via `/key/v2/server`.

Also: more tests for fetching keys via perspectives: it would be nice if we actually tested when our fetcher can't talk to our notary impl.
 2020-01-08 13:24:10 +00:00
Brendan Abolivier
cff1cb8685
Merge pull request #6624 from matrix-org/babolivier/retention_doc 
Add complete documentation of the message retention policies support
 2020-01-08 11:24:47 +00:00
Fabian Meyer
dd57715de2 contrib/docker-compose: fixing mount that overrides containers' /etc (#6656) 
The mount in the form of ./matrix-config:/etc overwrites the contents of the container /etc folder. Since all valid ca certificates are stored in /etc, the synapse.push.httppusher, for example, cannot validate the certificate from matrix.org.
 2020-01-08 07:25:05 +00:00
Erik Johnston
be29ed7ad8
Correctly proxy remote group HTTP errors. (#6654) 
e.g. if remote returns a 404 then that shouldn't be treated as an error
but should be proxied through.
 2020-01-07 15:36:41 +00:00
Brendan Abolivier
2b6b7f482a
Merge pull request #6621 from matrix-org/babolivier/purge_job_config_typo 
Fix a typo in the purge jobs configuration example
 2020-01-07 16:17:40 +01:00
Brendan Abolivier
4be582d7c8 Merge branch 'develop' into babolivier/retention_doc 2020-01-07 15:07:19 +00:00
Brendan Abolivier
03edfc5850
Update changelog.d/6624.doc 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-01-07 15:59:05 +01:00
Erik Johnston
7f0e706ebf 1.8.0rc1 2020-01-07 14:31:13 +00:00
Brendan Abolivier
0ab5853ec9
Merge pull request #6652 from matrix-org/babolivier/depth_missing_events 
Fix conditions failing if min_depth = 0
 2020-01-07 15:22:07 +01:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory (#6648) 
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
 2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates (#6647) 
so that bg update routines can be async
 2020-01-07 14:12:42 +00:00
Richard van der Hoff
d20c346544
port BackgroundUpdateTestCase to HomeserverTestCase (#6653) 2020-01-07 14:09:07 +00:00
Richard van der Hoff
1ff5491117
Merge pull request #6645 from matrix-org/rav/fix_synchrotron_error 
Fix exceptions in the synchrotron worker log when events are rejected.
 2020-01-07 14:02:14 +00:00
Brendan Abolivier
1f2a5923d4
Changelog 2020-01-07 13:12:17 +00:00
Richard van der Hoff
1807db5e73
Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashes 
Remove a bunch of unused code from event creation
 2020-01-06 17:46:31 +00:00
Richard van der Hoff
055e6fbaa2 changelog 2020-01-06 17:17:40 +00:00
Richard van der Hoff
ba897a7590
Fix some test failures when frozen_dicts are enabled (#6642) 
Fixes #4026
 2020-01-06 15:22:46 +00:00
Erik Johnston
9f6c1befbb
Add experimental 'databases' config (#6580) 2020-01-06 14:44:01 +00:00
Richard van der Hoff
ab4b4ee6a7
Fix an error which was thrown by the PresenceHandler _on_shutdown handler. (#6640) 2020-01-06 14:34:02 +00:00
Richard van der Hoff
550b2946d8 changelog 2020-01-06 13:45:33 +00:00
Richard van der Hoff
4b36b482e0
Fix exception when fetching notary server's old keys (#6625) 
Lift the restriction that *all* the keys used for signing v2 key responses be
present in verify_keys.

Fixes #6596.
 2020-01-06 12:33:56 +00:00
Richard van der Hoff
18674eebb1
Workaround for error when fetching notary's own key (#6620) 
* Kill off redundant SynapseRequestFactory

We already get the Site via the Channel, so there's no need for a dedicated
RequestFactory: we can just use the right constructor.

* Workaround for error when fetching notary's own key

As a notary server, when we return our own keys, include all of our signing
keys in verify_keys.

This is a workaround for #6596.
 2020-01-06 12:28:58 +00:00
Andrew Morgan
01c3c6c929 Fix power levels being incorrectly set in old and new rooms after a room upgrade (#6633) 
Modify a copy of an upgraded room's PL before sending to the new room
 2020-01-06 09:53:07 +00:00
Richard van der Hoff
08815566bc
Automate generation of the sample and debian log configs (#6627) 2020-01-03 17:14:00 +00:00
Richard van der Hoff
e484101306
Raise an error if someone tries to use the log_file config option (#6626) 
This has caused some confusion for people who didn't notice it going away.
 2020-01-03 17:11:29 +00:00
Richard van der Hoff
98247c4a0e
Remove unused, undocumented "content repo" resource (#6628) 
This looks like it got half-killed back in #888.

Fixes #6567.
 2020-01-03 17:10:52 +00:00
Richard van der Hoff
b6b57ecb4e
Kill off redundant SynapseRequestFactory (#6619) 
We already get the Site via the Channel, so there's no need for a dedicated
RequestFactory: we can just use the right constructor.
 2020-01-03 14:19:48 +00:00
Richard van der Hoff
6964ea095b
Reduce the reconnect time when replication fails. (#6617) 2020-01-03 14:19:09 +00:00
Brendan Abolivier
51b8a21f0c
Rename changelog 2020-01-03 13:49:12 +01:00
Brendan Abolivier
9279a2c4e4
Add a complete documentation of the message retention policies support 2020-01-03 13:45:03 +01:00
Brendan Abolivier
9c59bc59c8
Changelog 2020-01-03 13:00:32 +01:00
ewaf1
0495097a7f Added the section 'Configuration' in /docs/turn-howto.md (#6614) 
put the 2nd part of the "source installation"-section into a new section, because it also applies to Debian packages
 2020-01-02 10:41:30 +00:00
Aaron Raimist
32779b59fa Reword sections of federate.md that explained delegation at time of Synapse 1.0 transition (#6601) 
* Remove sections of federate.md explaining delegation at time of Synapse 1.0 transition

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2020-01-02 10:28:20 +00:00
Erik Johnston
75d8f26ac8
Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Erik Johnston
fa780e9721
Change EventContext to use the Storage class (#6564) 2019-12-20 10:32:02 +00:00
Erik Johnston
0b5dbadd96
Explode on duplicate delta file names. (#6565) 2019-12-19 15:07:37 +00:00
Erik Johnston
3d46124ad0
Port some admin handlers to async/await (#6559) 2019-12-19 15:07:28 +00:00
Richard van der Hoff
bca30cefee
Improve diagnostics on database upgrade failure (#6570) 
`Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst
has anything useful.
 2019-12-19 14:53:15 +00:00
Richard van der Hoff
0b794cbd7b
Fix sdnotify with acme enabled (#6571) 
If acme was enabled, the sdnotify startup hook would never be run because we
would try to add it to a hook which had already fired.

There's no need to delay it: we can sdnotify as soon as we've started the
listeners.
 2019-12-19 14:52:52 +00:00
Richard van der Hoff
b95b762560
Add an export_signing_key script (#6546) 
I want to do some key rotation, and it is silly that we don't have a way to do
this.
 2019-12-19 11:11:14 +00:00
Richard van der Hoff
d6752ce5da
Clean up startup for the pusher (#6558) 
* Remove redundant python2 support code

`str.decode()` doesn't exist on python3, so presumably this code was doing
nothing

* Filter out pushers with corrupt data

When we get a row with unparsable json, drop the row, rather than returning a
row with null `data`, which will then cause an explosion later on.

* Improve logging when we can't start a pusher

Log the ID to help us understand the problem

* Make email pusher setup more robust

We know we'll have a `data` member, since that comes from the database. What we
*don't* know is if that is a dict, and if that has a `brand` member, and if
that member is a string.
 2019-12-18 14:26:58 +00:00
Andrew Morgan
7963ca83cb
Add delta file to fix missing default table data (#6555) 2019-12-18 11:13:33 +00:00
Erik Johnston
2284eb3a53
Add database config class (#6513) 
This encapsulates config for a given database and is the way to get new
connections.
 2019-12-18 10:45:12 +00:00
Richard van der Hoff
91ccfe9f37 Merge branch 'master' into develop 2019-12-18 10:16:39 +00:00
Richard van der Hoff
6e8f8e14f2 Merge release-v1.7.1 into develop 2019-12-18 09:51:51 +00:00
Erik Johnston
5029422530
Fix bug where we added duplicate event IDs as auth_events (#6560) 2019-12-17 15:06:08 +00:00
Erik Johnston
02553901ce
Remove unused get_pagination_rows methods. (#6557) 
Remove unused get_pagination_rows methods
 2019-12-17 11:44:32 +00:00
Erik Johnston
5ca2cfadc3 Add auth events as per spec. (#6556) 
Previously we tried to be clever and filter out some unnecessary event
IDs to keep the auth chain small, but that had some annoying
interactions with state res v2 so we stop doing that for now.
 2019-12-16 17:05:01 +00:00
Erik Johnston
3fbe5b7ec3
Add auth events as per spec. (#6556) 
Previously we tried to be clever and filter out some unnecessary event
IDs to keep the auth chain small, but that had some annoying
interactions with state res v2 so we stop doing that for now.
 2019-12-16 16:59:32 +00:00
Brendan Abolivier
6316530366
Merge pull request #6553 from matrix-org/babolivier/fix-context-filter 
Use the filtered version of an event when responding to /context requests for that event
 2019-12-16 16:24:10 +00:00
Will Hunt
bfb95654c9 Add option to allow profile queries without sharing a room (#6523) 2019-12-16 16:11:55 +00:00
Brendan Abolivier
284e690aa0
Update changelog.d/6553.bugfix 
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2019-12-16 14:56:05 +00:00
Richard van der Hoff
bbb75ff6ee Exclude rejected state events when calculating state at backwards extrems (#6527) 
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
 2019-12-16 14:07:29 +00:00
Richard van der Hoff
ff773ff724 Persist auth/state events at backwards extremities when we fetch them (#6526) 
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
 2019-12-16 14:07:11 +00:00
Richard van der Hoff
83895316d4 sanity-checking for events used in state res (#6531) 
When we perform state resolution, check that all of the events involved are in
the right room.
 2019-12-16 13:53:53 +00:00
Richard van der Hoff
6577f2d887 Sanity-check room ids in event auth (#6530) 
When we do an event auth operation, check that all of the events involved are
in the right room.
 2019-12-16 13:47:34 +00:00
Richard van der Hoff
35bbe4ca79 Check the room_id of events when fetching room state/auth (#6524) 
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
 2019-12-16 13:47:07 +00:00
Richard van der Hoff
20d5ba16e6 Add include_event_in_state to _get_state_for_room (#6521) 
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
 2019-12-16 13:26:12 +00:00
Richard van der Hoff
be294d6fde Move get_state methods into FederationHandler (#6503) 
This is a non-functional refactor as a precursor to some other work.
 2019-12-16 13:20:21 +00:00
Richard van der Hoff
4c7b1bb6cc Refactor get_events_from_store_or_dest to return a dict (#6501) 
There was a bunch of unnecessary conversion back and forth between dict and
list going on here. We can simplify a bunch of the code.
 2019-12-16 13:19:32 +00:00
Richard van der Hoff
6920d88892
Exclude rejected state events when calculating state at backwards extrems (#6527) 
This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
 2019-12-16 13:14:37 +00:00
Richard van der Hoff
bc7de87650
Persist auth/state events at backwards extremities when we fetch them (#6526) 
The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
 2019-12-16 12:26:28 +00:00
Brendan Abolivier
8b9f5c21c3
Changelog 2019-12-16 12:19:35 +00:00
Werner Sembach
9d173b312c Automatically delete empty groups/communities (#6453) 
Signed-off-by: Werner Sembach <werner.sembach@fau.de>
 2019-12-16 12:12:40 +00:00
Andrew Morgan
0b90fc6ed2
Document Shutdown Room admin API (#6541) 2019-12-13 15:28:48 +00:00
Richard van der Hoff
1da15f05f5
sanity-checking for events used in state res (#6531) 
When we perform state resolution, check that all of the events involved are in
the right room.
 2019-12-13 12:55:32 +00:00
Richard van der Hoff
971a0702b5
Sanity-check room ids in event auth (#6530) 
When we do an event auth operation, check that all of the events involved are
in the right room.
 2019-12-13 11:44:41 +00:00
Erik Johnston
5cadbd9ebb
Merge pull request #6537 from matrix-org/erikj/bump_mypy_version 
Bump mypy version
 2019-12-13 11:32:53 +00:00
Richard van der Hoff
4ce05ec171
Adjust the sytest blacklist for worker mode (#6538) 
Remove tests that got blacklisted while torturing was enabled, and add one that
fails.
 2019-12-13 10:15:20 +00:00
Erik Johnston
caa52836e4
Merge pull request #6496 from matrix-org/erikj/initial_sync_asnyc 
Port synapse.handlers.initial_sync to async/await
 2019-12-13 10:01:51 +00:00
Erik Johnston
7e67cfc87d
Merge pull request #6534 from matrix-org/erikj/extend_mypy 
Include more folders in mypy
 2019-12-12 17:21:42 +00:00
Hubert Chathi
cb2db17994
look up cross-signing keys from the DB in bulk (#6486) 2019-12-12 12:03:28 -05:00
Andrew Morgan
5bfd8855d6
Fix redacted events being returned in search results ordered by "recent" (#6522) 2019-12-12 15:53:49 +00:00
Erik Johnston
5056d6d90a Newsfile 2019-12-12 15:22:46 +00:00
Erik Johnston
c965253e4b Newsfile 2019-12-12 14:54:03 +00:00
Richard van der Hoff
25f1244329
Check the room_id of events when fetching room state/auth (#6524) 
When we request the state/auth_events to populate a backwards extremity (on
backfill or in the case of missing events in a transaction push), we should
check that the returned events are in the right room rather than blindly using
them in the room state or auth chain.

Given that _get_events_from_store_or_dest takes a room_id, it seems clear that
it should be sanity-checking the room_id of the requested events, so let's do
it there.
 2019-12-12 12:57:45 +00:00
Erik Johnston
b8e4b39b69
Merge pull request #6511 from matrix-org/erikj/remove_db_config_from_apps 
Move database config from apps into HomeServer object
 2019-12-12 10:37:56 +00:00
Erik Johnston
6828b47c45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/initial_sync_asnyc 2019-12-11 17:01:41 +00:00
Richard van der Hoff
2045356517
Add include_event_in_state to _get_state_for_room (#6521) 
Make it return the state *after* the requested event, rather than the one
before it. This is a bit easier and requires fewer calls to
get_events_from_store_or_dest.
 2019-12-11 16:37:51 +00:00
Richard van der Hoff
894d2addac
Merge pull request #6517 from matrix-org/rav/event_auth/13 
Port some of FederationHandler to async/await
 2019-12-11 16:36:06 +00:00
Erik Johnston
31905a518e
Merge pull request #6504 from matrix-org/erikj/account_validity_async_await 
Port handlers.account_validity to async/await.
 2019-12-11 14:49:26 +00:00
Richard van der Hoff
5324bc20a6 changelog 2019-12-11 14:39:26 +00:00
Richard van der Hoff
7c429f92d6
Clean up some logging (#6515) 
This just makes some of the logging easier to follow when things start going
wrong.
 2019-12-11 14:32:25 +00:00
Erik Johnston
adb3a873fd Synapse 1.7.0rc2 (2019-12-11) 
=============================
 
 Bugfixes
 --------
 
 - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
 - Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
 - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl3w+RwQHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHijfD/43EQ58jqKvD+qKZwVFOE6JJ3SCS7UJbi3f
 zq9KWDuCB6EAFjAgmJikbBBqHPO5qq2WtNXaCpexx44s8Mk8SZKHg56dP6Fk651C
 assAb/Nsh6CAlPUcRkx8I0L/kYXMPDyATlLVBHVOi3pFDJ093mdOQ4q8yP9iUTM+
 OPsbT8k/pMhrhCH951bGmB6/SEcju+ubObW+bRFe8o3v1KE9jVYQjUGMhuoXp3pM
 z/OB8idZcqOvCc6HMo83tg9FuI613Jy80PMIc1ofyJgvnu+aDBepWuldvFEMnmSR
 D862jMor7+WdnDOTeWZrC+DXjl0qCP8F6ahs5rEllRglt/Ep2wEDPA/8YrRoEI3V
 RWe9W7XDdFCXdzlvXheOfETqTu9kdsurTwBEeJrWQ0vOLY86hxt9KKKcHVhy5eKq
 kNfRtvSVLmRhIssp7hVBcywRwnaxN7R2OoRq/TWnTZz+xEOPzYFU6r0l9kk5dbg6
 fqYYxIXbgZlhjihLWNIMNwwZH9ll/eoPiinkRTZNz40THP/VDTR9DM4tQ6jrhrr6
 sP0qM7LljvrdiimXtV00tUDyUspNgJl6xDJyGDHWM9uoCB7uorEpMQZSzlZhZe8s
 6q+fQPHlfW4JYOejfihPkjrV1ViawvEucqWPaIsD+v26C4RP7qCTtYj3NPiA65of
 zhOjomWWcg==
 =ABoZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.0rc2' into develop

Synapse 1.7.0rc2 (2019-12-11)
=============================

Bugfixes
--------

- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
- Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
 2019-12-11 14:14:30 +00:00
Erik Johnston
d156912c4c 1.7.0rc2 2019-12-11 13:56:50 +00:00
Andrew Morgan
fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Andrew Morgan
6676ee9c4a
Add dev script to generate full SQL schema files (#6394) 2019-12-11 13:16:01 +00:00
Andrew Morgan
ea0f0ad414
Prevent message search in upgraded rooms we're not in (#6385) 2019-12-11 13:07:25 +00:00
Erik Johnston
72acca6a32 Back out change preventing setting null avatar URLs 2019-12-11 11:46:55 +00:00
Brendan Abolivier
54ae52ba96
Merge pull request #6349 from matrix-org/babolivier/msc1802 
Implement v2 APIs for send_join and send_leave
 2019-12-11 11:41:47 +00:00
Erik Johnston
d21577bdcb Merge branch 'erikj/fix_sqlite_7' of github.com:matrix-org/synapse into release-v1.7.0 2019-12-11 11:34:50 +00:00
Erik Johnston
239d86a134
Merge pull request #6512 from matrix-org/erikj/silence_mypy 
Silence mypy errors for files outside those specified
 2019-12-11 10:39:57 +00:00
Richard van der Hoff
f8bc2ae883
Move get_state methods into FederationHandler (#6503) 
This is a non-functional refactor as a precursor to some other work.
 2019-12-10 17:42:46 +00:00
Andrew Morgan
4947de5a14
Allow SAML username provider plugins (#6411) 2019-12-10 17:30:16 +00:00
Erik Johnston
b2dcddc413
Merge pull request #6510 from matrix-org/erikj/phone_home_stats_db 
Phone home stats DB reporting should not assume a single DB.
 2019-12-10 16:31:24 +00:00
Richard van der Hoff
40eda84933
Fix race which caused deleted devices to reappear (#6514) 
Stop the `update_client_ips` background job from recreating deleted devices.
 2019-12-10 16:22:29 +00:00
Richard van der Hoff
c3dda2874d
Refactor get_events_from_store_or_dest to return a dict (#6501) 
There was a bunch of unnecessary conversion back and forth between dict and
list going on here. We can simplify a bunch of the code.
 2019-12-10 16:22:00 +00:00
Richard van der Hoff
424fd58237
Remove redundant code from event authorisation implementation. (#6502) 2019-12-10 15:09:45 +00:00
Erik Johnston
3f97b4c16b Newsfile 2019-12-10 14:40:15 +00:00
Erik Johnston
d630c82349 Newsfile 2019-12-10 14:34:17 +00:00
Erik Johnston
35f3c366ef
Merge pull request #6505 from matrix-org/erikj/make_deferred_yiedable 
Fix `make_deferred_yieldable` to work with coroutines
 2019-12-10 14:20:26 +00:00
Brendan Abolivier
ae49d29ef1
Fixup changelogs 2019-12-10 13:55:03 +00:00
Erik Johnston
e726e18737
Merge pull request #6499 from matrix-org/erikj/fix_sqlite_7 
Fix support for SQLite 3.7.
 2019-12-10 13:43:52 +00:00
Erik Johnston
4643bb2a37 Newsfile 2019-12-10 13:36:00 +00:00
Erik Johnston
accd343f91 Newsfile 2019-12-10 13:22:42 +00:00
Brendan Abolivier
451ec9b8b9
Changelog 2019-12-10 13:06:41 +00:00
Erik Johnston
e3f528c544
Merge pull request #6506 from matrix-org/erikj/remove_snapshot_cache 
Remove SnapshotCache in favour of ResponseCache
 2019-12-10 13:04:50 +00:00
Brendan Abolivier
ec5fdd1333
Changelog 2019-12-10 12:34:33 +00:00
Erik Johnston
b1e7012dee Newsfile 2019-12-10 11:29:44 +00:00
Erik Johnston
f5bb1531b7 Newsfile 2019-12-10 11:23:52 +00:00
Erik Johnston
52fe9788bc Newsfile 2019-12-09 15:19:32 +00:00
Neil Johnson
0f3614f0f6 Synapse 1.7.0rc1 (2019-12-09) 
=============================
 
 Features
 --------
 
 - Implement per-room message retention policies. ([\#5815](https://github.com/matrix-org/synapse/issues/5815), [\#6436](https://github.com/matrix-org/synapse/issues/6436))
 - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](https://github.com/matrix-org/synapse/issues/5858))
 - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
 - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](https://github.com/matrix-org/synapse/issues/6119))
 - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](https://github.com/matrix-org/synapse/issues/6176))
 - Configure privacy-preserving settings by default for the room directory. ([\#6354](https://github.com/matrix-org/synapse/issues/6354))
 - Add ephemeral messages support by partially implementing [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). ([\#6409](https://github.com/matrix-org/synapse/issues/6409))
 - Add support for [MSC 2367](https://github.com/matrix-org/matrix-doc/pull/2367), which allows specifying a reason on all membership events. ([\#6434](https://github.com/matrix-org/synapse/issues/6434))
 
 Bugfixes
 --------
 
 - Transfer non-standard power levels on room upgrade. ([\#6237](https://github.com/matrix-org/synapse/issues/6237))
 - Fix error from the Pillow library when uploading RGBA images. ([\#6241](https://github.com/matrix-org/synapse/issues/6241))
 - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](https://github.com/matrix-org/synapse/issues/6329))
 - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](https://github.com/matrix-org/synapse/issues/6332))
 - Prevent account data syncs getting lost across TCP replication. ([\#6333](https://github.com/matrix-org/synapse/issues/6333))
 - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](https://github.com/matrix-org/synapse/issues/6406))
 - Fix an intermittent exception when handling read-receipts. ([\#6408](https://github.com/matrix-org/synapse/issues/6408))
 - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](https://github.com/matrix-org/synapse/issues/6420))
 - Fix startup error when http proxy is defined. ([\#6421](https://github.com/matrix-org/synapse/issues/6421))
 - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](https://github.com/matrix-org/synapse/issues/6449))
 - Fix uploading multiple cross signing signatures for the same user. ([\#6451](https://github.com/matrix-org/synapse/issues/6451))
 - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](https://github.com/matrix-org/synapse/issues/6462))
 - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](https://github.com/matrix-org/synapse/issues/6470))
 - Improve sanity-checking when receiving events over federation. ([\#6472](https://github.com/matrix-org/synapse/issues/6472))
 - Fix inaccurate per-block Prometheus metrics. ([\#6491](https://github.com/matrix-org/synapse/issues/6491))
 - Fix small performance regression for sending invites. ([\#6493](https://github.com/matrix-org/synapse/issues/6493))
 - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](https://github.com/matrix-org/synapse/issues/6494))
 
 Improved Documentation
 ----------------------
 
 - Update documentation and variables in user contributed systemd reference file. ([\#6369](https://github.com/matrix-org/synapse/issues/6369), [\#6490](https://github.com/matrix-org/synapse/issues/6490))
 - Fix link in the user directory documentation. ([\#6388](https://github.com/matrix-org/synapse/issues/6388))
 - Add build instructions to the docker readme. ([\#6390](https://github.com/matrix-org/synapse/issues/6390))
 - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](https://github.com/matrix-org/synapse/issues/6443))
 - Write some docs for the quarantine_media api. ([\#6458](https://github.com/matrix-org/synapse/issues/6458))
 - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](https://github.com/matrix-org/synapse/issues/6461))
 
 Deprecations and Removals
 -------------------------
 
 - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
 - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](https://github.com/matrix-org/synapse/issues/6488))
 
 Internal Changes
 ----------------
 
 - Add benchmarks for structured logging and improve output performance. ([\#6266](https://github.com/matrix-org/synapse/issues/6266))
 - Improve the performance of outputting structured logging. ([\#6322](https://github.com/matrix-org/synapse/issues/6322))
 - Refactor some code in the event authentication path for clarity. ([\#6343](https://github.com/matrix-org/synapse/issues/6343), [\#6468](https://github.com/matrix-org/synapse/issues/6468), [\#6480](https://github.com/matrix-org/synapse/issues/6480))
 - Clean up some unnecessary quotation marks around the codebase. ([\#6362](https://github.com/matrix-org/synapse/issues/6362))
 - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](https://github.com/matrix-org/synapse/issues/6379))
 - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](https://github.com/matrix-org/synapse/issues/6392))
 - Clarifications for the email configuration settings. ([\#6423](https://github.com/matrix-org/synapse/issues/6423))
 - Add more tests to the blacklist when running in worker mode. ([\#6429](https://github.com/matrix-org/synapse/issues/6429))
 - Refactor data store layer to support multiple databases in the future. ([\#6454](https://github.com/matrix-org/synapse/issues/6454), [\#6464](https://github.com/matrix-org/synapse/issues/6464), [\#6469](https://github.com/matrix-org/synapse/issues/6469), [\#6487](https://github.com/matrix-org/synapse/issues/6487))
 - Port synapse.rest.client.v1 to async/await. ([\#6482](https://github.com/matrix-org/synapse/issues/6482))
 - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](https://github.com/matrix-org/synapse/issues/6483))
 - Port SyncHandler to async/await. ([\#6484](https://github.com/matrix-org/synapse/issues/6484))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAl3uYYkACgkQCTxDAAxP
 S/QWMQ/8CKgO5cFl6kMxS6EjZJ6C8rA2QxP0iEcveqMj++yT3pQmdtQ5zzO100tS
 ZZYAdQbHkk/HWW8TIUtlxC+pcb9ZL0sZejY6dXzhD0dBVkFfhXQhJeFAkZnlf5lu
 CR8/kKTJ9IMgL6WtseOCoZQV/N9ifurWqqb8d0J5H5w1i//CP8opUQTn1a7lUzgm
 +1KleBv+A+gfltcm+NuC3+Q0GJNqfmtjcneOGwXXiuS3Ux8r2uLpbQsQfuNMMRkP
 ItmLJLy8hkP2yncrM9NE1Jkh0uOwcBJczbmT1OpXDIJiueFl7FxLKFXQM17QrpOT
 AqVDnVTlXfTDvWFeFstNz84uceXz98tLKZIl0X0inkN/wo3ZxMmzzxQw2Ob55ZPY
 pT6oC1gm0bGNQ/smEonyMM3z0ir7OfAce/tFQigfo59qeA08uzeckWkWpiXTbmy5
 xu2lomhpQABBNBhEnVpm4+dMXeXWzz1NtkeC6auBn+eX19JqlWZM9WcuNfv4rC+M
 eg5mwv2qERBA+qLsZQLy0dEMUy9fIrjd0GklMl6GUKYiSAsbDIptYMZl504D9Emc
 adTY63BT7TTiu64lr+CiQ9BbWQkGJxM1MQ8BoYyaKvgtahmOi+0Y5R0RGmrG6Hju
 UhgMdjmt5HnN0wiepBFCth9nJq+XDdyls8xAKGtSzGt129krBP4=
 =queo
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.0rc1' into develop

Synapse 1.7.0rc1 (2019-12-09)
=============================

Features
--------

- Implement per-room message retention policies. ([\#5815](https://github.com/matrix-org/synapse/issues/5815), [\#6436](https://github.com/matrix-org/synapse/issues/6436))
- Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](https://github.com/matrix-org/synapse/issues/5858))
- Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
- Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](https://github.com/matrix-org/synapse/issues/6119))
- Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](https://github.com/matrix-org/synapse/issues/6176))
- Configure privacy-preserving settings by default for the room directory. ([\#6354](https://github.com/matrix-org/synapse/issues/6354))
- Add ephemeral messages support by partially implementing [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). ([\#6409](https://github.com/matrix-org/synapse/issues/6409))
- Add support for [MSC 2367](https://github.com/matrix-org/matrix-doc/pull/2367), which allows specifying a reason on all membership events. ([\#6434](https://github.com/matrix-org/synapse/issues/6434))

Bugfixes
--------

- Transfer non-standard power levels on room upgrade. ([\#6237](https://github.com/matrix-org/synapse/issues/6237))
- Fix error from the Pillow library when uploading RGBA images. ([\#6241](https://github.com/matrix-org/synapse/issues/6241))
- Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](https://github.com/matrix-org/synapse/issues/6329))
- Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](https://github.com/matrix-org/synapse/issues/6332))
- Prevent account data syncs getting lost across TCP replication. ([\#6333](https://github.com/matrix-org/synapse/issues/6333))
- Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](https://github.com/matrix-org/synapse/issues/6406))
- Fix an intermittent exception when handling read-receipts. ([\#6408](https://github.com/matrix-org/synapse/issues/6408))
- Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](https://github.com/matrix-org/synapse/issues/6420))
- Fix startup error when http proxy is defined. ([\#6421](https://github.com/matrix-org/synapse/issues/6421))
- Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](https://github.com/matrix-org/synapse/issues/6449))
- Fix uploading multiple cross signing signatures for the same user. ([\#6451](https://github.com/matrix-org/synapse/issues/6451))
- Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](https://github.com/matrix-org/synapse/issues/6462))
- Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](https://github.com/matrix-org/synapse/issues/6470))
- Improve sanity-checking when receiving events over federation. ([\#6472](https://github.com/matrix-org/synapse/issues/6472))
- Fix inaccurate per-block Prometheus metrics. ([\#6491](https://github.com/matrix-org/synapse/issues/6491))
- Fix small performance regression for sending invites. ([\#6493](https://github.com/matrix-org/synapse/issues/6493))
- Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](https://github.com/matrix-org/synapse/issues/6494))

Improved Documentation
----------------------

- Update documentation and variables in user contributed systemd reference file. ([\#6369](https://github.com/matrix-org/synapse/issues/6369), [\#6490](https://github.com/matrix-org/synapse/issues/6490))
- Fix link in the user directory documentation. ([\#6388](https://github.com/matrix-org/synapse/issues/6388))
- Add build instructions to the docker readme. ([\#6390](https://github.com/matrix-org/synapse/issues/6390))
- Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](https://github.com/matrix-org/synapse/issues/6443))
- Write some docs for the quarantine_media api. ([\#6458](https://github.com/matrix-org/synapse/issues/6458))
- Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](https://github.com/matrix-org/synapse/issues/6461))

Deprecations and Removals
-------------------------

- Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
- Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](https://github.com/matrix-org/synapse/issues/6488))

Internal Changes
----------------

- Add benchmarks for structured logging and improve output performance. ([\#6266](https://github.com/matrix-org/synapse/issues/6266))
- Improve the performance of outputting structured logging. ([\#6322](https://github.com/matrix-org/synapse/issues/6322))
- Refactor some code in the event authentication path for clarity. ([\#6343](https://github.com/matrix-org/synapse/issues/6343), [\#6468](https://github.com/matrix-org/synapse/issues/6468), [\#6480](https://github.com/matrix-org/synapse/issues/6480))
- Clean up some unnecessary quotation marks around the codebase. ([\#6362](https://github.com/matrix-org/synapse/issues/6362))
- Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](https://github.com/matrix-org/synapse/issues/6379))
- Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](https://github.com/matrix-org/synapse/issues/6392))
- Clarifications for the email configuration settings. ([\#6423](https://github.com/matrix-org/synapse/issues/6423))
- Add more tests to the blacklist when running in worker mode. ([\#6429](https://github.com/matrix-org/synapse/issues/6429))
- Refactor data store layer to support multiple databases in the future. ([\#6454](https://github.com/matrix-org/synapse/issues/6454), [\#6464](https://github.com/matrix-org/synapse/issues/6464), [\#6469](https://github.com/matrix-org/synapse/issues/6469), [\#6487](https://github.com/matrix-org/synapse/issues/6487))
- Port synapse.rest.client.v1 to async/await. ([\#6482](https://github.com/matrix-org/synapse/issues/6482))
- Port synapse.rest.client.v2_alpha to async/await. ([\#6483](https://github.com/matrix-org/synapse/issues/6483))
- Port SyncHandler to async/await. ([\#6484](https://github.com/matrix-org/synapse/issues/6484))
 2019-12-09 15:04:08 +00:00
Andrew Morgan
5e8abe9013
Better errors regarding changing avatar_url (#6497) 2019-12-09 14:54:33 +00:00
Neil Johnson
24da1ffcb6 1.7.0rc1 2019-12-09 14:46:20 +00:00
Clifford Garwood II
96d35f1028 Systemd documentation (#6490) 2019-12-09 14:42:29 +00:00
Neil Johnson
adfdd82b21
Back out perf regression from get_cross_signing_keys_from_cache. (#6494) 
Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression.
 2019-12-09 13:59:27 +00:00
Erik Johnston
30e9adf32f
Merge pull request #6487 from matrix-org/erikj/pass_in_db 
Pass in Database object to data stores.
 2019-12-09 13:53:21 +00:00
Erik Johnston
aeaeb72ee4 Newsfile 2019-12-09 13:48:14 +00:00
Erik Johnston
e1544b0af8
Merge pull request #6493 from matrix-org/erikj/invite_state_config 
Pull out room_invite_state_types config option once.
 2019-12-09 12:23:21 +00:00
Richard van der Hoff
18660a34d8
Fix inaccurate per-block metrics (#6491) 
`Measure` incorrectly assumed that it was the only thing being done by the parent `LoggingContext`. For instance, during a "renew group attestations" operation, hundreds of `outbound_request` calls could take place in parallel, all using the same `LoggingContext`. This would mean that any resources used during *any* of those calls would be reported against *all* of them, producing wildly inaccurate results.

Instead, we now give each `Measure` block its own `LoggingContext` (using the parent `LoggingContext` mechanism to ensure that the log lines look correct and that the metrics are ultimately propogated to the top level for reporting against requests/backgrond tasks).
 2019-12-09 11:55:30 +00:00
Erik Johnston
4a161a29ac Newsfile 2019-12-09 11:54:43 +00:00
Richard van der Hoff
e519489fc4
Remove fallback for missing /federation/v1/state_ids API (#6488) 
This API was added way back in 0.17.0; the code here is annoying to maintain
and entirely redundant.
 2019-12-09 11:37:26 +00:00
Erik Johnston
a9b393340f
Merge pull request #6484 from matrix-org/erikj/port_sync_handler 
Port SyncHandler to async/await
 2019-12-09 11:32:44 +00:00
Erik Johnston
5e35f69ac3 Newsfile 2019-12-06 16:13:41 +00:00
Erik Johnston
2ace775d88 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_database_class 2019-12-06 11:33:34 +00:00
Brendan Abolivier
e126d83f74 Merge branch 'develop' into babolivier/msc1802 2019-12-05 21:00:43 +00:00
Manuel Stahl
649b6bc088 Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925) 2019-12-05 18:12:23 +00:00
Erik Johnston
b2ee65ea8c Newsfile 2019-12-05 17:59:37 +00:00
Erik Johnston
edb8b6af9a Newsfile 2019-12-05 16:56:23 +00:00
Erik Johnston
af5d0ebc72
Merge pull request #6482 from matrix-org/erikj/port_rest_v1 
Port rest/v1 to async/await
 2019-12-05 16:40:06 +00:00
Erik Johnston
410bfd035a Newsfile 2019-12-05 15:57:28 +00:00
Andrew Morgan
09623446d4 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-12-05 15:46:39 +00:00
Andrew Morgan
ff119879d6 Revert "Modify systemd unit file reference to align with installation instruction (#6369)" 
This reverts commit dc8747895e.
 2019-12-05 15:46:19 +00:00
Andrew Morgan
e2cce15af1 Remove #6369 changelog 2019-12-05 15:44:02 +00:00
Clifford Garwood II
dc8747895e Modify systemd unit file reference to align with installation instruction (#6369) 
Signed-off-by: Clifford Garwood II cliff@cigii.com
 2019-12-05 15:43:32 +00:00
Richard van der Hoff
63d6ad1064
Stronger typing in the federation handler (#6480) 
replace the event_info dict with an attrs thing
 2019-12-05 15:02:35 +00:00
Richard van der Hoff
e1f4c83f41
Sanity-check the rooms of auth events before pulling them in. (#6472) 2019-12-05 14:14:45 +00:00
Clifford Garwood II
ba7af15d4e Modify systemd unit file reference to align with installation instruction (#6369) 
Signed-off-by: Clifford Garwood II cliff@cigii.com
 2019-12-05 13:13:47 +00:00
Erik Johnston
6dcd6c40a0 Newsfile 2019-12-05 11:11:30 +00:00
Erik Johnston
ddbbfc9512
Merge pull request #6464 from matrix-org/erikj/make_public_sql_base 
Clean up SQLBaseStore private function usage
 2019-12-05 10:43:49 +00:00
Brendan Abolivier
fe799f353d
Merge pull request #6470 from matrix-org/babolivier/port_db_ci_failure 
Make synapse_port_db exit with a non-0 code if something failed
 2019-12-04 18:20:36 +00:00
Brendan Abolivier
02c1f36ccd
Changelog 2019-12-04 17:49:28 +00:00
Richard van der Hoff
e203874caa
get rid of (most of) have_events from _update_auth_events_and_context_for_auth (#6468) 
have_events was a map from event_id to rejection reason (or None) for events
which are in our local database. It was used as filter on the list of
event_ids being passed into get_events_as_list. However, since
get_events_as_list will ignore any event_ids that are unknown or rejected, we
can equivalently just leave it to get_events_as_list to do the filtering.

That means that we don't have to keep `have_events` up-to-date, and can use
`have_seen_events` instead of `get_seen_events_with_rejection` in the one place
we do need it.
 2019-12-04 17:27:32 +00:00
Erik Johnston
685fae1ba5 Newsfile 2019-12-04 16:25:34 +00:00
Erik Johnston
3eb15c01d9
Merge pull request #6454 from matrix-org/erikj/clean_base_Store 
Move things out of SQLBaseStore
 2019-12-04 16:23:19 +00:00
Erik Johnston
b9449012db
Merge pull request #6441 from syamgk/fix-parameter-mismatch 
Fix issue #6406 parameter mismatch
 2019-12-04 15:42:33 +00:00
Brendan Abolivier
c530f9af4d
Merge pull request #6329 from matrix-org/babolivier/context_filters 
Filter state, events_before and events_after in /context requests
 2019-12-04 15:24:16 +00:00
Erik Johnston
a785a2febe Newsfile 2019-12-04 15:21:14 +00:00
Brendan Abolivier
9dc84b7989
Merge branch 'develop' into babolivier/context_filters 2019-12-04 14:23:44 +00:00
Brendan Abolivier
08a436ecb2
Incorporate review 2019-12-04 14:18:46 +00:00
Andrew Morgan
c1ae453932
Markdownification and other fixes to CONTRIBUTING (#6461) 2019-12-04 12:21:48 +00:00
Andrew Morgan
85901939c1
Fix error when using synapse_port_db on a vanilla synapse db (#6449) 2019-12-04 12:17:47 +00:00
Andrew Morgan
768b84409b
Update changelog.d/6449.bugfix 
Co-Authored-By: Erik Johnston <erik@matrix.org>
 2019-12-04 10:45:56 +00:00
Erik Johnston
2aa8943809
Merge pull request #6451 from matrix-org/uhoreg/cross_signing_signatures_index 
make cross signing signature index non-unique
 2019-12-04 09:57:09 +00:00
Neil Johnson
cb0aeb147e
privacy by default for room dir (#6355) 
Ensure that the the default settings for the room directory are that the it is hidden from public view by default.
 2019-12-04 09:46:16 +00:00
Richard van der Hoff
0120875462
Fix exception when a cross-signed device is deleted (#6462) 
(hopefully)

... and deobfuscate the relevant bit of code.
 2019-12-04 07:38:35 +00:00
Syam G Krishnan
b62c9db8d7 Add changelog file 
Signed-off-by: Syam G Krishnan <syamgk@gmail.com>
 2019-12-04 13:03:54 +05:30
Hubert Chathi
418813b205 apply changes from review 2019-12-03 15:27:00 -05:00
Brendan Abolivier
54dd5dc12b
Add ephemeral messages support (MSC2228) (#6409) 
Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are:

* the feature is hidden behind a configuration flag (`enable_ephemeral_messages`)
* self-destruction doesn't happen for state events
* only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one)
* doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
 2019-12-03 19:19:45 +00:00
Richard van der Hoff
620f98b65b
write some docs for the quarantine_media api (#6458) 2019-12-03 18:20:39 +00:00
Amber Brown
fdec84aa42
Add benchmarks for structured logging performance (#6266) 2019-12-03 20:21:25 +11:00
Hubert Chathi
e57567f994 add changelog 2019-12-02 17:10:57 -05:00
Andrew Morgan
8ee62e4b98 Add changelog 2019-12-02 18:43:25 +00:00
Andrew Morgan
72078e4be5
Transfer power level state events on room upgrade (#6237) 2019-12-02 15:11:32 +00:00
Andrew Morgan
0ad75fd98e
Use python3 packages for Ubuntu (#6443) 2019-12-02 15:09:57 +00:00
Filip Štědronský
81731c6e75 Fix: Pillow error when uploading RGBA image (#3325) (#6241) 
Signed-Off-By: Filip Štědronský <g@regnarg.cz>
 2019-12-02 12:12:55 +00:00
Andrew Morgan
23ea572125
Add User-Interactive Auth to /account/3pid/add (#6119) 2019-11-29 13:51:14 +00:00
Erik Johnston
1c3a61529f
Merge pull request #6434 from matrix-org/erikj/msc2367_membership_reasons 
Implement MSC 2367 - Membership Reasons
 2019-11-29 13:30:36 +00:00
Brendan Abolivier
5ee2beeddb
Changelog 2019-11-28 19:32:49 +00:00
Erik Johnston
19ba7c142e Newsfile 2019-11-28 13:59:32 +00:00
Andrew Morgan
e7777f3668 1.6.1 2019-11-28 11:29:50 +00:00
Andrew Morgan
2030193e55 Remove local threepids on account deactivation (#6426) 2019-11-28 11:07:45 +00:00
Andrew Morgan
a9c44d4008
Remove local threepids on account deactivation (#6426) 2019-11-28 10:40:42 +00:00
Richard van der Hoff
c48ea98007
Clarifications for the email configuration settings. (#6423) 
Cf #6422
 2019-11-28 09:29:18 +00:00
Amber Brown
0f87b912ab
Implementation of MSC2314 (#6176) 2019-11-28 08:54:07 +11:00
Hubert Chathi
0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Richard van der Hoff
6f4a63df00 Add more tests to the worker blacklist (#6429) 2019-11-27 18:18:33 +00:00
Brendan Abolivier
d31f69afa0
Merge pull request #6358 from matrix-org/babolivier/message_retention 
Implement message retention policies (MSC1763)
 2019-11-27 15:04:38 +00:00
Richard van der Hoff
9b9ee75666 Fix startup error when http proxy is defined. (#6421) 
Guess I only tested this on python 2 :/

Fixes #6419.
 2019-11-27 10:29:10 +00:00
Andrew Morgan
ce578031f4
Remove assertion and provide a clear warning on startup for missing public_baseurl (#6379) 2019-11-26 18:42:27 +00:00
Richard van der Hoff
651d930f16
Merge pull request #6343 from matrix-org/rav/event_auth/4 
Refactor _update_auth_events_and_context_for_auth
 2019-11-26 18:15:03 +00:00
Richard van der Hoff
ef1a85e773
Fix startup error when http proxy is defined. (#6421) 
Guess I only tested this on python 2 :/

Fixes #6419.
 2019-11-26 18:10:50 +00:00
Brendan Abolivier
9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Erik Johnston
ba110a2030 Newsfile 2019-11-26 15:54:48 +00:00
Andrew Morgan
a8175d0f96
Prevent account_data content from being sent over TCP replication (#6333) 2019-11-26 13:58:39 +00:00
Andrew Morgan
b98971e8a4 1.6.0 2019-11-26 13:28:40 +00:00
Erik Johnston
65d54c5e8c Fix phone home stats (#6418) 
Fix phone home stats
 2019-11-26 13:28:40 +00:00
Erik Johnston
f9f1c8acbb
Merge pull request #6332 from matrix-org/erikj/query_devices_fix 
Fix caching devices for remote servers in worker.
 2019-11-26 12:56:05 +00:00
Erik Johnston
35f9165e96 Fixup docs 2019-11-26 12:04:48 +00:00
Richard van der Hoff
c01d543584
Make sure that we close cursors before returning from a query (#6408) 
There are lots of words in the comment as to why this is a good idea.

Fixes #6403.
 2019-11-25 21:03:17 +00:00
Amber Brown
9eebd46048
Improve the performance of structured logging (#6322) 2019-11-26 03:45:50 +11:00
Richard van der Hoff
f9c9e1f076 1.6.0rc2 2019-11-25 13:28:12 +00:00
Richard van der Hoff
b7367c339d
Fix exceptions from background database update for event labels. (#6407) 
Add some exception handling here so that events whose json cannot be parsed are
ignored rather than getting us stuck in a loop.

Fixes #6404.
 2019-11-25 13:26:59 +00:00
Brendan Abolivier
78cfc05fc4
Merge pull request #6392 from matrix-org/babolivier/fix-1623 
Test if a purge can make /messages return 500 responses
 2019-11-25 12:59:36 +00:00
Andrew Morgan
265c0bd2fe Add working build command for docker image (#6390) 
* Add working build command for docker image

* Add changelog
 2019-11-23 06:54:05 +11:00
Aaron Raimist
24cc31ee96 Fix link to user_dir_populate.sql in the user directory docs (#6388) 2019-11-21 17:38:14 +00:00
Andrew Morgan
3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Brendan Abolivier
b2f8c21a9b
Format changelog 2019-11-20 16:10:27 +00:00
Brendan Abolivier
49243c55a4
Update changelog since this isn't going to be featured in 1.6.0 2019-11-20 16:09:11 +00:00
Brendan Abolivier
486be06f48
Changelog 2019-11-20 15:08:03 +00:00
Andrew Morgan
41e4566682 1.6.0rc1 2019-11-20 14:12:42 +00:00
Andrew Morgan
234f55f3c4
Docker: Change permissions for data dir before attempting to write to it (#6389) 2019-11-20 13:32:31 +00:00
Manuel Stahl
4f5ca455bf Move admin endpoints into separate files (#6308) 2019-11-20 11:49:11 +00:00
Brendan Abolivier
83446a18fb
Merge pull request #6335 from matrix-org/erikj/rc_login_cleanups 
Only do `rc_login` ratelimiting on succesful login.
 2019-11-20 09:52:38 +00:00
Richard van der Hoff
870c00e278 Merge remote-tracking branch 'origin/develop' into rav/event_auth/4 2019-11-18 12:05:36 +00:00
Andrew Morgan
657d614f6a
Replace UPDATE with UPSERT on device_max_stream_id table (#6363) 2019-11-15 14:02:34 +00:00
James
53b6559a89 Add optional python dependencies to snap packaging (#6317) 
Signed-off-by: James Hebden <james@ec0.io>
 2019-11-14 18:42:46 +00:00
Andrew Morgan
745a48625d
Fix guest -> real account upgrade with account validity enabled (#6359) 2019-11-14 12:02:05 +00:00
Andrew Morgan
a65a5ea125 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/fix_account_data_sync 
* 'develop' of github.com:matrix-org/synapse:
  Blacklist PurgeRoomTestCase (#6361)
  Set room version default to 5
 2019-11-14 10:26:56 +00:00
Andrew Morgan
473acedcdd Merge branch 'develop' of github.com:matrix-org/synapse into anoa/homeserver_copy 
* 'develop' of github.com:matrix-org/synapse:
  Blacklist PurgeRoomTestCase (#6361)
  Set room version default to 5
 2019-11-14 10:26:27 +00:00
Brendan Abolivier
a42567e4a8
Merge pull request #6220 from matrix-org/neilj/set_room_version_default_to_5 
Set room version default to 5
 2019-11-14 10:21:00 +00:00
Brendan Abolivier
c8444b4152 Merge branch 'develop' into babolivier/msc1802 2019-11-14 10:12:11 +00:00
Andrew Morgan
c350bc2f92
Blacklist PurgeRoomTestCase (#6361) 2019-11-13 19:09:20 +00:00
Andrew Morgan
85f172ef96 Add changelog 2019-11-12 13:13:19 +00:00
Brendan Abolivier
21056ad12a
Changelog 2019-11-11 16:53:29 +00:00
Richard van der Hoff
20d687516f newsfile 2019-11-08 16:17:02 +00:00
Andrew Morgan
318dd21b47 Add changelog 2019-11-08 15:45:08 +00:00
Brendan Abolivier
963ffb60b9
Merge pull request #6340 from matrix-org/babolivier/pagination_query 
Fix the SQL SELECT query in _paginate_room_events_txn
 2019-11-08 11:12:24 +00:00
Erik Johnston
f713c01e2b
Merge pull request #6295 from matrix-org/erikj/split_purge_history 
Split purge API into events vs state and add PurgeEventsStorage
 2019-11-08 10:19:15 +00:00
Brendan Abolivier
46e5db9eb2
Merge pull request #6310 from matrix-org/babolivier/msc2326_bg_update 
MSC2326: Add background update to take previous events into account
 2019-11-07 22:54:56 +00:00
Richard van der Hoff
c5abb67e43
Python 3.8 for tox (#6341) 
... and update INSTALL.md to include py3.8.

We'll also have to update the buildkite pipeline to run it
 2019-11-07 17:14:13 +00:00
Brendan Abolivier
4f519d556e
Changelog 2019-11-07 11:51:54 +00:00
Andrew Morgan
e914cf12f6
Merge pull request #6235 from matrix-org/anoa/room_upgrade_groups 2019-11-07 11:12:22 +00:00
Richard van der Hoff
b03cddaeb9 tweak changelog 2019-11-07 09:46:25 +00:00
V02460
affcc2cc36 Fix LruCache callback deduplication (#6213) 2019-11-07 09:43:51 +00:00
Andrew Morgan
a6ebef1bfd
Make numeric user_id checker start at @0, and don't ratelimit on checking (#6338) 2019-11-06 17:21:20 +00:00
Richard van der Hoff
915903eada Merge branch 'master' into develop 2019-11-06 13:51:11 +00:00
Andrew Morgan
d2f6a67cb4 Add changelog 2019-11-06 12:03:12 +00:00
Erik Johnston
4fc53bf1fb Newsfile 2019-11-06 11:08:58 +00:00
Richard van der Hoff
807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320) 
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
 2019-11-06 10:01:39 +00:00
Richard van der Hoff
0e3ab8afdc
Add some checks that we aren't using state from rejected events (#6330) 
* Raise an exception if accessing state for rejected events

Add some sanity checks on accessing state_group etc for
rejected events.

* Skip calculating push actions for rejected events

It didn't actually cause any bugs, because rejected events get filtered out at
various later points, but there's not point in trying to calculate the push
actions for a rejected event.
 2019-11-05 22:13:37 +00:00
Erik Johnston
01ba7b38a7
Merge pull request #6336 from matrix-org/erikj/fix_phone_home_stats 
Fix phone home stats
 2019-11-05 18:29:57 +00:00
Erik Johnston
b437eb48b6 Newsfile 2019-11-05 17:45:29 +00:00
Richard van der Hoff
5570d1c93f
Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2 
Fix exception when OpenGraph tag values are ints
 2019-11-05 17:28:11 +00:00
Richard van der Hoff
81d49cbb07 Fix exception when OpenGraph tag values are ints 2019-11-05 17:22:58 +00:00
Richard van der Hoff
55a7da247a
Merge branch 'develop' into rav/url_preview_limit_title 2019-11-05 17:08:07 +00:00
Erik Johnston
248111bae8 Newsfile 2019-11-05 15:54:23 +00:00
Richard van der Hoff
e9bfe719ba Strip overlong OpenGraph data from url preview 
... to stop people causing DoSes with malicious web pages
 2019-11-05 15:51:18 +00:00
Brendan Abolivier
f5d8fdf0a7
Update changelog 2019-11-05 14:44:25 +00:00
Brendan Abolivier
d8d808db64
Changelog 2019-11-05 14:42:05 +00:00
Richard van der Hoff
4086002827
Improve documentation for EventContext fields (#6319) 2019-11-05 13:23:25 +00:00
Erik Johnston
ffe595381d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-05 10:27:41 +00:00
Andrew Morgan
506a63de67 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_groups 2019-11-04 18:22:41 +00:00
Brendan Abolivier
e252ffadbc Merge branch 'develop' into babolivier/msc2326_bg_update 2019-11-04 18:09:50 +00:00
Andrew Morgan
0287d033ee Transfer upgraded rooms on groups 2019-11-04 18:08:50 +00:00
Amber Brown
4e1c7b79fa
Remove the psutil dependency (#6318) 
* remove psutil and replace with resource
 2019-11-05 05:05:48 +11:00
Brendan Abolivier
09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Erik Johnston
6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Richard van der Hoff
cc6243b4c0
document the REPLICATE command a bit better (#6305) 
since I found myself wonder how it works
 2019-11-04 12:40:18 +00:00
Brendan Abolivier
416c7baee6
Changelog 2019-11-04 09:56:10 +00:00
Brendan Abolivier
f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326 
Implement MSC2326 (label based filtering)
 2019-11-01 17:04:45 +00:00
Richard van der Hoff
c6516adbe0
Factor out an _AsyncEventContextImpl (#6298) 
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
 2019-11-01 16:19:09 +00:00
Hubert Chathi
fa7e52caf1
Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schema 
fix hidden field in devices table for older sqlite
 2019-11-01 10:52:46 -04:00
Jason Robinson
67a65918ad Add contributer docs for using the provided linters script (#6164) 
* Add lint dependencies black, flake8 and isort

These are required when running the `lint.sh` dev scripts.

Signed-off-by: Jason Robinson <jasonr@matrix.org>

* Add contributer docs for using the providers linters script

Add also to the pull request template to avoid build failures due
to people not knowing that linters need running.

Signed-off-by: Jason Robinson <jasonr@matrix.org>

* Fix mention of linter errors correction

Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Add mention for installing linter dependencies

Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Remove linters from python dependencies as per PR review

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-11-02 01:45:09 +11:00
Richard van der Hoff
1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239) 
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
 2019-11-01 14:07:44 +00:00
Andrew Morgan
fe1f2b4520
Remove last usages of deprecated logging.warn method (#6314) 2019-11-01 12:03:44 +00:00
Neil Pilgrim
befd58f47b Document lint.sh & allow application to specified files only (#6312) 2019-11-01 10:52:20 +00:00
Brendan Abolivier
c6dbca2422
Incorporate review 2019-11-01 10:30:51 +00:00
Andrew Morgan
ace947e8da
Depublish a room from the public rooms list when it is upgraded (#6232) 2019-11-01 10:28:09 +00:00
Hubert Chathi
1f156398b9 add changelog 2019-10-31 23:02:20 -04:00
Hubert Chathi
6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Hubert Chathi
3b4216f961
Merge pull request #6254 from matrix-org/uhoreg/cross_signing_fix_workers_notify 
make notification of signatures work with workers
 2019-10-31 22:35:03 -04:00
Will Hunt
42e707c663 rstrip slashes from url on appservice (#6306) 2019-10-31 17:32:25 +00:00
Hubert Chathi
9c94b48bf1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-31 12:32:07 -04:00
Amber Brown
020add5099
Update black to 19.10b0 (#6304) 
* update version of black and also fix the mypy config being overridden
 2019-11-01 02:43:24 +11:00
Erik Johnston
cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Erik Johnston
dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage 
Add StateGroupStorage interface
 2019-10-31 16:17:53 +01:00
Travis Ralston
3a74c03ffb
Expose some homeserver functionality to spam checkers (#6259) 
* Offer the homeserver instance to the spam checker

* Newsfile

* Linting

* Expose a Spam Checker API instead of passing the homeserver object

* Alter changelog

* s/hs/api
 2019-10-31 09:16:14 -06:00
Erik Johnston
69489f8eb1
Merge pull request #6307 from matrix-org/erikj/fix_purge_room 
Fix /purge_room admin API
 2019-10-31 16:08:34 +01:00
Erik Johnston
b2ff8c305f Newsfile 2019-10-31 11:32:53 +00:00
Richard van der Hoff
eb9a0d9e48 Merge remote-tracking branch 'origin/master' into develop 2019-10-31 11:17:05 +00:00
Andrew Morgan
54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271) 
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
 2019-10-31 10:23:24 +00:00
Hubert Chathi
670972c0e1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-30 16:46:31 -04:00
Richard van der Hoff
0467f33584
fix delete_existing for _persist_events (#6300) 
this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
 2019-10-30 18:05:00 +00:00
Brendan Abolivier
62588eae4a
Changelog 2019-10-30 17:54:40 +00:00
Erik Johnston
ecfba89a78 Newsfile 2019-10-30 15:23:37 +00:00
Erik Johnston
d3f694d628 Newsfile 2019-10-30 14:53:09 +00:00
Hubert Chathi
7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Erik Johnston
ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Erik Johnston
e5c3a99091
Merge pull request #6291 from matrix-org/erikj/fix_cache_descriptor 
Make ObservableDeferred.observe() always return deferred.
 2019-10-30 14:06:34 +01:00
Yash Jipkate
9677613e9c Modify doc to update Google ReCaptcha terms (#6257) 2019-10-30 12:30:20 +00:00
Andrew Morgan
7e17959984
Update email section of INSTALL.md about account_threepid_delegates (#6272) 2019-10-30 11:37:56 +00:00
Erik Johnston
1de28183cb Newsfile 2019-10-30 11:37:56 +00:00
Andrew Morgan
a2276d4d3c
Fix log line that was printing undefined value (#6278) 2019-10-30 11:28:48 +00:00
Andrew Morgan
2cab02f9d1
Update CI to run isort on scripts and scripts-dev (#6270) 2019-10-30 11:17:14 +00:00
Andrew Morgan
7955abeaac
Fix small typo in comment (#6269) 2019-10-30 11:16:19 +00:00
Andrew Morgan
46c12918ad
Fix typo in domain name in account_threepid_delegates config option (#6273) 2019-10-30 11:07:42 +00:00
Andrew Morgan
9178ac1b6a
Remove redundant arguments to CI's flake8 (#6277) 2019-10-30 11:07:18 +00:00
Andrew Morgan
b39ca49db1
Handle FileNotFound error in checking git repository version (#6284) 2019-10-30 11:00:15 +00:00
Erik Johnston
770d1ef673
Merge pull request #6280 from matrix-org/erikj/receipts_async_await 
Port receipt and read markers to async/wait
 2019-10-30 11:44:18 +01:00
Erik Johnston
ba4cc5541c
Merge pull request #6274 from matrix-org/erikj/replication_async 
Port replication http server endpoints to async/await
 2019-10-30 11:44:08 +01:00
Erik Johnston
72bc6294ed
Merge pull request #6275 from matrix-org/erikj/port_rest_events 
Port room rest handlers to async/await
 2019-10-30 11:44:02 +01:00
Erik Johnston
38474707b9 Merge branch 'erikj/federation_server_async_await' of github.com:matrix-org/synapse into erikj/receipts_async_await 2019-10-29 15:53:17 +00:00
Erik Johnston
74c1e16106 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/federation_server_async_await 2019-10-29 15:52:39 +00:00
Erik Johnston
307e313ef4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/port_rest_events 2019-10-29 15:51:12 +00:00
Erik Johnston
d6e40e7cbd Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_async 2019-10-29 15:42:58 +00:00
Brendan Abolivier
d79151921a
Fix CI for synapse_port_db (#6276) 
* Don't use a virtualenv

* Generate the server's signing key to allow it to start

* Add signing key paths to CI configuration files

* Use a Python script to create the postgresql database

* Improve logging
 2019-10-29 15:39:44 +00:00
Erik Johnston
7dd7a385f9 Newsfile 2019-10-29 15:09:48 +00:00
Richard van der Hoff
65cb307e19 Merge branch 'master' into develop 2019-10-29 14:40:57 +00:00
Richard van der Hoff
9ffcf0f7ba 1.5.0 2019-10-29 14:28:54 +00:00
Erik Johnston
fec7d88645 Newsfile 2019-10-29 14:27:18 +00:00
Brendan Abolivier
cc80968f62 Merge branch 'babolivier/changelog-name' into develop 2019-10-29 14:05:49 +00:00
Brendan Abolivier
20ebd24973
Fix changelog name 2019-10-29 14:04:02 +00:00
Erik Johnston
387324688e Newsfile 2019-10-29 13:10:45 +00:00
Erik Johnston
1a7ed37149 Newsfile 2019-10-29 13:01:50 +00:00
Erik Johnston
561133c3c5
Merge pull request #6263 from matrix-org/erikj/caches_return_deferreds 
Quick fix to ensure cache descriptors always return deferreds
 2019-10-29 12:53:21 +01:00
Erik Johnston
e419c44ba4 Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-29 11:41:27 +00:00
Erik Johnston
b7a0ea686f Newsfile 2019-10-29 10:25:25 +00:00
Brendan Abolivier
14504ad573
Add CI for synapse_port_db (#6140) 
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
 2019-10-28 17:45:32 +00:00
Erik Johnston
a8aced58df Newsfile 2019-10-28 13:36:52 +00:00
Richard van der Hoff
bcfc647e4d Synapse 1.5.0rc2 (2019-10-28) 
=============================
 
 Bugfixes
 --------
 
 - Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
 - Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
 - Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))
 
 Internal Changes
 ----------------
 
 - Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
 - Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl225VkACgkQOSor00I9
 eP/7wwf6AkQE1dGYDPpmS29XYocfD3EbjT8FojaHTz1gbkIt5rEF88QHzwRo4P/x
 1S1kQZpEKxKrwmLMDmSbw9STK/76LKGk9k8VycVHfTdy8SICCiL50pVPpSTJ4nXV
 905E7nY0zau3wunWJD4gEjqll967W0AtNKwCHO0D6vNP4pcQNhTkSjhnmypXCFGo
 bvV9r0wq8mtq/FS1GX0Y2CEwbbV1MoPrvlXW37t/gBPk3P1QB/vPjakxO8LgeuZf
 JzAamP5jOYZJpPCpZxwDm6ZNEQ7T4jORdA623wp1gh1nTG88cObCLXJe2jCnpjnJ
 QVb1YlJiwcehXkclIoywgIzj13iIsw==
 =7ctT
 -----END PGP SIGNATURE-----

Merge tag 'v1.5.0rc2' into develop

Synapse 1.5.0rc2 (2019-10-28)
=============================

Bugfixes
--------

- Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
- Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
- Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))

Internal Changes
----------------

- Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
- Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
 2019-10-28 12:59:13 +00:00
Richard van der Hoff
c482d45822 1.5.0rc2 2019-10-28 12:48:18 +00:00
Richard van der Hoff
172f264ed3
Improve signature checking on some federation APIs (#6262) 
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
 2019-10-28 12:43:23 +00:00
Richard van der Hoff
9aee28927b Convert EventContext to attrs (#6218) 
* make EventContext use an attr
 2019-10-28 14:29:55 +02:00
Hubert Chathi
da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys 
delete keys when deleting backup versions
 2019-10-25 11:28:11 -04:00
Erik Johnston
64c2cfda8a Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:34:49 +01:00
Erik Johnston
87259b3a3a
Merge pull request #6255 from matrix-org/erikj/stats_config 
Remove repeated calls to config.stats_enabled.
 2019-10-25 11:34:38 +01:00
Erik Johnston
a71b8c87ec Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:32:24 +01:00
Erik Johnston
deaa9db5f2 Newsfile 2019-10-25 11:11:38 +01:00
Erik Johnston
a52b23d413 Newsfile 2019-10-25 10:34:10 +01:00
Erik Johnston
44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging 
Reduce debug logging overhead
 2019-10-25 10:05:44 +01:00
Hubert Chathi
0417ca1a64 add changelog 2019-10-24 22:49:55 -04:00
Hubert Chathi
c40d7244f8 Merge branch 'develop' into cross-signing_federation 2019-10-24 22:31:25 -04:00
Hubert Chathi
608947eedf add changelog 2019-10-24 21:33:35 -04:00
Michael Kaye
47c02f82e3 Add missing '.' 2019-10-24 18:39:15 +01:00
Michael Kaye
0d7e9523e5 Reduce impact of debug logging 2019-10-24 18:37:55 +01:00
Michael Kaye
c3cd977fff Add changelog.d 2019-10-24 17:58:50 +01:00
Erik Johnston
e39bc62b8a
Merge pull request #6248 from matrix-org/erikj/move_schema_files 
Move schema delta files to the correct data store.
 2019-10-24 17:08:16 +01:00
Brendan Abolivier
0ef0b09d68
Merge pull request #6247 from matrix-org/babolivier/port_db_column 
Add new boolean column to synapse_port_db
 2019-10-24 17:07:34 +01:00
Erik Johnston
19be9b703a Newsfile 2019-10-24 16:46:39 +01:00
Andrew Morgan
8831b04a53 1.5.0rc1 2019-10-24 12:47:58 +01:00
Brendan Abolivier
ef8d76be99
Changelog 2019-10-24 12:40:13 +01:00
Neil Johnson
2794b79052 Option to suppress resource exceeded alerting (#6173) 
The expected use case is to suppress MAU limiting on small instances
 2019-10-24 11:48:46 +01:00
Andrew Morgan
92e88a71d3
Cleanup extra quotes from IDEs (#6236) 2019-10-23 16:49:05 +01:00
Erik Johnston
9fb96889a4 Newsfile 2019-10-23 16:15:03 +01:00
Brendan Abolivier
2f859e865a
Changelog 2019-10-23 15:56:50 +01:00
Brendan Abolivier
c97ed64db3
Make synapse_port_db correctly create indexes (#6102) 
Make `synapse_port_db` correctly create indexes in the PostgreSQL database, by having it run the background updates on the database before migrating the data.

To ensure we're migrating the right data, also block the port if the SQLite3 database still has pending or ongoing background updates.

Fixes #4877
 2019-10-23 15:31:59 +01:00
Andrew Morgan
409c62b27b
Add config linting script that checks for bool casing (#6203) 
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
 2019-10-23 13:22:54 +01:00
Hubert Chathi
0563839535 add news file 2019-10-22 21:51:01 -04:00
Erik Johnston
c17efdc01c Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 18:44:29 +01:00
Hubert Chathi
2761731634
Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2 
Cross-signing [3/4] -- uploading signatures edition
 2019-10-22 17:05:06 +01:00
Andrew Morgan
b2945d2672
Fix demo script on ipv6-supported boxes (#6229) 
The synapse demo was a bit flakey in terms of supporting federation. It turns out that if your computer resolved `localhost` to `::1` instead of `127.0.0.1`, the built-in federation blacklist specified in `start.sh` would still block it, since it contained an entry for `::/127`. Removing this no longer prevents Synapse from contacting `::1`, federation works again on these boxes.
 2019-10-22 13:52:25 +01:00
Erik Johnston
bb6264be0b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 10:41:18 +01:00
Erik Johnston
3c304aaaeb Newsfile 2019-10-21 16:10:37 +01:00
Andrew Morgan
22a9f75097 Delete format_tap.py (#6219) 
* Delete format_tap.py

This python implementation of a tap formatting library for buildkite has been
replaced with a perl implementation as part of the matrix-org/sytest repo,
which is specific to sytest's language, not that of any one homeserver's.
 2019-10-19 20:42:10 +03:00
Neil Johnson
82c8799ec7 Set room version default to 5 2019-10-19 09:06:15 +01:00
Hubert Chathi
6493ed5723 Add changelog entry 
... again?  How did you make it disappear, git?
 2019-10-18 18:45:36 +01:00
Richard van der Hoff
93eaeec75a
Remove Auth.check method (#6217) 
This method was somewhat redundant, and confusing.
 2019-10-18 19:43:36 +02:00
Hubert Chathi
0a9d22808c Merge branch 'uhoreg/e2e_cross-signing_merged' into develop 2019-10-18 15:57:40 +01:00
Richard van der Hoff
2b22faded7 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 12:35:13 +01:00
Richard van der Hoff
560c122267
Fix logging config for the docker image (#6197) 
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
 2019-10-18 13:34:33 +02:00
Hubert Chathi
cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Hubert Chathi
770a6053a0 add note about database upgrade 2019-10-18 11:38:27 +01:00
Erik Johnston
d98029ea89
Merge pull request #6196 from matrix-org/erikj/await 
Move rest/admin to use async/await.
 2019-10-18 11:53:02 +02:00
Bart Noordervliet
dc4bec885d Add missing BOOLEAN_COLUMNs to synapse_port_db (#6216) 
Small fix to synapse_port_db to be able to convert from database schema v56.
 2019-10-18 11:13:59 +02:00
Richard van der Hoff
80003dfcd5 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 10:11:40 +01:00
Erik Johnston
5859a5c569 Fix presence timeouts when synchrotron restarts. (#6212) 
* Fix presence timeouts when synchrotron restarts.

Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.

Fixes #3715.
 2019-10-18 06:42:26 +01:00
Richard van der Hoff
1594de856c changelog 2019-10-17 21:45:59 +01:00
Richard van der Hoff
6fb0a3da07 Remove dead changelog file 
This is part of 1.4.1
 2019-10-17 18:03:28 +01:00
Richard van der Hoff
423f7ae397 Fix up changelogs 2019-10-17 17:06:07 +01:00
Hubert Chathi
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage 
make storage layer in charge of interpreting the device key data
 2019-10-11 15:19:06 -04:00
Hubert Chathi
691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version 
make version optional in body of e2e backup version update
 2019-10-11 10:11:59 -04:00
Erik Johnston
6f5c6c8f60
Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url 
Trace non-JSON APIs, /media, /key etc
 2019-10-11 13:23:52 +01:00
Valérian Rousset
be9b55e0d2 cas: support setting display name (#6114) 
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
 2019-10-11 12:33:12 +01:00
Erik Johnston
c3b0fbe9c3 Newsfile 2019-10-11 12:24:52 +01:00
Erik Johnston
de3a176426 Newsfile 2019-10-11 11:58:52 +01:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi
2208891ace add changelog 2019-10-10 19:22:10 -04:00
Erik Johnston
83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any 
Use Postgres ANY for selecting many values.
 2019-10-10 16:41:36 +01:00
Erik Johnston
9d06fb9cb1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any 2019-10-10 14:38:21 +01:00
Erik Johnston
8bc529c04d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 14:30:51 +01:00
Richard van der Hoff
430dc2c67b Fix python packaging 
... after it got borked by #6081
 2019-10-10 14:05:30 +01:00
Erik Johnston
933034e2fe
Merge pull request #6127 from matrix-org/erikj/patch_inner 
Add more log context checks when patching inlineCallbacks
 2019-10-10 13:47:50 +01:00
Erik Johnston
203ccdac5f Newsfile 2019-10-10 13:15:49 +01:00
Erik Johnston
91f43dca39 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 13:10:57 +01:00
werner291
b5b03b7079 Add domain validation when creating room with list of invitees (#6121) 2019-10-10 13:05:48 +01:00
Michael Telatynski
9a84d74417 before fulfilling a group invite,check if user is already joined/invited (#3436) 
Fixes vector-im/riot-web#5645
 2019-10-10 13:03:44 +01:00
krombel
2efd050c9d send 404 as http-status when filter-id is unknown to the server (#2380) 
This fixed the weirdness of 400 vs 404 as http status code in the case
the filter id is not known by the server.
As e.g. matrix-js-sdk expects 404 to catch this situation this leads
to unwanted behaviour.
 2019-10-10 12:59:55 +01:00
James
0aee490013 Add snapcraft packaging information (#6084) 2019-10-10 11:59:06 +01:00
Erik Johnston
9970f955ce Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_inner 2019-10-10 11:51:50 +01:00
Richard van der Hoff
a139420a3c
Fix races in room stats (and other) updates. (#6187) 
Hopefully this will fix the occasional failures we were seeing in the room directory.

The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.

We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
 2019-10-10 11:29:01 +01:00
Richard van der Hoff
562b4e51dd
Rewrite the user_filter migration again (#6184) 
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
 2019-10-10 11:28:23 +01:00
Erik Johnston
f4571a0497 Newsfile 2019-10-10 11:17:00 +01:00
Andrew Morgan
da815c1f69 Move tag/push rules room upgrade checking ealier (#6155) 
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.

The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
 2019-10-10 10:06:45 +01:00
Amber Brown
f743108a94
Refactor HomeserverConfig so it can be typechecked (#6137) 2019-10-10 09:39:35 +01:00
Hubert Chathi
b46cc856ec add changelog 2019-10-09 18:03:40 -04:00
Erik Johnston
c3b34dc32f Newsfile 2019-10-09 16:32:04 +01:00
Erik Johnston
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets 
Fix inserting bytes as text in `censor_redactions`
 2019-10-09 15:39:13 +01:00
Erik Johnston
de26678724
Update changelog.d/6185.bugfix 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-10-09 15:13:02 +01:00
Brendan Abolivier
59d6290ed9
Merge pull request #6178 from matrix-org/babolivier/factor_out_bg_updates 
Factor out backgroung updates
 2019-10-09 12:29:01 +01:00
Erik Johnston
6b72508d15 Newsfile 2019-10-08 16:18:43 +01:00
Anshul Angaria
474abf1eb6 add M_TOO_LARGE error code for uploading a too large file (#6151) 
Fixes #6109
 2019-10-08 13:55:16 +01:00
Andrew Morgan
ea7d938bca Remove unused public room list timeout param (#6179) 
* Remove unused public room list timeout param

* Add changelog
 2019-10-08 13:51:25 +01:00
Richard van der Hoff
b94a401852 Fix /federation/v1/state for recent room versions (#6170) 
* Fix /federation/v1/state for recent room versions

Turns out this endpoint was completely broken for v3 rooms. Hopefully this
re-signing code is irrelevant nowadays anyway.
 2019-10-08 09:35:37 +01:00
Brendan Abolivier
21b5d8b107
Changelog 2019-10-07 18:00:31 +01:00
Richard van der Hoff
cc2e19ad4b fix changelog 2019-10-07 17:37:55 +01:00
Erik Johnston
2cb7466993
Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure 
Don't regenerate numeric user ID if registration fails.
 2019-10-07 16:56:23 +01:00
Michael Kaye
dc795ba709 Log responder we are using. (#6139) 
This prevents us logging "Responding to media request with responder %s".
 2019-10-07 15:41:25 +01:00
Brendan Abolivier
1992f21a9f
Fix changelog for PR #6175 2019-10-07 14:54:36 +01:00
Richard van der Hoff
276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test (#6167) 2019-10-07 14:41:39 +01:00
Brendan Abolivier
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index 
Fix unique_user_filter_index schema update
 2019-10-07 13:48:43 +01:00
Brendan Abolivier
aa7a003074
Changelog 2019-10-07 13:16:54 +01:00
Erik Johnston
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships 
Cache room membership lookups in _get_joined_users_from_context
 2019-10-07 13:15:00 +01:00
Brendan Abolivier
ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked 
Don't 500 when trying to exchange a revoked 3PID invite
 2019-10-04 12:09:05 +01:00
Alexander Maznev
13c4345c84 Update user_filters table to have a unique index, and non-null columns (#1172) 2019-10-04 10:34:16 +01:00
Robert Swain
39b40d6d99 media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142) 
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
 2019-10-04 09:34:52 +01:00
Erik Johnston
6511071837 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships 2019-10-03 17:51:06 +01:00
Richard van der Hoff
66537e10ce
add some metrics on the federation sender (#6160) 2019-10-03 17:47:20 +01:00
Erik Johnston
0186ec9df7 Fixup newsfile 2019-10-03 17:46:43 +01:00
Erik Johnston
4fc60f12de Newsfile 2019-10-03 17:35:50 +01:00
Erik Johnston
0ccf0ffc85 Newsfile 2019-10-03 17:12:24 +01:00
Andrew Morgan
6018bec919 Merge branch 'master' into develop 2019-10-03 13:48:45 +01:00
Erik Johnston
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination 
Fix appservice room list pagination
 2019-10-02 16:50:32 +01:00
Erik Johnston
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable 
Fix not showing non-federatable rooms to remote room list queries
 2019-10-02 16:09:15 +01:00
Erik Johnston
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated 
Bound find_next_generated_user_id DB query.
 2019-10-02 16:09:02 +01:00
Erik Johnston
5be4083306 Newsfile 2019-10-02 15:48:36 +01:00
Erik Johnston
ed73f04bef Newsfile 2019-10-02 15:24:33 +01:00
Erik Johnston
8e32240e6b Newsfile 2019-10-02 15:12:17 +01:00
Erik Johnston
a5166e4d5f
Land improved room list based on room stats (#6019) 
Use room_stats and room_state for room directory search
 2019-10-02 14:08:35 +01:00
Amber Brown
864f144543
Fix up some typechecking (#6150) 
* type checking fixes

* changelog
 2019-10-02 05:29:01 -07:00
Brendan Abolivier
24efea338d
Changelog 2019-10-02 12:20:03 +01:00
Andrew Morgan
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade (#6144) 
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.

For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.

Also fix some missing yields that probably broke things quite a bit.
 2019-10-02 12:04:22 +01:00
Erik Johnston
de1823b521 Newsfile 2019-10-02 11:45:31 +01:00
Erik Johnston
ecd254bc49 Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop 2019-10-02 11:08:07 +01:00
Erik Johnston
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings 
Fix errors storing large retry intervals.
 2019-10-02 11:05:22 +01:00
Erik Johnston
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions 
Fix fetching censored redactions from DB
 2019-10-02 10:57:30 +01:00
Erik Johnston
2bc027ab71 Newsfile 2019-10-02 10:41:29 +01:00
Erik Johnston
33d4ebdf78 Newsfile 2019-10-02 10:18:17 +01:00
Erik Johnston
2b8352e638 Newsfile 2019-10-01 13:47:06 +01:00
Erik Johnston
1d349fb159 Merge branch 'erikj/fixup_devices_last_seen_query' of github.com:matrix-org/synapse into develop 2019-10-01 10:17:24 +01:00
Erik Johnston
129343cd8a Newsfile 2019-09-30 12:00:34 +01:00
Richard van der Hoff
16cb9a71b8
Drop unused tables (#6115) 
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
 2019-09-30 09:38:41 +01:00
Andrew Morgan
f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes (#6077) 2019-09-27 17:59:18 +01:00
Brendan Abolivier
25a0a36ad9
Update changelog.d/6125.feature 
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2019-09-27 16:10:24 +01:00
Brendan Abolivier
0804a27c8c
Changelog 2019-09-27 15:14:34 +01:00
Andrew Morgan
8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978) 
Just to have all the methods that make calls to identity services in one place.
 2019-09-27 10:36:20 +01:00
Erik Johnston
67ee18daea
Merge pull request #6108 from matrix-org/erikj/remove_get_user_by_req-span 
Add some helpful opentracing tags and remove get_user_by_req span
 2019-09-27 10:02:57 +01:00
Erik Johnston
9d99eade7c Newsfile 2019-09-26 16:46:21 +01:00
Richard van der Hoff
54569c787b
Kill off half-implemented password-reset via sms (#6101) 
Doing a password reset via SMS has never worked, and in any case is a silly
idea because msisdn recycling is a thing.

See also matrix-org/matrix-doc#2303.
 2019-09-26 15:38:25 +01:00
Richard van der Hoff
3fbca80a8d changelog 2019-09-26 13:23:48 +01:00
Neil Johnson
8b8f8c7b3c Explicitly log when a homeserver does not have a trusted key server configured (#6090) 2019-09-26 12:57:01 +01:00
Neil Johnson
1b23f991ab
Clarify upgrade notes ahead of 1.4.0 release 2019-09-26 12:30:10 +01:00
Neil Johnson
034db2ba21 Fix dummy event insertion consent bug (#6053) 
Fixes #5905
 2019-09-26 11:47:53 +01:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston
9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips 
Move last seen info into devices table
 2019-09-25 17:42:39 +01:00
Neil Johnson
a4f3ca48b5 Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. (#5884) 2019-09-25 17:27:35 +01:00
Erik Johnston
d2bd0bc6b1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips 2019-09-25 17:16:28 +01:00
Neil Johnson
77dc7093a7 Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104) 
Fixes #6103
 2019-09-25 12:29:35 +01:00
Richard van der Hoff
990928abde
Stop advertising unsupported flows for registration (#6107) 
If email or msisdn verification aren't supported, let's stop advertising them
for registration.

Fixes #6100.
 2019-09-25 12:10:26 +01:00
Erik Johnston
5c1af6d1b8 Newsfile 2019-09-25 11:42:00 +01:00
Richard van der Hoff
2cd98812ba
Refactor the user-interactive auth handling (#6105) 
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.

This is also preparation for some more intelligent advertising of flows, as per #6100
 2019-09-25 11:33:03 +01:00
Richard van der Hoff
8004d6ca2f
Refactor code for calculating registration flows (#6106) 
because, frankly, it looked like it was written by an axe-murderer.

This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
 2019-09-25 11:32:05 +01:00
Brendan Abolivier
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return 
Fix the return value in the users_set_deactivated_flag background job
 2019-09-24 17:11:12 +01:00
Richard van der Hoff
9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
Richard van der Hoff
566ac40939
remove unused parameter to get_user_id_by_threepid (#6099) 
Added in #5377, apparently in error
 2019-09-24 17:01:09 +01:00
Erik Johnston
f8b02c5420 Newsfile 2019-09-24 15:59:43 +01:00
Richard van der Hoff
bb82be9851
Merge pull request #6069 from matrix-org/rav/fix_attribute_mapping 
Fix a bug with saml attribute maps.
 2019-09-24 15:07:26 +01:00
J. Ryan Stinnett
40fb00f5b7
Add sid to next_link for email validation (#6097) 2019-09-24 14:39:50 +01:00
Richard van der Hoff
ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Andrew Morgan
50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079) 
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
 2019-09-23 21:21:03 +01:00
Andrew Morgan
2b071a2ff1 Add an unstable feature flag for separate add/bind 3pid APIs (#6044) 
Add a m.separate_add_and_bind flag set to True. See MSC2290's Backward Compatibility section for details.
 2019-09-23 20:46:34 +01:00
Richard van der Hoff
78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup 
Make the sample saml config closer to our standards
 2019-09-23 20:36:51 +01:00
Andrew Morgan
e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000) 
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
 2019-09-23 20:23:20 +01:00
Andrew Morgan
1ea3ed7620 Add m.id_access_token to /versions unstable_features (MSC2264) (#5974) 
Adds a flag to /versions' unstable_features section indicating that this Synapse understands what an id_access_token is, as per MSC2264.

Fixes #5927
 2019-09-23 20:19:25 +01:00
Andrew Morgan
b38aa82b83 Add m.require_identity_server to /versions unstable_flags (#5972) 
As MSC2263 states, m.require_identity_server must be set to false when it does not require an identity server to be provided by the client for the purposes of email registration or password reset.

Adds an m.require_identity_server flag to /versionss unstable_flags section. This will advertise that Synapse no longer needs id_server as a parameter.
 2019-09-23 18:52:43 +01:00
Andrew Morgan
2c99c63453 Add POST submit_token endpoint for MSISDN (#6078) 
First part of solving #6076
 2019-09-23 17:49:00 +01:00
Brendan Abolivier
a2a09d42dd
Changelog 2019-09-23 17:22:55 +01:00
Erik Johnston
691a70190b Newsfile 2019-09-23 16:59:45 +01:00
Andrew Morgan
30af161af2 Implement MSC2290 (#6043) 
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
 2019-09-23 16:50:27 +01:00
Andrew Morgan
1b519e0272 Disable /register/available if registration is disabled (#6082) 
Fixes #6066

This register endpoint should be disabled if registration is disabled, otherwise we're giving anyone the ability to check if a username exists on a server when we don't need to be.

Error code is 403 (Forbidden) as that's the same returned by /register when registration is disabled.
 2019-09-23 15:38:38 +01:00
Andrew Morgan
1c9feadf4b Generalize email sending logging (#6075) 
In ancient times Synapse would only send emails when it was notifying a user about a message they received...

Now it can do all sorts of neat things!

Change the logging so it's not just about notifications.
 2019-09-23 14:38:19 +01:00
Andrew Morgan
885a4726b7 Return timeout error to user for identity server calls (#6073) 2019-09-23 14:37:23 +01:00
Richard van der Hoff
0660f8e367
Merge pull request #6072 from matrix-org/rav/fix_retry_reset 
Fix exception when resetting retry timings
 2019-09-23 14:31:47 +01:00
Andrew Morgan
df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Andrew Morgan
7763dd3e95
Remove trailing slash ability from password reset's submit_token endpoint (#6074) 
Remove trailing slash ability from the password reset submit_token endpoint. Since we provide the link in an email, and have never sent it with a trailing slash, there's no point for us to accept them on the endpoint.
 2019-09-20 14:58:37 +01:00
Richard van der Hoff
9d94313209 Fix exception when resetting retry timings 
Fixes:
> TypeError: set_destination_retry_timings() missing 1 required positional
argument: 'retry_interval'

Introduced in #6016.
 2019-09-20 12:09:39 +01:00
Andrew Morgan
aeb40f355c
Ensure email validation link parameters are URL-encoded (#6063) 
The validation links sent via email had their query parameters inserted without any URL-encoding. Surprisingly this didn't seem to cause any issues, but if a user were to put a `/` in their client_secret it could lead to problems.
 2019-09-20 10:46:59 +01:00
Andrew Morgan
3ac614eb6c
Drop support for bind param on POST /account/3pid (MSC2290) (#6067) 
As per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/files#diff-05cde9463e9209b701312b3baf2fb2ebR151), we're dropping the bind parameter from `/account/3pid`. This endpoint can now only be used for adding threepid's to the user's account on the homeserver.
 2019-09-20 10:46:34 +01:00
Michael Kaye
2def5ea0da Docker: support SYNAPSE_WORKER envvar (#6058) 
* Allow passing SYNAPSE_WORKER envvar

* changelog.d

* Document SYNAPSE_WORKER.

Attempting to imply that you don't need to change this default
unless you're in worker mode.

Also aware that there's a bigger problem of attempting to document
a complete working configuration of workers using docker, as we
currently only document to use `synctl` for worker mode, and synctl
doesn't work that way in docker.
 2019-09-19 22:29:47 +01:00
J. Ryan Stinnett
36015d68ef Use unstable prefix for 3PID unbind API (#6062) 2019-09-19 22:28:29 +01:00
Richard van der Hoff
284e1cb027 Merge branch 'develop' into rav/fix_attribute_mapping 2019-09-19 20:32:25 +01:00
Richard van der Hoff
b74606ea22 Fix a bug with saml attribute maps. 
Fixes a bug where the default attribute maps were prioritised over
user-specified ones, resulting in incorrect mappings.

The problem is that if you call SPConfig.load() multiple times, it adds new
attribute mappers to a list. So by calling it with the default config first,
and then the user-specified config, we would always get the default mappers
before the user-specified mappers.

To solve this, let's merge the config dicts first, and then pass them to
SPConfig.
 2019-09-19 20:32:14 +01:00
Richard van der Hoff
599f786e4e
Update 6037.feature 2019-09-19 18:52:17 +01:00
Richard van der Hoff
6db5adca0c Merge branch 'rav/saml_config_cleanup' into rav/saml_mapping_work 2019-09-19 18:20:23 +01:00
Richard van der Hoff
fe349b497e
Update the upgrade notes (#6050) 
* make it clear that if you installed from a package manager, you should use
   that to upgrade

 * Document the new way of getting the server version (cf #4878)

 * Write some words about downgrading.
 2019-09-19 18:20:01 +01:00
Richard van der Hoff
b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff
35ce3bda7a
Add some notes on rolling back to v1.3.1. (#6049) 2019-09-19 15:06:48 +01:00
Richard van der Hoff
bcd9132869
Undo the deletion of some tables (#6047) 
This is a partial revert of #5893. The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.

So let's leave the tables in place for now, and remember to drop them in a
subsequent release.

(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
 2019-09-19 15:06:27 +01:00
Richard van der Hoff
b789c7eb03 Merge branch 'develop' into rav/saml_config_cleanup 2019-09-19 15:05:31 +01:00
Richard van der Hoff
84a2743e2e Add changelog 2019-09-19 15:04:02 +01:00
Andrew Morgan
62e3ff92fd
Remove POST method from password reset submit_token endpoint (#6056) 
Removes the POST method from `/password_reset/<medium>/submit_token/` as it's only used by phone number verification which Synapse does not support yet.
 2019-09-19 10:53:14 +01:00
Matthew Hodgson
dd8e24f42e changelog 2019-09-19 01:14:17 +01:00
Jorik Schellekens
38fd1f8e3f Fix typo in account_threepid_delegates config (#6028) 2019-09-18 22:30:44 +01:00
Jorik Schellekens
d58cad635e Give appropriate exit codes when synctl fails (#5992) 2019-09-18 22:27:59 +01:00
J. Ryan Stinnett
a86a290850 Fix logcontext spam on non-Linux platforms (#6059) 
This checks whether the current platform supports thread resource usage tracking
before logging a warning to avoid log spam.

Fixes https://github.com/matrix-org/synapse/issues/6055
 2019-09-18 21:55:37 +01:00
Andrew Morgan
6670bd4072
v2 3PID Invites (part of MSC2140) (#5979) 
3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is.

These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens).

Requires: ~~#5976~~
 2019-09-17 18:05:13 +01:00
dstipp
379d2a8c39 (#5849) Convert rst to markdown (#6040) 
Converting some of the rst documentation to markdown.  Attempted to
preserve whitespace and line breaks to minimize cosmetic change.
 2019-09-17 12:55:29 +01:00
Erik Johnston
70c52821ce Fix race condition in room stats. (#6029) 
Broke in #5971

Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)
 2019-09-17 12:41:23 +01:00
Richard van der Hoff
1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016) 
Track the time that a server started failing at, for general analysis purposes.
 2019-09-17 11:41:54 +01:00
Amber Brown
850dcfd2d3
Fix well-known lookups with the federation certificate whitelist (#5997) 2019-09-14 04:58:38 +10:00
Richard van der Hoff
b9d57502da changelog 2019-09-13 16:06:03 +01:00
Travis Ralston
c755955f33 Add developer docs for using SAML without a server (#6032) 2019-09-13 08:58:18 +01:00
Amber Brown
b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Richard van der Hoff
3d882a7ba5
Remove the cap on federation retry interval. (#6026) 
Essentially the intention here is to end up blacklisting servers which never
respond to federation requests.

Fixes https://github.com/matrix-org/synapse/issues/5113.
 2019-09-12 13:00:13 +01:00
Richard van der Hoff
0388beafe4
Fix bug in calculating the federation retry backoff period (#6025) 
This was intended to introduce an element of jitter; instead it gave you a
30/60 chance of resetting to zero.
 2019-09-12 12:59:43 +01:00
David Baker
6db22e4702 changelog 2019-09-12 11:46:37 +01:00
Sorunome
dd2e5b0038 add report_stats_endpoint config option (#6012) 
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
 2019-09-12 11:24:57 +01:00
Jorik Schellekens
a8251da10f
Blow up config if opentracing is missing (#5985) 
* Blow up config if opentracing is missing
 2019-09-12 10:57:37 +01:00
Jason Robinson
6d847d8ce6 Ensure support users can be registered even if MAU limit is reached 
This allows support users to be created even on MAU limits via
the admin API. Support users are excluded from MAU after creation,
so it makes sense to exclude them in creation - except if the
whole host is in disabled state.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-09-11 20:48:31 +03:00
Andrew Morgan
9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976) 
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.

Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140).

Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
 2019-09-11 16:02:42 +01:00
Erik Johnston
cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction 
Allow use of different ratelimits for admin redactions.
 2019-09-11 15:39:38 +01:00
Richard van der Hoff
7902bf1e1d
Clean up some code in the retry logic (#6017) 
* remove some unused code
* make things which were constants into constants for efficiency and clarity
 2019-09-11 15:14:56 +01:00
Andrew Morgan
9c555f37e3
Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009) 
Some small fixes to `room_member.py` found while doing other PRs.

1. Add requester to the base `_remote_reject_invite` method.
2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.
 2019-09-11 14:23:24 +01:00
Jorik Schellekens
6604b64fae
Check dependencies on setup in the nicer way. (#5989) 2019-09-11 14:00:37 +01:00
Andrew Morgan
3505ffcda7
Fix existing v2 identity server calls (MSC2140) (#6013) 
Two things I missed while implementing [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R80).

1. Access tokens should be provided to the identity server as `access_token`, not `id_access_token`, even though the homeserver may accept the tokens as `id_access_token`.
2. Access tokens must be sent to the identity server in a query parameter, the JSON body is not allowed.

We now send the access token as part of an `Authorization: ...` header, which fixes both things.

The breaking code was added in https://github.com/matrix-org/synapse/pull/5892

Sytest PR: https://github.com/matrix-org/sytest/pull/697
 2019-09-11 11:59:45 +01:00
Erik Johnston
2434c0084b Newsfile 2019-09-11 10:48:52 +01:00
Andrew Morgan
cd17a2085e
Remove origin parameter from add_display_name_to_third_party_invite and add params to docstring (#6010) 
Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument.

Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
 2019-09-11 10:37:17 +01:00
Erik Johnston
5e9b05d7da
Merge pull request #6011 from matrix-org/anoa/fix_3pid_validation 
Use account_threepid_delegate for 3pid validation
 2019-09-10 18:15:07 +01:00
Andrew Morgan
b5833a2abf Add changelog 2019-09-10 17:56:10 +01:00
Jason Robinson
63f9317b8e
Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users 
Only count real users when checking for auto-creation of auto-join room
 2019-09-09 17:37:52 +03:00
Erik Johnston
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions 
Censor redactions in DB after a month
 2019-09-09 15:29:39 +01:00
Amber Brown
aeb9b2179e
Add a build info metric to Prometheus (#6005) 2019-09-10 00:14:58 +10:00
Jason Robinson
be618e0551 Only count real users when checking for auto-creation of auto-join room 
Previously if the first registered user was a "support" or "bot" user,
when the first real user registers, the auto-join rooms were not
created.

Fix to exclude non-real (ie users with a special user type) users
when counting how many users there are to determine whether we should
auto-create a room.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-09-09 14:48:08 +03:00
Erik Johnston
a852e93408 Newsfile 2019-09-09 10:24:14 +01:00
Hubert Chathi
8e86f5b65c Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-09-07 13:20:34 -04:00
Amber Brown
55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Andrew Morgan
78801e7f9e
Ensure a sid parameter is passed to bind_threepid (#5995) 
`sid` is required to be part of `three_pid_creds`. We were 500'ing if it wasn't provided instead of returning `M_MISSING_PARAM`.
 2019-09-06 15:36:50 +01:00
Erik Johnston
a2a695b7ec
Merge pull request #5998 from matrix-org/erikj/fixup_federate_flag 
Correctly handle non-bool m.federate flag
 2019-09-06 15:32:43 +01:00
Erik Johnston
85275c89d7 Newsfile 2019-09-06 14:21:14 +01:00
Erik Johnston
30b67e0f63
Merge pull request #5993 from matrix-org/anoa/worker_store_reg 
Move get_threepid_validation_session and delete_threepid_session into RegistrationWorkerStore
 2019-09-06 14:10:02 +01:00
Andrew Morgan
5d833f0923 Add changelog 2019-09-06 13:27:55 +01:00
Andrew Morgan
5a7e9fdd84 Change changelog 2019-09-06 13:18:03 +01:00
Andrew Morgan
1ab1479a92 Add changelog 2019-09-06 13:08:52 +01:00
Erik Johnston
146af7b47f
Merge pull request #5991 from matrix-org/erikj/fix_tracing_funcs 
Don't assume there is a 'self' arg in @trace decorator
 2019-09-06 11:42:45 +01:00
Andrew Morgan
0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987) 
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
 2019-09-06 11:35:28 +01:00
Erik Johnston
e5baf80237 Update changelog 2019-09-06 10:53:05 +01:00
Erik Johnston
4bc6b7130d Newsfile 2019-09-06 10:13:10 +01:00
Jorik Schellekens
f7c873a643
Trace how long it takes for the send trasaction to complete, including retrys (#5986) 2019-09-05 17:44:55 +01:00
Jorik Schellekens
bc604e7f94
Gracefully handle log context slips and missing opentracing import errors. (#5988) 2019-09-05 17:33:29 +01:00
Erik Johnston
591d82f06b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions 2019-09-05 17:27:46 +01:00
Erik Johnston
1a6ae33309
Merge pull request #5984 from matrix-org/joriks/opentracing_link_send_to_edu_contexts 
Link the send loop with the edus contexts
 2019-09-05 15:22:24 +01:00
Jorik Schellekens
7093790fbc
Bugfix phrasing 
Co-Authored-By: Erik Johnston <erik@matrix.org>
 2019-09-05 15:07:00 +01:00
Jorik Schellekens
909827b422
Add opentracing to all client servlets (#5983) 2019-09-05 14:46:04 +01:00
Jorik Schellekens
93bc9d73bf newsfile 2019-09-05 14:45:07 +01:00
Andrew Morgan
a0d294c306
Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (#5892) 2019-09-05 14:31:22 +01:00
Jorik Schellekens
b9cfd3c375
Fix opentracing contexts missing from outbound replication requests (#5982) 2019-09-05 14:22:15 +01:00
Andrew Morgan
90d17a3d28
Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (#5980) 
Implements `POST /_matrix/client/r0/account/3pid/unbind` from [MSC2140](https://github.com/matrix-org/matrix-doc/blob/dbkr/tos_2/proposals/2140-terms-of-service-2.md#post-_matrixclientr0account3pidunbind).
 2019-09-05 14:00:30 +01:00
Hubert Chathi
faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Andrew Morgan
b736c6cd3a
Remove bind_email and bind_msisdn (#5964) 
Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).
 2019-09-04 18:24:23 +01:00
Andrew Morgan
b09d443632
Cleanup event auth type initialisation (#5975) 
Very small code cleanup.
 2019-09-04 16:16:56 +01:00
Erik Johnston
6e834e94fc
Fix and refactor room and user stats (#5971) 
Previously the stats were not being correctly populated.
 2019-09-04 13:04:27 +01:00
Travis Ralston
2f416fc997
Ensure the list media admin API is always available (#5966) 
* Ensure the list media admin API is always available

This API is required for some external media repo implementations to operate (mostly for doing quarantine operations on a room).

* changelog
 2019-09-03 13:35:20 -06:00
Michael Kaye
894c1a5759
Docker packaging should not su-exec or chmod if already running as UID/GID (#5970) 
Adjust su-exec to only be used if needed.

If UID == getuid() and GID == getgid() then we do not need to su-exec, and chmod will not work.
 2019-09-03 16:36:01 +01:00
Travis Ralston
0eac7077c9
Ensure an auth instance is available to ListMediaInRoom (#5967) 
* Ensure an auth instance is available to ListMediaInRoom

Fixes https://github.com/matrix-org/synapse/issues/5737

* Changelog
 2019-09-03 09:01:30 -06:00
Andrew Morgan
2a44782666
Remove double return statements (#5962) 
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
 2019-09-03 11:42:45 +01:00
Jorik Schellekens
a90d16dabc
Opentrace device lists (#5853) 
Trace device list changes.
 2019-09-03 10:21:30 +01:00
Andrew Morgan
36f34e6f3d
Remove unused methods from c/s api v1 in register.py (#5963) 
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
 2019-09-02 18:29:21 +01:00
L0ric0
ce7803b8b0 fix thumbnail storage location (#5915) 
* fix thumbnail storage location

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Add changelog file.

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Update Changelog

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
 2019-09-02 12:18:41 +01:00
Aaron Raimist
cee00a3584 Update INSTALL.md to say that Python 2 is no longer supported (#5953) 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-09-02 11:27:39 +01:00
Andrew Morgan
2a012e8a04
Revert "Add m.id_access_token flag (#5930)" (#5945) 
This reverts commit 4765f0cfd9.
 2019-08-30 17:13:37 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931) 
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
 2019-08-30 16:28:26 +01:00
Amber Brown
4fca313389
Move buildkite config to the pipelines repo (#5943) 2019-08-31 01:01:57 +10:00
Andrew Morgan
4765f0cfd9
Add m.id_access_token flag (#5930) 
Adds a flag to `/versions`' `unstable_features` section indicating that this Synapse understands what an `id_access_token` is, as per https://github.com/matrix-org/synapse/issues/5927#issuecomment-523566043

Fixes #5927
 2019-08-30 15:22:51 +01:00
Amber Brown
d19505a8c1
Removed unused jenkins/ folder and script (#5938) 2019-08-30 23:13:16 +10:00
Andrew Morgan
3057095a5d Revert "Use the v2 lookup API for 3PID invites (#5897)" (#5937) 
This reverts commit 71fc04069a.

This broke 3PID invites as #5892 was required for it to work correctly.
 2019-08-30 12:00:20 +01:00
Erik Johnston
549f974897 Newsfile 2019-08-30 11:29:17 +01:00
Amber Brown
e7011280c7
Fix coverage in sytest and use plugins for buildkite (#5922) 2019-08-29 22:19:57 +10:00
Hubert Chathi
e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Hubert Chathi
72d296a7f3 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-08-28 17:33:44 -07:00
Jorik Schellekens
92c1550f4a Add a link to python's logging config schema (#5926) 2019-08-28 19:08:32 +01:00
Will Hunt
c8fa620d7a
Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent 
Exempt support users from consent
 2019-08-28 16:31:40 +01:00
Jorik Schellekens
deca277d09
Let synctl use a config directory. (#5904) 
* Let synctl use a config directory.
 2019-08-28 15:55:58 +01:00
Will Hunt
5798a134c0
Removing entry for 5903 2019-08-28 14:25:05 +01:00
Andrew Morgan
71fc04069a
Use the v2 lookup API for 3PID invites (#5897) 
Fixes https://github.com/matrix-org/synapse/issues/5861

Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites.

Sytest PR: https://github.com/matrix-org/sytest/pull/679
 2019-08-28 14:59:26 +02:00
Jorik Schellekens
6d97843793
Config templating (#5900) 
Template config files

* Imagine a system composed entirely of x, y, z etc and the basic operations..

Wait George, why XOR? Why not just neq?

George: Eh, I didn't think of that..

Co-Authored-By: Erik Johnston <erik@matrix.org>
 2019-08-28 13:12:22 +01:00
Amber Brown
7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff
49ef8ec399
Fix a cache-invalidation bug for worker-based deployments (#5920) 
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.

We need to make sure we send out cache invalidations even when no memberships
are changing.
 2019-08-28 10:18:16 +01:00
reivilibre
a3f0635686
Merge pull request #5914 from matrix-org/rei/admin_getadmin 
Add GET method to admin API /users/@user:dom/admin
 2019-08-28 09:44:22 +01:00
reivilibre
7ccc251415
Merge pull request #5859 from matrix-org/rei/msc2197 
MSC2197 Search Filters over Federation
 2019-08-28 09:00:21 +01:00
Erik Johnston
dfd10f5133
Merge pull request #5864 from matrix-org/erikj/reliable_lookups 
Refactor MatrixFederationAgent to retry SRV.
 2019-08-27 16:54:06 +01:00
Olivier Wilkinson (reivilibre)
c88a119259 Add GET method to admin API /users/@user:dom/admin 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-27 13:12:27 +01:00
reivilibre
322ccac33f
Allow schema deltas to be engine-specific (#5911) 
* Allow schema deltas to be engine-specific

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>

* Code style (Black)

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-27 11:53:21 +01:00
Richard van der Hoff
ccb15a5bbe
Merge pull request #5906 from matrix-org/neilj/increase_display_name_limit 
Increase profile display name limit
 2019-08-27 11:52:59 +01:00
Erik Johnston
f5b50d0871
Merge pull request #5895 from matrix-org/erikj/notary_key 
Add config option to sign remote key query responses with a separate key.
 2019-08-27 11:51:37 +01:00
Richard van der Hoff
e7577427c9
Update 5909.misc 2019-08-27 11:50:52 +01:00
Richard van der Hoff
7837a5f2ea
Merge pull request #5909 from aaronraimist/public_base_url 
public_base_url is actually public_baseurl
 2019-08-27 11:49:59 +01:00
reivilibre
1a7e6eb633
Add Admin API capability to set adminship of a user (#5878) 
Admin API: Set adminship of a user
 2019-08-27 10:14:00 +01:00
Olivier Wilkinson (reivilibre)
62a1639287 Newsfile 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-27 09:36:12 +01:00
Aaron Raimist
c25137a99f
Add changelog 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-08-26 21:06:10 -05:00
Neil Johnson
27d3fc421a Increase max display name limit 2019-08-24 22:33:43 +01:00
Half-Shot
4a2d2c2b6f Update changelog 2019-08-23 09:57:07 +01:00
Half-Shot
ffa5b757c7 Merge branch 'hs/bot-user-type' into hs/exempt-support-users-from-consent 2019-08-23 09:55:57 +01:00
Half-Shot
971c980c6e Add changelog 2019-08-23 09:53:48 +01:00
Half-Shot
80793e813c newsfile 5902 2019-08-23 09:20:31 +01:00
Jorik Schellekens
8767b63a82
Propagate opentracing contexts through EDUs (#5852) 
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-08-22 18:21:10 +01:00
Richard van der Hoff
0b39fa53b6
Merge pull request #5877 from Awesome-Technologies/remove_shared_secret_registration 
Remove shared secret registration
 2019-08-22 18:12:25 +01:00
Jorik Schellekens
812ed6b0d5
Opentracing across workers (#5771) 
Propagate opentracing contexts across workers


Also includes some Convenience modifications to opentracing for servlets, notably:
- Add boolean to skip the whitelisting check on inject
  extract methods. - useful when injecting into carriers
  locally. Otherwise we'd always have to include our
  own servername and whitelist our servername
- start_active_span_from_request instead of header
- Add boolean to decide whether to extract context
  from a request to a servlet
 2019-08-22 18:08:07 +01:00
Manuel Stahl
0bab582fd6 Remove shared secret registration from client/r0/register endpoint 
This type of registration was probably never used. It only includes the
user name in the HMAC but not the password.

Shared secret registration is still available via
client/r0/admin/register.

Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2019-08-22 18:04:08 +02:00
Erik Johnston
1e4b4d85e7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reliable_lookups 2019-08-22 13:41:57 +01:00
Erik Johnston
1b09cf8658
Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_fail 
Retry well known on fail
 2019-08-22 13:17:05 +01:00
Jorik Schellekens
9a6f2be572
Opentrace e2e keys (#5855) 
Add opentracing tags and logs for e2e keys
 2019-08-22 11:28:12 +01:00
Richard van der Hoff
c9f11d09fc
Add missing index on users_in_public_rooms. (#5894) 2019-08-22 10:43:13 +01:00
Richard van der Hoff
119aa31b10
Servlet to purge old rooms (#5845) 2019-08-22 10:42:59 +01:00
Richard van der Hoff
ef1c524bb3
Improve error msg when key-fetch fails (#5896) 
There's no point doing a raise_from here, because the exception is always
logged at warn with no stacktrace in the caller. Instead, let's try to give
better messages to reduce confusion.

In particular, this means that we won't log 'Failed to connect to remote
server' when we don't even attempt to connect to the remote server due to
blacklisting.
 2019-08-22 10:42:06 +01:00
Richard van der Hoff
4dab867288
Drop some unused tables. (#5893) 
These tables are never used, so we may as well drop them.
 2019-08-21 13:16:28 +01:00
Erik Johnston
62fb643cdc Newsfile 2019-08-21 11:21:58 +01:00
Richard van der Hoff
72bc285669
Refactor the Appservice scheduler code (#5886) 
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).

Add some docstrings too.
 2019-08-20 17:42:45 +01:00
Richard van der Hoff
baa3f4a80d
Avoid deep recursion in appservice recovery (#5885) 
Hopefully, this will fix a stack overflow when recovering an appservice.

The recursion here leads to a huge chain of deferred callbacks, which then
overflows the stack when the chain completes. `inlineCallbacks` makes a better
job of this if we use iteration instead.

Clean up the code a bit too, while we're there.
 2019-08-20 17:39:38 +01:00
Jorik Schellekens
c886f976e0
Opentracing doc update (#5776) 
Update opentracing docs to use the unified 'trace' method
 2019-08-20 13:56:03 +01:00
Erik Johnston
29763f01c6 Make changelog entry be a feature 2019-08-20 12:38:06 +01:00
Richard van der Hoff
5019945828 Refactor the Appservice scheduler code 
Get rid of the labyrinthine `recoverer_fn` code, and clean up the startup code
(it seemed to be previously inexplicably split between
`ApplicationServiceScheduler.start` and `_Recoverer.start`).

Add some docstrings too.
 2019-08-20 11:50:23 +01:00
Olivier Wilkinson (reivilibre)
502728777c Newsfile on one line 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-20 08:49:53 +01:00
Erik Johnston
d514dac0b2
Merge pull request #5860 from matrix-org/erikj/update_5704_comments 
Remove logging for #5407 and update comments
 2019-08-19 10:20:59 +01:00
Richard van der Hoff
74fb729213 1.3.1 2019-08-17 09:16:17 +01:00
Richard van der Hoff
412c6e21a8
Drop dependency on sdnotify (#5871) 
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
 2019-08-17 09:09:52 +01:00
Hubert Chathi
8a5f6ed130
Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index 
add the version field to the index for e2e_room_keys
 2019-08-16 17:45:50 -07:00
Richard van der Hoff
c188bd2c12 add attribution 2019-08-16 23:19:23 +01:00
Chris Moos
20402aa128 Add changelog entry. 2019-08-16 22:16:21 +01:00
Jorik Schellekens
87fa26006b
Opentracing misc (#5856) 
Add authenticated_entity and servlet_names tags.

Functionally:
- Add a tag for authenticated_entity
- Add a tag for servlet_names

Stylistically:
Moved to importing methods directly from opentracing.
 2019-08-16 16:13:25 +01:00
Erik Johnston
ebba15ee7f Newsfile 2019-08-16 13:29:41 +01:00
Erik Johnston
861d663c15 Fixup changelog and remove debug logging 2019-08-16 13:15:26 +01:00
Hubert Chathi
e132ba79ae fix changelog 2019-08-15 21:02:40 -07:00
Andrew Morgan
b13cac896d
Fix up password reset template config names (#5863) 
Fixes #5833

The emailconfig code was attempting to pull incorrect config file names. This corrects that, while also marking a difference between a config file variable that's a filepath versus a str containing HTML.
 2019-08-15 16:27:11 +01:00
Erik Johnston
c03e3e8301 Newsfile 2019-08-15 15:43:22 +01:00
Olivier Wilkinson (reivilibre)
a3df04a899 Newsfile 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2019-08-15 11:09:07 +01:00
Michael Telatynski
baee288fb4 Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633) 2019-08-15 09:45:57 +01:00
Erik Johnston
1771f0045d Newsfile 2019-08-15 09:28:58 +01:00
Hubert Chathi
81b8080acd add changelog 2019-08-14 17:53:33 -07:00
Erik Johnston
09f6152a11
Merge pull request #5844 from matrix-org/erikj/retry_well_known_lookup 
Retry well-known lookup before expiry.
 2019-08-14 09:53:33 +01:00
Brendan Abolivier
f70d0a1dd9 1.3.0rc1 2019-08-13 18:20:09 +01:00
Amber H. Brown
28bce1ac7c changelog 2019-08-14 02:08:24 +10:00
Erik Johnston
aedfec3ad7 Newsfile 2019-08-13 16:20:38 +01:00
Erik Johnston
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop 
Handle pusher being deleted during processing.
 2019-08-13 14:08:29 +01:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Erik Johnston
e9906b0772
Merge pull request #5836 from matrix-org/erikj/lower_bound_ttl_well_known 
Add a lower bound to well-known TTL.
 2019-08-13 12:41:16 +01:00
Erik Johnston
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses 
Don't unnecessarily block notifying of new events.
 2019-08-13 09:36:25 +01:00
Hubert Chathi
f63ba7a795
Cross-signing [1/4] -- hidden devices (#5759) 
* allow devices to be marked as "hidden"

This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
 2019-08-12 15:14:37 -07:00
Erik Johnston
9d9cf3583b
Merge pull request #5843 from matrix-org/erikj/workers_hist_vis 
Whitelist history visbility sytests for worker mode
 2019-08-12 18:02:19 +01:00
Erik Johnston
2bec3a4953
Merge pull request #5839 from tcitworld/fix-purge-remote-media-script 
Fix curl command typo in purge_remote_media.sh
 2019-08-12 14:51:27 +01:00
Erik Johnston
3de6cc245f
Changelogs should end in '.' or '!' 2019-08-12 14:16:42 +01:00
Erik Johnston
156a461cbd Newsfile 2019-08-12 13:57:52 +01:00
Richard van der Hoff
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions 
Fix handling of redactions of redactions
 2019-08-12 12:25:19 +01:00
Erik Johnston
41546f946e Newsfile 2019-08-12 09:56:58 +01:00
Thomas Citharel
a7f0161276
Fix curl command typo in purge_remote_media.sh 
Was verbose option instead of -X, command didn't work

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
 2019-08-09 18:36:12 +02:00
Erik Johnston
d1b5b055be
Merge pull request #5825 from matrix-org/erikj/fix_empty_limited_sync 
Handle TimelineBatch being limited and empty.
 2019-08-06 15:39:44 +01:00
Andrew Morgan
edeae53221
Return 404 instead of 403 when retrieving an event without perms (#5798) 
Part of fixing matrix-org/sytest#652

Sytest PR: matrix-org/sytest#667
 2019-08-06 13:33:55 +01:00
Erik Johnston
c32d359094 Newsfile 2019-08-06 13:33:42 +01:00
Erik Johnston
977fa4a717 Newsfile 2019-08-06 13:00:45 +01:00
Hubert Chathi
8c9adcc95d fix formatting 2019-08-01 22:09:05 -04:00
Hubert Chathi
d28d1e2d1b add changelog 2019-08-01 21:52:35 -04:00
Brendan Abolivier
8ed9e63432
Account validity: allow defining HTML templates to serve the us… (#5807) 
Account validity: allow defining HTML templates to serve the user on account renewal attempt
 2019-08-01 16:09:25 +02:00
Erik Johnston
d55bc4a8bf
Merge pull request #5810 from matrix-org/erikj/no_server_reachable 
Return 502 not 500 when failing to reach any remote server.
 2019-08-01 14:19:39 +01:00
Andrew Morgan
5d018d23f0
Have ClientReaderSlavedStore inherit RegistrationStore (#5806) 
Fixes #5803
 2019-08-01 13:54:56 +01:00
Erik Johnston
93fd3cbc7a Newsfile 2019-08-01 13:48:52 +01:00
Erik Johnston
3c076c79c5
Merge pull request #5808 from matrix-org/erikj/parse_decode_error 
Handle incorrectly encoded query params correctly
 2019-08-01 13:48:10 +01:00
Erik Johnston
55a0c98d16
Merge pull request #5805 from matrix-org/erikj/validate_state 
Validate well known state events are state events.
 2019-08-01 13:45:48 +01:00
Erik Johnston
0b36decfb6
Merge pull request #5801 from matrix-org/erikj/recursive_tombstone 
Don't allow clients to send tombstones that reference the same room
 2019-08-01 13:45:35 +01:00
Erik Johnston
312cc48e2b Newsfile 2019-08-01 13:45:09 +01:00
Erik Johnston
da378af445 Newsfile 2019-08-01 13:24:00 +01:00
Erik Johnston
76a58fdcce Fix spelling. 
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2019-08-01 13:17:55 +01:00
Erik Johnston
58af30a6c7
Merge pull request #5802 from matrix-org/erikj/deny_redacting_different_room 
Deny redaction of events in a different room.
 2019-08-01 13:14:46 +01:00
Erik Johnston
0f632f3a57
Merge pull request #5790 from matrix-org/erikj/groups_request_errors 
Handle RequestSendFailed exception correctly in more places.
 2019-08-01 13:14:08 +01:00
Brendan Abolivier
f4a30d286f
Changelog 2019-08-01 12:08:06 +02:00
Erik Johnston
e5a0224837 Newsfile 2019-07-31 16:39:42 +01:00
Erik Johnston
c5288e9984 Newsfile 2019-07-31 16:32:03 +01:00
Erik Johnston
0eefb76fa1 Newsfile 2019-07-31 16:13:57 +01:00
Erik Johnston
02735e140f Newsfile 2019-07-31 15:53:52 +01:00
Andrew Morgan
72167fb394
Change user deactivated errcode to USER_DEACTIVATED and use it (#5686) 
This is intended as an amendment to #5674 as using M_UNKNOWN as the errcode makes it hard for clients to differentiate between an invalid password and a deactivated user (the problem we were trying to solve in the first place).

M_UNKNOWN was originally chosen as it was presumed than an MSC would have to be carried out to add a new code, but as Synapse often is the testing bed for new MSC implementations, it makes sense to try it out first in the wild and then add it into the spec if it is successful. Thus this PR return a new M_USER_DEACTIVATED code when a deactivated user attempts to login.
 2019-07-31 15:19:06 +01:00
Erik Johnston
8fde611a8c
Merge pull request #5794 from matrix-org/erikj/share_ssl_options_for_well_known 
Share SSL options for well-known requests
 2019-07-31 11:40:02 +01:00
Amber Brown
8f15832950
Remove DelayedCall debugging from test runs (#5787) 2019-07-31 20:39:22 +10:00
Erik Johnston
fe2f2fc530 Newsfile 2019-07-31 10:59:39 +01:00
Erik Johnston
3b7a35a59a Newsfile 2019-07-31 10:39:24 +01:00
Hubert Chathi
c1f0a5636c Merge branch 'develop' into cross-signing_hidden 2019-07-30 23:33:06 -04:00
Hubert Chathi
2997a91250 add changelog file 2019-07-30 23:14:00 -04:00
Erik Johnston
4037d3220a Newsfile 2019-07-30 16:43:59 +01:00
Erik Johnston
958d69f300 Newsfile 2019-07-30 14:53:52 +01:00
Erik Johnston
f92d05e254 Newsfile 2019-07-30 13:43:53 +01:00
Erik Johnston
e23ab7f41a Newsfile 2019-07-30 13:10:00 +01:00
Erik Johnston
63eb4a1b62
Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_state 
Add unit test for current state membership bg update
 2019-07-30 10:00:02 +01:00
Richard van der Hoff
8c97f6414c
Remove non-functional 'expire_access_token' setting (#5782) 
The `expire_access_token` didn't do what it sounded like it should do. What it
actually did was make Synapse enforce the 'time' caveat on macaroons used as
access tokens, but since our access token macaroons never contained such a
caveat, it was always a no-op.

(The code to add 'time' caveats was removed back in v0.18.5, in #1656)
 2019-07-30 08:25:02 +01:00
Richard van der Hoff
5c3eecc70f changelog 2019-07-30 00:00:34 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Erik Johnston
7c8c3b8437
Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership 
Fix room summary when rejected events are in state
 2019-07-29 17:15:15 +01:00
Erik Johnston
3e013b7c8e
Merge pull request #5752 from matrix-org/erikj/forgotten_user 
Remove some more joins on room_memberships
 2019-07-29 17:15:01 +01:00
Erik Johnston
2a12d76646
Merge pull request #5770 from matrix-org/erikj/fix_current_state_event_sqlite 
Fix current_state bg update to work on old SQLite
 2019-07-29 17:09:01 +01:00
Amber Brown
97a8b4caf7
Move some timeout checking logs to DEBUG #5785 2019-07-30 02:02:18 +10:00
Jorik Schellekens
85b0bd8fe0 Update the device list cache when keys/query is called (#5693) 2019-07-29 16:34:44 +01:00
Erik Johnston
3b476f5767 Fix debian packages for sid being called buster. (#5775) 
* Fix debian packages for sid being called buster.

I don't know why the sid images return buster as its codename in
`lsb_release` but it does, so lets just grab the codename from the
distro we pass into dockerfile

* Newsfile
 2019-07-30 00:33:32 +10:00
Erik Johnston
d94916852f Newsfile 2019-07-29 13:04:58 +01:00
Brendan Abolivier
bd083a5fcf
Changelog 2019-07-29 10:04:09 +02:00
Richard van der Hoff
d74595e2ca Merge branch 'master' into develop 2019-07-26 12:39:33 +01:00
Richard van der Hoff
1a93daf353
Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch 
Log when we receive a /make_* request from a different origin
 2019-07-26 12:38:37 +01:00
Richard van der Hoff
dde6ea7ff6 1.2.1 2019-07-26 11:33:16 +01:00
Erik Johnston
2e9cf7dda5 Newsfile 2019-07-26 10:14:31 +01:00
Richard van der Hoff
a0ee2ec458 Merge branch 'erikj/log_leave_origin_mismatch' into release-v1.2.1 2019-07-26 10:09:36 +01:00
Richard van der Hoff
d1020653fc Log when we receive a /make_* request from a different origin 2019-07-26 10:08:22 +01:00
Richard van der Hoff
1cad8d7b6f
Convert RedactionTestCase to modern test style (#5768) 2019-07-26 07:38:55 +01:00
Richard van der Hoff
26d742fed6
Merge pull request #5767 from matrix-org/rav/redactions/cross_room_id 
log when a redaction attempts to redact an event in a different room
 2019-07-25 18:49:56 +01:00
Richard van der Hoff
b1605cdd23 log when a redaction attempts to redact an event in a different room 2019-07-25 18:26:20 +01:00
Richard van der Hoff
618bd1ee76
Fix some error cases in the caching layer. (#5749) 
There was some inconsistent behaviour in the caching layer around how
exceptions were handled - particularly synchronously-thrown ones.

This seems to be most easily handled by pushing the creation of
ObservableDeferreds down from CacheDescriptor to the Cache.
 2019-07-25 15:59:45 +01:00
Andrew Morgan
2d573e2e2b 1.2.0rc2 2019-07-24 13:43:40 +01:00
Erik Johnston
2276936bac
Merge pull request #5743 from matrix-org/erikj/log_origin_receipts_mismatch 
Log when we receive receipt from a different origin
 2019-07-24 13:27:57 +01:00
Richard van der Hoff
f30a71a67b
Stop trying to fetch events with event_id=None. (#5753) 
`None` is not a valid event id, so queuing up a database fetch for it seems
like a silly thing to do.

I considered making `get_event` return `None` if `event_id is None`, but then
its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
 2019-07-24 13:16:18 +01:00
Jorik Schellekens
cf2972c818
Fix servlet metric names (#5734) 
* Fix servlet metric names

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Remove redundant check

* Cover all return paths
 2019-07-24 13:07:35 +01:00
Erik Johnston
c159803067 Newsfile 2019-07-24 11:51:44 +01:00
Richard van der Hoff
418635e68a
Add a prometheus metric for active cache lookups. (#5750) 
* Add a prometheus metric for active cache lookups.

* changelog
 2019-07-24 11:33:13 +01:00
Erik Johnston
adcd5368b0 Newsfile 2019-07-23 17:00:24 +01:00
Jorik Schellekens
3641784e8c
Make Jaeger fully configurable (#5694) 
* Allow Jaeger to be configured

* Update sample config
 2019-07-23 15:46:04 +01:00
Erik Johnston
65afc535a6
Update changelog.d/5743.bugfix 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-07-23 15:14:21 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Erik Johnston
fadfde9aaa Newsfile 2019-07-23 13:32:37 +01:00
Jorik Schellekens
18a466b84e
Opentracing Utils (#5722) 
* Add decerators for tracing functions

* Use the new clean contexts

* Context and edu utils

* Move opentracing setters

* Move whitelisting

* Sectioning comments

* Better args wrapper

* Docstrings

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Remove unused methods.

* Don't use global

* One tracing decorator to rule them all.
 2019-07-23 13:31:16 +01:00
Erik Johnston
841b12867e
Merge pull request #5732 from matrix-org/erikj/sdnotify 
Add process hooks to tell systemd our state.
 2019-07-23 13:06:53 +01:00
Erik Johnston
22d2338ace Newsfile 2019-07-23 10:27:53 +01:00
Erik Johnston
4f6984aa88
Merge pull request #5738 from matrix-org/erikj/faster_update 
Speed up current state background update.
 2019-07-23 10:23:12 +01:00
Erik Johnston
cf0006719d Newsfile 2019-07-23 10:01:30 +01:00
Erik Johnston
d9ea9881d2 Newsfile 2019-07-22 16:09:15 +01:00
Amber Brown
0d0f6d12bc Fix logging in workers (#5729) 
This also adds a worker blacklist.
 2019-07-22 16:05:00 +01:00
Erik Johnston
720d30469f
Merge pull request #5730 from matrix-org/erikj/cache_versions 
Cache get_version_string.
 2019-07-22 14:52:52 +01:00
Erik Johnston
79f689e6c2 Newsfile 2019-07-22 14:52:19 +01:00
Jason Robinson
8e513e7afc
Merge pull request #5731 from matrix-org/jaywink/admin-user-list-user-type 
Add `user_type` to returned fields in admin API user list endpoints
 2019-07-22 16:28:51 +03:00
Erik Johnston
22e862304a
Update changelog.d/5730.misc 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-07-22 14:09:56 +01:00
Richard van der Hoff
0cb72812f9
Fix stack overflow in Keyring (#5724) 
* Refactor Keyring._start_key_lookups

There's an awful lot of deferreds and dictionaries flying around here. The
whole thing can be made much simpler and achieve the same effect.

* Add a delay to key lookup lock release to fix stack overflow

A tactical call_later here should fix #5723

* changelog
 2019-07-22 13:51:22 +01:00
Andrew Morgan
f477ce4b1a v1.2.0rc1 
Features
 --------
 
 - Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
 - Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
 - Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
 - Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
 - Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
 - Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
 - Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
 - Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
 - Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))
 
 Bugfixes
 --------
 
 - Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
 - Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
 - Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
 - Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
 - Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
 - Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
 - Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
 - Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
 - Ignore redactions of m.room.create events. ([\#5701](https://github.com/matrix-org/synapse/issues/5701))
 
 Updates to the Docker image
 ---------------------------
 
 - Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
 - Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))
 
 Improved Documentation
 ----------------------
 
 - Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
 - --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
 - Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
 - Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))
 
 Deprecations and Removals
 -------------------------
 
 - Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))
 
 Internal Changes
 ----------------
 
 - Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
 - Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
 - Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
 - Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
 - Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
 - Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
 - Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
 - Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
 - Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
 - Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
 - Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
 - Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
 - Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
 - Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
 - Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
 - Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
 - Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
 - Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
 - Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
 - Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
 - Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
 - Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
 - Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl01rA4THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9GYqD/sHgmTXL33/TL6ytdWfcrngo3+zyqz3
 eMTvLggwFojTEzE2pO9R3UsaEZI+gMZEKfm+EEU4dXEnXxRxppMniHX3mMball7Q
 Xz7TgmediE6nW+m8FDbAuHsMYa7GY3p+u5ZyleO78zIjIu8YahcpKwX40JCJNCE1
 Vzz1T24i+lIsGB+3ahzabxkG1QHMppfClEFEpC8jYb+DL8BQzdMl0HAvdw4tAbDW
 ONPAdGkjtPUv6TcfBDXoMXVdoAA4CFyDuCWR0O57I3Re5b7ARy6Dx0vcQbMCTEgQ
 GnT1/QWtrpkX04OezMre2FAI+VQpQu/Hlhrl3+h8uQT1jBqYpWfR9jy/Y/iJWH0a
 Dam0XzvZrMM1wrzd6exLMWS76W8EWKT9IV7xR9kf4HwnOxGrmusshzkCEUAxwxJX
 jmFfTUw2f0yKt6lfNWaa67wP3suOlb++g33becFMKEM2DIb3iFxdk8U9p50406fL
 asCfMsNWNKrcIkAnsgvPXfZMhUX0bgHBtkiZxTZ8/cJK01N+9pVvlb6ZZ7j30aIH
 r4I6MAfaA57QOIla9Ml3LrT49dRskJpbReIqPiPxfH47W7hZhaCt0mF95SXzogF0
 cP6t36ths/ywJ85EQUYY+9VgYK26jObm2xFTN2rUnZ1hnCEhKY4/Lwz0jPRQ8au3
 2XHFZfqeAVpvvw==
 =ZrnB
 -----END PGP SIGNATURE-----

Merge tag 'v1.2.0rc1' into develop

v1.2.0rc1

Features
--------

- Add support for opentracing. ([\#5544](https://github.com/matrix-org/synapse/issues/5544), [\#5712](https://github.com/matrix-org/synapse/issues/5712))
- Add ability to pull all locally stored events out of synapse that a particular user can see. ([\#5589](https://github.com/matrix-org/synapse/issues/5589))
- Add a basic admin command app to allow server operators to run Synapse admin commands separately from the main production instance. ([\#5597](https://github.com/matrix-org/synapse/issues/5597))
- Add `sender` and `origin_server_ts` fields to `m.replace`. ([\#5613](https://github.com/matrix-org/synapse/issues/5613))
- Add default push rule to ignore reactions. ([\#5623](https://github.com/matrix-org/synapse/issues/5623))
- Include the original event when asking for its relations. ([\#5626](https://github.com/matrix-org/synapse/issues/5626))
- Implement `session_lifetime` configuration option, after which access tokens will expire. ([\#5660](https://github.com/matrix-org/synapse/issues/5660))
- Return "This account has been deactivated" when a deactivated user tries to login. ([\#5674](https://github.com/matrix-org/synapse/issues/5674))
- Enable aggregations support by default ([\#5714](https://github.com/matrix-org/synapse/issues/5714))

Bugfixes
--------

- Fix 'utime went backwards' errors on daemonization. ([\#5609](https://github.com/matrix-org/synapse/issues/5609))
- Various minor fixes to the federation request rate limiter. ([\#5621](https://github.com/matrix-org/synapse/issues/5621))
- Forbid viewing relations on an event once it has been redacted. ([\#5629](https://github.com/matrix-org/synapse/issues/5629))
- Fix requests to the `/store_invite` endpoint of identity servers being sent in the wrong format. ([\#5638](https://github.com/matrix-org/synapse/issues/5638))
- Fix newly-registered users not being able to lookup their own profile without joining a room. ([\#5644](https://github.com/matrix-org/synapse/issues/5644))
- Fix bug in #5626 that prevented the original_event field from actually having the contents of the original event in a call to `/relations`. ([\#5654](https://github.com/matrix-org/synapse/issues/5654))
- Fix 3PID bind requests being sent to identity servers as `application/x-form-www-urlencoded` data, which is deprecated. ([\#5658](https://github.com/matrix-org/synapse/issues/5658))
- Fix some problems with authenticating redactions in recent room versions. ([\#5699](https://github.com/matrix-org/synapse/issues/5699), [\#5700](https://github.com/matrix-org/synapse/issues/5700), [\#5707](https://github.com/matrix-org/synapse/issues/5707))
- Ignore redactions of m.room.create events. ([\#5701](https://github.com/matrix-org/synapse/issues/5701))

Updates to the Docker image
---------------------------

- Base Docker image on a newer Alpine Linux version (3.8 -> 3.10). ([\#5619](https://github.com/matrix-org/synapse/issues/5619))
- Add missing space in default logging file format generated by the Docker image. ([\#5620](https://github.com/matrix-org/synapse/issues/5620))

Improved Documentation
----------------------

- Add information about nginx normalisation to reverse_proxy.rst. Contributed by @skalarproduktraum - thanks! ([\#5397](https://github.com/matrix-org/synapse/issues/5397))
- --no-pep517 should be --no-use-pep517 in the documentation to setup the development environment. ([\#5651](https://github.com/matrix-org/synapse/issues/5651))
- Improvements to Postgres setup instructions. Contributed by @Lrizika - thanks! ([\#5661](https://github.com/matrix-org/synapse/issues/5661))
- Minor tweaks to postgres documentation. ([\#5675](https://github.com/matrix-org/synapse/issues/5675))

Deprecations and Removals
-------------------------

- Remove support for the `invite_3pid_guest` configuration setting. ([\#5625](https://github.com/matrix-org/synapse/issues/5625))

Internal Changes
----------------

- Move logging code out of `synapse.util` and into `synapse.logging`. ([\#5606](https://github.com/matrix-org/synapse/issues/5606), [\#5617](https://github.com/matrix-org/synapse/issues/5617))
- Add a blacklist file to the repo to blacklist certain sytests from failing CI. ([\#5611](https://github.com/matrix-org/synapse/issues/5611))
- Make runtime errors surrounding password reset emails much clearer. ([\#5616](https://github.com/matrix-org/synapse/issues/5616))
- Remove dead code for persiting outgoing federation transactions. ([\#5622](https://github.com/matrix-org/synapse/issues/5622))
- Add `lint.sh` to the scripts-dev folder which will run all linting steps required by CI. ([\#5627](https://github.com/matrix-org/synapse/issues/5627))
- Move RegistrationHandler.get_or_create_user to test code. ([\#5628](https://github.com/matrix-org/synapse/issues/5628))
- Add some more common python virtual-environment paths to the black exclusion list. ([\#5630](https://github.com/matrix-org/synapse/issues/5630))
- Some counter metrics exposed over Prometheus have been renamed, with the old names preserved for backwards compatibility and deprecated. See `docs/metrics-howto.rst` for details. ([\#5636](https://github.com/matrix-org/synapse/issues/5636))
- Unblacklist some user_directory sytests. ([\#5637](https://github.com/matrix-org/synapse/issues/5637))
- Factor out some redundant code in the login implementation. ([\#5639](https://github.com/matrix-org/synapse/issues/5639))
- Update ModuleApi to avoid register(generate_token=True). ([\#5640](https://github.com/matrix-org/synapse/issues/5640))
- Remove access-token support from `RegistrationHandler.register`, and rename it. ([\#5641](https://github.com/matrix-org/synapse/issues/5641))
- Remove access-token support from `RegistrationStore.register`, and rename it. ([\#5642](https://github.com/matrix-org/synapse/issues/5642))
- Improve logging for auto-join when a new user is created. ([\#5643](https://github.com/matrix-org/synapse/issues/5643))
- Remove unused and unnecessary check for FederationDeniedError in _exception_to_failure. ([\#5645](https://github.com/matrix-org/synapse/issues/5645))
- Fix a small typo in a code comment. ([\#5655](https://github.com/matrix-org/synapse/issues/5655))
- Clean up exception handling around client access tokens. ([\#5656](https://github.com/matrix-org/synapse/issues/5656))
- Add a mechanism for per-test homeserver configuration in the unit tests. ([\#5657](https://github.com/matrix-org/synapse/issues/5657))
- Inline issue_access_token. ([\#5659](https://github.com/matrix-org/synapse/issues/5659))
- Update the sytest BuildKite configuration to checkout Synapse in `/src`. ([\#5664](https://github.com/matrix-org/synapse/issues/5664))
- Add a `docker` type to the towncrier configuration. ([\#5673](https://github.com/matrix-org/synapse/issues/5673))
- Convert `synapse.federation.transport.server` to `async`. Might improve some stack traces. ([\#5689](https://github.com/matrix-org/synapse/issues/5689))
- Documentation for opentracing. ([\#5703](https://github.com/matrix-org/synapse/issues/5703))
 2019-07-22 13:49:16 +01:00
Jason Robinson
66f5ff72fd Add user_type to returned fields in admin API user list endpoints 
Mostly user type will be empty (normal user) but there is also the
"support" user type.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2019-07-22 15:29:18 +03:00
Erik Johnston
2017369f7d Newsfile 2019-07-22 13:18:25 +01:00
Andrew Morgan
54437c48ca 1.2.0rc1 2019-07-22 12:59:04 +01:00
Jorik Schellekens
f337d2f0f0
Demo uses deprecated cli option (#5725) 
* Remove deprecated 'verbose' cli arg

* Create 5725.bugfix
 2019-07-22 11:31:05 +01:00
Jorik Schellekens
0fd171770a Merge branch 'release-v1.2.0' into develop 2019-07-22 11:18:50 +01:00
Jorik Schellekens
826e6ec3bd
Opentracing Documentation (#5703) 
* Opentracing survival guide

* Update decorator names in doc

* Doc cleanup

These are all alterations as a result of comments in #5703, it
includes mostly typos and clarifications. The most interesting
changes are:

- Split developer and user docs into two sections
- Add a high level description of OpenTracing

* newsfile

* Move contributer specific info to docstring.

* Sample config.

* Trailing whitespace.

* Update 5703.misc

* Apply suggestions from code review

Mostly just rewording parts of the docs for clarity.

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-07-22 11:15:21 +01:00
Richard van der Hoff
f214bff0c0 changelog 2019-07-19 17:58:17 +01:00
Erik Johnston
7704873cb8
Merge pull request #5720 from matrix-org/erikj/transactions_upsert 
Use upsert when updating destination retry interval
 2019-07-19 16:51:16 +01:00
Erik Johnston
d7bd9651bc
Merge pull request #5713 from matrix-org/erikj/use_cache_for_filtered_state 
Delegate to cached version when using get_filtered_current_state_ids
 2019-07-19 16:30:49 +01:00
Erik Johnston
5c07c97c09
Merge pull request #5706 from matrix-org/erikj/add_memberships_to_current_state 
Add membership column to current_state_events table
 2019-07-19 16:30:33 +01:00
Jorik Schellekens
7b8bc61834
Don't accept opentracing data from clients. (#5715) 
* Don't accept opentracing data from clients.

* newsfile
 2019-07-19 16:29:57 +01:00
Erik Johnston
ced4fdaa84 Newsfile 2019-07-19 13:40:26 +01:00
Neil Johnson
5c05ae7ba0 Add 'rel' attribute to default welcome page. (#5695) 
add rel attribute as a precaution against reverse tabnabbing in future
 2019-07-19 12:03:36 +01:00
Richard van der Hoff
b73ce4ba81
Update the coding style doc (#5719) 
A few fixes and removal of duplicated stuff, but mostly a bunch of the words on the config file.
 2019-07-19 11:55:14 +01:00
Amber Brown
356ed0438e
Speed up the PostgreSQL unit tests (#5717) 2019-07-19 19:01:23 +10:00
Amber Brown
6a85cb5ef7
Remove non-dedicated logging options and command line arguments (#5678) 2019-07-19 01:40:08 +10:00
Neil Johnson
a3e40bd5b4 towncrier 2019-07-18 16:02:09 +01:00
Erik Johnston
dd2851d576 Newsfile 2019-07-18 15:27:18 +01:00
Richard van der Hoff
82345bc09a
Clean up opentracing configuration options (#5712) 
Clean up config settings and dead code.

This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular:
 * There should be a blank line after `## Section ##' headings
 * There should be a blank line between each config setting
 * There should be a `#`-only line between a comment and the setting it describes
 * We don't really do the `#  #` style commenting-out of whole sections if we can help it
 * rename `tracer_enabled` to `enabled`

While we're here, do more config parsing upfront, which makes it easier to use
later on.

Also removes redundant code from LogContextScopeManager.

Also changes the changelog fragment to a `feature` - it's exciting!
 2019-07-18 15:06:54 +01:00
Amber Brown
7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Andrew Morgan
b2a382efdb
Remove the ability to query relations when the original event was redacted. (#5629) 
Fixes #5594

Forbid viewing relations on an event once it has been redacted.
 2019-07-18 14:41:42 +01:00
Erik Johnston
89c885909a Newsfile 2019-07-18 14:16:01 +01:00
Richard van der Hoff
fa8271c5ac
Convert synapse.federation.transport.server to async (#5689) 
* Convert BaseFederationServlet._wrap to async

Empirically, this fixes some lost stacktraces. It should be safe because the
wrapped function is called from JsonResource._async_render, which is already
async.

* Convert the rest of synapse.federation.transport.server to async

We may as well do the whole file while we're here.

* changelog

* flake8
 2019-07-18 11:46:47 +01:00
Richard van der Hoff
9c70a02a9c
Ignore redactions of m.room.create events (#5701) 2019-07-17 19:08:02 +01:00
Richard van der Hoff
1def298119
Improve Depends specs in debian package. (#5675) 
This is basically a contrived way of adding a `Recommends` on `libpq5`, to fix #5653.

The way this is supposed to happen in debhelper is to run
`dh_shlibdeps`, which in turn runs `dpkg-shlibdeps`, which spits things out
into `debian/<package>.substvars` whence they can later be included by
`control`.

Previously, we had disabled `dh_shlibdeps`, mostly because `dpkg-shlibdeps`
gets confused about PIL's interdependent objects, but that's not really the
right thing to do and there is another way to work around that.

Since we don't always use postgres, we don't necessarily want a hard Depends on
libpq5, so I've actually ended up adding an explicit invocation of
`dpkg-shlibdeps` for `psycopg2`.

I've also updated the build-depends list for the package, which was missing a
couple of entries.
 2019-07-17 17:47:07 +01:00
Richard van der Hoff
2091c91fde
More refactoring in get_events_as_list (#5707) 
We can now use `_get_events_from_cache_or_db` rather than going right back to
the database, which means that (a) we can benefit from caching, and (b) it
opens the way forward to more extensive checks on the original event.

We now always require the original event to exist before we will serve up a
redaction.
 2019-07-17 17:34:13 +01:00
Richard van der Hoff
375162b3c3
Fix redaction authentication (#5700) 
Ensures that redactions are correctly authenticated for recent room versions.

There are a few things going on here:

 * `_fetch_event_rows` is updated to return a dict rather than a list of rows.

 * Rather than returning multiple copies of an event which was redacted
   multiple times, it returns the redactions as a list within the dict.

 * It also returns the actual rejection reason, rather than merely the fact
   that it was rejected, so that we don't have to query the table again in
   `_get_event_from_row`.

 * The redaction handling is factored out of `_get_event_from_row`, and now
   checks if any of the redactions are valid.
 2019-07-17 16:52:02 +01:00
Richard van der Hoff
65c5592b8e
Refactor get_events_as_list (#5699) 
A couple of changes here:

* get rid of a redundant `allow_rejected` condition - we should already have filtered out any rejected
  events before we get to that point in the code, and the redundancy is confusing. Instead, let's stick in
  an assertion just to make double-sure we aren't leaking rejected events by mistake.

* factor out a `_get_events_from_cache_or_db` method, which is going to be important for a 
  forthcoming fix to redactions.
 2019-07-17 16:49:19 +01:00
Erik Johnston
e8c53b07f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmd 2019-07-15 14:13:22 +01:00
Andrew Morgan
18c516698e
Return a different error from Invalid Password when a user is deactivated (#5674) 
Return `This account has been deactivated` instead of `Invalid password` when a user is deactivated.
 2019-07-15 11:45:29 +01:00
Erik Johnston
d86321300a
Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data 
Add basic function to get all data for a user out of synapse
 2019-07-15 10:04:02 +01:00
Richard van der Hoff
d336b51331
Add a docker type to the towncrier configuration (#5673) 
... and certain other changelog-related fixes
 2019-07-12 17:27:07 +01:00
Richard van der Hoff
5f158ec039
Implement access token expiry (#5660) 
Record how long an access token is valid for, and raise a soft-logout once it
expires.
 2019-07-12 17:26:02 +01:00
Richard van der Hoff
4c17a87606 fix changelog name 2019-07-12 11:47:24 +01:00
Ulrik Günther
d445b3ae57 Update reverse_proxy.rst (#5397) 
Updates reverse_proxy.rst with information about nginx' URI normalisation.
 2019-07-12 11:46:18 +01:00
Slavi Pantaleev
59f15309ca Add missing space in default logging file format generated by the Docker image (#5620) 
This adds a missing space, without which log lines appear uglier.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>
 2019-07-12 11:43:42 +01:00
Slavi Pantaleev
f369164761 Upgrade Alpine Linux used in the Docker image (3.8 -> 3.10) (#5619) 
Alpine Linux 3.8 is still supported, but it seems like
it's quite outdated now.

While Python should be the same on both, all other libraries, etc.,
are much newer in Alpine 3.9 and 3.10.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>
 2019-07-12 11:38:25 +01:00
Richard van der Hoff
6bb0357c94
Add a mechanism for per-test configs (#5657) 
It's useful to be able to tweak the homeserver config to be used for each
test. This PR adds a mechanism to do so.
 2019-07-12 10:16:23 +01:00
Amber Brown
a83577d64f
Use /src for checking out synapse during sytests (#5664) 2019-07-11 23:43:41 +10:00
Lrizika
39e9839a04 Improved docs on setting up Postgresql (#5661) 
Added that synapse_user needs a database to access before it can auth
Noted you'll need to enable password auth, linked to pg_hba.conf docs
 2019-07-11 14:31:36 +01:00
Andrew Morgan
78a1cd36b5 small typo fix (#5655) 2019-07-11 13:33:23 +01:00
Richard van der Hoff
0a4001eba1
Clean up exception handling for access_tokens (#5656) 
First of all, let's get rid of `TOKEN_NOT_FOUND_HTTP_STATUS`. It was a hack we
did at one point when it was possible to return either a 403 or a 401 if the
creds were missing. We always return a 401 in these cases now (thankfully), so
it's not needed.

Let's also stop abusing `AuthError` for these cases. Honestly they have nothing
that relates them to the other places that `AuthError` is used, other than the
fact that they are loosely under the 'Auth' banner. It makes no sense for them
to share exception classes.

Instead, let's add a couple of new exception classes: `InvalidClientTokenError`
and `MissingClientTokenError`, for the `M_UNKNOWN_TOKEN` and `M_MISSING_TOKEN`
cases respectively - and an `InvalidClientCredentialsError` base class for the
two of them.
 2019-07-11 11:06:23 +01:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544) 
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
 2019-07-11 10:36:03 +01:00
Richard van der Hoff
1890cfcf82 Inline issue_access_token (#5659) 
this is only used in one place, so it's clearer if we inline it and reduce the
API surface.

Also, fixes a buglet where we would create an access token even if we were
about to block the user (we would never return the AT, so the user could never
use it, but it was still created and added to the db.)
 2019-07-11 04:10:07 +10:00
Brendan Abolivier
8ab3444fdf
Merge pull request #5658 from matrix-org/babolivier/is-json 
Send 3PID bind requests as JSON data
 2019-07-10 17:01:26 +01:00
Richard van der Hoff
953dbb7980
Remove access-token support from RegistrationStore.register (#5642) 
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
 2019-07-10 16:26:49 +01:00
Brendan Abolivier
351d9bd317
Rename changelog file 2019-07-10 15:48:50 +01:00
Brendan Abolivier
f77e997619
Send 3PID bind requests as JSON data 2019-07-10 15:46:42 +01:00
Andrew Morgan
f281714583
Don't bundle aggregations when retrieving the original event (#5654) 
A fix for PR #5626, which returned the original event content as part of a call to /relations.

Only problem was that we were attempting to aggregate the relations on top of it when we did so. We now set bundle_aggregations to False in the get_event call.

We also do this when pulling the relation events as well, because edits of edits are not something we'd like to support here.
 2019-07-10 14:43:11 +01:00
Andrew Morgan
3dd61d12cd
Add a linting script (#5627) 
Add a dev script to cover all the different linting steps.
 2019-07-10 14:03:18 +01:00
Bruno Windels
4d122d295c Correct pep517 flag in readme (#5651) 2019-07-10 13:55:24 +01:00
Brendan Abolivier
65434da75d
Merge pull request #5638 from matrix-org/babolivier/invite-json 
Use JSON when querying the IS's /store-invite endpoint
 2019-07-09 18:48:38 +01:00
Hubert Chathi
7b3bc755a3 remove unused and unnecessary check for FederationDeniedError (#5645) 
FederationDeniedError is a subclass of SynapseError, which is a subclass of
CodeMessageException, so if e is a FederationDeniedError, then this check for
FederationDeniedError will never be reached since it will be caught by the
check for CodeMessageException above.  The check for CodeMessageException does
almost the same thing as this check (since FederationDeniedError initialises
with code=403 and msg="Federation denied with %s."), so may as well just keep
allowing it to handle this case.
 2019-07-09 18:37:39 +01:00
Andrew Morgan
d88421ab03
Include the original event in /relations (#5626) 
When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.
 2019-07-09 13:43:08 +01:00
Brendan Abolivier
af67c7c1de
Merge pull request #5644 from matrix-org/babolivier/profile-allow-self 
Allow newly-registered users to lookup their own profiles
 2019-07-09 10:25:40 +01:00
Richard van der Hoff
824707383b
Remove access-token support from RegistrationHandler.register (#5641) 
Nothing uses this now, so we can remove the dead code, and clean up the
API.

Since we're changing the shape of the return value anyway, we take the
opportunity to give the method a better name.
 2019-07-08 19:01:08 +01:00
Brendan Abolivier
f3615a8aa5
Changelog 2019-07-08 17:31:58 +01:00
Richard van der Hoff
43d175d17a Unblacklist some user_directory sytests (#5637) 
Fixes https://github.com/matrix-org/synapse/issues/2306
 2019-07-09 02:15:17 +10:00
Richard van der Hoff
b70e080b59
Better logging for auto-join. (#5643) 
It was pretty unclear what was going on, so I've added a couple of log lines.
 2019-07-08 17:14:51 +01:00
Brendan Abolivier
57eacee4f4 Merge branch 'develop' into babolivier/invite-json 2019-07-08 15:49:23 +01:00
Brendan Abolivier
c142e5d16a Changelog 2019-07-08 15:28:38 +01:00
Richard van der Hoff
4b1f7febc7 Update ModuleApi to avoid register(generate_token=True) (#5640) 
* Update ModuleApi to avoid register(generate_token=True)

This is the only place this is still used, so I'm trying to kill it off.

* changelog
 2019-07-08 23:55:34 +10:00
Richard van der Hoff
f9e99f9534 Factor out some redundant code in the login impl (#5639) 
* Factor out some redundant code in the login impl

Also fixes a redundant access_token which was generated during jwt login.

* changelog
 2019-07-08 23:54:22 +10:00
Richard van der Hoff
1af2fcd492 Move get_or_create_user to test code (#5628) 
This is only used in tests, so...
 2019-07-08 23:52:26 +10:00
Andrew Morgan
589d43d9cd Add a few more common environment directory names to black exclusion (#5630) 
* Add a few more common environment directory names to black exclusion

* Add changelog
 2019-07-08 21:53:33 +10:00
J. Ryan Stinnett
9b1b79f3f5
Add default push rule to ignore reactions (#5623) 
This adds a default push rule following the proposal in
[MSC2153](https://github.com/matrix-org/matrix-doc/pull/2153).

See also https://github.com/vector-im/riot-web/issues/10208
See also https://github.com/matrix-org/matrix-js-sdk/pull/976
 2019-07-05 17:37:52 +01:00
Andrew Morgan
ad8b909ce9
Add origin_server_ts and sender fields to m.replace (#5613) 
Riot team would like some extra fields as part of m.replace, so here you go.

Fixes: #5598
 2019-07-05 17:20:02 +01:00
Richard van der Hoff
80cc82a445
Remove support for invite_3pid_guest. (#5625) 
This has never been documented, and I'm not sure it's ever been used outside
sytest.

It's quite a lot of poorly-maintained code, so I'd like to get rid of it.

For now I haven't removed the database table; I suggest we leave that for a
future clearout.
 2019-07-05 16:47:58 +01:00
Erik Johnston
7f0d8e4288 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data 2019-07-05 14:08:21 +01:00
Richard van der Hoff
a6a776f3d8
remove dead transaction persist code (#5622) 
this hasn't done anything for years
 2019-07-05 12:59:42 +01:00
Richard van der Hoff
9481707a52
Fixes to the federation rate limiter (#5621) 
- Put the default window_size back to 1000ms (broken by #5181)
- Make the `rc_federation` config actually do something
- fix an off-by-one error in the 'concurrent' limit
- Avoid creating an unused `_PerHostRatelimiter` object for every single
  incoming request
 2019-07-05 11:10:19 +01:00
Andrew Morgan
0e5434264f
Make errors about email password resets much clearer (#5616) 
The runtime errors that dealt with local email password resets talked about config options that users may not even have in their config file yet (if upgrading). Instead, the cryptic errors are now replaced with hopefully much more helpful ones.
 2019-07-05 10:44:12 +01:00
Amber Brown
1ee268d33d
Improve the backwards compatibility re-exports of synapse.logging.context (#5617) 
* Improve the backwards compatibility re-exports of synapse.logging.context.

* reexport logformatter too
 2019-07-05 02:32:02 +10:00
Andrew Morgan
ee91ac179c Add a sytest blacklist file (#5611) 
* Add a sytest blacklist file

* Add changelog

* Add blacklist to manifest
 2019-07-05 01:24:13 +10:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Richard van der Hoff
cb8d568cf9 Fix 'utime went backwards' errors on daemonization. (#5609) 
* Fix 'utime went backwards' errors on daemonization.

Fixes #5608

* remove spurious debug
 2019-07-03 22:40:45 +10:00
Richard van der Hoff
463d5a8fde 1.1.0rc2 2019-07-03 10:51:47 +01:00
Richard van der Hoff
91753cae59
Fix a number of "Starting txn from sentinel context" warnings (#5605) 
Fixes #5602, #5603
 2019-07-03 09:31:27 +01:00
Andrew Morgan
c7b48bd42d Remove SMTP_* env var functionality from docker conf (#5596) 
Removes any `SMTP_*` docker container environment variables from having any effect on the default config.

Fixes https://github.com/matrix-org/synapse/issues/5430
 2019-07-03 07:14:48 +01:00
Amber Brown
0ee9076ffe Fix media repo breaking (#5593) 2019-07-02 19:01:28 +01:00
Erik Johnston
10fe904d88 Newsfile 2019-07-02 17:21:27 +01:00
Erik Johnston
65dd5543f6 Newsfile 2019-07-02 12:10:23 +01:00
Richard van der Hoff
7085e8c0fb Merge remote-tracking branch 'origin/master' into release-v1.1.0 2019-07-02 11:56:17 +01:00
Richard van der Hoff
a0fa4641c4 prepare v1.1.0rc1 2019-07-02 11:51:55 +01:00
Richard van der Hoff
6eecb6e500
Complete the SAML2 implementation (#5422) 
* SAML2 Improvements and redirect stuff

Signed-off-by: Alexander Trost <galexrt@googlemail.com>

* Code cleanups and simplifications.

Also: share the saml client between redirect and response handlers.

* changelog

* Revert redundant changes to static js

* Move all the saml stuff out to a centralised handler

* Add support for tracking SAML2 sessions.

This allows us to correctly handle `allow_unsolicited: False`.

* update sample config

* cleanups

* update sample config

* rename BaseSSORedirectServlet for consistency

* Address review comments
 2019-07-02 11:18:11 +01:00
Erik Johnston
c3863ad6bf
Merge pull request #5587 from matrix-org/erikj/fix_synctl 
Fix --no-daemonize flag for synctl
 2019-07-02 11:17:55 +01:00
Erik Johnston
8134c49cad Newsfile 2019-07-02 10:36:04 +01:00
Amir Zarrinkafsh
de8077a164 Add ability to set timezone for Docker container (#5383) 
Signed-off-by: Amir Zarrinkafsh <nightah@me.com>
 2019-07-02 10:31:06 +01:00
PauRE
948488e115 Fix JWT login with new users (#5586) 
Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>
 2019-07-02 10:25:37 +01:00
Amber Brown
b4914681a5
fix async/await consentresource (#5585) 
Fixes #5582
 2019-07-01 23:33:52 +10:00
Richard van der Hoff
b4fd86a9b4 Merge branch 'develop' into rav/saml2_client 2019-07-01 14:21:03 +01:00
Erik Johnston
04196a4dae
Merge pull request #5507 from matrix-org/erikj/presence_sync_tighloop 
Fix sync tightloop bug.
 2019-07-01 11:43:10 +01:00
Amber Brown
f40a7dc41f
Make the http server handle coroutine-making REST servlets (#5475) 2019-06-29 17:06:55 +10:00
Brendan Abolivier
c7ff297dde
Merge pull request #5576 from matrix-org/babolivier/3pid-invite-ratelimit 
Don't update the ratelimiter before sending a 3PID invite
 2019-06-28 17:43:48 +01:00
Brendan Abolivier
b339f6489f
Changelog 2019-06-28 15:24:59 +01:00
Amber Brown
071150ce19
Don't log GC 0s at INFO (#5557) 2019-06-28 21:45:33 +10:00
Amber Brown
be3b901ccd
Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550) 2019-06-28 18:19:09 +10:00
Daniel Hoffend
9646a593ac Added possibilty to disable local password authentication (#5092) 
Signed-off-by: Daniel Hoffend <dh@dotlan.net>
 2019-06-27 18:37:29 +01:00
Andrew Morgan
c548dbc4b1
Make it clearer that the template dir is relative to synapse's root dir (#5543) 
Helps address #5444
 2019-06-27 18:20:17 +01:00
Erik Johnston
e79ec03165
Merge pull request #5559 from matrix-org/erikj/refactor_changed_devices 
Refactor devices changed query to pull less from DB
 2019-06-27 16:53:15 +01:00
Richard van der Hoff
555b6fa0d5
Docker image: Add a migrate_config mode (#5567) 
... to help people escape env var hell
 2019-06-27 13:52:40 +01:00
Richard van der Hoff
1ddc7b39c9
Docker image: open the non-TLS port by default. (#5568) 
There's not much point in binding to localhost when it's in a docker container.
 2019-06-27 13:50:10 +01:00
Richard van der Hoff
2f7ebc2a55
Deprecate the env var way of running the docker image (#5566) 
This is mostly a documentation change, but also adds a default value for
SYNAPSE_CONFIG_PATH, so that running from the generated config is the default,
and will Just Work provided your config is in the right place.
 2019-06-27 13:49:48 +01:00
PauRE
856ea04eb3 Fix JWT login (#5555) 
* Fix JWT login with register

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Add pyjwt conditional dependency

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Added changelog file

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>

* Improved changelog description

Signed-off-by: Pau Rodriguez-Estivill <prodrigestivill@gmail.com>
 2019-06-27 12:02:41 +01:00
Richard van der Hoff
b4db70e167
Merge pull request #5565 from matrix-org/rav/docker/fix_log_config 
Docker: generate our own log config
 2019-06-27 11:19:37 +01:00
Richard van der Hoff
b2d2617c0d
Reduce the amount of stuff we send in the docker context (#5564) 
this makes docker builds a bit faster.
 2019-06-27 11:18:51 +01:00
Richard van der Hoff
b1b8a24b63
Merge pull request #5563 from matrix-org/rav/docker/data_dir 
Docker image: add support for SYNAPSE_DATA_DIR parameter
 2019-06-27 11:17:44 +01:00
Richard van der Hoff
53faa6a429
Merge pull request #5562 from matrix-org/rav/docker/no-generate-keys 
Docker: only run --generate-keys when generating config on-the-fly.
 2019-06-27 11:17:21 +01:00
Richard van der Hoff
02aeb5a98a
Merge pull request #5561 from matrix-org/rav/docker/refactor 
Refactor the docker/start.py script
 2019-06-27 11:16:37 +01:00
Richard van der Hoff
42a3619ef4
Merge pull request #5570 from almereyda/patch-2 
Update purge_api README
 2019-06-27 00:58:17 +01:00
Richard van der Hoff
a11475f396
fix changelog 2019-06-27 00:57:59 +01:00
Richard van der Hoff
47fa836abb
Merge pull request #5313 from twrist/patch-1 
Update HAProxy example rules
 2019-06-27 00:53:48 +01:00
jon r
536820e572
Create 5570.misc 
Signed-off-by: Jon Richter <jon@allmende.io>
 2019-06-27 00:13:34 +02:00
Richard van der Hoff
a4daa899ec Merge branch 'develop' into rav/saml2_client 2019-06-26 22:34:41 +01:00
Erik Johnston
82028d723b Move changelog 2019-06-26 19:39:49 +01:00
Richard van der Hoff
b051cefc75
Merge pull request #5552 from matrix-org/rav/github_templates 
Update github templates
 2019-06-26 16:15:41 +01:00
Richard van der Hoff
befa116b31 changelog 2019-06-26 15:52:00 +01:00
Richard van der Hoff
a0f2921ccf changelog 2019-06-26 15:41:10 +01:00
Richard van der Hoff
7c453472e4 changelog 2019-06-26 15:32:55 +01:00
Richard van der Hoff
043ab6da13 changelog 2019-06-26 15:28:28 +01:00
Richard van der Hoff
2d91988799
Improve docs on choosing server_name (#5558) 
Fixes #4901
 2019-06-26 15:15:04 +01:00
Erik Johnston
508c3ce3d7 Newsfile 2019-06-26 12:03:49 +01:00
Amber Brown
0e97284dfa
Remove & changelog (#5548) 2019-06-26 04:36:34 +10:00
Andrew Morgan
3eb8c7b0eb Merge branch 'master' into develop 
* master:
  Fix broken link in MSC1711 FAQ
  Update changelog to better expain password reset change (#5545)
 2019-06-25 18:08:56 +01:00
Andrew Morgan
ef8c62758c
Prevent multiple upgrades on the same room at once (#5051) 
Closes #4583

Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time.

Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.
 2019-06-25 14:19:21 +01:00
Richard van der Hoff
a5222b386e changelog 2019-06-25 12:24:47 +01:00
Richard van der Hoff
fe2d876e2a Increase default log level for docker image to INFO. (#5547) 
Fixes #3370.
 2019-06-25 14:38:38 +10:00
Richard van der Hoff
f817fc9ad5 Update docker image to use Python 3.7. (#5546) 
Python 3.7 is apparently faster than 3.6, and should be mature enough.
 2019-06-25 14:20:53 +10:00
Andrew Morgan
78c00ad3ff
Update changelog to better expain password reset change (#5545) 
Updates the v1.0.0 changelog to provide more information to those upgrading to v1.0 on why they may receive errors or find their password reset abilities have now been disabled.

Helps address #5444
 2019-06-24 17:52:15 +01:00
Andrew Morgan
28604ab03d
Add info about black to code_style.rst (#5537) 
Fixes #5533

Adds information about how to install and run black on the codebase.
 2019-06-24 17:48:05 +01:00
Richard van der Hoff
4ac7ef4b67
Merge pull request #5524 from matrix-org/rav/new_cmdline_options 
Add --data-dir and --open-private-ports options.
 2019-06-24 17:25:57 +01:00
Richard van der Hoff
af8a962905
Merge pull request #5523 from matrix-org/rav/arg_defaults 
Stop conflating generated config and default config
 2019-06-24 17:24:35 +01:00
Richard van der Hoff
e59a8cd2e5
Merge pull request #5499 from matrix-org/rav/cleanup_metrics 
Cleanups and sanity-checking in cpu and db metrics
 2019-06-24 17:12:54 +01:00
Brendan Abolivier
deb4fe6ef3
Merge pull request #5534 from matrix-org/babolivier/federation-publicrooms 
Split public rooms directory auth config in two
 2019-06-24 16:08:02 +01:00
Brendan Abolivier
bfe84e051e Split public rooms directory auth config in two 2019-06-24 15:42:31 +01:00
Erik Johnston
25433f212d
Merge pull request #5531 from matrix-org/erikj/workers_pagination_token 
Fix /messages on workers when no from param specified.
 2019-06-24 15:30:10 +01:00
Richard van der Hoff
e6b2ccbb51 changelog 2019-06-24 14:15:34 +01:00
Richard van der Hoff
c783294549 changelog 2019-06-24 14:14:52 +01:00
Richard van der Hoff
5d9c101551 changelog 2019-06-24 13:51:22 +01:00
Richard van der Hoff
c3c6b00d95
Pass config_dir_path and data_dir_path into Config.read_config. (#5522) 
* Pull config_dir_path and data_dir_path calculation out of read_config_files

* Pass config_dir_path and data_dir_path into read_config
 2019-06-24 11:34:45 +01:00
Richard van der Hoff
21bf4318b5
Factor acme bits out to a separate file (#5521) 
This makes some of the conditional-import hoop-jumping easier.
 2019-06-24 11:33:56 +01:00
Erik Johnston
14aff5cc0d Newsfile 2019-06-24 10:22:34 +01:00
Richard van der Hoff
5097aee740 Merge branch 'develop' into rav/cleanup_metrics 2019-06-24 10:00:13 +01:00
Richard van der Hoff
c753c098dd
Merge pull request #5498 from matrix-org/rav/fix_clock_reversal 
Use monotonic clock where possible for metrics
 2019-06-24 09:55:12 +01:00
Richard van der Hoff
6cda36777b Drop support for cpu_affinity (#5525) 
This has no useful purpose on python3, and is generally a source of confusion.
 2019-06-22 11:01:55 +10:00
Richard van der Hoff
e1a795758c
Improve help and cmdline option names for --generate-config options (#5512) 
* group the arguments together into a group
* add new names "--generate-missing-config" and "--config-directory" for
  existing cmdline options "--generate-keys" and "--keys-dir", which better
  reflect their purposes.
 2019-06-21 18:50:43 +01:00
Richard van der Hoff
03cea2b0fe
Refactor Config parser and add some comments. (#5511) 
Add some comments, and simplify `read_config_files`.
 2019-06-21 17:43:38 +01:00
Richard van der Hoff
37933a3bf8
Improve logging when generating config files (#5510) 
Make it a bit clearer what's going on.
 2019-06-21 17:14:56 +01:00
Andrew Morgan
5d6644efe0
Only import jinja2 when needed (#5514) 
Fixes https://github.com/matrix-org/synapse/issues/5431

`jinja2` was being imported even when it wasn't strictly necessary. This made it required to run Synapse, even if the functionality that required it wasn't enabled. This was causing new Synapse installations to crash on startup.

Email modules are now required.
 2019-06-21 16:52:09 +01:00
Richard van der Hoff
0e8b35f7b0
Fix "Unexpected entry in 'full_schemas'" log warning (#5509) 
There is a README.txt which always sets off this warning, which is a bit
alarming when you first start synapse. I don't think we need to warn about
this.
 2019-06-21 15:11:57 +01:00
Richard van der Hoff
2f8491daef
Fix logging error when a tampered event is detected. (#5500) 2019-06-21 15:11:42 +01:00
Erik Johnston
2b0bde935a Newsfile 2019-06-21 14:16:03 +01:00
Erik Johnston
8d452e0ca5 Newsfile 2019-06-21 11:13:13 +01:00
Erik Johnston
626a26dbb8 Newsfile 2019-06-21 10:46:11 +01:00
Erik Johnston
7456698241
Merge pull request #5476 from matrix-org/erikj/histogram_extremities 
Add metrics for length of new extremities persisted.
 2019-06-21 09:58:54 +01:00
Neil Johnson
f47969f42a
Improve email notification logging (#5502) 2019-06-20 15:14:26 +01:00
Erik Johnston
45f28a9d2f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/histogram_extremities 2019-06-20 11:59:14 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Richard van der Hoff
ae4d97bf05 changelog 2019-06-19 21:18:38 +01:00
Richard van der Hoff
15bf32e062 Use monotonic clock where possible for metrics 
Fixes intermittent errors observed on Apple hardware which were caused by
time.clock() appearing to go backwards when called from different threads.

Also fixes a bug where database activity times were logged as 1/1000 of their
correct ratio due to confusion between milliseconds and seconds.
 2019-06-19 21:09:43 +01:00
Erik Johnston
7dcf984075
Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_error 
Handle the case of `get_missing_events` failing
 2019-06-19 13:20:09 +01:00
Erik Johnston
e0be8d7016
Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events 
Add experimental option to reduce extremities.
 2019-06-19 13:19:18 +01:00
Richard van der Hoff
ceb2fa60a5
Merge pull request #5490 from matrix-org/rav/xmlsec_in_docker 
Include xmlsec in the docker image
 2019-06-19 11:33:06 +01:00
Brendan Abolivier
092e36457a
Fix typo in deactivation background job 2019-06-19 11:09:41 +01:00
Richard van der Hoff
8fcd2ca907
Merge pull request #4276 from Ralith/performance-advice 
Improve advice regarding poor performance
 2019-06-18 23:29:58 +01:00
David Baker
f2d2ae03da Add some logging to 3pid invite sig verification (#5015) 
I had to add quite a lot of logging to diagnose a problem with 3pid
invites - we only logged the one failure which isn't all that
informative.

NB. I'm not convinced the logic of this loop is right: I think it
should just accept a single valid signature from a trusted source
rather than fail if *any* signature is invalid. Also it should
probably not skip the rest of middle loop if a check fails? However,
I'm deliberately not changing the logic here.
 2019-06-18 22:51:24 +01:00
Richard van der Hoff
b093cfb02c changelog 2019-06-18 22:36:40 +01:00
Erik Johnston
19b80fe68a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_get_missing_events_error 2019-06-18 16:11:43 +01:00
Erik Johnston
fc51e21326 Newsfile 2019-06-18 15:03:12 +01:00
cclauss
82d9d524bd Fix seven contrib files with Python syntax errors (#5446) 
* Fix seven contrib files with Python syntax errors

Signed-off-by: cclauss <cclauss@me.com>
 2019-06-18 03:21:30 +10:00
Brendan Abolivier
d6328e03fd
Merge pull request #5477 from matrix-org/babolivier/third_party_rules_3pid 
Add third party rules hook for 3PID invites
 2019-06-17 18:08:31 +01:00
Brendan Abolivier
9ce4220d6c Changelog 2019-06-17 17:39:09 +01:00
Erik Johnston
8353ddd951
Merge pull request #5479 from matrix-org/erikj/add_create_room_hook_develop 
Add third party rules hook into create room
 2019-06-17 17:30:05 +01:00
Jorik Schellekens
160c52d0d4
Merge pull request #5478 from matrix-org/joriks/demo_python3 
Joriks/demo python3
 2019-06-17 17:13:01 +01:00
Erik Johnston
4c4877bbc1 Newsfile 2019-06-17 17:02:53 +01:00
Jorik Schellekens
25d16fea78 Changelog 2019-06-17 16:50:31 +01:00
Erik Johnston
2d6308a043 Newsfile 2019-06-17 16:41:19 +01:00
Amber Brown
eba7caf09f
Remove Postgres 9.4 support (#5448) 2019-06-18 00:59:00 +10:00
Erik Johnston
6840ebeef8
Merge pull request #5385 from matrix-org/erikj/reduce_http_exceptions 
Handle HttpResponseException when using federation client.
 2019-06-17 13:54:47 +01:00
Erik Johnston
dd927b29e1
Merge pull request #5388 from matrix-org/erikj/fix_email_push 
Fix email notifications for unnamed rooms with multiple people
 2019-06-17 13:54:35 +01:00
Erik Johnston
414d2ca3a6
Merge pull request #5389 from matrix-org/erikj/renew_attestations_on_master 
Only start background group attestation renewals on master
 2019-06-17 13:54:29 +01:00
Amber Brown
97d7e4c7b7
Move SyTest to Buildkite (#5459) 
Including workers!
 2019-06-17 21:08:15 +10:00
Erik Johnston
a9dab970b8
Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_state 
Fix 3PID invite room state over federation.
 2019-06-17 10:18:28 +01:00
Brendan Abolivier
f12e1f029c
Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules 
Allow server admins to define implementations of extra rules for allowing or denying incoming events
 2019-06-14 19:37:59 +01:00
Erik Johnston
9ca4ae7131
Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive 
Prometheus histograms are cumalative
 2019-06-14 18:21:42 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Brendan Abolivier
14db086428
Merge pull request #5465 from matrix-org/babolivier/fix_deactivation_bg_job 
Fix background job for deactivated flag
 2019-06-14 18:12:56 +01:00
Brendan Abolivier
5cec6d1845
Fix changelog 2019-06-14 17:18:21 +01:00
Brendan Abolivier
4024520ff8
Changelog 2019-06-14 16:38:44 +01:00
Erik Johnston
3c9bb86fde Newsfile 2019-06-14 16:19:11 +01:00
Brendan Abolivier
9b14a810d2
Merge pull request #5462 from matrix-org/babolivier/account_validity_deactivated_accounts_2 
Don't send renewal emails to deactivated users (second attempt)
 2019-06-14 15:35:31 +01:00
Jorik Schellekens
1e7864c929
Merge pull request #5460 from matrix-org/joriks/demo_python3 
Use python3 in the demo
 2019-06-14 15:23:57 +01:00
Brendan Abolivier
6d56a694f4 Don't send renewal emails to deactivated users 2019-06-14 15:05:56 +01:00
Erik Johnston
e9344e0dee
Merge pull request #5390 from matrix-org/erikj/dont_log_on_fail_to_get_file 
Don't log exception when failing to fetch remote content.
 2019-06-14 14:25:14 +01:00
Erik Johnston
9fd4f83f1a Newsfile 2019-06-14 14:19:37 +01:00
Jorik Schellekens
cc7cc853b1 Changelog 2019-06-14 14:07:47 +01:00
Brendan Abolivier
d0530382ee
Track deactivated accounts in the database (#5378) 2019-06-14 13:18:24 +01:00
Amber H. Brown
b2a6f90a67 changelog 2019-06-14 21:10:21 +10:00
Richard van der Hoff
b59a4eba64
Updates to the federation_client script (#5447) 
* py3 fixes for federation_client
* .well-known support for federation_client
 2019-06-13 14:49:25 +01:00
Richard van der Hoff
5c15039e06
Clean up code for sending federation EDUs. (#5381) 
This code confused the hell out of me today. Split _get_new_device_messages
into its two (unrelated) parts.
 2019-06-13 13:52:08 +01:00
Amber Brown
6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Amber Brown
09e9a26b71
Remove Python 2.7 support. (#5425) 
* remove 2.7 from CI and publishing

* fill out classifiers and also make it not be installed on 3.5

* some minor bumps so that the old deps work on python 3.5
 2019-06-12 21:31:59 +10:00
Erik Johnston
7e68691ce9 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-06-11 17:25:16 +01:00
Erik Johnston
97174780ce 1.0.0 2019-06-11 17:10:01 +01:00
Erik Johnston
9532eb55ec
Merge pull request #5424 from matrix-org/erikj/change_password_reset_links 
Change password reset links to /_matrix.
 2019-06-11 13:29:42 +01:00
Neil Johnson
426218323b
Neilj/improve federation docs (#5419) 
Add FAQ questions to federate.md. Add a health warning making it clear that the 1711 upgrade FAQ is now out of date.
 2019-06-11 12:17:43 +01:00
Erik Johnston
453aaaadc0 Newsfile 2019-06-11 11:34:38 +01:00
Erik Johnston
5bc9484537 Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop 2019-06-11 10:37:43 +01:00
Richard van der Hoff
880005c3b7 changelog 2019-06-11 00:40:31 +01:00
Richard van der Hoff
b55e8840e4 Merge tag 'v1.0.0rc3' into rav/saml2_client 
Synapse 1.0.0rc3 (2019-06-10)
=============================

Security: Fix authentication bug introduced in 1.0.0rc1. Please upgrade to rc3 immediately
 2019-06-11 00:28:57 +01:00
Andrew Morgan
2ddc13577c
Don't warn user about password reset disabling through config code (#5387) 
Moves the warning about password resets being disabled to the point where a user actually tries to reset their password. Is this an appropriate place for it to happen?

Also removed the disabling of msisdn password resets when you don't have an email config, as that just doesn't make sense.

Also change the error a user receives upon disabled passwords to specify that only email-based password reset is disabled.
 2019-06-11 00:25:07 +01:00
Neil Johnson
94dac0f3e5
add monthly active users to phonehome stats (#5252) 
* add monthly active users to phonehome stats
 2019-06-10 23:33:59 +01:00
Erik Johnston
78e74ab8a1
Merge pull request #5418 from matrix-org/erikj/fix_send_fed_with_limit_zero 
Fix bug sending federation transactions with lots of EDUs
 2019-06-10 20:19:49 +01:00
Erik Johnston
abce00fc6a Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop 2019-06-10 18:33:55 +01:00
Erik Johnston
0167447965 1.0.0rc2 2019-06-10 18:17:43 +01:00
Erik Johnston
a6b1817940
Merge pull request #5417 from matrix-org/rav/shared_ssl_context 
Share an SSL context object between SSL connections
 2019-06-10 18:16:12 +01:00
Erik Johnston
48748c00c4
Update changelog.d/5418.bugfix 
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2019-06-10 16:28:45 +01:00
Erik Johnston
1fb6f68616 Newsfile 2019-06-10 16:26:36 +01:00
Richard van der Hoff
8d0bd9bb60 fix build fails 2019-06-10 16:23:39 +01:00
Erik Johnston
0382b0ffee
Merge pull request #5415 from matrix-org/erikj/fix_null_valid_until_ms 
Fix key verification when key stored with null valid_until_ms
 2019-06-10 15:56:36 +01:00
Erik Johnston
8da0d83a54 Merge branch 'erikj/fix_null_valid_until_ms' of github.com:matrix-org/synapse into release-v1.0.0 2019-06-10 15:56:28 +01:00
Erik Johnston
9bc7768ad3 Newsfile 2019-06-10 15:37:10 +01:00
Erik Johnston
6d1e699b5c
Merge pull request #5412 from SohamG/fix-4130 
Add --no-daemonize option to synctl
 2019-06-10 15:09:49 +01:00
Brendan Abolivier
843dd714cb
Merge pull request #5325 from matrix-org/babolivier/port_db_account_validity 
Add account_validity's email_sent column to the list of boolean columns in synapse_port_db
 2019-06-10 13:51:46 +01:00
sohamg
b56a224e22
Added changelog file. 2019-06-10 17:54:29 +05:30
Brendan Abolivier
26b62796c2
Merge pull request #5363 from matrix-org/babolivier/account_validity_send_mail_auth 
Don't check whether the user's account is expired on /send_mail requests
 2019-06-10 11:57:02 +01:00
Richard van der Hoff
88d7182ada
Improve startup checks for insecure notary configs (#5392) 
It's not really a problem to trust notary responses signed by the old key so
long as we are also doing TLS validation.

This commit adds a check to the config parsing code at startup to check that
we do not have the insecure matrix.org key without tls validation, and refuses
to start without it.

This allows us to remove the rather alarming-looking warning which happens at
runtime.
 2019-06-10 10:33:00 +01:00
Benjamin Saunders
047486a384 Improve advice regarding poor performance 
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
 2019-06-09 15:20:28 -07:00
Andrew Morgan
2decc92e2f
Liberapay is now officially recognised, update FUNDING.yml (#5386) 2019-06-09 02:20:23 +01:00
Erik Johnston
5009d988da Newsfile 2019-06-07 12:39:12 +01:00
Erik Johnston
2cca90dd40 Newsfile 2019-06-07 12:26:59 +01:00
Erik Johnston
a099926fcc Newsfile 2019-06-07 12:15:33 +01:00
Neil Johnson
8dc1eb6697 Synapse 1.0.0rc1 (2019-06-07) 
=============================
 
 Features
 --------
 
 - Synapse now more efficiently collates room statistics. ([\#4338](https://github.com/matrix-org/synapse/issues/4338), [\#5260](https://github.com/matrix-org/synapse/issues/5260), [\#5324](https://github.com/matrix-org/synapse/issues/5324))
 - Add experimental support for relations (aka reactions and edits). ([\#5220](https://github.com/matrix-org/synapse/issues/5220))
 - Ability to configure default room version. ([\#5223](https://github.com/matrix-org/synapse/issues/5223), [\#5249](https://github.com/matrix-org/synapse/issues/5249))
 - Allow configuring a range for the account validity startup job. ([\#5276](https://github.com/matrix-org/synapse/issues/5276))
 - CAS login will now hit the r0 API, not the deprecated v1 one. ([\#5286](https://github.com/matrix-org/synapse/issues/5286))
 - Validate federation server TLS certificates by default (implements [MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md)). ([\#5359](https://github.com/matrix-org/synapse/issues/5359))
 - Update /_matrix/client/versions to reference support for r0.5.0. ([\#5360](https://github.com/matrix-org/synapse/issues/5360))
 - Add a script to generate new signing-key files. ([\#5361](https://github.com/matrix-org/synapse/issues/5361))
 - Update upgrade and installation guides ahead of 1.0. ([\#5371](https://github.com/matrix-org/synapse/issues/5371))
 - Replace the `perspectives` configuration section with `trusted_key_servers`, and make validating the signatures on responses optional (since TLS will do this job for us). ([\#5374](https://github.com/matrix-org/synapse/issues/5374))
 - Add ability to perform password reset via email without trusting the identity server. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
 - Set default room version to v4. ([\#5379](https://github.com/matrix-org/synapse/issues/5379))
 
 Bugfixes
 --------
 
 - Fixes client-server API not sending "m.heroes" to lazy-load /sync requests when a rooms name or its canonical alias are empty. Thanks to @dnaf for this work! ([\#5089](https://github.com/matrix-org/synapse/issues/5089))
 - Prevent federation device list updates breaking when processing multiple updates at once. ([\#5156](https://github.com/matrix-org/synapse/issues/5156))
 - Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo. ([\#5200](https://github.com/matrix-org/synapse/issues/5200))
 - Fix race when backfilling in rooms with worker mode. ([\#5221](https://github.com/matrix-org/synapse/issues/5221))
 - Fix appservice timestamp massaging. ([\#5233](https://github.com/matrix-org/synapse/issues/5233))
 - Ensure that server_keys fetched via a notary server are correctly signed. ([\#5251](https://github.com/matrix-org/synapse/issues/5251))
 - Show the correct error when logging out and access token is missing. ([\#5256](https://github.com/matrix-org/synapse/issues/5256))
 - Fix error code when there is an invalid parameter on /_matrix/client/r0/publicRooms ([\#5257](https://github.com/matrix-org/synapse/issues/5257))
 - Fix error when downloading thumbnail with missing width/height parameter. ([\#5258](https://github.com/matrix-org/synapse/issues/5258))
 - Fix schema update for account validity. ([\#5268](https://github.com/matrix-org/synapse/issues/5268))
 - Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. ([\#5274](https://github.com/matrix-org/synapse/issues/5274), [\#5278](https://github.com/matrix-org/synapse/issues/5278), [\#5291](https://github.com/matrix-org/synapse/issues/5291))
 - Fix "db txn 'update_presence' from sentinel context" log messages. ([\#5275](https://github.com/matrix-org/synapse/issues/5275))
 - Fix dropped logcontexts during high outbound traffic. ([\#5277](https://github.com/matrix-org/synapse/issues/5277))
 - Fix a bug where it is not possible to get events in the federation format with the request `GET /_matrix/client/r0/rooms/{roomId}/messages`. ([\#5293](https://github.com/matrix-org/synapse/issues/5293))
 - Fix performance problems with the rooms stats background update. ([\#5294](https://github.com/matrix-org/synapse/issues/5294))
 - Fix noisy 'no key for server' logs. ([\#5300](https://github.com/matrix-org/synapse/issues/5300))
 - Fix bug where a notary server would sometimes forget old keys. ([\#5307](https://github.com/matrix-org/synapse/issues/5307))
 - Prevent users from setting huge displaynames and avatar URLs. ([\#5309](https://github.com/matrix-org/synapse/issues/5309))
 - Fix handling of failures when processing incoming events where calling `/event_auth` on remote server fails. ([\#5317](https://github.com/matrix-org/synapse/issues/5317))
 - Ensure that we have an up-to-date copy of the signing key when validating incoming federation requests. ([\#5321](https://github.com/matrix-org/synapse/issues/5321))
 - Fix various problems which made the signing-key notary server time out for some requests. ([\#5333](https://github.com/matrix-org/synapse/issues/5333))
 - Fix bug which would make certain operations (such as room joins) block for 20 minutes while attemoting to fetch verification keys. ([\#5334](https://github.com/matrix-org/synapse/issues/5334))
 - Fix a bug where we could rapidly mark a server as unreachable even though it was only down for a few minutes. ([\#5335](https://github.com/matrix-org/synapse/issues/5335), [\#5340](https://github.com/matrix-org/synapse/issues/5340))
 - Fix a bug where account validity renewal emails could only be sent when email notifs were enabled. ([\#5341](https://github.com/matrix-org/synapse/issues/5341))
 - Fix failure when fetching batches of events during backfill, etc. ([\#5342](https://github.com/matrix-org/synapse/issues/5342))
 - Add a new room version where the timestamps on events are checked against the validity periods on signing keys. ([\#5348](https://github.com/matrix-org/synapse/issues/5348), [\#5354](https://github.com/matrix-org/synapse/issues/5354))
 - Fix room stats and presence background updates to correctly handle missing events. ([\#5352](https://github.com/matrix-org/synapse/issues/5352))
 - Include left members in room summaries' heroes. ([\#5355](https://github.com/matrix-org/synapse/issues/5355))
 - Fix `federation_custom_ca_list` configuration option. ([\#5362](https://github.com/matrix-org/synapse/issues/5362))
 - Fix missing logcontext warnings on shutdown. ([\#5369](https://github.com/matrix-org/synapse/issues/5369))
 
 Improved Documentation
 ----------------------
 
 - Fix docs on resetting the user directory. ([\#5282](https://github.com/matrix-org/synapse/issues/5282))
 - Fix notes about ACME in the MSC1711 faq. ([\#5357](https://github.com/matrix-org/synapse/issues/5357))
 
 Internal Changes
 ----------------
 
 - Synapse will now serve the experimental "room complexity" API endpoint. ([\#5216](https://github.com/matrix-org/synapse/issues/5216))
 - The base classes for the v1 and v2_alpha REST APIs have been unified. ([\#5226](https://github.com/matrix-org/synapse/issues/5226), [\#5328](https://github.com/matrix-org/synapse/issues/5328))
 - Simplifications and comments in do_auth. ([\#5227](https://github.com/matrix-org/synapse/issues/5227))
 - Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2. ([\#5230](https://github.com/matrix-org/synapse/issues/5230))
 - Preparatory work for key-validity features. ([\#5232](https://github.com/matrix-org/synapse/issues/5232), [\#5234](https://github.com/matrix-org/synapse/issues/5234), [\#5235](https://github.com/matrix-org/synapse/issues/5235), [\#5236](https://github.com/matrix-org/synapse/issues/5236), [\#5237](https://github.com/matrix-org/synapse/issues/5237), [\#5244](https://github.com/matrix-org/synapse/issues/5244), [\#5250](https://github.com/matrix-org/synapse/issues/5250), [\#5296](https://github.com/matrix-org/synapse/issues/5296), [\#5299](https://github.com/matrix-org/synapse/issues/5299), [\#5343](https://github.com/matrix-org/synapse/issues/5343), [\#5347](https://github.com/matrix-org/synapse/issues/5347), [\#5356](https://github.com/matrix-org/synapse/issues/5356))
 - Specify the type of reCAPTCHA key to use. ([\#5283](https://github.com/matrix-org/synapse/issues/5283))
 - Improve sample config for monthly active user blocking. ([\#5284](https://github.com/matrix-org/synapse/issues/5284))
 - Remove spurious debug from MatrixFederationHttpClient.get_json. ([\#5287](https://github.com/matrix-org/synapse/issues/5287))
 - Improve logging for logcontext leaks. ([\#5288](https://github.com/matrix-org/synapse/issues/5288))
 - Clarify that the admin change password API logs the user out. ([\#5303](https://github.com/matrix-org/synapse/issues/5303))
 - New installs will now use the v54 full schema, rather than the full schema v14 and applying incremental updates to v54. ([\#5320](https://github.com/matrix-org/synapse/issues/5320))
 - Improve docstrings on MatrixFederationClient. ([\#5332](https://github.com/matrix-org/synapse/issues/5332))
 - Clean up FederationClient.get_events for clarity. ([\#5344](https://github.com/matrix-org/synapse/issues/5344))
 - Various improvements to debug logging. ([\#5353](https://github.com/matrix-org/synapse/issues/5353))
 - Don't run CI build checks until sample config check has passed. ([\#5370](https://github.com/matrix-org/synapse/issues/5370))
 - Automatically retry buildkite builds (max twice) when an agent is lost. ([\#5380](https://github.com/matrix-org/synapse/issues/5380))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlz6Lx0ACgkQCTxDAAxP
 S/QQ7Q//bhS5t9Sxm1dKBPPPZwaOr6kG5tmIiq+W51ReHaL9aLQHKSi2ZY3wL4dR
 4bHdd1Lc7Xk5b2s0JH//Z4+iCx8+5jbBSkHOJZ90rMPAe14ajfIie6tUv1GQTBNG
 na8zA+0gZ+BGIRe3du3+bpaSsPygpUJ11z/AtZ02d4PK+UxmObWHM9/qxU4OcFdw
 H0/z1uej4DXc1PrwJUrIye+8kil/JuSKb6T2LvOxKON3xz7aImBlWt9MKcFKAo45
 EEXZMLR8JJoS8NrUVKYaR0oHooyUQSkJJyAFpy5yekeuxh5Q7OUufPKdk6z9vhuX
 CqxN2geMTnozhsBsiFRSS/nhC/c0pSXaKiSJua6t22tP/ZDrdZMqQ0QXY2yLV3+6
 JEbzPE96XxdrN2cVi0nowIMwUf48IQCko+ofU2KVco5kSCb4UHqq55KVB0+J4iaP
 zEL37p1cyU/pWT7pYISpYTJig7RZZtU8X2qtwfTo0y6MCFtNmIXZWCEHbos0XTk9
 UGSJWQE+aizupLJfR8R6AKtmQStCJijpahr81FdR2p1l1dfZcyYue+hxnegYHmmx
 HwlrZolTHx0bYztlGqmvy1NLK93e+IfnlCRbkF6yBR/fRilT8IDuE1Hp+6amVPNi
 j0fIs2eIGnV91hHv6WLnyqT5vWJvD0z4a3jz7cI1QWVPcJH5+sM=
 =bwjf
 -----END PGP SIGNATURE-----

Merge tag 'v1.0.0rc1' into develop

Synapse 1.0.0rc1 (2019-06-07)
=============================

Features
--------

- Synapse now more efficiently collates room statistics. ([\#4338](https://github.com/matrix-org/synapse/issues/4338), [\#5260](https://github.com/matrix-org/synapse/issues/5260), [\#5324](https://github.com/matrix-org/synapse/issues/5324))
- Add experimental support for relations (aka reactions and edits). ([\#5220](https://github.com/matrix-org/synapse/issues/5220))
- Ability to configure default room version. ([\#5223](https://github.com/matrix-org/synapse/issues/5223), [\#5249](https://github.com/matrix-org/synapse/issues/5249))
- Allow configuring a range for the account validity startup job. ([\#5276](https://github.com/matrix-org/synapse/issues/5276))
- CAS login will now hit the r0 API, not the deprecated v1 one. ([\#5286](https://github.com/matrix-org/synapse/issues/5286))
- Validate federation server TLS certificates by default (implements [MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md)). ([\#5359](https://github.com/matrix-org/synapse/issues/5359))
- Update /_matrix/client/versions to reference support for r0.5.0. ([\#5360](https://github.com/matrix-org/synapse/issues/5360))
- Add a script to generate new signing-key files. ([\#5361](https://github.com/matrix-org/synapse/issues/5361))
- Update upgrade and installation guides ahead of 1.0. ([\#5371](https://github.com/matrix-org/synapse/issues/5371))
- Replace the `perspectives` configuration section with `trusted_key_servers`, and make validating the signatures on responses optional (since TLS will do this job for us). ([\#5374](https://github.com/matrix-org/synapse/issues/5374))
- Add ability to perform password reset via email without trusting the identity server. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
- Set default room version to v4. ([\#5379](https://github.com/matrix-org/synapse/issues/5379))

Bugfixes
--------

- Fixes client-server API not sending "m.heroes" to lazy-load /sync requests when a rooms name or its canonical alias are empty. Thanks to @dnaf for this work! ([\#5089](https://github.com/matrix-org/synapse/issues/5089))
- Prevent federation device list updates breaking when processing multiple updates at once. ([\#5156](https://github.com/matrix-org/synapse/issues/5156))
- Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo. ([\#5200](https://github.com/matrix-org/synapse/issues/5200))
- Fix race when backfilling in rooms with worker mode. ([\#5221](https://github.com/matrix-org/synapse/issues/5221))
- Fix appservice timestamp massaging. ([\#5233](https://github.com/matrix-org/synapse/issues/5233))
- Ensure that server_keys fetched via a notary server are correctly signed. ([\#5251](https://github.com/matrix-org/synapse/issues/5251))
- Show the correct error when logging out and access token is missing. ([\#5256](https://github.com/matrix-org/synapse/issues/5256))
- Fix error code when there is an invalid parameter on /_matrix/client/r0/publicRooms ([\#5257](https://github.com/matrix-org/synapse/issues/5257))
- Fix error when downloading thumbnail with missing width/height parameter. ([\#5258](https://github.com/matrix-org/synapse/issues/5258))
- Fix schema update for account validity. ([\#5268](https://github.com/matrix-org/synapse/issues/5268))
- Fix bug where we leaked extremities when we soft failed events, leading to performance degradation. ([\#5274](https://github.com/matrix-org/synapse/issues/5274), [\#5278](https://github.com/matrix-org/synapse/issues/5278), [\#5291](https://github.com/matrix-org/synapse/issues/5291))
- Fix "db txn 'update_presence' from sentinel context" log messages. ([\#5275](https://github.com/matrix-org/synapse/issues/5275))
- Fix dropped logcontexts during high outbound traffic. ([\#5277](https://github.com/matrix-org/synapse/issues/5277))
- Fix a bug where it is not possible to get events in the federation format with the request `GET /_matrix/client/r0/rooms/{roomId}/messages`. ([\#5293](https://github.com/matrix-org/synapse/issues/5293))
- Fix performance problems with the rooms stats background update. ([\#5294](https://github.com/matrix-org/synapse/issues/5294))
- Fix noisy 'no key for server' logs. ([\#5300](https://github.com/matrix-org/synapse/issues/5300))
- Fix bug where a notary server would sometimes forget old keys. ([\#5307](https://github.com/matrix-org/synapse/issues/5307))
- Prevent users from setting huge displaynames and avatar URLs. ([\#5309](https://github.com/matrix-org/synapse/issues/5309))
- Fix handling of failures when processing incoming events where calling `/event_auth` on remote server fails. ([\#5317](https://github.com/matrix-org/synapse/issues/5317))
- Ensure that we have an up-to-date copy of the signing key when validating incoming federation requests. ([\#5321](https://github.com/matrix-org/synapse/issues/5321))
- Fix various problems which made the signing-key notary server time out for some requests. ([\#5333](https://github.com/matrix-org/synapse/issues/5333))
- Fix bug which would make certain operations (such as room joins) block for 20 minutes while attemoting to fetch verification keys. ([\#5334](https://github.com/matrix-org/synapse/issues/5334))
- Fix a bug where we could rapidly mark a server as unreachable even though it was only down for a few minutes. ([\#5335](https://github.com/matrix-org/synapse/issues/5335), [\#5340](https://github.com/matrix-org/synapse/issues/5340))
- Fix a bug where account validity renewal emails could only be sent when email notifs were enabled. ([\#5341](https://github.com/matrix-org/synapse/issues/5341))
- Fix failure when fetching batches of events during backfill, etc. ([\#5342](https://github.com/matrix-org/synapse/issues/5342))
- Add a new room version where the timestamps on events are checked against the validity periods on signing keys. ([\#5348](https://github.com/matrix-org/synapse/issues/5348), [\#5354](https://github.com/matrix-org/synapse/issues/5354))
- Fix room stats and presence background updates to correctly handle missing events. ([\#5352](https://github.com/matrix-org/synapse/issues/5352))
- Include left members in room summaries' heroes. ([\#5355](https://github.com/matrix-org/synapse/issues/5355))
- Fix `federation_custom_ca_list` configuration option. ([\#5362](https://github.com/matrix-org/synapse/issues/5362))
- Fix missing logcontext warnings on shutdown. ([\#5369](https://github.com/matrix-org/synapse/issues/5369))

Improved Documentation
----------------------

- Fix docs on resetting the user directory. ([\#5282](https://github.com/matrix-org/synapse/issues/5282))
- Fix notes about ACME in the MSC1711 faq. ([\#5357](https://github.com/matrix-org/synapse/issues/5357))

Internal Changes
----------------

- Synapse will now serve the experimental "room complexity" API endpoint. ([\#5216](https://github.com/matrix-org/synapse/issues/5216))
- The base classes for the v1 and v2_alpha REST APIs have been unified. ([\#5226](https://github.com/matrix-org/synapse/issues/5226), [\#5328](https://github.com/matrix-org/synapse/issues/5328))
- Simplifications and comments in do_auth. ([\#5227](https://github.com/matrix-org/synapse/issues/5227))
- Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2. ([\#5230](https://github.com/matrix-org/synapse/issues/5230))
- Preparatory work for key-validity features. ([\#5232](https://github.com/matrix-org/synapse/issues/5232), [\#5234](https://github.com/matrix-org/synapse/issues/5234), [\#5235](https://github.com/matrix-org/synapse/issues/5235), [\#5236](https://github.com/matrix-org/synapse/issues/5236), [\#5237](https://github.com/matrix-org/synapse/issues/5237), [\#5244](https://github.com/matrix-org/synapse/issues/5244), [\#5250](https://github.com/matrix-org/synapse/issues/5250), [\#5296](https://github.com/matrix-org/synapse/issues/5296), [\#5299](https://github.com/matrix-org/synapse/issues/5299), [\#5343](https://github.com/matrix-org/synapse/issues/5343), [\#5347](https://github.com/matrix-org/synapse/issues/5347), [\#5356](https://github.com/matrix-org/synapse/issues/5356))
- Specify the type of reCAPTCHA key to use. ([\#5283](https://github.com/matrix-org/synapse/issues/5283))
- Improve sample config for monthly active user blocking. ([\#5284](https://github.com/matrix-org/synapse/issues/5284))
- Remove spurious debug from MatrixFederationHttpClient.get_json. ([\#5287](https://github.com/matrix-org/synapse/issues/5287))
- Improve logging for logcontext leaks. ([\#5288](https://github.com/matrix-org/synapse/issues/5288))
- Clarify that the admin change password API logs the user out. ([\#5303](https://github.com/matrix-org/synapse/issues/5303))
- New installs will now use the v54 full schema, rather than the full schema v14 and applying incremental updates to v54. ([\#5320](https://github.com/matrix-org/synapse/issues/5320))
- Improve docstrings on MatrixFederationClient. ([\#5332](https://github.com/matrix-org/synapse/issues/5332))
- Clean up FederationClient.get_events for clarity. ([\#5344](https://github.com/matrix-org/synapse/issues/5344))
- Various improvements to debug logging. ([\#5353](https://github.com/matrix-org/synapse/issues/5353))
- Don't run CI build checks until sample config check has passed. ([\#5370](https://github.com/matrix-org/synapse/issues/5370))
- Automatically retry buildkite builds (max twice) when an agent is lost. ([\#5380](https://github.com/matrix-org/synapse/issues/5380))
 2019-06-07 10:37:34 +01:00
Erik Johnston
a2419b27fe Newsfile 2019-06-07 10:31:53 +01:00
Andrew Morgan
8e0cee90d2
Add a sponsor button (#5382) 
Add a sponsor button with links to matrixdotorg's patreon and liberapay accounts.
 2019-06-07 10:31:48 +01:00
Neil Johnson
7c455a86bc 1.0.0rc1 2019-06-07 10:29:32 +01:00
Andrew Morgan
4f581faa98
Automatically retry builds when a buildkite agent is lost (#5380) 
Sometimes the build agents get lost or die (error codes -1 and 2). Retry automatically a maximum of 2 times if this happens.

Error code reference:

* -1: Agent was lost
* 0: Build successful
* 1: There was an error in your code
* 2: The build stopped abruptly
* 255: The build was cancelled
 2019-06-07 00:20:17 +01:00
Andrew Morgan
2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156) 
fixes #5153
 2019-06-06 23:54:00 +01:00
Neil Johnson
a11865016e
Set default room version to v4. (#5379) 
Set default room version to v4.
 2019-06-06 20:13:47 +01:00
Erik Johnston
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1 
Make a full SQL schema
 2019-06-06 18:35:38 +01:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377) 
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
 2019-06-06 17:34:07 +01:00
Richard van der Hoff
9fbb20a531
Stop hardcoding trust of old matrix.org key (#5374) 
There are a few changes going on here:

* We make checking the signature on a key server response optional: if no
  verify_keys are specified, we trust to TLS to validate the connection.

* We change the default config so that it does not require responses to be
  signed by the old key.

* We replace the old 'perspectives' config with 'trusted_key_servers', which
  is also formatted slightly differently.

* We emit a warning to the logs every time we trust a key server response
  signed by the old key.
 2019-06-06 17:33:11 +01:00
Neil Johnson
833c406b9b
Neilj/1.0 upgrade notes (#5371) 
1.0 upgrade/install notes
 2019-06-06 17:23:02 +01:00
Andrew Morgan
f868c8df03
Regen sample config before kicking off agents (#5370) 
* Regen sample config before kicking off agents

* Add changelog
 2019-06-06 16:36:28 +01:00
Richard van der Hoff
71063a69b8
Fix missing logcontext for PresenceHandler.on_shutdown. (#5369) 
Fixes some warnings, and a scary-looking stacktrace when sytest kills the
process.
 2019-06-06 14:45:17 +01:00
Erik Johnston
89d3d7b2c0
Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest 
Fix get_max_topological_token to never return None
 2019-06-06 13:54:46 +01:00
Brendan Abolivier
8f06344e11
Merge pull request #5089 from dnaf/m-heroes-empty-room-name 
Make /sync return heroes if room name or canonical alias are empty
 2019-06-06 11:18:13 +01:00
Richard van der Hoff
cb3b381fcb
Merge pull request #5359 from matrix-org/rav/enable_tls_verification 
Validate federation server TLS certificates by default.
 2019-06-06 10:50:42 +01:00
Richard van der Hoff
42555bc18b
Merge pull request #5361 from matrix-org/rav/generate_signing_key 
add a script to generate new signing_key files
 2019-06-06 10:50:22 +01:00
Brendan Abolivier
7898a1a48d
Add credit in the changelog 2019-06-06 10:34:33 +01:00
Brendan Abolivier
b9c43c8463
Merge pull request #5355 from matrix-org/babolivier/heroes_left_members 
Include left members in room summaries' heroes
 2019-06-06 10:05:27 +01:00
Richard van der Hoff
99d3497949
Merge pull request #5354 from matrix-org/rav/server_keys/99-room-v5 
Implement room v5 which enforces signing key validity
 2019-06-06 09:42:13 +01:00
Richard van der Hoff
2eb47e5ee7
Merge pull request #5353 from matrix-org/rav/verify_key_logging 
Associate a request_name with each verify request, for logging
 2019-06-06 09:33:09 +01:00
Brendan Abolivier
ccbc9e5e17
Gah towncrier 2019-06-05 16:41:26 +01:00
Brendan Abolivier
d51ca9d9b3
Changelog 2019-06-05 16:38:51 +01:00
Richard van der Hoff
7603a706eb Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification 2019-06-05 16:32:35 +01:00
Richard van der Hoff
f8a45302c9 Fix federation_custom_ca_list configuration option. 
Previously, setting this option would cause an exception at startup.
 2019-06-05 16:19:07 +01:00
Neil Johnson
94f6c674df
Neilj/add r0.5 to versions (#5360) 
* Update _matrix/client/versions to reference support for r0.5.0
 2019-06-05 16:11:31 +01:00
Erik Johnston
75538813fc Fix background updates to handle redactions/rejections (#5352) 
* Fix background updates to handle redactions/rejections

In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
 2019-06-06 00:45:46 +10:00
Richard van der Hoff
fb98c05e03 add a script to generate new signing_key files 2019-06-05 15:33:02 +01:00
Richard van der Hoff
95ab2eb4a1
Fix notes about well-known and acme (#5357) 
fixes #4951
 2019-06-05 15:12:33 +01:00
Richard van der Hoff
e2dfb922e1 Validate federation server TLS certificates by default. 2019-06-05 14:17:50 +01:00
Erik Johnston
d53faa40e9
Merge pull request #5317 from matrix-org/erikj/make_do_auth_non_essential 
Fix handling of failures when calling /event_auth.
 2019-06-05 14:01:32 +01:00
Brendan Abolivier
a4cf2c1184
Rewrite changelog 2019-06-05 14:00:18 +01:00
Neil Johnson
4650526b5e
Neilj/changelog clean up (#5356) 
* group together key validity refactors
 2019-06-05 13:47:03 +01:00
Brendan Abolivier
40596aec0e Merge branch 'develop' into m-heroes-empty-room-name 2019-06-05 13:38:01 +01:00
Neil Johnson
26713515de
Neilj/mau tracking config explainer (#5284) 
Improve documentation of monthly active user blocking and mau_trial_days
 2019-06-05 13:16:23 +01:00
Brendan Abolivier
804f26a9ff
Properly format the changelog 2019-06-05 12:03:01 +01:00
Brendan Abolivier
a412be2bc7
Changelog 2019-06-05 11:53:50 +01:00
Richard van der Hoff
d18e4ea0d4 Implement room v5 which enforces signing key validity 
Implements [MSC2077](https://github.com/matrix-org/matrix-doc/pull/2077) and
fixes #5247 and #4364.
 2019-06-05 11:00:47 +01:00
Richard van der Hoff
cea9750d11 Associate a request_name with each verify request, for logging 
Also:
* rename VerifyKeyRequest->VerifyJsonRequest
* calculate key_ids on VerifyJsonRequest construction
* refactor things to pass around VerifyJsonRequests instead of 4-tuples
 2019-06-05 10:46:26 +01:00
Richard van der Hoff
14f13babb0
Add a test room version where we enforce key validity (#5348) 2019-06-05 10:38:25 +01:00
Richard van der Hoff
2615c6bd9e
Clean up debug logging (#5347) 
Remove some spurious stuff, clarify some other stuff
 2019-06-05 10:35:40 +01:00
Richard van der Hoff
016af01598
Rename VerifyKeyRequest.deferred field (#5343) 
it's a bit confusing
 2019-06-05 10:35:13 +01:00
Richard van der Hoff
aa530e6800
Call RetryLimiter correctly (#5340) 
Fixes a regression introduced in #5335.
 2019-06-04 22:02:53 +01:00
Richard van der Hoff
dae224a73f
Fix failure to fetch batches of PDUs (#5342) 
FederationClient.get_pdu is called in a loop to fetch a batch of PDUs. A
failure to fetch one should not result in a failure of the whole batch. Add the
missing `continue`.
 2019-06-04 18:05:06 +01:00
Richard van der Hoff
b4189b112f
Rename get_events->get_events_from_store_or_dest (#5344) 
We have too many things called get_event, and it's hard to figure out what we
mean. Also remove some unused params from the signature, and add some logging.
 2019-06-04 18:01:09 +01:00
Brendan Abolivier
f6dd12d1e2
Merge pull request #5341 from matrix-org/babolivier/email_config 
Make account validity renewal emails work when email notifs are disabled
 2019-06-04 14:49:06 +01:00
Erik Johnston
d1d38081a7
Merge pull request #5324 from matrix-org/erikj/ignore_null 
Ignore room state with null bytes in for room stats
 2019-06-04 14:20:08 +01:00
Brendan Abolivier
ac3cc32367
Make account validity renewal emails work when email notifs are disabled 2019-06-04 13:47:44 +01:00
Richard van der Hoff
df9c100542
Avoid rapidly backing-off a server if we ignore the retry interval (#5335) 2019-06-04 11:53:29 +01:00
Richard van der Hoff
4d08b8f30c
Don't do long retries when calling the key notary server. (#5334) 
It takes at least 20 minutes to work through the long_retries schedule (11
attempts, each with a 60 second timeout, and 60 seconds between each request),
so if the notary server isn't returning within the timeout, we'll just end up
blocking whatever request is happening for 20 minutes.

Ain't nobody got time for that.
 2019-06-04 11:53:07 +01:00
Richard van der Hoff
cb683d3e3c
Merge pull request #5333 from matrix-org/rav/server_keys/09_improve_notary_server 
Fixes for the key-notary server
 2019-06-04 11:48:18 +01:00
Richard van der Hoff
5bdb189f86
Improve docstrings on MatrixFederationClient. (#5332) 2019-06-04 11:14:16 +01:00
Brendan Abolivier
aeb2263320 Merge branch 'develop' into babolivier/port_db_account_validity 2019-06-04 09:13:42 +01:00
Amber Brown
b2b90b7d34
Hawkowl/fix missing auth (#5328) 2019-06-04 15:54:27 +10:00
Richard van der Hoff
a3f2d000e0 changelog 2019-06-04 00:16:56 +01:00
Richard van der Hoff
dce6e9e0c1 Avoid rapidly backing-off a server if we ignore the retry interval 2019-06-03 23:58:42 +01:00
Richard van der Hoff
fec2dcb1a5
Enforce validity period on server_keys for fed requests. (#5321) 
When handling incoming federation requests, make sure that we have an
up-to-date copy of the signing key.

We do not yet enforce the validity period for event signatures.
 2019-06-03 22:59:51 +01:00
Erik Johnston
0d67a8cd9d Newsfile 2019-06-03 17:17:57 +01:00
Erik Johnston
fe2294ec8d Revert "Newsfile" 
This reverts commit 4bd67db100.
 2019-06-03 17:17:35 +01:00
Brendan Abolivier
deca87ddf2
Changelog 2019-06-03 17:11:28 +01:00
Erik Johnston
4bd67db100 Newsfile 2019-06-03 17:08:33 +01:00
Amber Brown
6f9f08005c Merge remote-tracking branch 'origin/develop' into hawkowl/full-schema-v1 2019-06-04 01:16:01 +10:00
Amber Brown
ed6138461b more fix 2019-06-03 22:29:19 +10:00
Richard van der Hoff
862b2f9ad5
Merge pull request #5307 from matrix-org/rav/server_keys/07-fix-notary-cache-poison 
Stop overwriting server keys with other keys
 2019-06-03 13:19:20 +01:00
Amber Brown
2889b05554
Unify v1 and v2 REST client APIs (#5226) 2019-06-03 21:28:59 +10:00
Erik Johnston
fde37e4e98 Newsfile 2019-06-03 10:22:03 +01:00
Ike Johnson
0df5b41759
Create 5313.misc 2019-06-02 23:23:58 +08:00
Richard van der Hoff
d828d1dc57
Merge pull request #5309 from matrix-org/rav/limit_displayname_length 
Limit displaynames and avatar URLs
 2019-06-01 11:34:50 +01:00
Richard van der Hoff
d16c6375fe Limit displaynames and avatar URLs 
These end up in join events everywhere, so let's limit them.

Fixes #5079
 2019-06-01 10:44:36 +01:00
Richard van der Hoff
37b165620d
Merge pull request #5299 from matrix-org/rav/server_keys/05-rewrite-gsvk-again 
Rewrite get_server_verify_keys, again.
 2019-05-31 17:07:31 +01:00
Richard van der Hoff
3600f5568b Stop overwriting server keys with other keys 
Fix a bug where we would discard a key result which the origin server is no
longer returning. Fixes #5305.
 2019-05-31 15:58:35 +01:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta 
Allow configuring a range for the account validity startup job
 2019-05-31 12:11:56 +01:00
Richard van der Hoff
c605da97bf Merge remote-tracking branch 'origin/develop' into rav/server_keys/05-rewrite-gsvk-again 2019-05-31 11:38:13 +01:00
Richard van der Hoff
fe79b5e521
Merge pull request #5300 from matrix-org/rav/server_keys/06-fix-serverkeys-handling 
Remove some pointless exception handling
 2019-05-31 11:35:29 +01:00
Richard van der Hoff
2ae3cc287e
Merge pull request #5296 from matrix-org/rav/server_keys/04-use-attrs-for_verify-request 
use attr.s for VerifyKeyRequest
 2019-05-31 11:34:09 +01:00
Erik Johnston
e9981d58ca
Merge pull request #5293 from Kagamihime/messages-federation-format 
Fix ignored filter field in `/messages` endpoint
 2019-05-31 10:52:59 +01:00
Erik Johnston
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats 
Speed up room stats background update
 2019-05-31 10:48:51 +01:00
Erik Johnston
39bbf6a4a5 Newsfile 2019-05-31 10:26:59 +01:00
Travis Ralston
3e1af5109c Clarify that the admin change password endpoint logs them out (#5303) 2019-05-31 09:45:46 +01:00
Richard van der Hoff
8ea2f756a9 Remove some pointless exception handling 
The verify_request deferred already returns a suitable SynapseError, so I don't
really know what we expect to achieve by doing more wrapping, other than log
spam.

Fixes #4278.
 2019-05-30 18:29:56 +01:00
Richard van der Hoff
a82c96b87f Rewrite get_server_verify_keys, again. 
Attempt to simplify the logic in get_server_verify_keys by splitting it into
two methods.
 2019-05-30 18:20:40 +01:00
Richard van der Hoff
099829d5a9 use attr.s for VerifyKeyRequest 
because namedtuple is awful
 2019-05-30 17:39:28 +01:00
Erik Johnston
c831748f4d 0.99.5.2 2019-05-30 16:29:47 +01:00
Erik Johnston
f5c7f90d72 Newsfile 2019-05-30 16:18:40 +01:00
Erik Johnston
9f5268388a Newsfile 2019-05-30 16:15:37 +01:00
Erik Johnston
df9d900544 Correctly filter out extremities with soft failed prevs (#5274) 
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
 2019-05-30 16:12:50 +01:00
Eisha Chen-yen-su
0b6bc36402 Add changelog 2019-05-30 17:07:21 +02:00
Erik Johnston
06675db684 Newsfile 2019-05-30 15:05:26 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities 
Add DB bg update to cleanup extremities.
 2019-05-30 14:28:26 +01:00
Erik Johnston
45f5d8f3fd
Merge pull request #5256 from aaronraimist/logout-correct-error 
Show correct error when logging out and access token is missing
 2019-05-30 13:33:44 +01:00
Richard van der Hoff
8d92329214
Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) 
This is just unhelpful spam
 2019-05-29 19:31:52 +01:00
Richard van der Hoff
3dcf2feba8
Improve logging for logcontext leaks. (#5288) 2019-05-29 19:27:50 +01:00
Erik Johnston
8541db741a
Merge pull request #5283 from aaronraimist/captcha-docs 
Specify the type of reCAPTCHA key to use (#5013)
 2019-05-29 19:02:27 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Erik Johnston
67e0631f8f Newsfile 2019-05-29 15:59:10 +01:00
Amber Brown
532b825ed9
Serve CAS login over r0 (#5286) 2019-05-30 00:55:18 +10:00
Aaron Raimist
30858ff461 Fix error when downloading thumbnail with width/height param missing (#5258) 
Fix error when downloading thumbnail with width/height param missing

Fixes #2748

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-29 14:27:41 +01:00
Erik Johnston
58c8ed5b0d
Correctly filter out extremities with soft failed prevs (#5274) 
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
 2019-05-29 11:56:24 +01:00
Richard van der Hoff
f76d407ef3
Fix dropped logcontexts during high outbound traffic. (#5277) 
Fixes #5271.
 2019-05-29 09:17:33 +01:00
Amber Brown
ecaa299cab
Rename 5282.misc to 5282.doc 2019-05-29 16:32:30 +10:00
Aaron Raimist
2ec2809460
Add changelog 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-28 22:05:53 -05:00
Aaron Raimist
878b00c395
Add changelog 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-28 20:58:22 -05:00
Richard van der Hoff
540f40f0cd
Merge pull request #5251 from matrix-org/rav/server_keys/01-check_sig 
Ensure that server_keys fetched via a notary server are correctly signed.
 2019-05-28 21:32:17 +01:00
Richard van der Hoff
5726378ece
Fix "db txn 'update_presence' from sentinel context" log messages (#5275) 
Fixes #4414.
 2019-05-28 21:20:11 +01:00
Brendan Abolivier
4aba561c65
Config and changelog 2019-05-28 16:55:10 +01:00
Brendan Abolivier
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema 
Fix schema update for account validity
 2019-05-28 10:30:07 +01:00
Brendan Abolivier
ddd30f44a0
Changelog 2019-05-28 10:14:21 +01:00
Erik Johnston
d0bba35197
Merge pull request #5260 from matrix-org/travis/fix-room-bg-task 
Fix logging for room stats background update
 2019-05-25 19:59:42 +01:00
Travis Ralston
4ccdbfcdb1 Changelog 2019-05-25 12:21:21 -06:00
Aaron Raimist
0b4f4cb0b4
Add changelog 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-25 11:39:18 -05:00
Erik Johnston
338dca58c0
Merge pull request #5257 from aaronraimist/fix-error-code-publicrooms 
Fix error code for invalid parameter
 2019-05-25 14:09:27 +01:00
Aaron Raimist
6dac0e738c
Add changelog 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-05-24 17:15:30 -05:00
Richard van der Hoff
fa1b293da2
Simplification to Keyring.wait_for_previous_lookups. (#5250) 
The list of server names was redundant, since it was equivalent to the keys on
the server_to_deferred map. This reduces the number of large lists being passed
around, and has the benefit of deduplicating the entries in `wait_on`.
 2019-05-24 22:17:18 +01:00
Richard van der Hoff
cbcfd642a0 changelog 2019-05-24 15:47:30 +01:00
Tulir Asokan
dd64b9dbdd Fix appservice timestamp massaging (#5233) 
Signed-off-by: Tulir Asokan <tulir@maunium.net>
 2019-05-24 14:44:04 +01:00
Richard van der Hoff
dba9152d15
Add missing blank line in config (#5249) 2019-05-24 14:12:38 +01:00
Erik Johnston
d16f5574b6
Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events 
Don't bundle aggregations with events in /sync or /events or state queries
 2019-05-24 10:36:31 +01:00
Erik Johnston
8c41c04ee4
Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers 
Factor out KeyFetchers from KeyRing
 2019-05-23 16:28:51 +01:00
Andrew Morgan
6368150a74
Add config option for setting homeserver's default room version (#5223) 
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
 2019-05-23 15:00:20 +01:00
Richard van der Hoff
895b79ac2e Factor out KeyFetchers from KeyRing 
Rather than have three methods which have to have the same interface,
factor out a separate interface which is provided by three implementations.

I find it easier to grok the code this way.
 2019-05-23 13:46:47 +01:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer 
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
 2019-05-23 11:52:22 +01:00
Richard van der Hoff
84660d91b2
Simplify process_v2_response (#5236) 
* Pass time_added_ms into process_v2_response

* Simplify process_v2_response

We can merge old_verify_keys into verify_keys, and reduce the number of dicts
flying around.
 2019-05-23 11:51:39 +01:00
Richard van der Hoff
cc187f9337
Remove unused VerifyKey.expired and .time_added fields (#5235) 
These were never used, and poking arbitary data into objects from other
packages seems confusing at best.
 2019-05-23 11:46:05 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234) 
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
 2019-05-23 11:45:39 +01:00
Richard van der Hoff
85d1e03b9d
Simplifications and comments in do_auth (#5227) 
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
 2019-05-23 11:17:42 +01:00
Richard van der Hoff
1a94de60e8
Run black on synapse.crypto.keyring (#5232) 2019-05-22 18:39:33 +01:00
Marcus Hoffmann
62388a1e44 remove urllib3 pin (#5230) 
requests 2.22.0 as been released supporting urllib3 1.25.2

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
 2019-05-22 16:48:12 +01:00
Neil Johnson
66b75e2d81
Neilj/ensure get profileinfo available in client reader slaved store (#5213) 
* expose SlavedProfileStore to ClientReaderSlavedStore
 2019-05-22 13:55:32 +01:00
Richard van der Hoff
b898a5600a Merge branch 'master' into develop 2019-05-22 11:38:27 +01:00
Richard van der Hoff
e26e6b3230 update changelog 2019-05-21 17:37:19 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Richard van der Hoff
f3ff64e000 Merge commit 'f4c80d70f' into release-v0.99.5 2019-05-21 17:35:31 +01:00
Erik Johnston
f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender 
Only count aggregations from distinct senders
 2019-05-21 17:10:48 +01:00
Erik Johnston
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions 
Block attempts to annotate the same event twice
 2019-05-21 17:08:14 +01:00
Erik Johnston
9259cd4bee Newsfile 2019-05-21 17:06:21 +01:00
Richard van der Hoff
959550b645 0.99.5rc1 2019-05-21 16:51:49 +01:00
Richard van der Hoff
17f6804837
Introduce room v4 which updates event ID format. (#5217) 
Implements https://github.com/matrix-org/matrix-doc/pull/2002.
 2019-05-21 16:22:54 +01:00
Richard van der Hoff
c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146) 
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
 2019-05-21 16:10:54 +01:00
Richard van der Hoff
bab3eddac4
Pin eliot to <1.8 on python 3.5.2 (#5218) 
* Pin eliot to <1.8 on python 3.5.2

Fixes https://github.com/matrix-org/synapse/issues/5199

* Add support for 'markers' to python_dependencies

* tell xargs not to strip quotes
 2019-05-21 15:58:01 +01:00
Erik Johnston
c448f35de2 Newsfile 2019-05-21 15:35:13 +01:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date 
Add startup background job for account validity
 2019-05-21 14:55:15 +01:00
Erik Johnston
ef13dc4846 Newsfile 2019-05-21 13:59:09 +01:00
Richard van der Hoff
04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219) 
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
 2019-05-21 13:47:25 +01:00
Erik Johnston
0620dd49db Newsfile 2019-05-20 17:40:24 +01:00
Richard van der Hoff
24b93b9c76 Revert "expose SlavedProfileStore to ClientReaderSlavedStore (#5200)" 
This reverts commit ce5bcefc60.

This caused:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module>
    from synapse.replication.slave.storage import SlavedProfileStore
ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py)
error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs
```
 2019-05-20 16:21:34 +01:00
Richard van der Hoff
5206648a4a
Add a test room version which updates event ID format (#5210) 
Implements MSC1884
 2019-05-20 15:54:42 +01:00
Erik Johnston
d642178654 Newsfile 2019-05-20 14:32:16 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base 
Land basic reaction and edit support.
 2019-05-20 14:06:40 +01:00
Erik Johnston
06671057b6 Newsfile 2019-05-20 12:39:07 +01:00
ReidAnderson
3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
Brendan Abolivier
99c4ec1eef
Changelog 2019-05-17 19:38:41 +01:00
bytepoets-blo
291e1eea5e fix mapping of return values for get_or_register_3pid_guest (#5177) 
* fix mapping of return values for get_or_register_3pid_guest
 2019-05-17 17:27:14 +01:00
Erik Johnston
85ece3df46
Merge pull request #5191 from matrix-org/erikj/refactor_pagination_bounds 
Make generating SQL bounds for pagination generic
 2019-05-17 17:24:36 +01:00
Neil Johnson
ce5bcefc60
expose SlavedProfileStore to ClientReaderSlavedStore (#5200) 
* expose SlavedProfileStore to ClientReaderSlavedStore
 2019-05-17 13:27:19 +01:00
Richard van der Hoff
da5ef0bb42 Merge remote-tracking branch 'origin/master' into develop 2019-05-17 12:39:48 +01:00
Richard van der Hoff
7ce1f97a13
Stop telling people to install the optional dependencies. (#5197) 
* Stop telling people to install the optional dependencies.

They're optional.

Also update the postgres docs a bit for clarity(?)
 2019-05-17 12:38:03 +01:00
Brendan Abolivier
fdeac1e984
Merge pull request #5196 from matrix-org/babolivier/per_room_profiles 
Add an option to disable per-room profiles
 2019-05-17 12:10:49 +01:00
PauRE
f89f688a55 Fix image orientation when generating thumbnail (#5039) 2019-05-16 19:04:26 +01:00
David Baker
07cff7b121
Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only 
Re-order registration stages to do msisdn & email auth last
 2019-05-16 17:27:39 +01:00
Brendan Abolivier
a5fe16c5a7
Changelog + sample config 2019-05-16 15:11:37 +01:00
Matthew Hodgson
4a6d5de98c Make /sync attempt to return device updates for both joined and invited users (#3484) 2019-05-16 13:23:43 +01:00
David Baker
fafb936de5
Merge pull request #5187 from matrix-org/dbkr/only_check_threepid_not_in_use_if_actually_registering 
Only check 3pids not in use when registering
 2019-05-16 10:58:09 +01:00
David Baker
cd0faba7cd
Make newsfile clearer 2019-05-15 20:53:48 +01:00
Amber Brown
f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Richard van der Hoff
5f027a315f
Drop support for v2_alpha API prefix (#5190) 2019-05-15 17:37:46 +01:00
Erik Johnston
5fb72e6888 Newsfile 2019-05-15 13:36:51 +01:00
Erik Johnston
7155162844 Newsfile 2019-05-15 11:33:22 +01:00
Erik Johnston
0aba6c8251
Merge pull request #5183 from matrix-org/erikj/async_serialize_event 
Allow client event serialization to be async
 2019-05-15 10:36:30 +01:00
Erik Johnston
d94544051b
Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array 
Expose DataStore._get_events as get_events_as_list
 2019-05-15 10:17:38 +01:00
David Baker
efefb5bda2 Have I got newsfile for you 2019-05-14 19:18:42 +01:00
Erik Johnston
db3046f565 Newsfile 2019-05-14 14:39:27 +01:00
Erik Johnston
53788a447f Newsfile 2019-05-14 13:41:36 +01:00
Erik Johnston
a80e6b53f9 Newsfile 2019-05-14 13:12:23 +01:00
Amber Brown
df2ebd75d3
Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171) 2019-05-13 15:01:14 -05:00
Andrew Morgan
5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Brendan Abolivier
2e1129b5f7 0.99.4rc1 2019-05-13 16:11:21 +01:00
David Baker
516a5fb64b Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:54:25 +01:00
David Baker
9e99143c47 Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:37:03 +01:00
Brendan Abolivier
2725cd2290 Fix changelog 2019-05-13 15:32:07 +01:00
Brendan Abolivier
1a536699fd Changelog 2019-05-13 15:21:23 +01:00
Andrew Morgan
2f48c4e1ae
URL preview blacklisting fixes (#5155) 
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
 2019-05-10 10:32:44 -07:00
David Baker
c9f811c5d4 Update changelog 2019-05-10 14:01:19 +01:00
David Baker
7a3eb8657d Thanks, automated grammar pedantry. 2019-05-10 11:18:35 +01:00
David Baker
a18f93279e Add changelog entry 2019-05-10 11:11:59 +01:00
Christoph Müller
ee90c06e38 Set syslog identifiers in systemd units (#5023) 2019-05-10 09:09:25 +01:00
Amber Brown
b36c82576e
Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00
Quentin Dufour
11ea16777f Limit the number of EDUs in transactions to 100 as expected by receiver (#5138) 
Fixes #3951.
 2019-05-09 11:01:41 +01:00
Brendan Abolivier
d216a36b37
Fix bogus imports in tests (#5154) 2019-05-08 21:57:03 +01:00
Matthew Hodgson
c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083) 
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
 2019-05-08 18:26:56 +01:00
Erik Johnston
c8c069db92
Merge pull request #5037 from matrix-org/erikj/limit_inflight_dns 
Limit in flight DNS requests
 2019-05-08 17:11:03 +01:00
Brendan Abolivier
1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128) 
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
 2019-05-08 17:01:30 +01:00
Erik Johnston
de655e669a
Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite 
Ratelimit 3pid invites
 2019-05-07 10:12:49 +01:00
Richard van der Hoff
59e2d2694d
Remove the requirement to authenticate for /admin/server_version. (#5122) 
This endpoint isn't much use for its intended purpose if you first need to get
yourself an admin's auth token.

I've restricted it to the `/_synapse/admin` path to make it a bit easier to
lock down for those concerned about exposing this information. I don't imagine
anyone is using it in anger currently.
 2019-05-07 09:29:30 +01:00
Travis Ralston
3fdff14207 Fix spelling in server notices admin API docs (#5142) 2019-05-06 22:15:02 +01:00
Richard van der Hoff
12f9d51e82
Add admin api for sending server_notices (#5121) 2019-05-02 11:59:16 +01:00
Brendan Abolivier
c193b39134
Merge pull request #5124 from matrix-org/babolivier/aliases 
Add some limitations to alias creation
 2019-05-02 11:22:40 +01:00
Brendan Abolivier
84196cb231 Add some limitations to alias creation 2019-05-02 11:05:11 +01:00
Richard van der Hoff
0836cbb9f5
Factor out an "assert_requester_is_admin" function (#5120) 
Rather than copying-and-pasting the same four lines hundreds of times
 2019-05-02 10:45:52 +01:00
Richard van der Hoff
03ad6bd483 changelog 2019-05-01 15:44:30 +01:00
Brendan Abolivier
c1799b0f85
Merge pull request #5116 from matrix-org/babolivier/account_expiration 
Fix path in account validity admin route's doc
 2019-05-01 11:59:56 +01:00
Brendan Abolivier
6aad81ec0c Rename changelog file 2019-05-01 11:50:15 +01:00
Brendan Abolivier
803a28fd1d Add changelog 2019-05-01 11:43:31 +01:00
Travis Ralston
8c5b1e30d4
Add a default .m.rule.tombstone push rule (#4867) 
* Add a default .m.rule.tombstone push rule

In support of MSC1930: https://github.com/matrix-org/matrix-doc/pull/1930

* changelog

* Appease the changelog linter
 2019-04-29 15:40:31 -06:00
Richard van der Hoff
b31cc1c613
Merge pull request #5100 from matrix-org/rav/verification_hackery 
Improve logging when event-signature checking fails
 2019-04-29 13:19:32 +01:00
Erik Johnston
d6118c5be6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ratelimit_3pid_invite 2019-04-26 18:14:23 +01:00
Erik Johnston
19f0722b2c Newsfile 2019-04-26 18:08:33 +01:00
Richard van der Hoff
bd0d45ca69 Fix infinite loop in presence handler 
Fixes #5102
 2019-04-26 11:14:49 +01:00
Richard van der Hoff
0962d3cdff changelog 2019-04-25 23:05:06 +01:00
Richard van der Hoff
2ebf7d56fa
Merge pull request #5098 from matrix-org/rav/fix_pep_517 
Workarounds for pep-517 errors
 2019-04-25 15:11:27 +01:00
Richard van der Hoff
e86d74d748 Changelog 2019-04-25 14:56:06 +01:00
*=0=1=4=*
4a9a118a94 Fix handling of SYNAPSE_NO_TLS in docker image (#5005) 2019-04-25 14:47:22 +01:00
Andrew Morgan
6824ddd93d Config option for verifying federation certificates (MSC 1711) (#4967) 2019-04-25 14:22:49 +01:00
Michael Kaye
788163e204 Remove log error for .well-known/matrix/client (#4972) 2019-04-24 17:44:06 +01:00
Amber Brown
6b2b9a58c4 Prevent "producer not unregistered" message (#5009) 2019-04-24 17:37:32 +01:00
Katie Wolfe
60041eac4b
Add full stop to 5084.bugfix 
Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:26:51 -04:00
Katie Wolfe
0a4c135f68
Add changelog.d/5084.bugfix 
Signed-off-by: Katie Wolfe <katie@dnaf.moe>
 2019-04-24 12:26:47 -04:00
Brendan Abolivier
f8826d31cd Don't crash on lack of expiry templates 2019-04-18 14:50:05 +01:00
Erik Johnston
ca90336a69 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration 2019-04-17 19:44:40 +01:00
Brendan Abolivier
eaf41a943b Add management endpoints for account validity 2019-04-17 19:34:45 +01:00
Brendan Abolivier
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration 
Send out emails with links to extend an account's validity period
 2019-04-17 14:57:39 +01:00
Brendan Abolivier
20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Brendan Abolivier
49ff74da9b
Merge pull request #5071 from matrix-org/babolivier/3pid-check 
Make sure we're not registering the same 3pid twice
 2019-04-17 14:37:42 +01:00
Brendan Abolivier
600ec04739 Make sure we're not registering the same 3pid twice 2019-04-17 14:23:01 +01:00
Erik Johnston
fd2fcb817c
Merge pull request #5070 from matrix-org/erikj/postpath 
Remove usage of request.postpath
 2019-04-17 09:03:26 +01:00
Erik Johnston
a1eb4c6d2f
Merge pull request #5065 from matrix-org/erikj/fix_versions 
VersionRestServlet doesn't take a param
 2019-04-16 17:52:36 +01:00
Erik Johnston
14d5ad7d2b Newsfile 2019-04-16 17:52:00 +01:00
Travis Ralston
3f22e993f0
Use packages.matrix.org for packages (#5067) 
* Use packages.matrix.org for packages

See https://github.com/vector-im/riot-web/issues/9497 (applies to more than just Olm)

* changelog
 2019-04-16 08:31:59 -06:00
Silke Hofstra
a137f4eac0 Add systemd-python to optional dependencies (#4339) 
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
 2019-04-16 20:41:17 +10:00
Erik Johnston
468b2bcb2e Newsfile 2019-04-15 19:41:25 +01:00
Erik Johnston
6e27a8620f
Merge pull request #5063 from matrix-org/erikj/move_endpoints 
Move some rest endpoints to client reader
 2019-04-15 18:55:01 +01:00
Erik Johnston
208251956d Newsfile 2019-04-15 17:21:08 +01:00
Andrew Morgan
caa76e6021
Remove periods from copyright headers (#5046) 2019-04-11 17:08:13 +01:00
Erik Johnston
42e1aa5b2c Newsfile 2019-04-10 10:43:47 +01:00
Erik Johnston
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta 
Fix schema upgrade when dropping tables
 2019-04-10 10:22:35 +01:00
Richard van der Hoff
329688c161
Fix disappearing exceptions in manhole. (#5035) 
Avoid sending syntax errors from the manhole to sentry.
 2019-04-10 07:23:48 +01:00
Erik Johnston
02491e009d Newsfile 2019-04-09 17:23:46 +01:00
Brendan Abolivier
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration 
Add time-based account expiration
 2019-04-09 17:02:41 +01:00
Brendan Abolivier
747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Erik Johnston
5e45b558b0 Newsfile 2019-04-09 14:39:36 +01:00
Richard van der Hoff
644b86677f
Merge pull request #5030 from matrix-org/rav/rewrite_g_s_v_k 
Rewrite Datastore.get_server_verify_keys
 2019-04-09 11:30:23 +01:00
Richard van der Hoff
4abf5aa81a
Bump psycopg requirement (#5032) 2019-04-09 11:29:50 +01:00
Richard van der Hoff
1f1e8dd8ec changelog 2019-04-09 00:00:10 +01:00
Richard van der Hoff
3352baac4b
Remove unused server_tls_certificates functions (#5028) 
These have been unused since #4120, and with the demise of perspectives, it is
unlikely that they will ever be used again.
 2019-04-08 21:50:18 +01:00
Neil Johnson
b25e387c0d
add context to phonehome stats (#5020) 
add context to phonehome stats
 2019-04-08 15:47:39 +01:00
Richard van der Hoff
67d7b44784
Merge pull request #5024 from matrix-org/rav/record_correct_server_in_serverkeys 
Fix from_server buglet in get_keys_from_perspectives
 2019-04-08 15:40:37 +01:00
Neil Johnson
2d951686a7
drop tables listed in #1830 (#4992) 
Tables dropped: 
 * application_services, 
 * application_services_regex, 
 * transaction_id_to_pdu, 
 * stats_reporting
 * current_state_resets
 * event_content_hashes
 * event_destinations
 * event_edge_hashes
 * event_signatures
 * feedback
 * room_hosts
 * state_forward_extremities
 2019-04-08 15:37:26 +01:00
Richard van der Hoff
7d2a0c848e Fix from_server buglet in get_keys_from_perspectives 
make sure we store the name of the server the keys came from, rather than the
origin server, after doing a fetch-from-perspectives.
 2019-04-08 12:51:16 +01:00
Richard van der Hoff
7fc1e17f4c
Merge pull request #5001 from matrix-org/rav/keyring_cleanups 
Cleanups in the Keyring
 2019-04-08 12:47:09 +01:00
Richard van der Hoff
b78aac5582 changelog 2019-04-04 19:12:54 +01:00
Brendan Abolivier
8e85493b0c
Add config option to block users from looking up 3PIDs (#5010) 2019-04-04 17:25:47 +01:00
Amber Brown
a33a5abc4c
Clean up the database pagination code (#5007) 
* rewrite & simplify

* changelog

* cleanup potential sql injection
 2019-04-05 00:21:16 +11:00
Erik Johnston
616e6a10bd
Merge pull request #5002 from matrix-org/erikj/delete_group 
Add delete group admin API
 2019-04-04 14:15:41 +01:00
Andrew Morgan
db265f0642
Prevent kicking users who aren't in the room (#4999) 
Prevent kick events from succeeding if the user is not currently in the room.
 2019-04-04 13:05:51 +01:00
Marcel Krüger
9f5d206c4a Avoid redundant URL encoding (#4555) 
* Do not double encode fallback redirect URL

Signed-off-by: Marcel Fabian Krüger <zauguin@gmail.com>
 2019-04-04 12:05:56 +01:00
Erik Johnston
43c707a010
Merge pull request #5004 from ajensenwaud/develop 
Fix issue #4596
 2019-04-04 10:44:46 +01:00
*=0=1=4=*
40810b81d2 Correct default POSTGRES_USER in Docker README (#4987) 
Correct default POSTGRES_USER in Docker README
 2019-04-04 10:38:16 +01:00
Anders
2a59e8e429 Fix issue #4596 
Make synapse_port_db --curses work with Python 3.
Signed-off-by: Anders Jensen-Waud <anders@jensenwaud.com>
 2019-04-03 21:59:48 +00:00
Erik Johnston
bd3435e982 Newsfile 2019-04-03 16:35:33 +01:00
Andrew Morgan
4a4d5c4fd6
Fix grammar and document get_current_users_in_room (#4998) 2019-04-03 14:32:20 +01:00
Richard van der Hoff
e4d473d855
Rewrite KeyringTestCase as a HomeServerTestCase (#4986) 
This is a bit fiddly due to the keyring doing weird things with logcontexts.
 2019-04-03 14:11:27 +01:00
Neil Johnson
e8419554ff
Remove presence lists (#4989) 
Remove presence list support as per MSC 1819
 2019-04-03 11:11:15 +01:00
Erik Johnston
8f549c1177
Merge pull request #4982 from matrix-org/erikj/msc1915 
Implement MSC1915 - 3PID unbind APIs
 2019-04-03 11:07:09 +01:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Erik Johnston
3039d61baf
Merge pull request #4991 from matrix-org/erikj/stagger_push_startup 
Make starting pushers faster during start up
 2019-04-02 18:23:32 +01:00
Erik Johnston
6f226eed42 s/misc/feature/ 2019-04-02 18:22:28 +01:00
Andrew Morgan
66e78700a2
Transfer related groups on room upgrade (#4990) 
Transfers the m.room.related_groups state event on room upgrade.
 2019-04-02 17:15:24 +01:00
Erik Johnston
ac45b0df0b Newsfile 2019-04-02 17:00:18 +01:00
Richard van der Hoff
8530090b16
Add config.signing_key_path. (#4974) 
As requested by @andrewshadura
 2019-04-02 16:59:27 +01:00
Richard van der Hoff
297bf2547e
Fix sync bug when accepting invites (#4956) 
Hopefully this time we really will fix #4422.

We need to make sure that the cache on
`get_rooms_for_user_with_stream_ordering` is invalidated *before* the
SyncHandler is notified for the new events, and we can now do so reliably via
the `events` stream.
 2019-04-02 12:42:39 +01:00
Neil Johnson
4c552ed78a
Neilj/fix threepid auth check (with tests) (#4474) 
test threepid checking
 2019-04-01 17:42:18 +01:00
Erik Johnston
39fb971e85 Newsfile 2019-04-01 15:31:47 +01:00
manuroe
d461c65465
Merge pull request #4981 from matrix-org/manuroe/demo_bypass_account_rate_limiting 
start.sh: Fix the --no-rate-limit option for messages
 2019-04-01 16:00:08 +02:00
Neil Johnson
62988f73fd Merge branch 'master' into develop 2019-04-01 14:08:53 +01:00
manuroe
bb925b1bd7 start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too. 2019-04-01 14:27:28 +02:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969) 
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
 2019-04-01 10:24:38 +01:00
Jurrie Overgoor
50b5f08740 Add changelog.d entry 2019-03-29 10:30:24 +01:00
Richard van der Hoff
902cdc63b6
Merge pull request #4955 from matrix-org/rav/merge_state_into_events 
Combine the CurrentStateDeltaStream into the EventStream
 2019-03-28 18:32:13 +00:00
Richard van der Hoff
d688a51736
Merge pull request #4954 from matrix-org/rav/refactor_parse_row 
Refactors to replication stream row update/parsing
 2019-03-28 18:31:17 +00:00
Neil Johnson
c7296bcb98
remove log line for password (#4965) 
Remove log line for password.
 2019-03-28 17:38:01 +00:00
Andrew Morgan
7a91b9d81c
Allow password providers to bind emails (#4947) 
This PR allows password provider modules to bind email addresses when a user is registering and is motivated by matrix-org/matrix-synapse-ldap3#58
 2019-03-28 15:48:07 +00:00
Erik Johnston
248014379e
Merge pull request #4942 from matrix-org/erikj/fix_presence 
Use event streams to calculate presence
 2019-03-28 14:38:31 +00:00
Richard van der Hoff
d035d62f6b Merge remote-tracking branch 'origin/develop' into rav/refactor_parse_row 2019-03-28 13:45:14 +00:00
Richard van der Hoff
4eeb2c2f07
Merge pull request #4953 from matrix-org/rav/refactor_replication_streams 
Split up replication.tcp.streams into smaller files
 2019-03-28 13:43:25 +00:00
Amber Brown
2e060774ad
Run black on some storage modules that the stats branch touches (#4959) 2019-03-29 00:37:16 +11:00
Richard van der Hoff
17d7bacbcf changelog 2019-03-27 22:08:39 +00:00
Richard van der Hoff
a65763a5d6 changelog 2019-03-27 22:04:01 +00:00
Richard van der Hoff
91c3513668 changelog 2019-03-27 21:30:01 +00:00
Richard van der Hoff
acaa18f7dd
Fix/improve some docstrings in the replication code. (#4949) 2019-03-27 21:12:36 +00:00
Erik Johnston
d5a5d1c632 Newsfile 2019-03-27 13:41:36 +00:00
Neil Johnson
6a69bf67db 0.99.3rc1 2019-03-27 10:24:24 +00:00
Andrew Morgan
bbd244c7b2
Support 3PID login in password providers (#4931) 
Adds a new method, check_3pid_auth, which gives password providers
the chance to allow authentication with third-party identifiers such
as email or msisdn.
 2019-03-26 17:48:30 +00:00
Amber Brown
903f04c21f
Use the state event amount for userdir import batching, not room count (#4944) 2019-03-27 02:49:28 +11:00
Andrew Morgan
4a125be138
Make federation endpoints more tolerant of trailing slashes v2 (#4935) 
Redo of https://github.com/matrix-org/synapse/pull/4840
 2019-03-26 11:35:29 +00:00
Richard van der Hoff
8cbbedaa2b
Fix ClientReplicationStreamProtocol.__str__ (#4929) 
`__str__` depended on `self.addr`, which was absent from
ClientReplicationStreamProtocol, so attempting to call str on such an object
would raise an exception.

We can calculate the peer addr from the transport, so there is no need for addr
anyway.
 2019-03-25 16:41:51 +00:00
Richard van der Hoff
9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927) 
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
 2019-03-25 16:38:05 +00:00
Richard van der Hoff
a54a44734f
Use an explicit dbname for postgres connections in the tests. (#4928) 
I don't have a database with the same name as my user, so leaving the database
name unset fails.

While we're at it, clear out some unused stuff in the test setup.
 2019-03-25 16:36:56 +00:00
Richard van der Hoff
7105057cf2
Fix nginx example in ACME doc. (#4923) 2019-03-25 09:59:36 +00:00
Amber Brown
ac396a0d32
Refactor out state delta handling into its own class (#4917) 2019-03-25 20:37:08 +11:00
Erik Johnston
5fee9d8067
Merge pull request #4869 from matrix-org/erikj/yaml_load 
Fix yaml warnings by using safe_load
 2019-03-22 11:58:13 +00:00
Erik Johnston
d21a4d6be6 Newsfile 2019-03-22 10:24:58 +00:00
Richard van der Hoff
224783a73f
Allow newsfragments to end with exclamation marks! (#4912) 2019-03-21 15:28:19 +00:00
Amber Brown
a68e00fca8
Some more porting to HomeserverTestCase and remove old RESTHelper (#4913) 2019-03-22 02:10:21 +11:00
Andrew Morgan
7bef97dfb7
Remove trailing slashes from outbound federation requests and retry on 400 (#4840) 
As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.
 2019-03-21 15:07:28 +00:00
Erik Johnston
01e6b405be
Merge pull request #4908 from matrix-org/erikj/block_peek_on_blocked_rooms 
Deny peeking into rooms that have been blocked
 2019-03-21 14:07:17 +00:00
Richard van der Hoff
27813b4ca1
Update changelog.d/4908.bugfix 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-03-21 14:05:59 +00:00
Colin W
ab4e4c6c2f Update Apache Setup To Remove Location Syntax (#4870) 
This one should close #4841. Many thanks to @dev4223 for bringing it up and finding a solution.

Signed-off-by: Colin White
 2019-03-21 14:05:56 +00:00
Erik Johnston
017ed9d423 Newsfile 2019-03-21 11:26:47 +00:00
Erik Johnston
3959858eaa
Merge pull request #4904 from matrix-org/erikj/fix_shutdown 
Fixup shutdown room API
 2019-03-21 11:24:42 +00:00
Erik Johnston
09f991a63d
Merge pull request #4896 from matrix-org/erikj/disable_room_directory 
Add option to disable search room lists
 2019-03-21 10:16:54 +00:00
Richard van der Hoff
a6f2d3053d
Log requests which are simulated by the unit tests. (#4905) 
Rather than stubbing out the access_log, make it actually log the requests,
which makes it a lot more obvious what is going on during tests.
 2019-03-20 18:00:02 +00:00
Erik Johnston
30e69ff9b6 Newsfile 2019-03-20 16:56:55 +00:00
Erik Johnston
263f2c9ce1
Merge pull request #4895 from matrix-org/erikj/disable_user_search 
Add option to disable searching in the user dir
 2019-03-20 16:47:15 +00:00
Amber Brown
4d53017432
Batching in the user directory import (#4900) 2019-03-21 03:06:36 +11:00
Richard van der Hoff
cdb8036161
Add a config option for torture-testing worker replication. (#4902) 
Setting this to 50 or so makes a bunch of sytests fail in worker mode.
 2019-03-20 16:04:35 +00:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890) 
Rate-limit outgoing read-receipts as per #4730.
 2019-03-20 16:02:25 +00:00
Erik Johnston
bf5876990f Newsfile 2019-03-19 17:10:54 +00:00
Erik Johnston
0891202629 Newsfile 2019-03-19 16:41:57 +00:00
Erik Johnston
38ae23d5c2 Newsfile 2019-03-19 16:07:07 +00:00
Erik Johnston
4aa0b707d2
Merge pull request #4879 from matrix-org/erikj/test_old_deps 
Add py27-old test case to buildkite
 2019-03-19 13:05:39 +00:00
Richard van der Hoff
07f057ac80 changelog 2019-03-19 11:44:43 +00:00
Richard van der Hoff
45c4e19c74 Merge remote-tracking branch 'origin/develop' into HEAD 2019-03-19 11:43:46 +00:00
Neil Johnson
88f0675967 fix test_auto_create_auto_join_where_no_consent (#4886) 2019-03-19 11:38:59 +00:00
Richard van der Hoff
0dbfae03f9 Enforce hs_disabled_message correctly 
Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests
if there was no server_notices_mxid configured.
 2019-03-19 11:30:54 +00:00
Erik Johnston
e9eeca1314 Fix user directory background update (#4887) 2019-03-19 22:13:53 +11:00
Erik Johnston
cfc5a442ac Update newsfile 2019-03-19 10:56:11 +00:00
Richard van der Hoff
d2a537ea60 Merge remote-tracking branch 'origin/master' into develop 2019-03-19 10:37:50 +00:00
Michael Kaye
9482a84c0a Repoint docs for federation (#4881) 2019-03-19 10:37:18 +00:00
Richard van der Hoff
fd463b4f5d
Comment out most options in the generated config. (#4863) 
Make it so that most options in the config are optional, and commented out in
the generated config.

The reasons this is a good thing are as follows:

* If we decide that we should change the default for an option, we can do so,
  and only those admins that have deliberately chosen to override that option
  will be stuck on the old setting.

* It moves us towards a point where we can get rid of the super-surprising
  feature of synapse where the default settings for the config come from the
  generated yaml.

* It makes setting up a test config for unit testing an order of magnitude
  easier (see forthcoming PR).

* It makes the generated config more consistent, and hopefully easier for users
  to understand.
 2019-03-19 10:06:40 +00:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Erik Johnston
9ef1107b33 Newsfile 2019-03-18 17:19:49 +00:00
Brendan Abolivier
651ad8bc96 Add ratelimiting on failed login attempts (#4865) 2019-03-18 12:57:20 +00:00
Brendan Abolivier
899e523d6d
Add ratelimiting on login (#4821) 
Add two ratelimiters on login (per-IP address and per-userID).
 2019-03-15 17:46:16 +00:00
Richard van der Hoff
3b7ceb2c69
Merge pull request #4855 from matrix-org/rav/refactor_transaction_queue 
Split TransactionQueue up
 2019-03-15 12:32:11 +00:00
Richard van der Hoff
2dee441bdb
Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker 
Move client receipt processing to federation sender worker.
 2019-03-15 12:30:30 +00:00
Erik Johnston
b0fa3f6ff3
Merge pull request #4853 from matrix-org/erikj/worker_docker_ci 
Allow passing --daemonize to workers
 2019-03-15 10:35:38 +00:00
Luca Corbatto
a6d84190eb Add systemd setup that supports workers (#4662) 
This setup is a way to manage workers with systemd. It does however not
require workers. You can use this setup without workers. You just have
to make sure that the homeserver is forking and writes its PID file
to the location the service is looking in.

The currently distributed setup in the debian package does not work in
conjunction with workers.

* Adds changelog

* Lets systemd handle the forking

Sets all services to `type=simple` and disables daemonizing on the
synapse side.

* Formats readme to 80 columns per line

* Allows for full restart of all workers

* Changes README to reflect the new setup

* Adds dot to end of changelog file

* Removes surplus word

Co-Authored-By: targodan <targodan@users.noreply.github.com>

* Adds missing word

Co-Authored-By: targodan <targodan@users.noreply.github.com>

* Fixes linebreak

Co-Authored-By: targodan <targodan@users.noreply.github.com>

* Fixes unit type
 2019-03-15 09:51:46 +00:00
Aaron Raimist
2fb4ff8c89 Add some stuff back to the .gitignore (#4843) 
* Add some stuff back to the .gitignore

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Reorder and remove old items from .gitignore

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-03-15 09:22:29 +00:00
Erik Johnston
271cb1998b Revert "Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793)" 
This reverts commit 290552fd83.
 2019-03-14 14:30:54 +00:00
Erik Johnston
9073cfc8bd
Merge pull request #4846 from matrix-org/hawkowl/userdir-search 
Improve searching in the userdir
 2019-03-14 13:54:15 +00:00
Richard van der Hoff
ec3a59de50 changelog 2019-03-13 22:08:37 +00:00
Erik Johnston
f5d57d4848 Newsfile 2019-03-13 17:38:58 +00:00
Richard van der Hoff
6accbd25bc changelog 2019-03-13 17:24:10 +00:00
Andrew Morgan
7998ca3a66
Document using a certificate with a full chain (#4849) 2019-03-13 15:26:29 +00:00
Neil Johnson
332b60ec68 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-03-12 17:15:21 +00:00
Andrew Morgan
d42c81d724
Transfer local user's push rules on room upgrade (#4838) 
Transfer push rules (notifications) on room upgrade
 2019-03-12 14:42:53 +00:00
Neil Johnson
8b692bf7c2
Neilj/improved delegation doc 2 (#4832) 
Improved federation configuration docs.  Specifically detailing  .well-known and SRV based delegation methods. 

Inspiration Valentin Lab <valentin.lab@kalysto.org> for https://github.com/matrix-org/synapse/pull/4781
 2019-03-12 14:23:28 +00:00
Aaron Raimist
8ea1b41a0e Clarify what registration_shared_secret allows for (#2885) (#4844) 
* Clarify what registration_shared_secret allows for (#2885)

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2019-03-11 18:21:52 +00:00
Andrew Morgan
f18dca26da Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:44:29 +00:00
Andrew Morgan
290552fd83
Make federation endpoints more tolerant of trailing slashes for some endpoints (#4793) 
Server side of a solution towards #3622.
 2019-03-11 17:44:03 +00:00
Andrew Morgan
a8a028dfce Merge branch 'develop' into anoa/trailing_slashes_client 2019-03-11 17:07:31 +00:00
Erik Johnston
33dfd9a288 Newsfile 2019-03-11 14:13:58 +00:00
Amber Brown
41a5ba1682 changelog 2019-03-12 00:38:17 +11:00
Andrew Morgan
b61ac9660a
Merge pull request #4839 from matrix-org/anoa/no_captcha_tests 
Disable captcha registration by default in tests
 2019-03-11 09:58:43 +00:00
Matthew Hodgson
2326e00bc4 fix incorrect encoding of filenames with spaces in (#2090) 
fixes https://github.com/vector-im/riot-web/issues/3155
 2019-03-11 09:53:45 +00:00
Andrew Morgan
f8740d57de Add changelog 2019-03-08 18:23:54 +00:00
Andrew Morgan
50924ee34a Add changelog 2019-03-08 16:49:16 +00:00
Erik Johnston
cac4723afe Newsfile 2019-03-08 15:08:34 +00:00
Erik Johnston
8c4896668f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl 2019-03-08 11:44:20 +00:00
Erik Johnston
469b66c102
Merge pull request #4829 from matrix-org/erikj/device_list_seen_updates 
When re-syncing device lists reset the state
 2019-03-08 09:54:54 +00:00
Erik Johnston
cef80da903 Newsfile 2019-03-07 16:18:02 +00:00
Erik Johnston
7404fb3cdb Newsfile 2019-03-07 16:00:09 +00:00
Richard van der Hoff
fde26e47d6
Merge pull request #4824 from matrix-org/rav/docker_docs 
Document the `generate` option for the docker image.
 2019-03-07 14:12:44 +00:00
Richard van der Hoff
32471d63b7
Update example_log_config.yaml (#4820) 2019-03-07 14:12:10 +00:00
Richard van der Hoff
685704536f
Fix check-newsfragment for debian-only changes. (#4825) 2019-03-07 14:03:05 +00:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Richard van der Hoff
5580616235 Document the generate option for the docker image. 2019-03-07 07:35:42 +00:00
Matthew Hodgson
8f4b9f5210 Reword the sample config header to be less scary (#4801) 2019-03-07 07:09:01 +00:00
Travis Ralston
06cbf79fe3
Merge pull request #4779 from matrix-org/travis/icons 
Use static locations for Riot icons
 2019-03-06 15:11:41 -07:00
Richard van der Hoff
898378c9b5
Update changelog.d/4779.misc 
Co-Authored-By: turt2live <travpc@gmail.com>
 2019-03-06 14:29:56 -07:00
Erik Johnston
1f85c2c0ce
Merge pull request #4818 from matrix-org/erikj/prefill_client_ips 
Prefill client IPs cache on workers
 2019-03-06 19:17:39 +00:00
Erik Johnston
366877c579 Update changelog 2019-03-06 19:04:52 +00:00
Erik Johnston
0a6e716600
Merge pull request #4815 from matrix-org/erikj/docstrings 
Add docstrings from matrix-org-hotfixes
 2019-03-06 18:59:28 +00:00
Erik Johnston
c665b637de
Merge pull request #4816 from matrix-org/erikj/4422_debug 
Port #4422 debug logging from hotfixes
 2019-03-06 18:59:12 +00:00
Erik Johnston
7791c5194e Newsfile 2019-03-06 17:40:51 +00:00
Erik Johnston
03dce32019 Newsfile 2019-03-06 17:38:19 +00:00
Erik Johnston
4238f63545 Newsfile 2019-03-06 17:32:48 +00:00
Erik Johnston
9c50074c21 Newsfile 2019-03-06 17:24:53 +00:00
Erik Johnston
4c473ba088 Newsfile 2019-03-06 16:24:03 +00:00
Andrew Morgan
7b8a157b79
Merge pull request #4792 from matrix-org/anoa/replication_tokens 
Support batch updates in the worker sender
 2019-03-06 15:48:29 +00:00
Brendan Abolivier
6fcecb4859 Add changelog 2019-03-05 18:55:29 +00:00
Erik Johnston
16c8b4ecbd
Merge pull request #4772 from jbweston/jbweston/server-version-api 
Add 'server_version' endpoint to admin API
 2019-03-05 16:31:00 +00:00
Brendan Abolivier
a4c3a361b7
Add rate-limiting on registration (#4735) 
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
 2019-03-05 14:25:33 +00:00
Andrew Morgan
3887e0cd80
Merge pull request #4795 from matrix-org/anoa/configinatoractoring 
Remove reference to that no longer exists in README
 2019-03-05 09:47:14 +00:00
Andrew Morgan
ae90531036
Merge pull request #4794 from matrix-org/anoa/erroneous_dollahs 
Remove unnecessary dollar signs
 2019-03-05 09:46:59 +00:00
Erik Johnston
b050a10871
Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room 
Stop backpaginating when events not visible
 2019-03-05 09:32:33 +00:00
Erik Johnston
9e8bca5667
Merge pull request #4799 from matrix-org/rav/clean_up_replication_code 
Clean ups in replication notifier
 2019-03-05 09:19:48 +00:00
Erik Johnston
aa06d26ae0 clarify comments 2019-03-05 09:16:35 +00:00
Erik Johnston
c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys 
Allow /keys/{changes,query} API to run on worker
 2019-03-05 09:06:25 +00:00
Richard van der Hoff
eaa9f43603 changelog 2019-03-04 19:01:19 +00:00
Erik Johnston
00b0e8b7df Newsfile 2019-03-04 18:30:07 +00:00
Richard van der Hoff
2db49ea476 Add some debug about processing read receipts. 
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
 2019-03-04 18:19:40 +00:00
Richard van der Hoff
b29693a30b Clean up read-receipt handling. 
Remove a call to run_as_background_process: there is no need to run this as a
background process, because build_and_send_edu does not block.

We may as well inline the whole of _push_remotes.
 2019-03-04 18:16:43 +00:00
Richard van der Hoff
8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Andrew Morgan
be18073692 Add changelog 2019-03-04 16:01:17 +00:00
Andrew Morgan
3a438c24a6 Add changelog 2019-03-04 15:28:22 +00:00
Andrew Morgan
0bc50fb60a Add changelog 2019-03-04 14:05:16 +00:00
Seebi
aba5eeabd5 Fix v4v6 option in HAProxy example config (#4790) 
The v4v6 option only has a usage one ipv6 socket: https://serverfault.com/q/747895

Signed-off-by: Flakebi <flakebi@t-online.de>
 2019-03-04 13:19:41 +00:00
Richard van der Hoff
856c83f5f8
Avoid rebuilding Edu objects in worker mode (#4770) 
In worker mode, on the federation sender, when we receive an edu for sending
over the replication socket, it is parsed into an Edu object. There is no point
extracting the contents of it so that we can then immediately build another Edu.
 2019-03-04 12:57:44 +00:00
Erik Johnston
fbc047f2a5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room 2019-03-04 11:54:58 +00:00
Richard van der Hoff
2c3548d9d8
Update test_typing to use HomeserverTestCase. (#4771) 2019-03-04 10:05:39 +00:00
Travis Ralston
13c18853b9 Changelog 2019-03-01 15:15:00 -07:00
Richard van der Hoff
3064952939
Fix incorrect log about not persisting duplicate state event. (#4776) 
We were logging this when it was not true.
 2019-03-01 16:47:12 +00:00
Joseph Weston
d3dcb64501
Add changelog and AUTHORS file entry 
Signed-off-by: Joseph Weston <joseph@weston.cloud>
 2019-03-01 10:44:40 +01:00
Andrew Morgan
ac61b45a75 Minor docstring fixes for MatrixFederationAgent (#4765) 2019-02-28 16:24:01 +00:00
Amber Brown
b131cc77df
Make 'event_id' a required parameter in federated state requests (#4741) 
* make 'event_id' a required parameter in federated state requests

As per the spec: https://matrix.org/docs/spec/server_server/r0.1.1.html#id40

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* add changelog entry for bugfix

Signed-off-by: Joseph Weston <joseph@weston.cloud>

* Update server.py
 2019-02-27 14:35:47 -08:00
Richard van der Hoff
68f47d6744 Fix parsing of Content-Disposition headers (#4763) 
* Fix parsing of Content-Disposition headers

TIL: filenames in content-dispostion headers can contain semicolons, and aren't
%-encoded.

* fix python2 incompatibility

* Fix docstrings
 2019-02-27 14:29:10 -08:00
Amber Brown
f2a753ea38
Move from TravisCI to BuildKite (#4752) 2019-02-27 13:03:14 -08:00
Erik Johnston
76550c58d2
Merge pull request #4759 from matrix-org/erikj/3pid_client_reader 
Move /account/3pid to client_reader
 2019-02-27 16:11:21 +00:00
Erik Johnston
8267034a63
Merge pull request #4758 from matrix-org/erikj/use_presence_replication 
When presence is disabled don't send over replication
 2019-02-27 15:46:26 +00:00
Richard van der Hoff
3134964054
Update changelog.d/4759.feature 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-02-27 15:32:21 +00:00
Erik Johnston
95840d84d4 Newsfile 2019-02-27 14:28:52 +00:00
Erik Johnston
1b2940b3bd Newsfile 2019-02-27 13:54:45 +00:00
Erik Johnston
b5c13df0c4 Newsfile 2019-02-27 13:46:29 +00:00
Erik Johnston
71ef5fc411
Update newsfile to have a full stop 2019-02-27 13:22:23 +00:00
Erik Johnston
7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff 
Fix tightloop over connecting to replication server
 2019-02-27 11:00:59 +00:00
Richard van der Hoff
44a4d65586 0.99.2rc1 2019-02-27 10:48:34 +00:00
Erik Johnston
6bb1c028f1 Limit cache invalidation replication line length (#4748) 2019-02-27 10:28:37 +00:00
Richard van der Hoff
f191be822b
Add database version to phonehome stats. (#4753) 2019-02-27 10:21:49 +00:00
Richard van der Hoff
57426ec6a3 Fix check-newsfragment script (#4750) 
* Fix check-newsfragment script

I previously broke this so that it always succeeded...

* more fixes

* fix newsfiles
 2019-02-26 13:13:41 -08:00
Erik Johnston
09fc34c935 Newsfile 2019-02-26 15:13:55 +00:00
Andrew Morgan
802884d4ee Merge branch 'develop' of github.com:matrix-org/synapse into anoa/public_rooms_federate_develop 2019-02-26 14:23:40 +00:00
Andrew Morgan
b8e6ed36c1 Add changelog 2019-02-26 14:21:20 +00:00
Erik Johnston
899a119c2b Don't log stack trace when client has gone away during media download (#4738) 
* Don't log stack trace when client has gone away during media download

* Newsfile

* Fixup newsfile
 2019-02-25 11:17:22 -08:00
Richard van der Hoff
641c409e4e Fix ACME config for python 2. (#4717) 
Fixes #4675.
 2019-02-25 11:16:33 -08:00
Erik Johnston
65f3fbfbf7
Merge pull request #4737 from matrix-org/erikj/failure_log_tb 
Log tracebacks correctly
 2019-02-25 17:26:30 +00:00
Erik Johnston
1d0f2ec812 Newsfile 2019-02-25 17:02:31 +00:00
Erik Johnston
69efe6fb16
Merge pull request #4718 from matrix-org/erikj/fix_backfill_state_shred 
Fix backfill storing incorrect state for events
 2019-02-25 16:48:52 +00:00
Erik Johnston
108d5fb20d Fixup changelog 2019-02-25 16:32:19 +00:00
Erik Johnston
b1a90da82e
Merge pull request #4721 from matrix-org/erikj/msc_1866 
MSC 1866 - Use M_UNSUPPORTED_ROOM_VERSION for invite API
 2019-02-25 15:53:10 +00:00
Erik Johnston
16c7afa94c
Merge pull request #4722 from matrix-org/erikj/correctly_handle_keyring_exceptions 
Handle errors when fetching remote server keys
 2019-02-25 15:53:02 +00:00
Erik Johnston
f5050e148c Newsfile 2019-02-25 10:02:58 +00:00
Erik Johnston
21d3f82344 Newsfile 2019-02-23 15:18:38 +00:00
Erik Johnston
65bf9f1119 Newsfile 2019-02-23 15:10:59 +00:00
Erik Johnston
71304bfc8d Newsfile 2019-02-23 14:53:15 +00:00
Erik Johnston
d14e94bae4
Merge pull request #4716 from matrix-org/erikj/pusher_logging 
Fix up pusher logging a bit
 2019-02-22 15:29:16 +00:00
Erik Johnston
5d3e3c051d Newsfile 2019-02-22 14:51:33 +00:00
Erik Johnston
e28ef831e6 Newsfile 2019-02-22 14:40:08 +00:00
Richard van der Hoff
e07384c4e1
Add prometheus metrics for number of badge update pushes. (#4709) 
We're counting the number of push notifications, but not the number of badges;
I'd like to see if they are significant.
 2019-02-22 10:57:15 +00:00
Richard van der Hoff
e1666af9be
Better checks on newsfragments (#4698) 
* You need an entry in the debian changelog (and not a regular newsfragment)
  for debian packaging changes.
* Regular newsfragments must end in full stops.
 2019-02-22 10:56:59 +00:00
Richard van der Hoff
fcd6f01dc7
Minor tweaks to acme docs (#4689) 2019-02-22 10:56:42 +00:00
Richard van der Hoff
0abb094f1a
bail out early in on_new_receipts if no pushers (#4706) 2019-02-21 17:51:21 +00:00
Richard van der Hoff
6d65659b62
Run push_receipts_to_remotes as background job (#4707) 
I suspect the CPU usage metrics for this are going to /dev/null at the moment.
 2019-02-21 17:50:30 +00:00
Benoît S
16e0680498 Added HAProxy example (#4660) 
* Added HAProxy example

Proposal of an example with HAProxy. Asked by #4541.

Signed-off-by: Benoît S. (“Benpro”) <gitlab@benpro.fr>

* Following suggestions of @richvdh
 2019-02-21 17:44:10 +00:00
Erik Johnston
b9d6756b14
Merge pull request #4263 from rkfg/develop 
Prevent crash on pagination.
 2019-02-21 17:42:15 +00:00
Eric
8184ae8a09 Consider e2e_room_keys.is_verified column as boolean 
This column was considered as an int, crashing the whole
migration process

Signed-off-by: Eric <eric@pedr0.net>
 2019-02-20 23:18:00 +01:00
Erik Johnston
56f4ece778 Newsfile 2019-02-20 18:15:13 +00:00
Erik Johnston
82fca11fc1
Merge pull request #4694 from matrix-org/erikj/fix_sentry_config_format 
Fixup generated metrics config
 2019-02-20 14:13:38 +00:00
Richard van der Hoff
82ca6d1f9f
Add metrics for number of outgoing EDUs, by type (#4695) 2019-02-20 14:13:14 +00:00
Erik Johnston
633e5c933b Newsfile 2019-02-20 13:41:43 +00:00
Amber Brown
a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Richard van der Hoff
b2200a8690
Logging improvements for the pusher (#4691) 2019-02-20 11:36:50 +00:00
Richard van der Hoff
c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Richard van der Hoff
8d98dc8ffe
Clean up gitignores (#4688) 
I just got bitten by a file being caught by the .gitignore, which shouldn't
have been, and am now pissed off with the .gitignore. I have basically declared
bankruptcy on it and started again.
 2019-02-20 11:10:50 +00:00
Richard van der Hoff
86920ac266
Merge pull request #4678 from matrix-org/rav/tls_install_instructions 
Attempt to clarify installation/config instructions
 2019-02-20 11:10:33 +00:00
Erik Johnston
dbdc565dfd Fix registration on workers (#4682) 
* Move RegistrationHandler init to HomeServer

* Move post registration actions to RegistrationHandler

* Add post regisration replication endpoint

* Newsfile
 2019-02-20 18:47:31 +11:00
Richard van der Hoff
c594cc8076 Run unit tests against python 3.7 (#4677) 
* Run unit tests against python 3.7

... so that we span the full range of our supported python versions

* Switch to xenial

* fix psql fail

* pep8 etc want python 3.6
 2019-02-20 18:42:24 +11:00
Richard van der Hoff
ae753fed8c changelog 2019-02-19 13:56:44 +00:00
Erik Johnston
c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation 
Batch cache invalidation over replication
 2019-02-19 13:14:30 +00:00
Richard van der Hoff
1c0eb8bbb2
Merge pull request #4676 from matrix-org/rav/pg95 
Test against Postgres 9.5 as well as 9.4
 2019-02-19 11:19:21 +00:00
Brendan Abolivier
a288bdf0b1
Merge pull request #4652 from matrix-org/babolivier/acme-delegated 
Support .well-known delegation when issuing certificates through ACME
 2019-02-19 11:15:38 +00:00
Erik Johnston
32590b7139
Merge pull request #4670 from matrix-org/erikj/register_login_split 
Split /login into client_reader
 2019-02-19 10:46:22 +00:00
Richard van der Hoff
7c70b8f8a6
Try and make TLS federation client code faster (#4674) 
* fix to use makeContext so that we don't need to rebuild the certificateoptions each time
 2019-02-19 10:19:16 +00:00
Richard van der Hoff
107aeb6915 misc->feature 2019-02-19 10:18:48 +00:00
Andrew Morgan
968a30a75c
Merge pull request #4642 from matrix-org/anoa/bans_room_upgrade 
Transfer bans on room upgrade
 2019-02-19 10:16:42 +00:00
Richard van der Hoff
0869f01e74 Test against Postgres 9.5 as well as 9.4 
Postgres 9.5 is the first to support UPSERTs, so we should really run against
it as well as 9.4.
 2019-02-19 09:54:31 +00:00
Amber Brown
2b2466f78b changelog 2019-02-19 16:18:48 +11:00
Erik Johnston
92e6fb5c89 Newsfile 2019-02-18 17:58:17 +00:00
Andrew Morgan
f8b9ca53ce Move member event processing and changelog fix 2019-02-18 17:28:52 +00:00
Erik Johnston
d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry 
Add basic optional sentry.io integration
 2019-02-18 17:22:45 +00:00
Erik Johnston
f3ab0b2390 Newsfile 2019-02-18 17:22:01 +00:00
Erik Johnston
128902d60a Update worker docs 2019-02-18 17:21:51 +00:00
Erik Johnston
fc2c245a1f
Merge pull request #4666 from matrix-org/erikj/register_login_split 
Split out registration to worker
 2019-02-18 17:18:06 +00:00
Erik Johnston
459d3d5046
Merge pull request #4668 from matrix-org/erikj/catch_exceptions 
Correctly handle HttpResponseException when handling device updates
 2019-02-18 16:55:43 +00:00
Erik Johnston
028267acd2
Merge pull request #4669 from matrix-org/erikj/log_exception 
Cleanup top level request exception logging
 2019-02-18 16:41:41 +00:00
Erik Johnston
d08bac4136
Merge pull request #4651 from matrix-org/matthew/well-known-cors 
set CORS on .well-known URI to unbreak modular
 2019-02-18 16:38:11 +00:00
Erik Johnston
c30f73c86a
Merge pull request #4667 from matrix-org/erikj/fix_revoke_guest_access_workers 
Fix kicking guest users in worker mode
 2019-02-18 15:55:48 +00:00
Erik Johnston
092b541401 Fixup 2019-02-18 15:52:26 +00:00
Erik Johnston
12ae64ce0d Newsfile 2019-02-18 15:23:10 +00:00
Erik Johnston
e85aabb030 Newsfile 2019-02-18 15:06:22 +00:00
Matthew Hodgson
d9713e916e changelog 2019-02-18 16:00:22 +01:00
Erik Johnston
2f16857ca9 Newsfile 2019-02-18 14:55:11 +00:00
Brendan Abolivier
68a53f825f Merge branch 'develop' into babolivier/acme-delegated 2019-02-18 14:52:23 +00:00
Erik Johnston
d1b060b492
Merge pull request #4643 from matrix-org/erikj/catch_exceptions 
Correctly handle RequestSendFailed exceptions
 2019-02-18 13:53:24 +00:00
Andrew Morgan
7033b05cad Add changelog 2019-02-18 13:52:56 +00:00
Will Hunt
e83a190643
Update changelog.d/4666.feature 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2019-02-18 13:46:13 +00:00