Commit graph

165 commits

Author SHA1 Message Date
Sean Quah 605d161d7d
Add cancellation support to ReadWriteLock (#12120)
Also convert `ReadWriteLock` to use async context managers.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 18:49:07 +00:00
Sean Quah 90b2327066
Add delay_cancellation utility function (#12180)
`delay_cancellation` behaves like `stop_cancellation`, except it
delays `CancelledError`s until the original `Deferred` resolves.
This is handy for unifying cleanup paths and ensuring that uncancelled
coroutines don't use finished logcontexts.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 17:52:15 +00:00
Patrick Cloke 690cb4f3b3
Allow for ignoring some arguments when caching. (#12189)
* `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key.
* Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity.
* Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
2022-03-09 18:07:41 +00:00
Olivier Wilkinson (reivilibre) 0dc9c5653c Merge branch 'master' into develop 2022-03-08 15:37:35 +00:00
reivilibre 2eef234ae3
Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to start if pre-release versions of dependencies were installed. (#12177)
* Add failing test to characterise the regression #12176

* Permit pre-release versions of specified packages

* Newsfile (bugfix)

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-03-08 10:47:28 +00:00
Richard van der Hoff 00a67f831a Merge remote-tracking branch 'origin/release-v1.54' into develop 2022-03-04 22:40:51 +00:00
Sean Quah 158e0937eb
Add test for ObservableDeferred's cancellation behaviour (#12149)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-04 13:10:05 +00:00
David Robertson cea1b58c4a
Don't impose version checks on dev extras at runtime (#12129)
* Fix incorrect argument in test case

* Add copyright header

* Docstring and __all__

* Exclude dev depenencies

* Use changelog from #12088

* Include version in error messages

This will hopefully distinguish between the version of the source code
and the version of the distribution package that is installed.

* Linter script is your friend
2022-03-03 12:47:55 +00:00
David Robertson 313581e4e9
Use importlib.metadata to read requirements (#12088)
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`

I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.

I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.

I hope you agree that this is clearer.

* Test cases
2022-03-01 17:44:41 +00:00
Sean Quah 4d6b6c17c8
Fix rare error in ReadWriteLock when writers complete immediately (#12105)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-01 15:27:15 +00:00
Sean Quah 91bc15c772
Add stop_cancellation utility function (#12106) 2022-03-01 13:51:03 +00:00
Richard van der Hoff 5458eb8551
Fix 'Unhandled error in Deferred' (#12089)
* Fix 'Unhandled error in Deferred'

Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when
a function wrapped with `@cachedList` failed

* Minor optimisation to cachedListDescriptor

we can avoid re-using `missing`, which saves looking up entries in
`deferreds_map`, and means we don't need to copy it.

* Improve type annotation on CachedListDescriptor
2022-03-01 09:51:38 +00:00
Richard van der Hoff 9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Patrick Cloke 02d708568b
Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
Richard van der Hoff e24ff8ebe3
Remove HomeServer.get_datastore() (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
reivilibre 84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Richard van der Hoff c3e38b88f2
Improve opentracing support for ResponseCache (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Sean Quah 0147b3de20
Add missing type hints to synapse.logging.context (#11556) 2021-12-14 17:35:28 +00:00
Sean Quah 088d748f2c
Revert "Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) (#11527)
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
Sean Quah a77c369897
Move glob_to_regex and re_word_boundary to matrix-python-common (#11505) 2021-12-06 11:36:08 +00:00
Sean Quah 7ff22d6da4
Fix LruCache corruption bug with a size_callback that can return 0 (#11454)
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Richard van der Hoff f36434590c
Additional test for cachedList (#11246)
I was trying to understand how `cachedList` works, and ended up writing this
extra test. I figure we may as well keep it.
2021-11-04 14:45:34 +00:00
Richard van der Hoff 46d0937447
ObservableDeferred: run observers in order (#11229) 2021-11-02 00:17:35 +00:00
Patrick Cloke 47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Jonathan de Jong 93729719b8
Use inline type hints in tests/ (#10350)
This PR is tantamount to running:

    python3.8 -m com2ann -v 6 tests/

(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
Erik Johnston 7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Richard van der Hoff d7808a2dde
Extend ResponseCache to pass a context object into the callback (#10157)
This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug.

The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
2021-06-14 10:26:09 +01:00
Erik Johnston fc3d2dc269
Rewrite the KeyRing (#10035) 2021-06-02 16:37:59 +01:00
Erik Johnston 78b5102ae7
Fix up BatchingQueue (#10078)
Fixes #10068
2021-05-27 14:32:31 +01:00
Richard van der Hoff 224f2f949b
Combine LruCache.invalidate and invalidate_many (#9973)
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
2021-05-27 10:33:56 +01:00
Patrick Cloke 7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Richard van der Hoff c0df6bae06
Remove keylen from LruCache. (#9993)
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
Erik Johnston 3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Erik Johnston 7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Richard van der Hoff 5090f26b63
Minor @cachedList enhancements (#9975)
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
Richard van der Hoff 7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Richard van der Hoff 03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Andrew Morgan 6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Patrick Cloke b076bc276e
Always use the name as the log ID. (#9829)
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke 0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Jonathan de Jong 2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Patrick Cloke 48d44ab142
Record more information into structured logs. (#9654)
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Patrick Cloke 01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Jonathan de Jong d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff 3b754aea27
Clean up caching/locking of OIDC metadata load (#9362)
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Erik Johnston 056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Erik Johnston 1a08e0cdab
Fix event chain bg update. (#9118)
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston 1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00