synapse/tests/util
Erik Johnston 81e0f57800
Fix perf when streams don't change often (#17767)
There is a bug with the `StreamChangeCache` where it would incorrectly
return that all entities had changed if asked for entities changed
*since* the earliest stream position.

Note that for streams we use the inequalities: `$min_stream_id <
stream_id <= $max_stream_id`, i.e. when we ask the stream change cache
for all things that have changed since `$stream_id` we don't care for
events that happened *at* `$stream_id`.

Specifically: `_earliest_known_stream_pos` is the position at which we
know that we'll have entries for all changes since that point, we can
use the cache for any stream IDs that equal
`_earliest_known_stream_pos`.

`_earliest_known_stream_pos` is set in three places:
- On startup we set it either to:
  - the current maximum stream ID, with not prefilled values; or
  - the minimum of the latest N values we pulled from the DB
- When we evict items from the bottom, we set it to the stream ID of the
evicted items.

This was changed in https://github.com/matrix-org/synapse/pull/14435,
but I think we were overly conservative there.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-09-30 13:52:33 +01:00
..
caches Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
__init__.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_async_helpers.py Update license headers 2023-11-21 15:29:58 -05:00
test_batching_queue.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_check_dependencies.py Bump mypy from 1.9.0 to 1.10.1 (#17445) 2024-07-16 12:08:06 +01:00
test_dict_cache.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_expiring_cache.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_file_consumer.py Update license headers 2023-11-21 15:29:58 -05:00
test_itertools.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_linearizer.py Bump black from 23.10.1 to 24.2.0 (#16936) 2024-03-13 16:46:44 +00:00
test_logcontext.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_logformatter.py Update license headers 2023-11-21 15:29:58 -05:00
test_lrucache.py Don't invalidate the entire event cache when we purge history (#16905) 2024-02-13 13:24:11 +00:00
test_macaroons.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_ratelimitutils.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_retryutils.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_rwlock.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_stream_change_cache.py Fix perf when streams don't change often (#17767) 2024-09-30 13:52:33 +01:00
test_stringutils.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_task_scheduler.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_threepids.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_treecache.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00
test_wheel_timer.py Correctly mention previous copyright (#16820) 2024-01-23 11:26:48 +00:00