Recommend poetry in docs (#12475)

* Recommend poetry in docs

- readme
- contributor guide
- upgrade notes
- new dev cheat sheet for poetry

Co-authored-by: Shay <hillerys@element.io>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
This commit is contained in:
David Robertson 2022-04-20 15:18:21 +01:00 committed by GitHub
parent d0c1f4ca4c
commit ecef741add
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 303 additions and 52 deletions

View file

@ -293,24 +293,27 @@ directory of your choice::
git clone https://github.com/matrix-org/synapse.git git clone https://github.com/matrix-org/synapse.git
cd synapse cd synapse
Synapse has a number of external dependencies, that are easiest Synapse has a number of external dependencies. We maintain a fixed development
to install using pip and a virtualenv:: environment using [poetry](https://python-poetry.org/). First, install poetry. We recommend
python3 -m venv ./env pip install --user pipx
source ./env/bin/activate pipx install poetry
pip install -e ".[all,dev]"
as described `here <https://python-poetry.org/docs/#installing-with-pipx>`_.
(See `poetry's installation docs <https://python-poetry.org/docs/#installation>`
for other installation methods.) Then ask poetry to create a virtual environment
from the project and install Synapse's dependencies::
poetry install --extras "all test"
This will run a process of downloading and installing all the needed This will run a process of downloading and installing all the needed
dependencies into a virtual env. If any dependencies fail to install, dependencies into a virtual env.
try installing the failing modules individually::
pip install -e "module-name"
We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082` We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082`
./demo/start.sh poetry run ./demo/start.sh
(to stop, you can use `./demo/stop.sh`) (to stop, you can use `poetry run ./demo/stop.sh`)
See the `demo documentation <https://matrix-org.github.io/synapse/develop/development/demo.html>`_ See the `demo documentation <https://matrix-org.github.io/synapse/develop/development/demo.html>`_
for more information. for more information.
@ -318,14 +321,14 @@ for more information.
If you just want to start a single instance of the app and run it directly:: If you just want to start a single instance of the app and run it directly::
# Create the homeserver.yaml config once # Create the homeserver.yaml config once
python -m synapse.app.homeserver \ poetry run synapse_homeserver \
--server-name my.domain.name \ --server-name my.domain.name \
--config-path homeserver.yaml \ --config-path homeserver.yaml \
--generate-config \ --generate-config \
--report-stats=[yes|no] --report-stats=[yes|no]
# Start the app # Start the app
python -m synapse.app.homeserver --config-path homeserver.yaml poetry run synapse_homeserver --config-path homeserver.yaml
Running the unit tests Running the unit tests
@ -334,7 +337,7 @@ Running the unit tests
After getting up and running, you may wish to run Synapse's unit tests to After getting up and running, you may wish to run Synapse's unit tests to
check that everything is installed correctly:: check that everything is installed correctly::
trial tests poetry run trial tests
This should end with a 'PASSED' result (note that exact numbers will This should end with a 'PASSED' result (note that exact numbers will
differ):: differ)::

1
changelog.d/12475.doc Normal file
View file

@ -0,0 +1 @@
Strongly recommend `poetry` for development.

View file

@ -48,19 +48,28 @@ can find many good git tutorials on the web.
# 4. Install the dependencies # 4. Install the dependencies
Once you have installed Python 3 and added the source, please open a terminal and Synapse uses the [poetry](https://python-poetry.org/) project to manage its dependencies
setup a *virtualenv*, as follows: and development environment. Once you have installed Python 3 and added the
source, you should install `poetry`.
Of their installation methods, we recommend
[installing `poetry` using `pipx`](https://python-poetry.org/docs/#installing-with-pipx),
```shell
pip install --user pipx
pipx install poetry
```
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
for other installation methods.
Next, open a terminal and install dependencies as follows:
```sh ```sh
cd path/where/you/have/cloned/the/repository cd path/where/you/have/cloned/the/repository
python3 -m venv ./env poetry install --extras all
source ./env/bin/activate
pip install wheel
pip install -e ".[all,dev]"
pip install tox
``` ```
This will install the developer dependencies for the project. This will install the runtime and developer dependencies for the project.
# 5. Get in touch. # 5. Get in touch.
@ -117,11 +126,10 @@ The linters look at your code and do two things:
- ensure that your code follows the coding style adopted by the project; - ensure that your code follows the coding style adopted by the project;
- catch a number of errors in your code. - catch a number of errors in your code.
The linter takes no time at all to run as soon as you've [downloaded the dependencies into your python virtual environment](#4-install-the-dependencies). The linter takes no time at all to run as soon as you've [downloaded the dependencies](#4-install-the-dependencies).
```sh ```sh
source ./env/bin/activate poetry run ./scripts-dev/lint.sh
./scripts-dev/lint.sh
``` ```
Note that this script *will modify your files* to fix styling errors. Note that this script *will modify your files* to fix styling errors.
@ -131,15 +139,13 @@ If you wish to restrict the linters to only the files changed since the last com
(much faster!), you can instead run: (much faster!), you can instead run:
```sh ```sh
source ./env/bin/activate poetry run ./scripts-dev/lint.sh -d
./scripts-dev/lint.sh -d
``` ```
Or if you know exactly which files you wish to lint, you can instead run: Or if you know exactly which files you wish to lint, you can instead run:
```sh ```sh
source ./env/bin/activate poetry run ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
``` ```
## Run the unit tests (Twisted trial). ## Run the unit tests (Twisted trial).
@ -148,16 +154,14 @@ The unit tests run parts of Synapse, including your changes, to see if anything
was broken. They are slower than the linters but will typically catch more errors. was broken. They are slower than the linters but will typically catch more errors.
```sh ```sh
source ./env/bin/activate poetry run trial tests
trial tests
``` ```
If you wish to only run *some* unit tests, you may specify If you wish to only run *some* unit tests, you may specify
another module instead of `tests` - or a test class or a method: another module instead of `tests` - or a test class or a method:
```sh ```sh
source ./env/bin/activate poetry run trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite
trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite
``` ```
If your tests fail, you may wish to look at the logs (the default log level is `ERROR`): If your tests fail, you may wish to look at the logs (the default log level is `ERROR`):
@ -169,7 +173,7 @@ less _trial_temp/test.log
To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`: To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`:
```sh ```sh
SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests SYNAPSE_TEST_LOG_LEVEL=DEBUG poetry run trial tests
``` ```
By default, tests will use an in-memory SQLite database for test data. For additional By default, tests will use an in-memory SQLite database for test data. For additional
@ -180,7 +184,7 @@ database state to be stored in a file named `test.db` under the trial process'
working directory. Typically, this ends up being `_trial_temp/test.db`. For example: working directory. Typically, this ends up being `_trial_temp/test.db`. For example:
```sh ```sh
SYNAPSE_TEST_PERSIST_SQLITE_DB=1 trial tests SYNAPSE_TEST_PERSIST_SQLITE_DB=1 poetry run trial tests
``` ```
The database file can then be inspected with: The database file can then be inspected with:

View file

@ -0,0 +1,239 @@
# Managing dependencies with Poetry
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/).
# Background
Synapse uses a variety of third-party Python packages to function as a homeserver.
Some of these are direct dependencies, listed in `pyproject.toml` under the
`[tool.poetry.dependencies]` section. The rest are transitive dependencies (the
things that our direct dependencies themselves depend on, and so on recursively.)
We maintain a locked list of all our dependencies (transitive included) so that
we can track exactly which version of each dependency appears in a given release.
See [here](https://github.com/matrix-org/synapse/issues/11537#issue-1074469665)
for discussion of why we wanted this for Synapse. We chose to use
[`poetry`](https://python-poetry.org/) to manage this locked list; see
[this comment](https://github.com/matrix-org/synapse/issues/11537#issuecomment-1015975819)
for the reasoning.
The locked dependencies get included in our "self-contained" releases: namely,
our docker images and our debian packages. We also use the locked dependencies
in development and our continuous integration.
Separately, our "broad" dependencies—the version ranges specified in
`pyproject.toml`—are included as metadata in our "sdists" and "wheels" [uploaded
to PyPI](https://pypi.org/project/matrix-synapse). Installing from PyPI or from
the Synapse source tree directly will _not_ use the locked dependencies; instead,
they'll pull in the latest version of each package available at install time.
## Example dependency
An example may help. We have a broad dependency on
[`phonenumbers`](https://pypi.org/project/phonenumbers/), as declared in
this snippet from pyproject.toml [as of Synapse 1.57](
https://github.com/matrix-org/synapse/blob/release-v1.57/pyproject.toml#L133
):
```toml
[tool.poetry.dependencies]
# ...
phonenumbers = ">=8.2.0"
```
In our lockfile this is
[pinned]( https://github.com/matrix-org/synapse/blob/dfc7646504cef3e4ff396c36089e1c6f1b1634de/poetry.lock#L679-L685)
to version 8.12.44, even though
[newer versions are available](https://pypi.org/project/phonenumbers/#history).
```toml
[[package]]
name = "phonenumbers"
version = "8.12.44"
description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
category = "main"
optional = false
python-versions = "*"
```
The lockfile also includes a
[cryptographic checksum](https://github.com/matrix-org/synapse/blob/release-v1.57/poetry.lock#L2178-L2181)
of the sdists and wheels provided for this version of `phonenumbers`.
```toml
[metadata.files]
# ...
phonenumbers = [
{file = "phonenumbers-8.12.44-py2.py3-none-any.whl", hash = "sha256:cc1299cf37b309ecab6214297663ab86cb3d64ae37fd5b88e904fe7983a874a6"},
{file = "phonenumbers-8.12.44.tar.gz", hash = "sha256:26cfd0257d1704fe2f88caff2caabb70d16a877b1e65b6aae51f9fbbe10aa8ce"},
]
```
We can see this pinned version inside the docker image for that release:
```
$ docker pull matrixdotorg/synapse:v1.57.0
...
$ docker run --entrypoint pip matrixdotorg/synapse:v1.57.0 show phonenumbers
Name: phonenumbers
Version: 8.12.44
Summary: Python version of Google's common library for parsing, formatting, storing and validating international phone numbers.
Home-page: https://github.com/daviddrysdale/python-phonenumbers
Author: David Drysdale
Author-email: dmd@lurklurk.org
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires:
Required-by: matrix-synapse
```
Whereas the wheel metadata just contains the broad dependencies:
```
$ cd /tmp
$ wget https://files.pythonhosted.org/packages/ca/5e/d722d572cc5b3092402b783d6b7185901b444427633bd8a6b00ea0dd41b7/matrix_synapse-1.57.0rc1-py3-none-any.whl
...
$ unzip -c matrix_synapse-1.57.0rc1-py3-none-any.whl matrix_synapse-1.57.0rc1.dist-info/METADATA | grep phonenumbers
Requires-Dist: phonenumbers (>=8.2.0)
```
# Tooling recommendation: direnv
[`direnv`](https://direnv.net/) is a tool for activating environments in your
shell inside a given directory. Its support for poetry is unofficial (a
community wiki recipe only), but works solidly in our experience. We thoroughly
recommend it for daily use. To use it:
1. [Install `direnv`](https://direnv.net/docs/installation.html) - it's likely
packaged for your system already.
2. Teach direnv about poetry. The [shell config here](https://github.com/direnv/direnv/wiki/Python#poetry)
needs to be added to `~/.config/direnv/direnvrc` (or more generally `$XDG_CONFIG_HOME/direnv/direnvrc`).
3. Mark the synapse checkout as a poetry project: `echo layout poetry > .envrc`.
4. Convince yourself that you trust this `.envrc` configuration and project.
Then formally confirm this to `direnv` by running `direnv allow`.
Then whenever you navigate to the synapse checkout, you should be able to run
e.g. `mypy` instead of `poetry run mypy`; `python` instead of
`poetry run python`; and your shell commands will automatically run in the
context of poetry's venv, without having to run `poetry shell` beforehand.
# How do I...
## ...reset my venv to the locked environment?
```shell
poetry install --extras all --remove-untracked
```
## ...run a command in the `poetry` virtualenv?
Use `poetry run cmd args` when you need the python virtualenv context.
To avoid typing `poetry run` all the time, you can run `poetry shell`
to start a new shell in the poetry virtualenv context. Within `poetry shell`,
`python`, `pip`, `mypy`, `trial`, etc. are all run inside the project virtualenv
and isolated from the rest o the system.
Roughly speaking, the translation from a traditional virtualenv is:
- `env/bin/activate` -> `poetry shell`, and
- `deactivate` -> close the terminal (Ctrl-D, `exit`, etc.)
See also the direnv recommendation above, which makes `poetry run` and
`poetry shell` unnecessary.
## ...inspect the `poetry` virtualenv?
Some suggestions:
```shell
# Current env only
poetry env info
# All envs: this allows you to have e.g. a poetry managed venv for Python 3.7,
# and another for Python 3.10.
poetry env list --full-path
poetry run pip list
```
Note that `poetry show` describes the abstract *lock file* rather than your
on-disk environment. With that said, `poetry show --tree` can sometimes be
useful.
## ...add a new dependency?
Either:
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else
- `poetry add packagename`. See `poetry add --help`; note the `--dev`,
`--extras` and `--optional` flags in particular.
- **NB**: this specifies the new package with a version given by a "caret bound". This won't get forced to its lowest version in the old deps CI job: see [this TODO](https://github.com/matrix-org/synapse/blob/4e1374373857f2f7a911a31c50476342d9070681/.ci/scripts/test_old_deps.sh#L35-L39).
Include the updated `pyproject.toml` and `poetry.lock` files in your commit.
## ...remove a dependency?
This is not done often and is untested, but
```shell
poetry remove packagename
```
ought to do the trick. Alternatively, manually update `pyproject.toml` and
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock`
files in your commit.
## ...update the version range for an existing dependency?
Best done by manually editing `pyproject.toml`, then `poetry lock --no-update`.
Include the updated `pyproject.toml` and `poetry.lock` in your commit.
## ...update a dependency in the locked environment?
Use
```shell
poetry update packagename
```
to use the latest version of `packagename` in the locked environment, without
affecting the broad dependencies listed in the wheel.
There doesn't seem to be a way to do this whilst locking a _specific_ version of
`packagename`. We can workaround this (crudely) as follows:
```shell
poetry add packagename==1.2.3
# This should update pyproject.lock.
# Now undo the changes to pyproject.toml. For example
# git restore pyproject.toml
# Get poetry to recompute the content-hash of pyproject.toml without changing
# the locked package versions.
poetry lock --no-update
```
Either way, include the updated `poetry.lock` file in your commit.
## ...export a `requirements.txt` file?
```shell
poetry export --extras all
```
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`.
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may
be required.
## ...build a test wheel?
I usually use
```shell
poetry run pip install build && poetry run python -m build
```
because [`build`](https://github.com/pypa/build) is a standardish tool which
doesn't require poetry. (It's what we use in CI too). However, you could try
`poetry build` too.

View file

@ -19,32 +19,36 @@ this document.
packages](setup/installation.md#prebuilt-packages), you will need to follow the packages](setup/installation.md#prebuilt-packages), you will need to follow the
normal process for upgrading those packages. normal process for upgrading those packages.
- If Synapse was installed using pip then upgrade to the latest
version by running:
```bash
pip install --upgrade matrix-synapse
```
- If Synapse was installed from source, then: - If Synapse was installed from source, then:
1. Activate the virtualenv before upgrading. For example, if 1. Obtain the latest version of the source code. Git users can run
Synapse is installed in a virtualenv in `~/synapse/env` then `git pull` to do this.
2. If you're running Synapse in a virtualenv, make sure to activate it before
upgrading. For example, if Synapse is installed in a virtualenv in `~/synapse/env` then
run: run:
```bash ```bash
source ~/synapse/env/bin/activate source ~/synapse/env/bin/activate
```
2. If Synapse was installed using pip then upgrade to the latest
version by running:
```bash
pip install --upgrade matrix-synapse
```
If Synapse was installed using git then upgrade to the latest
version by running:
```bash
git pull
pip install --upgrade . pip install --upgrade .
``` ```
Include any relevant extras between square brackets, e.g. `pip install --upgrade ".[postgres,oidc]"`.
3. Restart Synapse: 3. If you're using `poetry` to manage a Synapse installation, run:
```bash
poetry install
```
Include any relevant extras with `--extras`, e.g. `poetry install --extras postgres --extras oidc`.
It's probably easiest to run `poetry install --extras all`.
4. Restart Synapse:
```bash ```bash
synctl restart synctl restart