# Administrator Guide — Building and Deploying TomoTree This guide is intended for maintainers who need to build a new release, publish the Python package, build and push a Docker image, and update a running production server. --- ## Table of Contents - [Administrator Guide — Building and Deploying TomoTree](#administrator-guide--building-and-deploying-tomotree) - [Table of Contents](#table-of-contents) - [Prerequisites](#prerequisites) - [Bumping the Version](#bumping-the-version) - [Building and Publishing the Python Package](#building-and-publishing-the-python-package) - [Publish to GitHub (recommended)](#publish-to-github-recommended) - [Publish to PyPI](#publish-to-pypi) - [1. Build the distribution archives](#1-build-the-distribution-archives) - [2. Upload to PyPI](#2-upload-to-pypi) - [3. Upload to TestPyPI first (optional)](#3-upload-to-testpypi-first-optional) - [Building and Pushing the Docker Image](#building-and-pushing-the-docker-image) - [1. Build the image](#1-build-the-image) - [2. Tag for Docker Hub](#2-tag-for-docker-hub) - [3. Push to Docker Hub](#3-push-to-docker-hub) - [Updating a Production Server](#updating-a-production-server) - [Typical deployment directory layout](#typical-deployment-directory-layout) - [compose.yml on the server](#composeyml-on-the-server) - [Pulling the latest image and restarting](#pulling-the-latest-image-and-restarting) - [Converting datasets to the current format](#converting-datasets-to-the-current-format) - [Checking the running container](#checking-the-running-container) - [Rolling back to a previous version](#rolling-back-to-a-previous-version) - [Configuration Files on the Server](#configuration-files-on-the-server) - [Installation check](#installation-check) - [Whitelist and Blacklist](#whitelist-and-blacklist) - [Account expiry](#account-expiry) - [Cleaning Up Old Docker Images and Files](#cleaning-up-old-docker-images-and-files) - [Remove dangling images (untagged leftovers)](#remove-dangling-images-untagged-leftovers) - [Remove all unused images (including older tagged versions)](#remove-all-unused-images-including-older-tagged-versions) - [Remove stopped containers, unused networks, dangling images, and build cache at once](#remove-stopped-containers-unused-networks-dangling-images-and-build-cache-at-once) - [Remove a specific old image by tag](#remove-a-specific-old-image-by-tag) - [Typical post-upgrade cleanup sequence](#typical-post-upgrade-cleanup-sequence) - [Check disk usage before and after](#check-disk-usage-before-and-after) --- ## Prerequisites Your local machine needs: - **Git** with push access to the repository. - **Python ≥ 3.9** with `build` and `twine` installed (for PyPI releases): ```bash pip install build twine ``` - **Docker** with the Compose plugin. - A **Docker Hub account** with push access to `robertmarik/tomotree`. --- ## Bumping the Version The version of the library lives in one place: ``` src/tomotree/__init__.py ``` ```python __version__ = "1.3.2" # ← change this ``` `pyproject.toml` reads the version dynamically from that attribute. The web GUI has its own version in `src/tomotree/streamlit_app/__init__.py` (shown as `gui-v…` in the About tab); bump it separately when the GUI changes. Releases of the library are normally prepared by **release-please** (`.github/workflows/release-please.yml`): after every push to `main` it opens or updates a release pull request that bumps `src/tomotree/__init__.py` and `CHANGELOG.md` according to the conventional-commit messages (`feat:`, `fix:`, …). Merging that pull request creates the tag and the GitHub release. To bump the version manually instead, commit the change and tag the release in Git: ```bash git add src/tomotree/__init__.py git commit -m "Bump version to 1.2.0" git tag v1.2.0 git push origin main --tags ``` --- ## Building and Publishing the Python Package ### Publish to GitHub (recommended) The simplest distribution method is to push the tagged commit. Users install directly from the repository without any package index: ```bash # SSH (requires SSH key authorised for the repo) pip install git+ssh://git@github.com/robert-marik/tomotree.git # HTTPS (no SSH key required) pip install git+https://github.com/robert-marik/tomotree.git # Pin a specific tag pip install git+https://github.com/robert-marik/tomotree.git@v1.2.0 ``` No additional build step is required. Once the tag is pushed, users can install that exact version. **Creating a GitHub Release (optional but recommended):** Go to the repository page on GitHub → **Releases** → **Draft a new release**, select the tag you just pushed, add release notes, and publish. This makes the release visible in the GitHub UI and allows users to download a source archive. ### Publish to PyPI Use this method if you want users to be able to install with a plain `pip install tomotree`. #### 1. Build the distribution archives From the repository root: ```bash python -m build ``` This produces two files in `dist/`: ``` dist/ tomotree-1.2.0-py3-none-any.whl tomotree-1.2.0.tar.gz ``` #### 2. Upload to PyPI ```bash python -m twine upload dist/* ``` You will be prompted for your PyPI username and password (or an API token — recommended). To use a token, create `~/.pypirc`: ```ini [pypi] username = __token__ password = pypi-YOUR_API_TOKEN_HERE ``` #### 3. Upload to TestPyPI first (optional) ```bash python -m twine upload --repository testpypi dist/* pip install --index-url https://test.pypi.org/simple/ tomotree ``` --- ## Building and Pushing the Docker Image One `docker/Dockerfile` with three targets (`python:3.12-slim`, non-root): `lib` (the library), `jupyter` (lib + JupyterLab and the demos) and `gui` (lib + the Streamlit app). User documentation: [docker/README.md](../docker/README.md). ### 1. Build the images From the repository root: ```bash make docker-build # tomotree:lib, tomotree:jupyter, tomotree:latest (GUI) ``` Before a release, skip the layer cache: `docker builder prune -f && make docker-build`. The build context is limited by `.dockerignore` (a whitelist), so local datasets and secrets never enter an image. ### 2. Tag for Docker Hub The images are not published yet. When they are: ```bash VERSION=$(python -c "import tomotree; print(tomotree.__version__)") for tag in latest lib jupyter; do docker tag tomotree:$tag robertmarik/tomotree:$tag done docker tag tomotree:latest robertmarik/tomotree:$VERSION ``` ### 3. Push to Docker Hub ```bash docker login # enter Docker Hub credentials once for tag in latest lib jupyter $VERSION; do docker push robertmarik/tomotree:$tag; done ``` The server then uses them with `TOMOTREE_IMAGE=robertmarik/tomotree` in its `.env`. --- ## Updating a Production Server The production server runs the container defined by a `compose.yml` file in the deployment directory. It mounts persistent data directories as volumes so that user data, shared datasets, configuration, and logs survive container restarts. ### Typical deployment directory layout ``` /srv/tomotree/ ├── compose.yml # docker/compose.yml of the repository ├── .env # TOMOTREE_DIR=., TOMOTREE_UID, TOMOTREE_GID ├── .streamlit/ # secrets.toml (OIDC) ├── streamlit_data/ # preloaded datasets (read-only for users) ├── streamlit_user/ # per-user data: // ├── streamlit_shared/ # shared dataset symlinks ├── streamlit_session/ # automatic session saves ├── config/ │ ├── config.yaml │ └── login.yaml └── log/ ``` Inside a dataset, every kind of data has its own subdirectory (`tomogram/`, `backgrounds/`, `resistograph/`, `scans/`, `photos/`, `states/`); older datasets with all files in the dataset root keep working. See [Where Your Data Are Stored](user_guide.md#where-your-data-are-stored) in the User Guide. Storage quotas are counted over `streamlit_user//`. ### compose.yml on the server Use the `docker/compose.yml` of the repository unchanged (the same file runs in the repository and on the server). Copy it to the deployment directory and create `.env` next to it: ```bash cd /srv/tomotree mkdir -p streamlit_data streamlit_user streamlit_shared streamlit_session config log .streamlit cat > .env <> .env # or the UID of a dedicated service account echo "TOMOTREE_GID=$(id -g)" >> .env docker compose up -d docker compose exec tomotree id # shows the UID/GID the app runs as ``` Docker Compose reads `.env` from the directory of `compose.yml`. Do not use `$UID`: bash does not export it, so Compose would not see it. Without Compose use `docker run --user "$(id -u):$(id -g)" ...`. The image works with any UID, no rebuild is needed (`HOME` is `/tmp`, the only directories written inside the image, the data, config and log directories, are writable for everyone; a mounted volume has the owner and permissions of the host directory). **Upgrading from an image that ran as root.** Files created by the old container belong to root. Give them to the chosen user once, otherwise the app cannot write there (`setup_check` reports it): ```bash cd /srv/tomotree docker compose down sudo chown -R "$(id -u):$(id -g)" streamlit_data streamlit_user streamlit_shared \ streamlit_session jupyter_notebooks config log .streamlit docker compose up -d ``` A `secrets.toml` mounted separately (e.g. into `/root/.streamlit/`) now belongs to `.streamlit/secrets.toml` of the deployment directory (mounted to `/app/.streamlit/`). ### Pulling the latest image and restarting With a locally built image (current state): build it on the server from a clone of the repository (`git pull && make docker-build`), or copy it from the build machine with `docker save tomotree:latest | ssh server docker load`. Then: ```bash cd /srv/tomotree docker compose up -d # recreates the container when the image changed ``` Once the images are on Docker Hub: `docker compose pull && docker compose up -d`. ### Converting datasets to the current format Datasets created by older versions keep working, but they lack the data sources in `config.yaml` (`sources`, `active`), may still have `tomogram/config.yaml` and keep all files in the dataset root. Convert them after an upgrade, inside the running container. The datasets are on mounted volumes, so back them up first; neither step can be undone automatically. ```bash cd /srv/tomotree tar czf datasets-backup.tgz streamlit_data streamlit_user # 1. Data sources: config.yaml to the dataset root, sources and active added docker compose exec tomotree sh -c 'python -m tomotree.importers migrate --dry-run /app/streamlit_data/*/ /app/streamlit_user/*/*/' docker compose exec tomotree sh -c 'python -m tomotree.importers migrate /app/streamlit_data/*/ /app/streamlit_user/*/*/' # 2. Layout: files from the dataset root to their subdirectories, sources follow them docker compose exec tomotree sh -c 'python -m tomotree.importers layout /app/streamlit_data/*/ /app/streamlit_user/*/*/' # 3. Check: every dataset should report OK docker compose exec tomotree sh -c 'python -m tomotree.importers check /app/streamlit_data/*/ /app/streamlit_user/*/*/' ``` - `python -m tomotree.importers convert ...` does steps 1 and 2 at once. A single dataset can also be converted in the GUI: the sidebar expander *Data Info & Sources* shows the button *Move all to subdirectories* (with write permission) when files lie in the dataset root. - Run the steps in this order. `migrate` skips datasets that already have `sources`, `layout` prints *nothing to move* for converted datasets, so both can be repeated safely. - The `sh -c '...'` is needed: the `*` must be expanded inside the container, not on the host. - `layout` moves only files it recognizes (see [Where Your Data Are Stored](user_guide.md#where-your-data-are-stored)): tomograph files and `nodes`/`distances`/`speeds`/`velocities`/`times` CSV files to `tomogram/`, images except `thumbnail.png` to `backgrounds/`, `*.dpa` to `resistograph/`, 3D scans with their textures to `scans/`, `*.state` to `states/`. `config.yaml`, `thumbnail.png`, readme files, `keywords.txt`, `scan_geometry.csv` and unknown files stay in the root. A file already present in the subdirectory is not overwritten and stays in the root. - Shared datasets (`streamlit_shared/`) are links to user datasets and need no conversion. - Outside Docker the same commands run as `python -m tomotree.importers ...` (`make migrate-datasets DATASETS="..."`, `make check-datasets`). ### Checking the running container ```bash # List running containers docker compose ps # Follow live logs docker compose logs -f # Show the last 100 log lines docker compose logs --tail=100 ``` ### Rolling back to a previous version Keep the previous image under a tag before an upgrade (`docker tag tomotree:latest tomotree:previous`). To go back, tag it as `latest` again and recreate the container: ```bash docker tag tomotree:previous tomotree:latest docker compose up -d ``` With Docker Hub images use a version tag instead: `docker tag robertmarik/tomotree:1.1.0 robertmarik/tomotree:latest`. --- ## Configuration Files on the Server All persistent configuration lives in the `config/` volume mount and is **not** replaced by image updates. | File | Purpose | |------|---------| | `config/config.yaml` | Main application settings (directories, hostname, passphrase, storage quotas, …) | | `config/login.yaml` | User accounts and password hashes | | `config/state_secret` | Key for signing saved states and session files (created automatically, keep it with the backup) | | `config/titlepage.txt` | Markdown text shown on the login page | | `config/side_note.txt` | Warning box shown in the sidebar on every page | | `config/whitelist.txt` | (optional) Allow-list: only the users listed here can log in | | `config/blacklist.txt` | (optional) Block-list: users listed here are refused access | | `config/account_expiry.yaml` | (optional) Date until which an account may be used | `config.yaml` is read again on every page interaction (the Docker image starts the application without an explicit `Config` object), so most changes take effect without a restart. After editing `login.yaml`, or when in doubt, **restart the container**: ```bash docker compose restart tomotree ``` ### Editing in the admin console The files above (and the `extra_editable_files` of `config.yaml`) can be edited in the Admin console, tab **Configuration Files**. A change is checked before it is saved, and nothing is written when the check fails: - `config.yaml` and `login.yaml` are checked on a temporary copy by the same checks as `make check-config`. Only problems the saved file does not have already block the change, so an older problem does not stop the fix of another one. Unknown settings are a warning that needs **Save anyway**. In `login.yaml` every user needs a `password` and only known roles. - The account expiry file must contain valid dates; YAML and JSON files must parse; whitelist and blacklist are plain lists; no file may exceed 1 MB. - **Lockout:** a change that would lock out the admin who saves it is refused: being put on the blacklist or left off the whitelist, an expired account, losing the `admin`/`developer` role or the account in `login.yaml` (password login), a removed OIDC provider or role in `oidc_providers` (OIDC login), a `login_file` that does not exist. Such a change must be made by another admin or on the server. Each save keeps the previous content in `.backups/` next to the file (the last 10 versions; the directory is readable only by the app user, since backups of `login.yaml` hold password hashes) and replaces the file atomically. **Previous versions** in the editor shows and restores them (a restore is checked the same way and is itself backed up). When a broken file keeps everybody out, restore from the server: ```bash make restore-config FILE=config/login.yaml # list the backups make restore-config FILE=config/login.yaml RESTORE=1 # restore the newest docker compose exec tomotree python -m tomotree.streamlit_app.config_files config/login.yaml --restore ``` Files changed outside the admin console (editor on the server, file manager) are not checked; run `make check-config` after such a change. > **Security reminder:** Set a strong, unique `passphrase` value in `config.yaml` before going live. This key is used to encrypt secret sharing links. The app refuses to start with the template `passphrase`, the template `cookie_secret` in `.streamlit/secrets.toml` (it signs the OIDC login cookie), the template `cookie.key` or template passwords in `login.yaml` (fix: `setup_check --fix-login`, which replaces all of them by random values). A `passphrase` or `cookie_secret` shorter than 32 characters is a warning. ### Signed saved states Saved states (`*.state`) and session files (`session_*.pkl`) are Python pickles, and loading a pickle runs the code it contains. The app therefore signs each such file with HMAC-SHA256 and loads only files signed with its key; a state uploaded in a ZIP or edited in the file manager is refused with a message. The key is `state_secret` in `config.yaml` (hex string, at least 64 characters) or, by default, the file `config/state_secret`. The file is created with a random key on the first start (or by `make init-config`), so no setup is needed. Keep it with the backup of `config/`; installations that share data directories must use the same key file. States saved before the signing was introduced, or after the key changed, are refused until an administrator signs them. The command never unpickles them, but a signed file is trusted by the app. > **Warning:** Users can upload `*.state` files into their directories, and a crafted file runs its code on the server (in Docker as root) once it is signed. Run `make sign-states` only if you are 100% sure that none of the files has been compromised; otherwise sign only chosen directories or files (`STATES="..."`) after checking them. Before signing unsigned files the command lists them, prints this warning and asks you to type `yes` (`--yes` skips the question, e.g. in scripts). ```bash make sign-states SIGN_FLAGS=--check # list files the app would refuse make sign-states # sign dirname, userdir and shareddir of config.yaml make sign-states STATES="streamlit_data/oak" SIGN_FLAGS=--dry-run python -m tomotree.streamlit_app.sign_states --old-key-file old_state_secret # after a key change python -m tomotree.streamlit_app.sign_states --include-sessions # also session files ``` In Docker run it inside the container: `docker compose exec tomotree python -m tomotree.streamlit_app.sign_states`. ### Temporary uploads "Upload temporary data" stores each upload in its own directory `tomotree_upload_*` in the system temp directory (in Docker inside the container). Files over 10 MB are refused, and so is an upload of more than 100 files or more than 200 MB in total, ZIP archives counted as extracted (several ZIPs uploaded together count together, measured before anything is written). A new upload of a session replaces its previous one, uploads older than 24 hours are deleted, and all uploads together may not exceed 2 GB (`src/tomotree/streamlit/files/temp_uploads.py`). Write access applies only while the upload is the open dataset; "Save uploaded data" copies it to the user's directory, where the storage quota applies. ### Installation check `make check-config` (`python -m tomotree.streamlit_app.setup_check`) lists configuration problems with the commands that fix them; `make init-config` creates the missing files from the `*.dist` templates with random secrets (never overwrites). In Docker run it inside the container, e.g. `docker compose exec tomotree python -m tomotree.streamlit_app.setup_check`. The app runs the same check (`setup_check.py`, shown by `setup_report.py`) and creates the missing runtime directories at start: - **Errors** (no login method, unreadable `config.yaml`, `login.yaml` or `secrets.toml`, an OIDC provider listed in `oidc_providers` without the `[auth]` / `[auth.]` keys, a provider named `password` (reserved), `trust_email` of a provider that is not `true`/`false`, the public template `passphrase` in `config.yaml` or `cookie_secret` in `secrets.toml`, the public template `cookie.key` or the template accounts `demo`/`password` and `guest`/`Tomotree` with unchanged passwords in `login.yaml`) stop the app before login with a *Setup required* page. The page shows how to fix each error; for the template login run `docker compose exec tomotree python -m tomotree.streamlit_app.setup_check --fix-login`, which sets a random cookie key and random passwords and prints the passwords once. - **Warnings and notes** are shown only to admins and developers: collapsed in the sidebar (*Installation check: N warning(s)*), in the Admin console tab **Installation check** and summarized in the Health details. They cover a missing `config.yaml` (defaults used), unknown keys with a did-you-mean hint, removed OIDC keys (`authenticated_by_google`, `oauth_roles`, ...) with a pointer to `oidc_providers`, a `server_metadata_url` that is not an OIDC discovery address (`/.well-known/openid-configuration`), a `passphrase` or `cookie_secret` shorter than 32 characters, a provider with `trust_email: true` (accepts e-mail addresses the provider did not verify), a missing or empty data directory, non-writable directories, an unused `config/oauth_users.yaml` and an app not started from the project root. `allow_no_auth: true` in `config.yaml` lets everybody in as `developer` when no login method is configured and shows a red banner on every page. Never enable it on a server. ### Whitelist and Blacklist After a user successfully authenticates, TomoTree performs a second access check using optional allow-list and block-list files. The blacklist is evaluated first and always wins; then the whitelist; then the [account expiry](#account-expiry). **Whitelist** — if `config/whitelist.txt` exists and is referenced by `whitelist_file` in `config.yaml`, only the users listed in that file are allowed in. Everyone else is shown `whitelist_message` and the session is stopped. Use this to restrict access to a closed group of users. **Blacklist** — if `config/blacklist.txt` exists and is referenced by `blacklist_file` in `config.yaml`, any user whose name or e-mail appears in that file is blocked. A blocked user sees `blacklist_message` and the session is stopped. Both files use the same plain-text format: one username or e-mail address per line. Empty lines and lines starting with `#` are ignored. Matching is **case-insensitive**. Example `config/whitelist.txt`: ``` # Allowed users alice@example.com bob@example.com ``` The corresponding `config.yaml` entries (uncomment and adjust as needed): ```yaml whitelist_file: config/whitelist.txt whitelist_message: "You are not on the list of authorised users. Please contact the administrator." blacklist_file: config/blacklist.txt blacklist_message: "Your account has been blocked. Please contact the administrator." ``` The files are re-read at most every 30 seconds (cache `user_lists`), so changes take effect without restarting the container. ### Account expiry An account can be limited in time, independently of the whitelist and blacklist and equally for password and OIDC accounts. `account_expiry_file` in `config.yaml` names a YAML file that maps a username or e-mail address (case-insensitive) to the moment the account stops working: ```yaml student@example.com: 2026-12-31 # a date alone: the whole day is still valid guest: 2026-12-15 18:00 # valid until 18:00, local time of the server tester: 2026-12-15T18:00:00+01:00 # with an explicit time zone ``` Accounts that are not listed have no time limit. For a password account both the username and the e-mail are looked up; with two entries the earlier one applies. Order of the checks after login: 1. **Blacklist** — a listed user is refused, whatever else applies. 2. **Whitelist** — if active, a user who is not on it is refused. 3. **Account expiry** — an expired account is refused, also when the user is on the whitelist. The user sees `account_expired_message` (`{date}` is replaced by the last valid day or time). 4. **Maintenance mode**. The check runs on every interaction, so an account that expires while the user works is stopped at the next click (a change of the file takes up to 30 seconds). An entry whose value is not a valid date refuses that account; a file that cannot be parsed refuses everybody except admins until it is fixed. `make check-config` reports both. The file can be edited in the admin console (Configuration files → Account expiry). ```yaml account_expiry_file: config/account_expiry.yaml account_expired_message: "Your account was valid until {date}. Please contact the administrator." ``` --- ## Cleaning Up Old Docker Images and Files After pulling and starting a new image version, the old image layers remain on disk and can accumulate significant storage over time. Use the following commands to reclaim space. ### Remove dangling images (untagged leftovers) Dangling images are layers that are no longer referenced by any tag — these are the most common leftovers after a `docker compose build` or `docker pull`: ```bash docker image prune ``` ### Remove all unused images (including older tagged versions) This removes every image that is not referenced by a running or stopped container: ```bash docker image prune -a ``` > **Warning:** This also removes any locally built images you have not pushed yet. Run only when you are sure all valuable images are already pushed to Docker Hub. ### Remove stopped containers, unused networks, dangling images, and build cache at once ```bash docker system prune ``` To also include unused volumes (⚠ this will delete volume data not mounted by any container): ```bash docker system prune --volumes ``` ### Remove a specific old image by tag If you want to keep recent images but drop a specific version: ```bash docker rmi robertmarik/tomotree:1.1.0 ``` ### Typical post-upgrade cleanup sequence ```bash # 1. New image and restart (see "Updating a Production Server" above) cd /srv/tomotree docker compose up -d # 2. Remove dangling layers left by the pull docker image prune -f # 3. (Optional) Remove all images not in use by a running container docker image prune -a -f ``` The `-f` flag skips the confirmation prompt, which is useful in scripts. ### Check disk usage before and after ```bash docker system df ``` This shows how much space is used by images, containers, volumes, and build cache — and how much can be reclaimed.