Administrator Guide — Building and Deploying TomoTree

This guide is intended for maintainers who need to build a new release, publish the Python package, build and push a Docker image, and update a running production server.


Table of Contents


Prerequisites

Your local machine needs:

  • Git with push access to the repository.

  • Python ≥ 3.9 with build and twine installed (for PyPI releases):

    pip install build twine
    
  • Docker with the Compose plugin.

  • A Docker Hub account with push access to robertmarik/tomotree.


Bumping the Version

The version of the library lives in one place:

src/tomotree/__init__.py
__version__ = "1.3.2"   # ← change this

pyproject.toml reads the version dynamically from that attribute. The web GUI has its own version in src/tomotree/streamlit_app/__init__.py (shown as gui-v… in the About tab); bump it separately when the GUI changes.

Releases of the library are normally prepared by release-please (.github/workflows/release-please.yml): after every push to main it opens or updates a release pull request that bumps src/tomotree/__init__.py and CHANGELOG.md according to the conventional-commit messages (feat:, fix:, …). Merging that pull request creates the tag and the GitHub release.

To bump the version manually instead, commit the change and tag the release in Git:

git add src/tomotree/__init__.py
git commit -m "Bump version to 1.2.0"
git tag v1.2.0
git push origin main --tags

Building and Publishing the Python Package

Publish to PyPI

Use this method if you want users to be able to install with a plain pip install tomotree.

1. Build the distribution archives

From the repository root:

python -m build

This produces two files in dist/:

dist/
  tomotree-1.2.0-py3-none-any.whl
  tomotree-1.2.0.tar.gz

2. Upload to PyPI

python -m twine upload dist/*

You will be prompted for your PyPI username and password (or an API token — recommended). To use a token, create ~/.pypirc:

[pypi]
  username = __token__
  password = pypi-YOUR_API_TOKEN_HERE

3. Upload to TestPyPI first (optional)

python -m twine upload --repository testpypi dist/*
pip install --index-url https://test.pypi.org/simple/ tomotree

Building and Pushing the Docker Image

One docker/Dockerfile with three targets (python:3.12-slim, non-root): lib (the library), jupyter (lib + JupyterLab and the demos) and gui (lib + the Streamlit app). User documentation: docker/README.md.

1. Build the images

From the repository root:

make docker-build      # tomotree:lib, tomotree:jupyter, tomotree:latest (GUI)

Before a release, skip the layer cache: docker builder prune -f && make docker-build. The build context is limited by .dockerignore (a whitelist), so local datasets and secrets never enter an image.

2. Tag for Docker Hub

The images are not published yet. When they are:

VERSION=$(python -c "import tomotree; print(tomotree.__version__)")
for tag in latest lib jupyter; do
    docker tag tomotree:$tag robertmarik/tomotree:$tag
done
docker tag tomotree:latest robertmarik/tomotree:$VERSION

3. Push to Docker Hub

docker login          # enter Docker Hub credentials once
for tag in latest lib jupyter $VERSION; do docker push robertmarik/tomotree:$tag; done

The server then uses them with TOMOTREE_IMAGE=robertmarik/tomotree in its .env.


Updating a Production Server

The production server runs the container defined by a compose.yml file in the deployment directory. It mounts persistent data directories as volumes so that user data, shared datasets, configuration, and logs survive container restarts.

Typical deployment directory layout

/srv/tomotree/
├── compose.yml         # docker/compose.yml of the repository
├── .env                # TOMOTREE_DIR=., TOMOTREE_UID, TOMOTREE_GID
├── .streamlit/         # secrets.toml (OIDC)
├── streamlit_data/      # preloaded datasets (read-only for users)
├── streamlit_user/      # per-user data: <username>/<dataset>/
├── streamlit_shared/    # shared dataset symlinks
├── streamlit_session/   # automatic session saves
├── config/
│   ├── config.yaml
│   └── login.yaml
└── log/

Inside a dataset, every kind of data has its own subdirectory (tomogram/, backgrounds/, resistograph/, scans/, photos/, states/); older datasets with all files in the dataset root keep working. See Where Your Data Are Stored in the User Guide. Storage quotas are counted over streamlit_user/<username>/.

compose.yml on the server

Use the docker/compose.yml of the repository unchanged (the same file runs in the repository and on the server). Copy it to the deployment directory and create .env next to it:

cd /srv/tomotree
mkdir -p streamlit_data streamlit_user streamlit_shared streamlit_session config log .streamlit
cat > .env <<EOF
TOMOTREE_DIR=.
TOMOTREE_UID=$(id -u)
TOMOTREE_GID=$(id -g)
# TOMOTREE_IMAGE=robertmarik/tomotree   # once the images are on Docker Hub
# TOMOTREE_MEM_LIMIT=8g
EOF
docker compose run --rm tomotree python -m tomotree.streamlit_app.setup_check --init   # first time only
docker compose up -d

.streamlit/ holds secrets.toml (OIDC). The port listens on 127.0.0.1:8501; put a reverse proxy with HTTPS in front of it (TOMOTREE_BIND=0.0.0.0 exposes it without encryption). The container runs with cap_drop: ALL, no-new-privileges, pids_limit and mem_limit, and restarts automatically (restart: unless-stopped).

Running as your user (not root)

The container does not run as root. It runs as the UID:GID given by user: in compose.yml (default 1000:1000), so the files the app writes to the mounted directories (datasets, exports, logs, config/) belong to that user on the host and can be read, copied and deleted without sudo. Choose the owner of the deployment directory:

cd /srv/tomotree
echo "TOMOTREE_UID=$(id -u)" >> .env    # or the UID of a dedicated service account
echo "TOMOTREE_GID=$(id -g)" >> .env
docker compose up -d
docker compose exec tomotree id         # shows the UID/GID the app runs as

Docker Compose reads .env from the directory of compose.yml. Do not use $UID: bash does not export it, so Compose would not see it. Without Compose use docker run --user "$(id -u):$(id -g)" .... The image works with any UID, no rebuild is needed (HOME is /tmp, the only directories written inside the image, the data, config and log directories, are writable for everyone; a mounted volume has the owner and permissions of the host directory).

Upgrading from an image that ran as root. Files created by the old container belong to root. Give them to the chosen user once, otherwise the app cannot write there (setup_check reports it):

cd /srv/tomotree
docker compose down
sudo chown -R "$(id -u):$(id -g)" streamlit_data streamlit_user streamlit_shared \
    streamlit_session jupyter_notebooks config log .streamlit
docker compose up -d

A secrets.toml mounted separately (e.g. into /root/.streamlit/) now belongs to .streamlit/secrets.toml of the deployment directory (mounted to /app/.streamlit/).

Pulling the latest image and restarting

With a locally built image (current state): build it on the server from a clone of the repository (git pull && make docker-build), or copy it from the build machine with docker save tomotree:latest | ssh server docker load. Then:

cd /srv/tomotree
docker compose up -d          # recreates the container when the image changed

Once the images are on Docker Hub: docker compose pull && docker compose up -d.

Converting datasets to the current format

Datasets created by older versions keep working, but they lack the data sources in config.yaml (sources, active), may still have tomogram/config.yaml and keep all files in the dataset root. Convert them after an upgrade, inside the running container. The datasets are on mounted volumes, so back them up first; neither step can be undone automatically.

cd /srv/tomotree
tar czf datasets-backup.tgz streamlit_data streamlit_user

# 1. Data sources: config.yaml to the dataset root, sources and active added
docker compose exec tomotree sh -c 'python -m tomotree.importers migrate --dry-run /app/streamlit_data/*/ /app/streamlit_user/*/*/'
docker compose exec tomotree sh -c 'python -m tomotree.importers migrate /app/streamlit_data/*/ /app/streamlit_user/*/*/'

# 2. Layout: files from the dataset root to their subdirectories, sources follow them
docker compose exec tomotree sh -c 'python -m tomotree.importers layout /app/streamlit_data/*/ /app/streamlit_user/*/*/'

# 3. Check: every dataset should report OK
docker compose exec tomotree sh -c 'python -m tomotree.importers check /app/streamlit_data/*/ /app/streamlit_user/*/*/'
  • python -m tomotree.importers convert <dataset>... does steps 1 and 2 at once. A single dataset can also be converted in the GUI: the sidebar expander Data Info & Sources shows the button Move all to subdirectories (with write permission) when files lie in the dataset root.

  • Run the steps in this order. migrate skips datasets that already have sources, layout prints nothing to move for converted datasets, so both can be repeated safely.

  • The sh -c '...' is needed: the * must be expanded inside the container, not on the host.

  • layout moves only files it recognizes (see Where Your Data Are Stored): tomograph files and nodes/distances/speeds/velocities/times CSV files to tomogram/, images except thumbnail.png to backgrounds/, *.dpa to resistograph/, 3D scans with their textures to scans/, *.state to states/. config.yaml, thumbnail.png, readme files, keywords.txt, scan_geometry.csv and unknown files stay in the root. A file already present in the subdirectory is not overwritten and stays in the root.

  • Shared datasets (streamlit_shared/) are links to user datasets and need no conversion.

  • Outside Docker the same commands run as python -m tomotree.importers ... (make migrate-datasets DATASETS="...", make check-datasets).

Checking the running container

# List running containers
docker compose ps

# Follow live logs
docker compose logs -f

# Show the last 100 log lines
docker compose logs --tail=100

Rolling back to a previous version

Keep the previous image under a tag before an upgrade (docker tag tomotree:latest tomotree:previous). To go back, tag it as latest again and recreate the container:

docker tag tomotree:previous tomotree:latest
docker compose up -d

With Docker Hub images use a version tag instead: docker tag robertmarik/tomotree:1.1.0 robertmarik/tomotree:latest.


Configuration Files on the Server

All persistent configuration lives in the config/ volume mount and is not replaced by image updates.

File

Purpose

config/config.yaml

Main application settings (directories, hostname, passphrase, storage quotas, …)

config/login.yaml

User accounts and password hashes

config/state_secret

Key for signing saved states and session files (created automatically, keep it with the backup)

config/titlepage.txt

Markdown text shown on the login page

config/side_note.txt

Warning box shown in the sidebar on every page

config/whitelist.txt

(optional) Allow-list: only the users listed here can log in

config/blacklist.txt

(optional) Block-list: users listed here are refused access

config/account_expiry.yaml

(optional) Date until which an account may be used

config.yaml is read again on every page interaction (the Docker image starts the application without an explicit Config object), so most changes take effect without a restart. After editing login.yaml, or when in doubt, restart the container:

docker compose restart tomotree

Editing in the admin console

The files above (and the extra_editable_files of config.yaml) can be edited in the Admin console, tab Configuration Files. A change is checked before it is saved, and nothing is written when the check fails:

  • config.yaml and login.yaml are checked on a temporary copy by the same checks as make check-config. Only problems the saved file does not have already block the change, so an older problem does not stop the fix of another one. Unknown settings are a warning that needs Save anyway. In login.yaml every user needs a password and only known roles.

  • The account expiry file must contain valid dates; YAML and JSON files must parse; whitelist and blacklist are plain lists; no file may exceed 1 MB.

  • Lockout: a change that would lock out the admin who saves it is refused: being put on the blacklist or left off the whitelist, an expired account, losing the admin/developer role or the account in login.yaml (password login), a removed OIDC provider or role in oidc_providers (OIDC login), a login_file that does not exist. Such a change must be made by another admin or on the server.

Each save keeps the previous content in .backups/ next to the file (the last 10 versions; the directory is readable only by the app user, since backups of login.yaml hold password hashes) and replaces the file atomically. Previous versions in the editor shows and restores them (a restore is checked the same way and is itself backed up). When a broken file keeps everybody out, restore from the server:

make restore-config FILE=config/login.yaml              # list the backups
make restore-config FILE=config/login.yaml RESTORE=1    # restore the newest
docker compose exec tomotree python -m tomotree.streamlit_app.config_files config/login.yaml --restore

Files changed outside the admin console (editor on the server, file manager) are not checked; run make check-config after such a change.

Security reminder: Set a strong, unique passphrase value in config.yaml before going live. This key is used to encrypt secret sharing links. The app refuses to start with the template passphrase, the template cookie_secret in .streamlit/secrets.toml (it signs the OIDC login cookie), the template cookie.key or template passwords in login.yaml (fix: setup_check --fix-login, which replaces all of them by random values). A passphrase or cookie_secret shorter than 32 characters is a warning.

Signed saved states

Saved states (*.state) and session files (session_*.pkl) are Python pickles, and loading a pickle runs the code it contains. The app therefore signs each such file with HMAC-SHA256 and loads only files signed with its key; a state uploaded in a ZIP or edited in the file manager is refused with a message.

The key is state_secret in config.yaml (hex string, at least 64 characters) or, by default, the file config/state_secret. The file is created with a random key on the first start (or by make init-config), so no setup is needed. Keep it with the backup of config/; installations that share data directories must use the same key file.

States saved before the signing was introduced, or after the key changed, are refused until an administrator signs them. The command never unpickles them, but a signed file is trusted by the app.

Warning: Users can upload *.state files into their directories, and a crafted file runs its code on the server (in Docker as root) once it is signed. Run make sign-states only if you are 100% sure that none of the files has been compromised; otherwise sign only chosen directories or files (STATES="...") after checking them. Before signing unsigned files the command lists them, prints this warning and asks you to type yes (--yes skips the question, e.g. in scripts).

make sign-states SIGN_FLAGS=--check     # list files the app would refuse
make sign-states                         # sign dirname, userdir and shareddir of config.yaml
make sign-states STATES="streamlit_data/oak" SIGN_FLAGS=--dry-run
python -m tomotree.streamlit_app.sign_states --old-key-file old_state_secret   # after a key change
python -m tomotree.streamlit_app.sign_states --include-sessions                # also session files

In Docker run it inside the container: docker compose exec tomotree python -m tomotree.streamlit_app.sign_states.

Temporary uploads

“Upload temporary data” stores each upload in its own directory tomotree_upload_* in the system temp directory (in Docker inside the container). Files over 10 MB are refused, and so is an upload of more than 100 files or more than 200 MB in total, ZIP archives counted as extracted (several ZIPs uploaded together count together, measured before anything is written). A new upload of a session replaces its previous one, uploads older than 24 hours are deleted, and all uploads together may not exceed 2 GB (src/tomotree/streamlit/files/temp_uploads.py). Write access applies only while the upload is the open dataset; “Save uploaded data” copies it to the user’s directory, where the storage quota applies.

Installation check

make check-config (python -m tomotree.streamlit_app.setup_check) lists configuration problems with the commands that fix them; make init-config creates the missing files from the *.dist templates with random secrets (never overwrites). In Docker run it inside the container, e.g. docker compose exec tomotree python -m tomotree.streamlit_app.setup_check.

The app runs the same check (setup_check.py, shown by setup_report.py) and creates the missing runtime directories at start:

  • Errors (no login method, unreadable config.yaml, login.yaml or secrets.toml, an OIDC provider listed in oidc_providers without the [auth] / [auth.<name>] keys, a provider named password (reserved), trust_email of a provider that is not true/false, the public template passphrase in config.yaml or cookie_secret in secrets.toml, the public template cookie.key or the template accounts demo/password and guest/Tomotree with unchanged passwords in login.yaml) stop the app before login with a Setup required page. The page shows how to fix each error; for the template login run docker compose exec tomotree python -m tomotree.streamlit_app.setup_check --fix-login, which sets a random cookie key and random passwords and prints the passwords once.

  • Warnings and notes are shown only to admins and developers: collapsed in the sidebar (Installation check: N warning(s)), in the Admin console tab Installation check and summarized in the Health details. They cover a missing config.yaml (defaults used), unknown keys with a did-you-mean hint, removed OIDC keys (authenticated_by_google, oauth_roles, …) with a pointer to oidc_providers, a server_metadata_url that is not an OIDC discovery address (/.well-known/openid-configuration), a passphrase or cookie_secret shorter than 32 characters, a provider with trust_email: true (accepts e-mail addresses the provider did not verify), a missing or empty data directory, non-writable directories, an unused config/oauth_users.yaml and an app not started from the project root.

allow_no_auth: true in config.yaml lets everybody in as developer when no login method is configured and shows a red banner on every page. Never enable it on a server.

Whitelist and Blacklist

After a user successfully authenticates, TomoTree performs a second access check using optional allow-list and block-list files. The blacklist is evaluated first and always wins; then the whitelist; then the account expiry.

Whitelist — if config/whitelist.txt exists and is referenced by whitelist_file in config.yaml, only the users listed in that file are allowed in. Everyone else is shown whitelist_message and the session is stopped. Use this to restrict access to a closed group of users.

Blacklist — if config/blacklist.txt exists and is referenced by blacklist_file in config.yaml, any user whose name or e-mail appears in that file is blocked. A blocked user sees blacklist_message and the session is stopped.

Both files use the same plain-text format: one username or e-mail address per line. Empty lines and lines starting with # are ignored. Matching is case-insensitive.

Example config/whitelist.txt:

# Allowed users
alice@example.com
bob@example.com

The corresponding config.yaml entries (uncomment and adjust as needed):

whitelist_file: config/whitelist.txt
whitelist_message: "You are not on the list of authorised users. Please contact the administrator."

blacklist_file: config/blacklist.txt
blacklist_message: "Your account has been blocked. Please contact the administrator."

The files are re-read at most every 30 seconds (cache user_lists), so changes take effect without restarting the container.

Account expiry

An account can be limited in time, independently of the whitelist and blacklist and equally for password and OIDC accounts. account_expiry_file in config.yaml names a YAML file that maps a username or e-mail address (case-insensitive) to the moment the account stops working:

student@example.com: 2026-12-31          # a date alone: the whole day is still valid
guest: 2026-12-15 18:00                  # valid until 18:00, local time of the server
tester: 2026-12-15T18:00:00+01:00        # with an explicit time zone

Accounts that are not listed have no time limit. For a password account both the username and the e-mail are looked up; with two entries the earlier one applies.

Order of the checks after login:

  1. Blacklist — a listed user is refused, whatever else applies.

  2. Whitelist — if active, a user who is not on it is refused.

  3. Account expiry — an expired account is refused, also when the user is on the whitelist. The user sees account_expired_message ({date} is replaced by the last valid day or time).

  4. Maintenance mode.

The check runs on every interaction, so an account that expires while the user works is stopped at the next click (a change of the file takes up to 30 seconds). An entry whose value is not a valid date refuses that account; a file that cannot be parsed refuses everybody except admins until it is fixed. make check-config reports both. The file can be edited in the admin console (Configuration files → Account expiry).

account_expiry_file: config/account_expiry.yaml
account_expired_message: "Your account was valid until {date}. Please contact the administrator."

Cleaning Up Old Docker Images and Files

After pulling and starting a new image version, the old image layers remain on disk and can accumulate significant storage over time. Use the following commands to reclaim space.

Remove dangling images (untagged leftovers)

Dangling images are layers that are no longer referenced by any tag — these are the most common leftovers after a docker compose build or docker pull:

docker image prune

Remove all unused images (including older tagged versions)

This removes every image that is not referenced by a running or stopped container:

docker image prune -a

Warning: This also removes any locally built images you have not pushed yet. Run only when you are sure all valuable images are already pushed to Docker Hub.

Remove stopped containers, unused networks, dangling images, and build cache at once

docker system prune

To also include unused volumes (⚠ this will delete volume data not mounted by any container):

docker system prune --volumes

Remove a specific old image by tag

If you want to keep recent images but drop a specific version:

docker rmi robertmarik/tomotree:1.1.0

Typical post-upgrade cleanup sequence

# 1. New image and restart (see "Updating a Production Server" above)
cd /srv/tomotree
docker compose up -d

# 2. Remove dangling layers left by the pull
docker image prune -f

# 3. (Optional) Remove all images not in use by a running container
docker image prune -a -f

The -f flag skips the confirmation prompt, which is useful in scripts.

Check disk usage before and after

docker system df

This shows how much space is used by images, containers, volumes, and build cache — and how much can be reclaimed.