Administrator Guide — Building and Deploying TomoTree
This guide is intended for maintainers who need to build a new release, publish the Python package, build and push a Docker image, and update a running production server.
Table of Contents
Prerequisites
Your local machine needs:
Git with push access to the repository.
Python ≥ 3.9 with
buildandtwineinstalled (for PyPI releases):pip install build twine
Docker with the Compose plugin.
A Docker Hub account with push access to
robertmarik/tomotree.
Bumping the Version
The version of the library lives in one place:
src/tomotree/__init__.py
__version__ = "1.3.2" # ← change this
pyproject.toml reads the version dynamically from that attribute. The web GUI has its own version in src/tomotree/streamlit_app/__init__.py (shown as gui-v… in the About tab); bump it separately when the GUI changes.
Releases of the library are normally prepared by release-please (.github/workflows/release-please.yml): after every push to main it opens or updates a release pull request that bumps src/tomotree/__init__.py and CHANGELOG.md according to the conventional-commit messages (feat:, fix:, …). Merging that pull request creates the tag and the GitHub release.
To bump the version manually instead, commit the change and tag the release in Git:
git add src/tomotree/__init__.py
git commit -m "Bump version to 1.2.0"
git tag v1.2.0
git push origin main --tags
Building and Publishing the Python Package
Publish to GitHub (recommended)
The simplest distribution method is to push the tagged commit. Users install directly from the repository without any package index:
# SSH (requires SSH key authorised for the repo)
pip install git+ssh://git@github.com/robert-marik/tomotree.git
# HTTPS (no SSH key required)
pip install git+https://github.com/robert-marik/tomotree.git
# Pin a specific tag
pip install git+https://github.com/robert-marik/tomotree.git@v1.2.0
No additional build step is required. Once the tag is pushed, users can install that exact version.
Creating a GitHub Release (optional but recommended):
Go to the repository page on GitHub → Releases → Draft a new release, select the tag you just pushed, add release notes, and publish. This makes the release visible in the GitHub UI and allows users to download a source archive.
Publish to PyPI
Use this method if you want users to be able to install with a plain pip install tomotree.
1. Build the distribution archives
From the repository root:
python -m build
This produces two files in dist/:
dist/
tomotree-1.2.0-py3-none-any.whl
tomotree-1.2.0.tar.gz
2. Upload to PyPI
python -m twine upload dist/*
You will be prompted for your PyPI username and password (or an API token — recommended). To use a token, create ~/.pypirc:
[pypi]
username = __token__
password = pypi-YOUR_API_TOKEN_HERE
3. Upload to TestPyPI first (optional)
python -m twine upload --repository testpypi dist/*
pip install --index-url https://test.pypi.org/simple/ tomotree
Building and Pushing the Docker Image
One docker/Dockerfile with three targets (python:3.12-slim, non-root): lib (the library),
jupyter (lib + JupyterLab and the demos) and gui (lib + the Streamlit app). User documentation:
docker/README.md.
1. Build the images
From the repository root:
make docker-build # tomotree:lib, tomotree:jupyter, tomotree:latest (GUI)
Before a release, skip the layer cache: docker builder prune -f && make docker-build.
The build context is limited by .dockerignore (a whitelist), so local datasets and secrets never
enter an image.
2. Tag for Docker Hub
The images are not published yet. When they are:
VERSION=$(python -c "import tomotree; print(tomotree.__version__)")
for tag in latest lib jupyter; do
docker tag tomotree:$tag robertmarik/tomotree:$tag
done
docker tag tomotree:latest robertmarik/tomotree:$VERSION
3. Push to Docker Hub
docker login # enter Docker Hub credentials once
for tag in latest lib jupyter $VERSION; do docker push robertmarik/tomotree:$tag; done
The server then uses them with TOMOTREE_IMAGE=robertmarik/tomotree in its .env.
Updating a Production Server
The production server runs the container defined by a compose.yml file in the deployment directory. It mounts persistent data directories as volumes so that user data, shared datasets, configuration, and logs survive container restarts.
Typical deployment directory layout
/srv/tomotree/
├── compose.yml # docker/compose.yml of the repository
├── .env # TOMOTREE_DIR=., TOMOTREE_UID, TOMOTREE_GID
├── .streamlit/ # secrets.toml (OIDC)
├── streamlit_data/ # preloaded datasets (read-only for users)
├── streamlit_user/ # per-user data: <username>/<dataset>/
├── streamlit_shared/ # shared dataset symlinks
├── streamlit_session/ # automatic session saves
├── config/
│ ├── config.yaml
│ └── login.yaml
└── log/
Inside a dataset, every kind of data has its own subdirectory (tomogram/, backgrounds/, resistograph/, scans/, photos/, states/); older datasets with all files in the dataset root keep working. See Where Your Data Are Stored in the User Guide. Storage quotas are counted over streamlit_user/<username>/.
compose.yml on the server
Use the docker/compose.yml of the repository unchanged (the same file runs in the repository and
on the server). Copy it to the deployment directory and create .env next to it:
cd /srv/tomotree
mkdir -p streamlit_data streamlit_user streamlit_shared streamlit_session config log .streamlit
cat > .env <<EOF
TOMOTREE_DIR=.
TOMOTREE_UID=$(id -u)
TOMOTREE_GID=$(id -g)
# TOMOTREE_IMAGE=robertmarik/tomotree # once the images are on Docker Hub
# TOMOTREE_MEM_LIMIT=8g
EOF
docker compose run --rm tomotree python -m tomotree.streamlit_app.setup_check --init # first time only
docker compose up -d
.streamlit/ holds secrets.toml (OIDC). The port listens on 127.0.0.1:8501; put a reverse proxy
with HTTPS in front of it (TOMOTREE_BIND=0.0.0.0 exposes it without encryption). The container
runs with cap_drop: ALL, no-new-privileges, pids_limit and mem_limit, and restarts
automatically (restart: unless-stopped).
Running as your user (not root)
The container does not run as root. It runs as the UID:GID given by user: in compose.yml
(default 1000:1000), so the files the app writes to the mounted directories (datasets, exports,
logs, config/) belong to that user on the host and can be read, copied and deleted without sudo.
Choose the owner of the deployment directory:
cd /srv/tomotree
echo "TOMOTREE_UID=$(id -u)" >> .env # or the UID of a dedicated service account
echo "TOMOTREE_GID=$(id -g)" >> .env
docker compose up -d
docker compose exec tomotree id # shows the UID/GID the app runs as
Docker Compose reads .env from the directory of compose.yml. Do not use $UID: bash does not
export it, so Compose would not see it. Without Compose use docker run --user "$(id -u):$(id -g)" ....
The image works with any UID, no rebuild is needed (HOME is /tmp, the only directories written
inside the image, the data, config and log directories, are writable for everyone; a mounted volume
has the owner and permissions of the host directory).
Upgrading from an image that ran as root. Files created by the old container belong to root.
Give them to the chosen user once, otherwise the app cannot write there (setup_check reports it):
cd /srv/tomotree
docker compose down
sudo chown -R "$(id -u):$(id -g)" streamlit_data streamlit_user streamlit_shared \
streamlit_session jupyter_notebooks config log .streamlit
docker compose up -d
A secrets.toml mounted separately (e.g. into /root/.streamlit/) now belongs to
.streamlit/secrets.toml of the deployment directory (mounted to /app/.streamlit/).
Pulling the latest image and restarting
With a locally built image (current state): build it on the server from a clone of the repository
(git pull && make docker-build), or copy it from the build machine with
docker save tomotree:latest | ssh server docker load. Then:
cd /srv/tomotree
docker compose up -d # recreates the container when the image changed
Once the images are on Docker Hub: docker compose pull && docker compose up -d.
Converting datasets to the current format
Datasets created by older versions keep working, but they lack the data sources in config.yaml
(sources, active), may still have tomogram/config.yaml and keep all files in the dataset root.
Convert them after an upgrade, inside the running container. The datasets are on mounted volumes,
so back them up first; neither step can be undone automatically.
cd /srv/tomotree
tar czf datasets-backup.tgz streamlit_data streamlit_user
# 1. Data sources: config.yaml to the dataset root, sources and active added
docker compose exec tomotree sh -c 'python -m tomotree.importers migrate --dry-run /app/streamlit_data/*/ /app/streamlit_user/*/*/'
docker compose exec tomotree sh -c 'python -m tomotree.importers migrate /app/streamlit_data/*/ /app/streamlit_user/*/*/'
# 2. Layout: files from the dataset root to their subdirectories, sources follow them
docker compose exec tomotree sh -c 'python -m tomotree.importers layout /app/streamlit_data/*/ /app/streamlit_user/*/*/'
# 3. Check: every dataset should report OK
docker compose exec tomotree sh -c 'python -m tomotree.importers check /app/streamlit_data/*/ /app/streamlit_user/*/*/'
python -m tomotree.importers convert <dataset>...does steps 1 and 2 at once. A single dataset can also be converted in the GUI: the sidebar expander Data Info & Sources shows the button Move all to subdirectories (with write permission) when files lie in the dataset root.Run the steps in this order.
migrateskips datasets that already havesources,layoutprints nothing to move for converted datasets, so both can be repeated safely.The
sh -c '...'is needed: the*must be expanded inside the container, not on the host.layoutmoves only files it recognizes (see Where Your Data Are Stored): tomograph files andnodes/distances/speeds/velocities/timesCSV files totomogram/, images exceptthumbnail.pngtobackgrounds/,*.dpatoresistograph/, 3D scans with their textures toscans/,*.statetostates/.config.yaml,thumbnail.png, readme files,keywords.txt,scan_geometry.csvand unknown files stay in the root. A file already present in the subdirectory is not overwritten and stays in the root.Shared datasets (
streamlit_shared/) are links to user datasets and need no conversion.Outside Docker the same commands run as
python -m tomotree.importers ...(make migrate-datasets DATASETS="...",make check-datasets).
Checking the running container
# List running containers
docker compose ps
# Follow live logs
docker compose logs -f
# Show the last 100 log lines
docker compose logs --tail=100
Rolling back to a previous version
Keep the previous image under a tag before an upgrade (docker tag tomotree:latest tomotree:previous).
To go back, tag it as latest again and recreate the container:
docker tag tomotree:previous tomotree:latest
docker compose up -d
With Docker Hub images use a version tag instead:
docker tag robertmarik/tomotree:1.1.0 robertmarik/tomotree:latest.
Configuration Files on the Server
All persistent configuration lives in the config/ volume mount and is not replaced by image updates.
File |
Purpose |
|---|---|
|
Main application settings (directories, hostname, passphrase, storage quotas, …) |
|
User accounts and password hashes |
|
Key for signing saved states and session files (created automatically, keep it with the backup) |
|
Markdown text shown on the login page |
|
Warning box shown in the sidebar on every page |
|
(optional) Allow-list: only the users listed here can log in |
|
(optional) Block-list: users listed here are refused access |
|
(optional) Date until which an account may be used |
config.yaml is read again on every page interaction (the Docker image starts the application without an explicit Config object), so most changes take effect without a restart. After editing login.yaml, or when in doubt, restart the container:
docker compose restart tomotree
Editing in the admin console
The files above (and the extra_editable_files of config.yaml) can be edited in the Admin console, tab Configuration Files. A change is checked before it is saved, and nothing is written when the check fails:
config.yamlandlogin.yamlare checked on a temporary copy by the same checks asmake check-config. Only problems the saved file does not have already block the change, so an older problem does not stop the fix of another one. Unknown settings are a warning that needs Save anyway. Inlogin.yamlevery user needs apasswordand only known roles.The account expiry file must contain valid dates; YAML and JSON files must parse; whitelist and blacklist are plain lists; no file may exceed 1 MB.
Lockout: a change that would lock out the admin who saves it is refused: being put on the blacklist or left off the whitelist, an expired account, losing the
admin/developerrole or the account inlogin.yaml(password login), a removed OIDC provider or role inoidc_providers(OIDC login), alogin_filethat does not exist. Such a change must be made by another admin or on the server.
Each save keeps the previous content in .backups/ next to the file (the last 10 versions; the directory is readable only by the app user, since backups of login.yaml hold password hashes) and replaces the file atomically. Previous versions in the editor shows and restores them (a restore is checked the same way and is itself backed up). When a broken file keeps everybody out, restore from the server:
make restore-config FILE=config/login.yaml # list the backups
make restore-config FILE=config/login.yaml RESTORE=1 # restore the newest
docker compose exec tomotree python -m tomotree.streamlit_app.config_files config/login.yaml --restore
Files changed outside the admin console (editor on the server, file manager) are not checked; run make check-config after such a change.
Security reminder: Set a strong, unique
passphrasevalue inconfig.yamlbefore going live. This key is used to encrypt secret sharing links. The app refuses to start with the templatepassphrase, the templatecookie_secretin.streamlit/secrets.toml(it signs the OIDC login cookie), the templatecookie.keyor template passwords inlogin.yaml(fix:setup_check --fix-login, which replaces all of them by random values). Apassphraseorcookie_secretshorter than 32 characters is a warning.
Signed saved states
Saved states (*.state) and session files (session_*.pkl) are Python pickles, and loading a pickle runs the code it contains. The app therefore signs each such file with HMAC-SHA256 and loads only files signed with its key; a state uploaded in a ZIP or edited in the file manager is refused with a message.
The key is state_secret in config.yaml (hex string, at least 64 characters) or, by default, the file config/state_secret. The file is created with a random key on the first start (or by make init-config), so no setup is needed. Keep it with the backup of config/; installations that share data directories must use the same key file.
States saved before the signing was introduced, or after the key changed, are refused until an administrator signs them. The command never unpickles them, but a signed file is trusted by the app.
Warning: Users can upload
*.statefiles into their directories, and a crafted file runs its code on the server (in Docker as root) once it is signed. Runmake sign-statesonly if you are 100% sure that none of the files has been compromised; otherwise sign only chosen directories or files (STATES="...") after checking them. Before signing unsigned files the command lists them, prints this warning and asks you to typeyes(--yesskips the question, e.g. in scripts).
make sign-states SIGN_FLAGS=--check # list files the app would refuse
make sign-states # sign dirname, userdir and shareddir of config.yaml
make sign-states STATES="streamlit_data/oak" SIGN_FLAGS=--dry-run
python -m tomotree.streamlit_app.sign_states --old-key-file old_state_secret # after a key change
python -m tomotree.streamlit_app.sign_states --include-sessions # also session files
In Docker run it inside the container: docker compose exec tomotree python -m tomotree.streamlit_app.sign_states.
Temporary uploads
“Upload temporary data” stores each upload in its own directory tomotree_upload_* in the system temp directory (in Docker inside the container). Files over 10 MB are refused, and so is an upload of more than 100 files or more than 200 MB in total, ZIP archives counted as extracted (several ZIPs uploaded together count together, measured before anything is written). A new upload of a session replaces its previous one, uploads older than 24 hours are deleted, and all uploads together may not exceed 2 GB (src/tomotree/streamlit/files/temp_uploads.py). Write access applies only while the upload is the open dataset; “Save uploaded data” copies it to the user’s directory, where the storage quota applies.
Installation check
make check-config (python -m tomotree.streamlit_app.setup_check) lists configuration problems with the commands that fix them; make init-config creates the missing files from the *.dist templates with random secrets (never overwrites). In Docker run it inside the container, e.g. docker compose exec tomotree python -m tomotree.streamlit_app.setup_check.
The app runs the same check (setup_check.py, shown by setup_report.py) and creates the missing runtime directories at start:
Errors (no login method, unreadable
config.yaml,login.yamlorsecrets.toml, an OIDC provider listed inoidc_providerswithout the[auth]/[auth.<name>]keys, a provider namedpassword(reserved),trust_emailof a provider that is nottrue/false, the public templatepassphraseinconfig.yamlorcookie_secretinsecrets.toml, the public templatecookie.keyor the template accountsdemo/passwordandguest/Tomotreewith unchanged passwords inlogin.yaml) stop the app before login with a Setup required page. The page shows how to fix each error; for the template login rundocker compose exec tomotree python -m tomotree.streamlit_app.setup_check --fix-login, which sets a random cookie key and random passwords and prints the passwords once.Warnings and notes are shown only to admins and developers: collapsed in the sidebar (Installation check: N warning(s)), in the Admin console tab Installation check and summarized in the Health details. They cover a missing
config.yaml(defaults used), unknown keys with a did-you-mean hint, removed OIDC keys (authenticated_by_google,oauth_roles, …) with a pointer tooidc_providers, aserver_metadata_urlthat is not an OIDC discovery address (/.well-known/openid-configuration), apassphraseorcookie_secretshorter than 32 characters, a provider withtrust_email: true(accepts e-mail addresses the provider did not verify), a missing or empty data directory, non-writable directories, an unusedconfig/oauth_users.yamland an app not started from the project root.
allow_no_auth: true in config.yaml lets everybody in as developer when no login method is configured and shows a red banner on every page. Never enable it on a server.
Whitelist and Blacklist
After a user successfully authenticates, TomoTree performs a second access check using optional allow-list and block-list files. The blacklist is evaluated first and always wins; then the whitelist; then the account expiry.
Whitelist — if config/whitelist.txt exists and is referenced by whitelist_file in config.yaml, only the users listed in that file are allowed in. Everyone else is shown whitelist_message and the session is stopped. Use this to restrict access to a closed group of users.
Blacklist — if config/blacklist.txt exists and is referenced by blacklist_file in config.yaml, any user whose name or e-mail appears in that file is blocked. A blocked user sees blacklist_message and the session is stopped.
Both files use the same plain-text format: one username or e-mail address per line. Empty lines and lines starting with # are ignored. Matching is case-insensitive.
Example config/whitelist.txt:
# Allowed users
alice@example.com
bob@example.com
The corresponding config.yaml entries (uncomment and adjust as needed):
whitelist_file: config/whitelist.txt
whitelist_message: "You are not on the list of authorised users. Please contact the administrator."
blacklist_file: config/blacklist.txt
blacklist_message: "Your account has been blocked. Please contact the administrator."
The files are re-read at most every 30 seconds (cache user_lists), so changes take effect without restarting the container.
Account expiry
An account can be limited in time, independently of the whitelist and blacklist and equally for password and OIDC accounts. account_expiry_file in config.yaml names a YAML file that maps a username or e-mail address (case-insensitive) to the moment the account stops working:
student@example.com: 2026-12-31 # a date alone: the whole day is still valid
guest: 2026-12-15 18:00 # valid until 18:00, local time of the server
tester: 2026-12-15T18:00:00+01:00 # with an explicit time zone
Accounts that are not listed have no time limit. For a password account both the username and the e-mail are looked up; with two entries the earlier one applies.
Order of the checks after login:
Blacklist — a listed user is refused, whatever else applies.
Whitelist — if active, a user who is not on it is refused.
Account expiry — an expired account is refused, also when the user is on the whitelist. The user sees
account_expired_message({date}is replaced by the last valid day or time).Maintenance mode.
The check runs on every interaction, so an account that expires while the user works is stopped at the next click (a change of the file takes up to 30 seconds). An entry whose value is not a valid date refuses that account; a file that cannot be parsed refuses everybody except admins until it is fixed. make check-config reports both. The file can be edited in the admin console (Configuration files → Account expiry).
account_expiry_file: config/account_expiry.yaml
account_expired_message: "Your account was valid until {date}. Please contact the administrator."
Cleaning Up Old Docker Images and Files
After pulling and starting a new image version, the old image layers remain on disk and can accumulate significant storage over time. Use the following commands to reclaim space.
Remove dangling images (untagged leftovers)
Dangling images are layers that are no longer referenced by any tag — these are the most common leftovers after a docker compose build or docker pull:
docker image prune
Remove all unused images (including older tagged versions)
This removes every image that is not referenced by a running or stopped container:
docker image prune -a
Warning: This also removes any locally built images you have not pushed yet. Run only when you are sure all valuable images are already pushed to Docker Hub.
Remove stopped containers, unused networks, dangling images, and build cache at once
docker system prune
To also include unused volumes (⚠ this will delete volume data not mounted by any container):
docker system prune --volumes
Remove a specific old image by tag
If you want to keep recent images but drop a specific version:
docker rmi robertmarik/tomotree:1.1.0
Typical post-upgrade cleanup sequence
# 1. New image and restart (see "Updating a Production Server" above)
cd /srv/tomotree
docker compose up -d
# 2. Remove dangling layers left by the pull
docker image prune -f
# 3. (Optional) Remove all images not in use by a running container
docker image prune -a -f
The -f flag skips the confirmation prompt, which is useful in scripts.
Check disk usage before and after
docker system df
This shows how much space is used by images, containers, volumes, and build cache — and how much can be reclaimed.