Publishing datasets on Zenodo
Two pages of Data & Info & Share work together:
Metadata describes the open dataset (authors, species, place, date, devices). The description is saved in
config.yamlof the dataset with every change.Zenodo connector publishes the dataset on Zenodo with a DOI, or downloads the same package to your computer, for example to share it or to keep it. Each tree becomes one ZIP file, a FAIR data package. Several trees can share one Zenodo record, so a whole study gets one DOI.
Only the owner of a dataset can use the page Zenodo connector, and only after logging in. Others can read the metadata on the page Metadata but cannot change them.
Metadata
Every change of the form is saved at once in config.yaml of the dataset under metadata
(see below); the page shows the time of the last save. Reopening the page or the dataset
fills the form again. Just opening the page writes nothing.
The form is prefilled from the dataset:
the notes of PiCUS/Arbotom files: species, date, coordinates, locality, circumference;
the device kind;
the resistograph headers: date and version;
readme.mdandkeywords.txt.
The prefilled values are saved with your first change.
Look up in GBIF checks the species name in the GBIF Backbone Taxonomy. It fills the taxon URL, genus, family and kingdom.
Authors are written as “Family name, Given names”. An ORCID is optional and identifies the author unambiguously.
Find the ORCID of a person searches the names in the public ORCID registry (every word must match, with or without diacritics, e.g. Mařík finds Marik), or an ORCID. All people found are listed, up to 200; people with that family name, of the optional Institution and of the affiliations of the authors already in the table come first. With more matches (e.g. a common family name alone) the page asks for the given name or an institution. Add as an author adds the chosen person to the authors with name, ORCID and institution, or adds the ORCID to the author of the same name.
Pick the location on a map shows OpenStreetMap or a satellite image; a click sets the coordinates (WGS84).
Fill the locality and country from the coordinates asks OpenStreetMap (Nominatim) for the place, the country and the ISO country code. Only the point is sent, and only on this click.
Problems are listed below the form. Incomplete metadata are saved too; the errors marked as required block only the upload and the download.
Zenodo connector
The page has framed sections: Zenodo account, Record, Metadata, Readiness of the data, Contents of the package, Download the package, Upload and publish. A change in the contents, the download or the upload section redraws only that section. A change of the account or the record redraws the whole page. The page asks Zenodo only on a click: Refresh the list reads your records again (otherwise they are kept for 5 minutes), and Check on Zenodo reads the state of the draft, for example after you published it on the website. When Zenodo is switched off on the server (zenodo_enabled: false), only the download is offered.
Record. At the top of the page choose New record, or One of my records to add the tree to a record you already have on Zenodo.
Both need your Zenodo account: create a personal access token with the scopes
deposit:writeanddeposit:actions(Zenodo → Applications → Personal access tokens) and save it on the page. The token is stored encrypted on the server and used only for your uploads. Delete the token removes it. Zenodo sandbox (https://sandbox.zenodo.org) is a test server with its own accounts. Its records get no real DOI.For One of my records pick the record (filter them by words of the title or keywords). Take over the shared metadata of the record writes the metadata shared by all trees of the record into the metadata of this tree: title, licence, authors, keywords, related article and funding. The authors and keywords of this tree that the record lacks stay. Without it, the upload still keeps the title and the licence of the record.
Metadata. A summary of the metadata from the page Metadata and their problems. Edit them there.
Readiness. The page checks that the dataset holds everything needed for the reconstruction:
the data sources are registered and unchanged;
the sensor positions and the travel times are present.
Errors block the upload and the download. A warning recommends saving the tomogram settings (“Save to file” on the tomogram page), so that others can reproduce your tomogram.
Contents. Optionally add:
the tomograms computed in this session (image, data, parameters, a reproducing script);
photos;
background images;
3D scans.
Leave out of the package lists every file of the package (measured files such as
.f3d, photos, scans, generated tables, metadata files); the chosen files are not published and README, RO-Crate and the checksums do not list them. Leaving outconfig.yamlor a measured file shows a warning: the package then cannot be opened in TomoTree or checked against the measured data. The list of files and their size is shown before the upload.Download the package. Prepare the package builds the ZIP with the metadata and the contents chosen above, then Download saves it on your computer. Nothing is sent to Zenodo. If the dataset was published on Zenodo before, the package names its DOI.
Upload as a draft. Nothing is public yet.
A draft takes the ZIP directly.
A published record cannot take new files. A new version is created with the files of the current version plus the new tree. The new version gets its own DOI; the concept DOI of the record stays.
The metadata of the record are merged: authors, keywords, species, locations and dates are added, and the table of trees in the description is extended. The title and the licence of the record stay.
Publish. Check the draft on Zenodo, add more trees, then publish it with Publish the record (after a confirmation) or on the Zenodo website. Publishing cannot be undone.
The package (one ZIP per tree)
The folder in the ZIP is a TomoTree dataset. load_dataset() opens it after
unpacking:
Jicin_05/
├── config.yaml data sources, measured data (SI), all settings, metadata
├── tomogram/… original device files (.f3d, .pit, .abt, CSV), tap quality decisions
├── resistograph/*.dpa original resistograph profiles
├── scan_geometry.csv sensor positions and boundary from a 3D scan (if any)
├── scans/sensor_geometry.json sensor positions found on the scan (always); models only on request
├── photos/*.json photo annotations (always); photos only on request
├── readme.md, keywords.txt the author's description
├── derived/ the same data in open formats (Apache Parquet)
│ ├── sensors.parquet sensor number (from 1), x, y [m], used
│ ├── times.parquet source, receiver, distance [m], time [µs], velocity [m/s], disabled
│ ├── boundary.parquet trunk boundary polygon [m] and sensors on it
│ ├── resistograph/*.parquet depth [mm], amplitude
│ ├── settings.json reconstruction settings, scan cut plane/placement, drilling paths
│ ├── tomogram_<method>.parquet / .json / .png tomogram cells, parameters, sound wood speed, figure
│ └── reproduce_<method>.py recomputes the tomogram from this folder
├── dwc/ Darwin Core Event, Occurrence, MeasurementOrFact (CSV and Parquet)
├── ro-crate-metadata.json RO-Crate 1.1: every file, format, checksum, provenance
├── datapackage.json Frictionless Data Package: table schemas with units
├── metadata.json, README.md, LICENSE.txt
└── MANIFEST.sha256 checksums (sha256sum -c MANIFEST.sha256)
Never included:
the access list
allowedand the navigation keys ofconfig.yaml;saved states (
*.state,autoload.state,states/);lock files,
thumbnail.pngand symbolic links.
Each published tomogram is computed again by the library with the parameters
applied on the tomogram page. The Parquet table, the figure and
reproduce_<method>.py therefore give the same result as the GUI.
How the package follows FAIR
Findable:
DOI from Zenodo;
the species as a GBIF taxon URI;
AGROVOC subjects;
keywords, locations and the collection date in the Zenodo metadata;
Darwin Core fields (
custom), if the Zenodo server accepts them. Otherwise a warning is shown, and the Darwin Core data stay indwc/andmetadata.json.
Accessible: an open licence (CC BY 4.0 by default, CC0, CC BY-SA 4.0 or ODC-By), a plain ZIP, and every proprietary device file accompanied by open tables.
Interoperable:
Parquet with units and descriptions in the column metadata;
Frictionless schemas and RO-Crate (JSON-LD, schema.org plus Darwin Core);
coordinates in WGS84, dates in ISO 8601;
ORCID for people.
Reusable:
provenance of the derived files (
isBasedOn, TomoTree version, parameters, input checksums);the reproducing scripts, the README with a data dictionary, and checksums.
Metadata in config.yaml
metadata:
title: Acoustic tomography of Tilia cordata (Jičín 05)
creators:
- {name: "Mařík, Robert", affiliation: Mendel University in Brno, orcid: 0000-0002-1825-0097}
license: cc-by-4.0
keywords: [linden, alley]
dwc:scientificName: Tilia cordata
dwc:taxonID: https://www.gbif.org/species/3152047
dwc:eventDate: 2024-03-25
dwc:decimalLatitude: 50.437
dwc:decimalLongitude: 15.35
dwc:geodeticDatum: EPSG:4326
dwc:locality: Jičín, lime alley
dwc:countryCode: CZ
dwc:organismID: tree 5
device: {name: Fakopp ArborSonic 3D, version: "5.1"}
resistograph_device: {name: IML Resistograph, version: "117"}
measurements: {circumference_cm: 251, measuring_height_cm: 30, tree_height_m: 18}
related_doi: 10.1234/article
zenodo: {server: https://zenodo.org, record_id: 123, doi: 10.5281/zenodo.123, file: Jicin_05.zip, published: true}
The zenodo entry is written by the page after an upload and is never
published.
In Python
from tomotree import publication, zenodo
meta = publication.prefill_metadata("Jičín_05")
meta["creators"] = [{"name": "Mařík, Robert", "orcid": "0000-0002-1825-0097"}]
print([p.text() for p in publication.check_ready("Jičín_05", meta)])
publication.build_package("Jičín_05", "Jicin_05.zip", metadata=meta,
tomograms=[{"method": "ERSM", "params": {"nx": 41, "ny": 41}}])
client = zenodo.ZenodoClient(token, sandbox=True)
draft = client.create(publication.zenodo_metadata(meta, "Jicin_05.zip"))
client.upload_file(draft, "Jicin_05.zip")
# client.publish(draft["id"])
include={'photos': True, ...} adds the optional folders; exclude=[...] leaves out any files of the package (paths in the package folder, e.g. tomogram/x.f3d or derived/times.parquet; plan_package() lists them). publication.record_metadata(deposition['metadata']) returns the metadata shared by the trees of an existing record (title, licence, authors, keywords, related article, funding); publication.merge_shared(meta, shared) merges them into the metadata of a tree. publication.orcid_search('Carberry') looks up authors.
build_package needs pyarrow, from the extra publish (pip install tomotree[publish]) or the full gui install.
Server settings (config/config.yaml)
Key |
Default |
Meaning |
|---|---|---|
|
|
Show the page. |
|
|
Zenodo server. |
|
|
Test server. |
|
|
Personal tokens of the users. Encrypted with a key derived from the state secret ( |
|
empty |
Zenodo community the records are submitted to. |
|
empty |
DOI of TomoTree, linked from the records as |
make check-config warns about a tokens file readable by other users and refuses non-https server addresses.