The structure of QIIME 2 archives has evolved as QIIME 2 has been developed. This page describes each historical version of the QIIME 2 Archive format, and may be useful to interface developers whose code depends on guarantees made by that format (source code).
For general information about the structure of current QIIME 2 Archives, see Anatomy of an Archive. For a detailed description of the part of an Archive which holds Provenance data, see Decentralized Provenance Tracking.
Though there is significant variability in the format of QIIME 2 Archives across archive versions, all archive versions share some common traits.
These shared characteristics, defined in the
_Archive class in
must be consistent across all formats over time,
as they allow archive versions to be checked,
and archives with different formats to be dispatched to the appropriate version-specific tools.
All QIIME 2 Archives have:
a directory named with the Archive UUID, directly under the archive root at
/<UUID>/VERSIONwithin that directory, formatted as shown below
The Archive file system must look like this:
The VERSION file must look like this:
QIIME 2 archive: <archive version> framework: <framework version>
This file is intentionally not in YAML, INI, or any other common data serialization or configuration format. This is to discourage the situation where important archive files are reformatted from YAML to another format and VERSION is updated “for consistency”, breaking backwards compatibility.
The original QIIME 2 Archive format, there aren’t many V0 Archives “in the wild”. V0 Archives were produced by Alpha versions of the QIIME 2 framework, and were superseded in framework version 2.0.6 on 10/24/16.
Result data files are written in the directory
- Result UUID, Semantic Type, and Format information are saved in
ArchiveFormatclass in v0.py offers convenience methods for loading and parsing
V0 Archives look like this:
Released in QIIME 2 version 2.0.6,
Version 1 Archives introduce decentralized provenance tracking to QIIME 2.
inherits all traits of v0,
write() methods only to add provenance capture.
ArchiveFormat versions subclass their predecessor.
v1.py inherits from the
etc. This makes it easier for humans to interpret the version history.
Provenance data is stored in the directory
are captured for the current Result and each of its ancestors.
action.yaml file and associated data artifacts (e.g. sample metadata)
are stored in an
action directory alongside that Result’s
Considered together, we can describe these as “provenance files”.
V1 Archives look like this:
V0 Archives do not capture provenance data. As a result, if a V0 artifact is an ancestor to a V1 (or greater) artifact, it is possible for the action.yaml to list Artifact UUIDs which are not present in the artifacts directory.
In commit 4389a0b,
the Version 2
ArchiveFormat adds an
output-name key to the
action section of
(unless the action type is
assigning it the output name registered to the relevant action.
Prior to this change, if one action returned multiple artifacts of the same Semantic Type,
it was not possible to differentiate between them using provenance alone.
With this release, QIIME 2 Actions are able to take variadic arguments,
allowing users to pass collections of Artifacts (
List s and
A YAML representer has been added so that
action.yaml can represent
Set s of Artifact inputs.
These will show up in
action.yaml as custom
Released in QIIME 2 version 2018.4
this format adds citations to the directory format,
transformers section to
and aligns the structure of
environment:framework (also in
to match the structure of
Whenever an Action is run, its registered citations are captured.
When saved, they are written to a
inside the Archive’s
Citations for all of the current Result’s ancestors are stored in their respective <UUID> directories
The overall directory structure remains identical to a v1 archive, above.
Result-specific citation tags are also written to
environment sections of the
for the current Result and for all ancestors with registered citations.
A new custom
!cite '<citation key>' tag is use to support this in YAML.
transformers section is added between the
environment sections of
Because Pipelines do not use transformers,
transformers will be recorded only for Methods, Visualizers,
and when importing data (such as with
qiime tools import).
It looks like this:
transformers: inputs: demultiplexed_seqs: - from: SingleLanePerSamplePairedEndFastqDirFmt to: SingleLanePerSamplePairedEndFastqDirFmt output: - from: q2_types.feature_data._transformer:DNAIterator to: DNASequencesDirectoryFormat plugin: !ref 'environment:plugins:types'
environment::framework was previously only a version string,
and is now structured identically to each plugin action’s
with version, website, and citation sections:
framework: version: 2019.10.0 website: https://qiime2.org citations: - !cite 'framework|qiime2:2019.10.0|0' plugins: fragment-insertion: version: 2019.10.0 website: https://github.com/qiime2/q2-fragment-insertion citations: - !cite 'plugin|fragment-insertion:2019.10.0|0' ...
A new, md5sum-formatted checksum file has been added at
with one md5sum and one filename on each line. For a more detailed specification, see the
Checksums.md looks like this:
5a7118c14fd1bacc957ddf01e61491b7 VERSION 333fd63a2b4a102e58e364f37cd98b74 metadata.yaml 4373b96f26689f78889caeb1fbb94090 data/faith_pd-cat1.jsonp ... 7a40cff7855daffa28d4082194bdf60e provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/metadata.yaml ae0d0e26da5b84a6c0722148789c51e0 provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/action/action.yaml
V5 Archives look like this: