Library Publishing Tutorial¶
There are a few steps to publishing a QIIME 2 plugin, this tutorial will show you the more common aspects present in most cases, but if you get stuck please come find us at https://forum.qiime2.org/c/dev-discussion.
1. Write a plugin¶
See Developing a plugin for dummies for more details on how to do that. This is where you’ll spend the most time on this project! For this particular example, we will use the q2-cutadapt plugin as a reference:
2. Host the source code on GitHub¶
See https://github.com for more details. The reason you must host on GitHub is because our publishing tools are built using GitHub Actions, which is a great way for building automated tasks.
For our example plugin, it is already hosted on GitHub (see Step 1 above).
3. Define a conda recipe¶
In order to build a conda package you have to first start with a conda recipe. As the name implies, this is a set of instructions that tells conda how to package your project’s code up, what external dependencies are necessary to run the code, and how to run any tests to validate that the package was built successfully.
The entirety of the recipe file has been copied here, but please check the git repo for the latest copy.
{% set data = load_setup_py_data() %}
{% set version = data.get('version') or 'placehold' %}
This first block is a bit of inline python code that allows our recipe to dynamically read the plugin package’s version identifier, which is nice to keep things synchronized between the python code and the package recipe, but is not strictly necessary.
package:
name: q2-cutadapt
version: {{ version }}
Next, we specify some metadata about the package: the name, and the version that we parsed in the block above.
source:
path: ../..
This tells conda where to find the source code, relative to the recipe file.
This particular recipe file is stored in a directory called ci/meta
.
build:
script: make install
Next we actually tell conda what command to run in order to build this plugin.
For the plugins in the core distribution, we like to use Makefiles for
standardizing a lot of our commands, this particular make directive is short
for python setup.py install
.
requirements:
host:
- python {{ python }}
- setuptools
run:
- python {{ python }}
- cutadapt >=3
- pigz
- pandas {{ pandas }}
- numpy
- qiime2 {{ qiime2_epoch }}.*
- q2-types {{ qiime2_epoch }}.*
This block is where we specify the plugin’s dependencies. The topic of host vs
run dependencies is outside of the scope of this tutorial, but most QIIME 2
plugins are going to have a host
section that looks similar to this one.
The run
section is where we tell conda about our specific external
dependencies. Since conda is runtime-agnostic, we first start with telling
conda that we depend on python
. The {{ python }}
part is an inline
variable that tells conda to use the value assigned to the variable python
,
which allows the QIIME 2 team to dynamically set the version for you!
Next, cutadapt
is listed, which makes sense, since this plugin is
q2-cutadapt! This entry also includes a version specification of >=3
. which
tells conda to install any version of cutadapt, as long as it is at least v3 or
greater.
The remaining dependencies are left as an exercise for the reader.
test:
requires:
- qiime2 >={{ qiime2 }}
- q2-types >={{ q2_types }}
- pytest
imports:
- q2_cutadapt
- qiime2.plugins.cutadapt
This section tells conda how to test the built package. It has it’s own
requires
section for listing test requirements, - in this case we provide
version variables that will evaluate to slightly more precise version pins,
which allows us to ensure we’re always testing with the latest development
versions of a QIIME 2 plugin. pytest
is also listed because we use it as
the test runner, but you can use whatever runner you wish.
Then, the imports
subsection runs the list of python commands as simple
smoke tests. The plugin’s unit tests are added by action-library-packaging
,
which will be discussed in later steps.
about:
home: https://qiime2.org
license: BSD-3-Clause
license_family: BSD
This final block is optional, but it allows you to specify the plugin’s license and homepage. QIIME 2 has no requirements on software licensing.
4. Ask an admin for a token¶
This step is a placeholder for now - in future versions of Library we would like for this to be self-serve. In the meantime, come find us on the QIIME 2 Forum:
https://forum.qiime2.org/c/dev-discussion
Once we get a Library token, we need to set it as a GitHub Secret.
5. Write a GitHub workflow that uses the action-library-packaging action¶
Now that we have a conda recipe prepared, we need to tell Library about it.
The way we do that is by using a tool called action-library-packaging
,
which uses GitHub Actions to build, test, and publish QIIME 2 plugins.
The full copy of the GitHub workflow that uses this actions can be found here.
name: ci
on:
pull_request:
push:
branches:
- master
This first section instructs GitHub when to run this workflow. At the very least it needs to run on pushes to whatever the default branch is in your repository. For core distribution plugins, we like to also run the workflow any time a pull request is opened - this has the benefit of ensuring the conda package can successfully build, but won’t actually deploy the changes to Library.
jobs:
build-and-test:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
All QIIME 2 packages must be buildable on macOS and linux - this ensures that the workflow runs on both of those platforms.
steps:
- name: checkout source
uses: actions/[email protected]
with:
fetch-depth: 0
This first step is required, in order to build your plugin, you first have to ensure that GitHub checks out the latest copy of the source code.
- name: set up git repo for versioneer
run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
For core distribution plugins, we use a tool called versioneer which allows us to manage our version identifiers using git tags, but that is not a requirement for your plugin. Because of this, we have to tell this step how much of the repo to check out.
- uses: qiime2/[email protected]
with:
package-name: q2-cutadapt
build-target: dev
additional-tests: py.test --pyargs q2_cutadapt
library-token: ${{ secrets.LIBRARY_TOKEN }}
Finally, the moment we’ve been waiting for - this step actually runs
action-library-packaging
! This is where the Library token from above comes
into play - this is how we “communicate” back to Library to let it know that
the plugin is authentic.
As well, the ability to specify additional tests is really useful here - anything listed on that line will be run as part of the package building process.
6. Wait for Library to pick up the changes and publish them¶
This final step is hands-off - just sit back and wait for the GitHub workflow to successfully complete - once it does, Library should publish the package within the next 6-8 hours. You can follow the progress at this automated repo: https://github.com/qiime2/package-integration.
Congratulations!