Developing a plug-in for dummies

This tutorial is essentially a cleaned-up version of the notes I took as I was developing my first plugin, q2-perc-norm. I hope this is helpful to others like me, who aren’t trained computer scientists/developers, but who are keen and able to learn the programming stuff to make their tools more useful to more people.

A first reminder that for any qiime2 functions to work, you need to be in the qiime2 virtual environment. If you’re trying to run things and getting errors like ImportError: No module named qiime2, then you’re probably not in the right environment.

I recommend installing the qiime2 development environment, as this will have all of the latest features. Because qiime2 is still in active development, there may be some non-backward compatible changes with new releases, so you’ll want to have those right from the start!

It’s also very helpful to look through multiple existing plugins to get a sense of how other people have done things. I recommend looking at more than one example, since this will give you a sense of which parameters/setup/styles are required, and which are flexible. Plugins that were really helpful to me when I made this were the q2-ghost-tree user-created plugin, as well as the massive q2-diversity plugin (specifically the q2_diversity.beta_group_significance for specific examples using different types of QIIME 2 data types.)

Make the repo

As you poke around multiple plugins, you’ll notice that the general structure for a plugin repo is:

q2-myplugin/
|
|-- setup.py
|-- README.md
|-- LICENSE
|
|-- q2_myplugin/
    |
    |-- plugin_setup.py
    |-- __init__.py
    |-- _my_plugin_function_1.py
    |-- _my_plugin_function_2.py

Basic setup files

First, start populating your repo with some basic setup files. These files are fairly well-described in the original “Developing a QIIME 2 plugin” page on the general QIIME 2 documentation.

In the main directory, make a setup.py file that gives some broad info about your plugin to-be. This file should contain some basic information about the package you’re building. You can copy it from any existing plugin.

Next, make a directory with the same name as your plugin. In this folder, make your plugin_setup.py file.

plugin_setup.py is where you register all the functions/methods/visualizers that your plugin will have. In other words, these will be the things that your plugin does, the things you type after your plugin name (for example, perc-norm percentile-normalize for the registered percentile-normalize function). Again, the “Developing a QIIME 2 plugin” page describes the content of these files pretty well.

Basic debugging

Once you have your content set in plugin_setup.py, it’s good to start with some basic debugging to make sure there are no errors:

python plugin_setup.py

Note for debugging: if you want to run a script (e.g. plugin_setup.py) directly, make sure to add your main repo directory to your PYTHONPATH so that anything you import from your project in the scripts is discoverable.

export PYTHONPATH=~/github/q2-perc-norm/

For example, it looks like the common practice is to have a file for each method in the same directory as plugin_setup.py. In the plugins I used as examples, these files are typically named ._method_name.py, and are imported at the top of plugin_setup.py (e.g. from._method_name import method_name). This import statement only works if the folder is in your PYTHONPATH.

As an alternative to messing with your PYTHONPATH, which is easy to forget to do each time, you can use pip install -e . from the main directory which contains setup.py. This installs an editable version in development mode in your current directory (you’ll see an egg-info directory after you run this). With this method, any updates you make to the code should automatically be “re-installed” without needing to re-run pip install.

Install the plugin

When you’re ready to actually try doing stuff, you’ll need to run python setup.py install (from the main repo folder) for your qiime plugin to be callable from the command line.

python setup.py install

If you’ve made updates to your plugin’s command line interface, don’t forget to clear the cache (qiime dev refresh-cache) before running this!

Note that the name of your plugin is what you put in your plugin_setup.py file:

plugin = Plugin(
    name='perc-norm',
    ...
    )

You can double check that it worked by just typing qiime on the command line and seeing if your plugin shows up. Then, you can just try running the plugin:

qiime perc-norm

This will show you the general plugin info, and you should see all of the functions that you registered at the bottom:

(qiime2-dev) 19:24-claire:~/$ qiime perc-norm

Usage: qiime perc-norm [OPTIONS] COMMAND [ARGS]...

  Description: This QIIME 2 plugin performs a model-free normalization

  ...

Options:
  --help  Show this message and exit.

Commands:
  percentile-normalize  Percentile normalization

Then you can try running each function and see if the inputs are what you want them to be:

(qiime2-dev) 19:24-claire:~/$ qiime perc-norm percentile-normalize

Usage: qiime perc-norm percentile-normalize [OPTIONS]

  Converts OTUs in case samples to percentiles of their distribution in
  controls.

Options:
  --i-table ARTIFACT PATH FeatureTable[RelativeFrequency]
                                  The feature table containing the samples
                                  which will be percentile normalized.
                                  [required]
  --m-metadata-file MULTIPLE PATH
                                  Metadata file or artifact viewable as
                                  metadata. This option may be supplied
                                  multiple times to merge metadata.
                                  [required]
  --m-metadata-column MetadataColumn[Categorical]
                                  Column from metadata file or artifact
                                  viewable as metadata. Sample metadata column
                                  which has samples labeled as "case" or
                                  "control". Samples which are not labeled are
                                  not included in the output table.
                                  [required]

...

  --help                          Show this message and exit.

Woop! The plugin was set up correctly!

Side note on MetadataColumn[Categorical]

I specified a MetadataColumn[Categorical] required parameter in my plugin_setup.py function, and I wasn’t sure how this would be parsed or treated by the underlying code. It turns out that qiime automatically parses it and turned into the two inputs you see: --m-metadata-file and --m-metadata-column. This is how I made the metadata an input to my function (in _percentile_normalize.py):

def percentile_normalize(table: biom.Table,
                         metadata: qiime2.CategoricalMetadataColumn,
                         n_control_thresh: int=10,
                         otu_thresh: float=0.3) -> biom.Table:

I used some functions I found in another qiime plugin to ensure that sample IDs in the metadata and OTU table matched, and then converted the metadata column into a pandas Series object.

metadata = metadata.filter_ids(table.ids(axis='sample'))
metadata = metadata.drop_missing_values()
table = table.filter(metadata.ids)
metadata = metadata.to_series()

The developers on the qiime2 forum were REALLY helpful to figure this out, since there currently isn’t really much documentation on the different qiime2 data types.

Testing your plugin

Once you’ve written something that looks like it works, you’ll want to check that it actually does work! Since this is less generalizable, you can check out the original version of this tutorial for one example of what this process can look like.

Contributors

The original version of this post was adapted from https://cduvallet.github.io/posts/2018/03/qiime2-plugin with permission.

  • Claire Duvallet (github: cduvallet), June 2018