Tree-for-All hackathon series: Introduction

The Tree-for-All: Introduction

Welcome to the first in a series of posts featuring results from the recent “Tree-for-all” hackathon (Sept 15 to 19, 2014, U. Mich., Ann Arbor) aimed at leveraging data resources of the Open Tree of Life project.   This post is written by Arlin Stoltzfus (NIST)[1], one of the hackathon organizers (but not affiliated with Open Tree in any other way).  Below, I’m going to introduce the rationale and aims of the hackathon, describe the process, and summarize some of the projects.  In subsequent posts, we will discuss products and lessons learned.  The list of forward links will be updated as new posts appear:

Motivation: bridging the accessibility gap

The Open Tree of Life project aims to provide data resources for the scientific community, including

  • a grand synthetic tree covering millions of species, generated from thousands of source trees
  • a database of the source trees, published species trees used to generate the synthetic tree
  • a reference taxonomy used (among other things) to align names from different sources

The premise of synthesizing a grand Tree of Life, and making it available with source studies and a reference taxonomy, is that these resources are valuable.  To assess the value of these resources right now would be premature— we will  return to that question later.  For now, I will just point out that, until recently, when scientists in the bioinformatics community have needed a tree broadly covering the kingdoms of life, they have used the NCBI taxonomy hierarchy (multiple examples are cited by Stoltzfus, et al., 2012), an approach that causes phylogeneticists and systematists to groan.  Surely we are better off now, but determining how much better off we are probably will require further analysis.

For the present, it is important to understand that the value of a community resource is predicated on accessibility.  Most users would not know how to handle a tree with 3 million species, useful or not.  For the value of OpenTree’s resources to be realized, it is important to anticipate the needs of users, and support them with appropriate tools.

The aim of the recent Tree-for-all hackathon was to begin bridging this accessibility gap.  More specifically, the aim of the hackathon was to build capacity for the community to leverage Open Tree’s resources via their recently announced web services API (Application Programming Interface).   This enhanced capacity may take the form of end-user tools, library code, standards, and designs.

Technology: web services

Web services are a natural choice for accessibility, because they provide programmable access to a resource to anyone with a networked computer.  Most of the time when you use the web, you are sending a request for a specific page, and receiving results in HTML that are rendered by your browser.  But more generally, web services work by a standard protocol that allows you to send data and commands, and receive results.

Some services are so simple that you can access them just by typing in the URL box of your browser.  For instance, TreeBASE has a web-services API that allows you to access data with commands such as

which retrieves a particular tree in NEXUS format.  When that isn’t enough, you can use a command-line tool such as  cURL (command-line URL), found on most computer systems.   I’ll give an example using cURL, then explain how to use a Chrome extension called DHC that provides a graphical user interface.

Open Tree’s web API can do many things, but let’s start with something simple: find out what the synthetic tree implies about the relationships of a set of named species, “Panthera tigris”, “Sorex araneus”, “Erinaceus europaeus”.   To get the tree, we need to chain together a workflow based on 2 web services, the match_names service (click to read the docs) to convert species names to OT taxon identifiers, and the induced_tree service to get a tree for species designated by identifiers.  In the first step, using cURL, we issue this command:

curl -X POST \
-H "content-type:application/json" \
-d '{"names":["Panthera tigris","Sorex araneus","Erinaceus europaeus"]}'

This command matches our list of input names with the names in OpenTree’s taxonomy. If a species is in the tree, it will have an id in the taxonomy. The output of this command yields the matching identifiers 633213, 796660, and 42314.  To find them, scroll through the output and look for the “ottId” field, which refers to Open Tree taxonomy ids.  Once we have those ids, the next step is to use them to request the tree:

curl -X POST \
-H "content-type:application/json" \
-d '{"ott_ids":[633213, 796660, 42314]}'

which returns a Newick tree (embedded in JSON). OpenTree’s interface refers to this as the “induced” tree, though perhaps it is more appropriately called the implied tree: for any set of nodes in the synthetic tree, the structure of the larger tree immediately implies a topology for the subset, e.g., the tree of A, C and E implied by (A,(B,(C,(D,E)))) is (A,(C,E)).

To run these commands in DHC, start with the cURL command above, then copy and paste the service (the “http” part) and the body (after the -d), into the appropriate boxes, click on “JSON” below the body window (or set the header to content-type: application/json), choose “POST”, then hit “Send”.  The output will appear below.


DHC allows you to use web services in a one-off manner, interactively, but the real power of web services starts to emerge when they are invoked and processed in an automated way, within another program.

Process: Hackathon

Open Tree announced version 1 of its web services in May, at the same time we distributed an open call for participation in a “Tree-for-all” hackathon, which took place September 15 to 19 at University of Michigan, Ann Arbor.  The hackathon was organized and funded by Open Tree, the Arbor workflows project and NESCent’s HIP (Hackathons, Interoperability, Phylogenies) working group.

What, exactly, is a hackathon?  A hackathon is an intensive bout of computer programming, usually with a scope that allows for considerable creativity (when the objectives are pre-determined, the event might be called a “code sprint” instead).  Often it involves bringing together people who haven’t worked face-to-face before.

The tree-for-all hackathon followed a plan for a participant-driven 5-day meeting with ~30 people.  The participant pool is seeded with some hand-picked developers, but consists mainly of folks who have responded to an open call.  The people chosen to participate are not all elite super-coders— some are subject-matter experts without advanced coding skills.  On the morning of day 1, these participants hear informational presentations— in this case, about Open Tree’s data and services (above), the Arbor workflow project, and HIP’s vision of an interoperable web of evolutionary resources.  This is followed by open discussion of possible projects, a process that typically begins (via email list) long before the hackathon.

On the afternoon of Day 1 comes the make-or-break moment: pitching and team-formation.  Participants with ideas stand up, make a pitch for a software development target, and post it on the wall using a giant sticky note.  Others move from pitch to pitch, critiquing, suggesting ideas, and trying to find where they could contribute (or learn) the most.  Pitches evolve through this process, and eventually a set of teams emerges.  From this point on— days 2 to 5 of the hackathon— the meeting belongs to the teams.  The hackathon will succeed or fail, depending on the strength of the teams.

Hackathon participants gather to hear a progress report.  Left to right: Matt Yoder, Stephen Smith, Cody Hinchliff (standing), Andréa Matsunaga, Joseph Brown, Zack Galbreath (standing), Chodon Sass, Alex Harkess, Julienne Ng (eyes only),  Katie Lyons, Gaurav Vaidya (standing), Jorrit Poelen, Shan Kothari (facing left), David Winter, Julie Allen (standing), Karolis Ramanauskas, Nicky Nicolson, Josef Uyeda, Miranda Sinnott-Armstrong (standing), Rachel Warnock, François Michonneau, Luke Harmon, Kayce Bell, Jon Hill's right arm.

Hackathon participants gather to hear a progress report. Left to right: Matt Yoder, Stephen Smith, Cody Hinchliff (standing), Andréa Matsunaga, Joseph Brown, Zack Galbreath (standing), Chodon Sass, Alex Harkess, Julienne Ng (eyes only), Katie Lyons, Gaurav Vaidya (standing), Jorrit Poelen, Shan Kothari (facing left), David Winter, Julie Allen (standing), Karolis Ramanauskas, Nicky Nicolson, Josef Uyeda, Miranda Sinnott-Armstrong (standing), Rachel Warnock, François Michonneau, Luke Harmon, Kayce Bell, Jon Hill’s right arm.

Outcomes: Hackathon team projects

Over the coming weeks, I’m going to write about hackathon team projects and, ideally, provoke some other hackathon participants to do the same.  Hackathon teams are instructed (and cajoled) to focus on tangible outcomes, and the Tree-for-All hackathon produced a lot of them!  For now, here is a brief synopsis.

Integration of Trees and Traits involved hackathon participants Jeff Cavner (remote), Luke Harmon, Zack Galbreath, Jorrit Poelen, Julienne Ng, Alex Harkess, Chodon Sass, Shan Kothari, and Mark Westneat (remote).   They aimed to develop ways to integrate Open Tree’s resources into workflows for analysis of character data and other data.  They already have a nice presentation on their wiki.

Library wrappers for OT APIs involved Joseph Brown, Mark Holder (remote), Jon Hill, Matt Yoder, François Michonneau, Jeet Sukumaran, David Winter, and Karolis Ramanauskas.  The aim of this group was to develop programmable interfaces to Open Tree’s web services in Python, Ruby and R.  They developed an innovative test scheme in which all the libraries were subjected to the same tests.

Phylogeny visualization style-sheets were the focus of Peter Midford (remote), Jim Allman (remote), Pandurang Kolekar (remote), Daisie Huang, Gaurav Vaidya, Julie Allen, and Mike Rosenberg (remote).  Every year thousands of researchers generate  tree images, import them into a graphics editor, and add the same kinds of adornments (colored branches, numbers on nodes, images at the tips, brackets, etc).   The aim of this group was to develop and implement a scheme to treat graphical markups as styles in a separate document (because most tree formats don’t have room for markup), analogous to stylesheets for web pages.

The taxon sampling team included Andréa Matsunaga, Kayce Bell, Dilrini de Silva, Jonathan Rees, Nicky NIcolson and Arlin Stoltzfus.  This group focused on ways to get a phylogeny that represents a sample from a larger taxon— a sample that integrates some useful data, or is otherwise representative of the taxon.

The branch lengths team, including Lyndon Coghill (remote), Rachel Warnock, Josef Uyeda, Katie Lyons, Miranda Sinnott-Armstrong, Bob Thacker (remote), and Curt Lisle (remote) explored ways to address the challenge of adding branch lengths to the synthetic tree.  Like most supertrees, the synthetic tree lacks branch lengths, which limits its usefulness in many kinds of evolutionary studies.

A major knowledge engineering challenge for the Tree of Life community is to link knowledge to nodes in a comprehensive tree, and then ensure that this knowledge persists (as appropriate) when the tree is updated.  A scheme for addressing this challenge was developed and implemented by the annotation database group, including Cody Hinchliff, Karen Cranston, Stephen Smith, Joseph Brown, Mark Holder (remote), Hilmar Lapp (remote) and Temi Varghese.


Next week, I’ll start to describe the work of the taxon sampling team.  To be sure you hear about future posts, click “Follow” in the WordPress bar above this pane.


[1] The identification of any specific commercial products is for the purpose of specifying a protocol, and does not imply a recommendation or endorsement by the National Institute of Standards and Technology.


3 responses

  1. Pingback: Tree-for-All hackathon series: Taxon sampling, part 1  | OPEN TREE OF LIFE

  2. Pingback: Eating of Pudding | Global Biotic Interactions

  3. Pingback: Tree-for-All hackathon series: taxon sampling, part 2 | OPEN TREE OF LIFE