The new Mac Pro, Intel Phi, and the future of Bioinformatics

There was a time not too long ago where you could get a relatively easy publication in bioinformatics simply by slapping together a current algorithm and a GPU, and benchmark it against the same data set using a single-thread for the algorithm. I tended to dismiss all of these efforts, because once you look closely at what they actually did, you could find ways to easily outperform the GPU implementation using standard threading, or Map-Reduce over a cluster, at much less cost in the short and long terms. Data staging to the GPU was generally the Acheles heel of these algorithms, and there was just no getting around that step in high-throughput genomics.

The other detriment to these sorts of algorithms was that most bioinformatitians tended to stick to interpreted languages such as Perl, Ruby and Python. The programmer productivity gains were tremendous and the languages are “fast enought”.

There are two major changes in the ecosystem that are having me rethink my long-standing biases. First and foremost, properly conducted next-generation sequencing experiments are quite ridiculous in size. Certainly a lot bigger that any technology before it (in biology). New algorithms, like STAR for RNA-Seq alignment, show us the value of compiled languages paired with big RAM and big data pipes.

Second, the introduction of faster bus speeds (Thunderbolt 2), the Intel Phi architecture, and the new Mac Pro with its pair of standard pro-class GPUs, will drastically alter the economics and complexity of staging data to GPUs for computation.

In parallel to this, you have Continuum IO developing Python modules for fast scientific computation and I/O on GPUs (Numba and Blaze). Last but not least, we have C++11 and GNU gcc 4.8, with a slew of features being added to the standard library useful for bioiformatics (hello regular expressions ;) ).

It is my prediction that the future of NGS bioinformatics, or at least the cutting edge, will lie not with Hadoop and other “Big Data” infrastrucures, but rather with high-performance algorithms that take advantage of the new architectures coming to the pro-sumer market.

Don't do this

This past week I gave my boss and co-workers notice that I was moving on from UPENN. Normally you wouldn't do this sort of thing until you actually have an offer letter in hand, and for most people I would recommend the latter strategy.

So why did I do it? The main reason is that I love the group I work for. Specifically Garret FitzGerald and John Hogenesch are fantastic researchers, mentors, and all around great people. I have a broad role and I don't want them to be caught with their pants down when I do have that offer letter. Even with the four weeks notice that I will be giving them, it still won't be enough time to properly hand off my duties to other people. I specifically wanted them to have as much time to prepare and to prioritize my responsibilities.

 You may be thinking to yourself right about now "if you love it so much, why are you leaving?" Simply put, I currently don't see a long term future for me at UPENN. Once I realized that, I started really asking myself "what am I good at?", "what do I like to do the most?", and "what do I want to do in the future?" To answer the first two, I like and am extremely good at talking to researchers about their problems and about how to solve those problems with informatics. I like to build prototypes and design long-term scalable architectures that address actual research questions. In short, a Solutions Architect role fits my skillset very nicely.

 The last question has to do with the problem domain I want to solve. Genomics, my current research field, is poised to change medicine is ways we can't yet predict. We are already seeing the value of genetic sequencing in the diagnostics field, especially with respect to cancer. We are starting to see that our internal microbe populations play a larger part in our overall health than previously thought. Most of the current work studying the microbiome would not be possible without next generation sequencing. I want to be part of that sea change. I want to enable the use of high-throughput sequencing, and other 'omics technologies, in biomedical research that will have a huge and immediate impact on medicine. It should be noted that I could probably do that now in the group I am in, and I had already taken that into account when I thought long and hard about wanting to move on.

 So where to now? Good question. I'll have an answer for you, my captive audience, in about three weeks.

Getting Promise Pegasus to actually email you in case of drive failure

UPDATE: ContrarySheep (e.g. GriffithStudio) has posted an excellent update to these scripts that take advantage of launchd to run these processes. See this gist for more information.

So the Promise Pegasus thunderbolt external storage array is a super piece of hardware for the price. One thing they list as a feature is the ability to email a user in case of a drive failure. One problem, though, is that I could not get this to work for me.

A nice feature of the system, though, is that the included Promise Utility GUI that ships with the array also installs quite a full featured command line utility. For instance, you can get a quick report of the drives in the array by issuing the following:




w00t!


I wrote a quick little Ruby script to grab the drive information and send me an email in case the status is not "OK".



DISCLAMIER: This snippet is provided as-is. I am not guaranteeing this works at all, and is using the assumption that the status will change from "OK" to something else if there is an issue with a drive. Use at your own risk.

Compile BLAT on x86_64 Ubuntu 10.04

Recently I needed to compile BLAT for x86_64 on Ubuntu 10.04 server. The BLAT pre-compiled binaries for linux are 32-bit. Here is how you do it:

1. apt-get install build-essential
2. Download  BLAT source
3. remove the -Werror compiler flag to treat warnings as errors
4. mkdir -p ~/bin/x86_64
5. export MACHTYPE=x86_64
6. make

Step (3) above requires editing the /inc/common.mk file, line 17 goes from     


HG_WARN_ERR = -DJK_WARN -Wall -Werror


to 


HG_WARN_ERR = -DJK_WARN -Wall

Make places the executables into ~/bin/x86_64 directory. Either add this to $PATH or copy the files to /usr/local/bin

Ruby + AWS == Easy Map-Reduce

As read lengths grow, we have found that the current crop of short read aligners don't seem to do very well with RNA-seq data. In a recent experiment we were only able to map about 40% of the reads using Bowtie. BLAT was able to map an additional 45% percent of the reads, but was orders of magnitude slower than Bowtie (3 hours vs 8 days.)

I decided to try out my hand at using Amazon Web Services (AWS) to parallelize the BLAT search. Naturally I investigated Hadoop, and in particular AWS Elastic Map Reduce (EMR) service, which handles the particulars of setting up a Hadoop cluster for you. While Hadoop is a great project, and lots of people are having great success with it, there is a certain level of investment that must be made to learn how to mold your particular data analysis pipeline to fit into Hadoop's assumptions. For example, Hadoop assumes that the input file is large and that it will need to spilt the input for parallel processing on the worker nodes. The default method for splitting the input is on line breaks, which is not useful for FASTA sequence files. There are ways to customize this behavior, but I had to have results in two days for John Hogenesch to present at a cloud computing session for Science Online 2010.

Lucky for me, I had previously investigated another Map Reduce framework, CloudCrowd, which was developed by Document Cloud, a non-profit agency that archives source documents (scan PDF's, thumbnail pages, OCR) for news articles. The system is elegantly simple, built around the notion of a central work queue and worker nodes communicating over HTTP requests. A "job" is defined by specifying a set of input documents, an action to perform on those documents, and any other options that you want to pass along. The action is a simple Ruby class. CloudCrowd takes care of distributing the input files to worker nodes, and cleans up after itself pretty nicely. One last point is that the documentation is fairly complete, easy to use, and provides clear examples that work off the bat on install.

For our use case, the action we want to perform is the align all of the unmapped sequences using BLAT. In order for CloudCrowd to run BLAT, we must define a Ruby class that inherits from CloudCrowd::Action and implements a system call to BLAT:



CloudCrowd::Action takes care of a lot of the details for staging a work unit, such as creating working directories, downloading input files, launching the subprocesses, and cleaning up after itself when the job completes.

One difference between Hadoop and CloudCrowd is that while Hadoop will split the input and parcel out work items to nodes, CloudCrowd will send an entire input file to a single node. This means that I had to split the sequences into multiple FASTA files before sending them on to CloudCrowd. Using 100K sequences per file, we had 303 FASTA files to align. Note, that this is not lost work, as I would have had to define the split routine for Hadoop anyway. All of the input files, BLAT executable, and the search database were transferred to a S3 bucket.




Next I provisioned an small EC2 instance to act as the central CloudCrowd master, giving it a stable IP and configuring the connection options for CloudCrowd to connect to S3 and HTTP basic authentication for the nodes (see the CloudCrowd documentation). This manual process is fine for the master, but I worker nodes to automatically configure themselves, start the CloudCrowd node process, connect to the master and get jobs. You can provide EC2 nodes with a shell script that get's executed on start in what they call "user data". Here is my script (modified to protect my credentials and the data ;) ):



That script takes the public ubuntu 9.10 image (ami-55739e3c), adds the latest public repositories to the apt sources, updates all the packages and installs the latest Ruby and RubyGems packages. In addition it installs the CloudCrowd gem, downloads and installs BLAT, uses a pre-authenticated URL to fetch the pre-packaged CloudCrowd instance that I made (which has my AWS credentials) that actually defnes the BLAT job, and runs a CloudCrowd node process.

Now as nodes became available, they connected to the master and started working. If the nodes go down, the system would automatically cope with the node failures by resending the work to other nodes. We could also add new nodes as we see fit to decrease the time to completion. This type of fluid set up allowed me to take advantage of EC2 "spot instances" which have a variable pricing model based on demand and tends to be cheaper than the full price (12¢ vs. 34¢ per hour), with the caveat that your instance may come up at some random time and get shut down whenever the price exceeds your request maximum. For this particular job, I bid $0.25 and the actual price fluctuated from $0.12 to $0.18 per hour.

Finally we create a HTTP request using the following Ruby script, which designates the input files and database located in a public S3 bucket, as well as defines the parameters to use:



The system is not perfect and could use a lot of tweeking, but not bad for a day's work.

Heuristically Beneficial Artifacts

I've recently embarked on a study to see what effect clustering mass spectra has on the design of MRM experiments. Specifically one clusters spectra from the shotgun proteomics discovery phase of the experiment to see how it affects the selection of peptide species and the transitions that a mass spec looks for during the second targeted sequencing phase of the experiment.

While the work is ongoing, a very interesting (at least to me) result has come out of plotting the unfiltered SEQUEST results from the non-clustered and clustered version of the mass spec data. The say a picture is worth a thousand words, so:


The chart is showing the XCORR values of the clustered spectra plotted against the XCORR values for their respective members, independent of what peptide was identified for each. The color coding represent scores for peptides that are part of the decoy database, where:

• blue = a decoy hit was scored in the clustered spectra, but not the original
• green = a decoy hit was scored in the original spectra, but not the clustered spectra
• red = both spectra scored a decoy peptide

What you'll note is that spectra that all clustered score a different set of peptides higher than do the original spectra, and vice versa. At least for this data set, which is SILAC labeled, we note that each method is identifying a different population of peptides. This goes against all current publications, so I have to be careful in the interpretation of these results and will need a significant amount of validation experiments, but this is exciting stuff!

Particularly, it is going to be very interesting to see whether clustering actually helps or hinders ion selection when designing MRM experiments. If it does help ion selection, I have already come up with the catchy name, Heuristically Beneficial Artifacts ^TM. If not, well, we will at least have looked at the real-world effects of a methodology when applied to a new type of experimental goal.
More on this as it develops.

Not so much about science...

So in looking back over that last few entries, it seems I have strayed from my original mission, which was to cover issues regarding informatics and how it relates to science. The first part seems to have dominated the posts recently, so I am dropping all pretenses starting a new blog, over at Blogger competitor Wordpress.

Why a new platform? I figured I would give Wordpress a spin to see how it compares to blogger. Also I like the template they had for the blog ;) So far, the interface is similar, but looks a bit cleaner than Blogger's. Overall, it seems the same functionality and workflow, with the addition of one thing that I can see coming in very useful for a site like I envision, permanent pages.

In effect, If I write something that I want to be prominently displayed at all times, I create a page. This can come in handy say, if I give a set of instructions for setting up your development environment, or for providing a listing of resources. As a regular blog post these types of items are "discoverable" by searching, but they will eventually go away as new posts drown them out.

Another agument for the Page as opposed to a post is to combat a post from becoming stale. How many times have you thought you found an answer to a question in a blog post but it turns out said post is a bit old and the solution no longer applies? I guess stackoverflow seeks to remedy this, but bloggers can do their part by posting "important" and generally applicable posts with prominence as a page, thus they are always reminded to update it, since is it always visible.

Anywho, there I go again, talking strictly about web applications. Well, I guess I'll shut up now and you can check AppMecha for more posts related to these issues.