Getting Promise Pegasus to actually email you in case of drive failure

UPDATE: ContrarySheep (e.g. GriffithStudio) has posted an excellent update to these scripts that take advantage of launchd to run these processes. See this gist for more information.

So the Promise Pegasus thunderbolt external storage array is a super piece of hardware for the price. One thing they list as a feature is the ability to email a user in case of a drive failure. One problem, though, is that I could not get this to work for me.

A nice feature of the system, though, is that the included Promise Utility GUI that ships with the array also installs quite a full featured command line utility. For instance, you can get a quick report of the drives in the array by issuing the following:




w00t!


I wrote a quick little Ruby script to grab the drive information and send me an email in case the status is not "OK".



DISCLAMIER: This snippet is provided as-is. I am not guaranteeing this works at all, and is using the assumption that the status will change from "OK" to something else if there is an issue with a drive. Use at your own risk.

Compile BLAT on x86_64 Ubuntu 10.04

Recently I needed to compile BLAT for x86_64 on Ubuntu 10.04 server. The BLAT pre-compiled binaries for linux are 32-bit. Here is how you do it:

1. apt-get install build-essential
2. Download  BLAT source
3. remove the -Werror compiler flag to treat warnings as errors
4. mkdir -p ~/bin/x86_64
5. export MACHTYPE=x86_64
6. make

Step (3) above requires editing the /inc/common.mk file, line 17 goes from     


HG_WARN_ERR = -DJK_WARN -Wall -Werror


to 


HG_WARN_ERR = -DJK_WARN -Wall

Make places the executables into ~/bin/x86_64 directory. Either add this to $PATH or copy the files to /usr/local/bin

Ruby + AWS == Easy Map-Reduce

As read lengths grow, we have found that the current crop of short read aligners don't seem to do very well with RNA-seq data. In a recent experiment we were only able to map about 40% of the reads using Bowtie. BLAT was able to map an additional 45% percent of the reads, but was orders of magnitude slower than Bowtie (3 hours vs 8 days.)

I decided to try out my hand at using Amazon Web Services (AWS) to parallelize the BLAT search. Naturally I investigated Hadoop, and in particular AWS Elastic Map Reduce (EMR) service, which handles the particulars of setting up a Hadoop cluster for you. While Hadoop is a great project, and lots of people are having great success with it, there is a certain level of investment that must be made to learn how to mold your particular data analysis pipeline to fit into Hadoop's assumptions. For example, Hadoop assumes that the input file is large and that it will need to spilt the input for parallel processing on the worker nodes. The default method for splitting the input is on line breaks, which is not useful for FASTA sequence files. There are ways to customize this behavior, but I had to have results in two days for John Hogenesch to present at a cloud computing session for Science Online 2010.

Lucky for me, I had previously investigated another Map Reduce framework, CloudCrowd, which was developed by Document Cloud, a non-profit agency that archives source documents (scan PDF's, thumbnail pages, OCR) for news articles. The system is elegantly simple, built around the notion of a central work queue and worker nodes communicating over HTTP requests. A "job" is defined by specifying a set of input documents, an action to perform on those documents, and any other options that you want to pass along. The action is a simple Ruby class. CloudCrowd takes care of distributing the input files to worker nodes, and cleans up after itself pretty nicely. One last point is that the documentation is fairly complete, easy to use, and provides clear examples that work off the bat on install.

For our use case, the action we want to perform is the align all of the unmapped sequences using BLAT. In order for CloudCrowd to run BLAT, we must define a Ruby class that inherits from CloudCrowd::Action and implements a system call to BLAT:



CloudCrowd::Action takes care of a lot of the details for staging a work unit, such as creating working directories, downloading input files, launching the subprocesses, and cleaning up after itself when the job completes.

One difference between Hadoop and CloudCrowd is that while Hadoop will split the input and parcel out work items to nodes, CloudCrowd will send an entire input file to a single node. This means that I had to split the sequences into multiple FASTA files before sending them on to CloudCrowd. Using 100K sequences per file, we had 303 FASTA files to align. Note, that this is not lost work, as I would have had to define the split routine for Hadoop anyway. All of the input files, BLAT executable, and the search database were transferred to a S3 bucket.




Next I provisioned an small EC2 instance to act as the central CloudCrowd master, giving it a stable IP and configuring the connection options for CloudCrowd to connect to S3 and HTTP basic authentication for the nodes (see the CloudCrowd documentation). This manual process is fine for the master, but I worker nodes to automatically configure themselves, start the CloudCrowd node process, connect to the master and get jobs. You can provide EC2 nodes with a shell script that get's executed on start in what they call "user data". Here is my script (modified to protect my credentials and the data ;) ):



That script takes the public ubuntu 9.10 image (ami-55739e3c), adds the latest public repositories to the apt sources, updates all the packages and installs the latest Ruby and RubyGems packages. In addition it installs the CloudCrowd gem, downloads and installs BLAT, uses a pre-authenticated URL to fetch the pre-packaged CloudCrowd instance that I made (which has my AWS credentials) that actually defnes the BLAT job, and runs a CloudCrowd node process.

Now as nodes became available, they connected to the master and started working. If the nodes go down, the system would automatically cope with the node failures by resending the work to other nodes. We could also add new nodes as we see fit to decrease the time to completion. This type of fluid set up allowed me to take advantage of EC2 "spot instances" which have a variable pricing model based on demand and tends to be cheaper than the full price (12¢ vs. 34¢ per hour), with the caveat that your instance may come up at some random time and get shut down whenever the price exceeds your request maximum. For this particular job, I bid $0.25 and the actual price fluctuated from $0.12 to $0.18 per hour.

Finally we create a HTTP request using the following Ruby script, which designates the input files and database located in a public S3 bucket, as well as defines the parameters to use:



The system is not perfect and could use a lot of tweeking, but not bad for a day's work.

Heuristically Beneficial Artifacts

I've recently embarked on a study to see what effect clustering mass spectra has on the design of MRM experiments. Specifically one clusters spectra from the shotgun proteomics discovery phase of the experiment to see how it affects the selection of peptide species and the transitions that a mass spec looks for during the second targeted sequencing phase of the experiment.

While the work is ongoing, a very interesting (at least to me) result has come out of plotting the unfiltered SEQUEST results from the non-clustered and clustered version of the mass spec data. The say a picture is worth a thousand words, so:


The chart is showing the XCORR values of the clustered spectra plotted against the XCORR values for their respective members, independent of what peptide was identified for each. The color coding represent scores for peptides that are part of the decoy database, where:

• blue = a decoy hit was scored in the clustered spectra, but not the original
• green = a decoy hit was scored in the original spectra, but not the clustered spectra
• red = both spectra scored a decoy peptide

What you'll note is that spectra that all clustered score a different set of peptides higher than do the original spectra, and vice versa. At least for this data set, which is SILAC labeled, we note that each method is identifying a different population of peptides. This goes against all current publications, so I have to be careful in the interpretation of these results and will need a significant amount of validation experiments, but this is exciting stuff!

Particularly, it is going to be very interesting to see whether clustering actually helps or hinders ion selection when designing MRM experiments. If it does help ion selection, I have already come up with the catchy name, Heuristically Beneficial Artifacts ^TM. If not, well, we will at least have looked at the real-world effects of a methodology when applied to a new type of experimental goal.
More on this as it develops.

Not so much about science...

So in looking back over that last few entries, it seems I have strayed from my original mission, which was to cover issues regarding informatics and how it relates to science. The first part seems to have dominated the posts recently, so I am dropping all pretenses starting a new blog, over at Blogger competitor Wordpress.

Why a new platform? I figured I would give Wordpress a spin to see how it compares to blogger. Also I like the template they had for the blog ;) So far, the interface is similar, but looks a bit cleaner than Blogger's. Overall, it seems the same functionality and workflow, with the addition of one thing that I can see coming in very useful for a site like I envision, permanent pages.

In effect, If I write something that I want to be prominently displayed at all times, I create a page. This can come in handy say, if I give a set of instructions for setting up your development environment, or for providing a listing of resources. As a regular blog post these types of items are "discoverable" by searching, but they will eventually go away as new posts drown them out.

Another agument for the Page as opposed to a post is to combat a post from becoming stale. How many times have you thought you found an answer to a question in a blog post but it turns out said post is a bit old and the solution no longer applies? I guess stackoverflow seeks to remedy this, but bloggers can do their part by posting "important" and generally applicable posts with prominence as a page, thus they are always reminded to update it, since is it always visible.

Anywho, there I go again, talking strictly about web applications. Well, I guess I'll shut up now and you can check AppMecha for more posts related to these issues.

Why Thn.gs should send a chill down evey ISV's spine

Independent software vendors (ISV's) have recently been touting some of the most useful applications that I have seen in my entire career. From 37signals to GitHub to CulturedCode, the genius of these applications are simple interfaces to complex sets of requirements. That last one, CulturedCode, has a problem though. Their flagship about to launch product just got pwned by some russians.

While 37signals and GitHub are already web applications, with a service based revenue stream, CC's product is an OS X desktop application that works beautifully as a get-things-done task orginizer, with a perpetual licensing model. The aforementioned russians took every piece of this application and made it into what appears to be an almost complete equivilient of the desktop application. And if that we not bad enough, they integrated it into Google Gears for offline use.

What can CC do about this? Not much, unless they already have a huge bankroll for lawyers. And this situation should send chills down the spine of every small software shop releasing small useful tools. The very nature of these shops constrains feature creap, which in turn forces the designers and developers to squeeze the most they can out of what they have in place, which in turn makes the software simple and powerful at the same time. But it also makes these applications vulnerable to xeroxing using the web, (relatively) cheap labor pools, and robust distribution network with firewall-like immunity to legal action in the form of international borders ( "litigation-wall" ?).

Anyone want to take bets on who is next on the feed tray? My guess is a Delicious Library web clone.

So take home message is, design software that is not easily replicated, either by feature or connectivity, or realize that your app really is easy to xerox and have a large maerketing scheme to drown out any news of the enemy.

Speaking of Google, there was that little hiccup in relations related to releasing a clone 37signal's campfire with the release App Engine. And then there is Chrome+Gears, the browser-DB combination that makes web-applications even more desktop like. Not that they are alone in this, Adobe (AIR) and Microsoft (SilverLight will certainly "extend" the reach of IE) are walking the same browser-as-platform path, and trying to building their market share on the "everything should be free" web culture, and that really raises my mercury. Fucking piracy enablers.

Licensing makes Cloud Computing ... difficult

I have thought a lot in recent months about how to best leverage cloud computing resources, or utility computing, that are increasingly becoming available to the general development community and one issue in particular makes me cringe: LICENSING.

It just so happens that in my field, proteomics, the open source set of tools gather a lot of press, but really most submissions to publish research are still using commercial algorithms for the initial data analysis, even as plenty of research has been show (and published) that results from commercial and open source algorithms are comparable. There is an inherent level of trust manuscript reviewers have in the commercial offerings that is hard to overcome, hence most researchers still opt to use the commercial algorithms as the gold standard.

Not that this is a bad thing, mind you. As someone whom supports the informatics efforts for many researchers, I find that the commercial offerings are much more stable, and are subsequently easier to support and maintain than most of the current crop of open source offerings.

The trouble lies with rigid licensing models of commercial offerings. Specifically you must purchase perpetual licenses for a certain number of compute cores. Such a model is just not compatible with what I would like to do as a service center, namely to provide software-as-service billing to researchers. True the high up-front licensing can be amortized over the life of the support contract, but it assumes that the computers running the algorithm are already procured and dedicated to the software (also not a bad assumption in most cases, since the hardware costs pale in comparison to the licensing).

In effect, there is no way to make a utility computing model, such as one offered by Amazon Web Services, work with these sorts of license restrictions. The set-up and tear down of a compute job is too high to be a viable full time solution.

What I would like to do is augment my current computing capacity during crunch times. Dedicated licensing prevents this. As does the way most networked algorithms work, but that's another post.