tag:blogger.com,1999:blog-79407300100224851112024-03-14T00:04:07.353-04:00def science { }bioinformatics on the cloud.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comBlogger50125tag:blogger.com,1999:blog-7940730010022485111.post-78069855768072842852017-02-01T08:45:00.000-05:002017-02-01T08:45:29.930-05:00To be (AWS certified), or not to be (AWS certified)A couple of my bioinformatics colleagues have recently asked me about whether they should be <a href="https://aws.amazon.com/certification/" target="_blank">AWS certified</a>, and, if so, which certification would be best.<br />
<br />
My general answer is that AWS certification doesn't hurt to have, but it is should not be considered a "must have" requirement for bioinformatics positions. The main reason I say this is that the certifications test, at least at the Associates level, tend to cover a broad set of AWS services, and some will most certainly not apply to many bioinformatics positions. As a community, we tend to go very deep in certain technologies as a function of our projects. Bioinformatics spans a varied set of technology stacks including web applications, batch analysis of large corpuses of data, machine learning, etc. But any individual bioinformatics practitioner will tend to focus on only one of these types of projects for long stretches of time, usually measured in years.<br />
<br />
Still, it does not hurt to have a Associate level AWS certification in either<a href="https://aws.amazon.com/certification/certified-solutions-architect-associate/" target="_blank"> Solutions Architecture</a> or <a href="https://aws.amazon.com/certification/certified-developer-associate/" target="_blank">Software Developer</a>. These two in particular will teach the foundational services that one would encounter in using AWS for bioinformatics, as well as give you a grounding in how to navigate the documentation and leverage the AWS ecosystem when you need to dive deeper on some service or domain.<br />
<br />
<br />Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-84466630070490387252016-12-23T21:26:00.004-05:002016-12-23T21:26:42.479-05:00Post on BAM index file parsing. I posted a note on parsing <a href="http://delagoya.github.io/rusty-bio/binary-file-parsing-in-bioinformatics" target="_blank">binary files using Python</a> over on my <a href="http://delagoya.github.io/rusty-bio/" target="_blank">Rusty Bioinformatics</a> blog. The example uses BAM index files, which are well described but do not have a software library that specifically targets reading and writing these types of files. Enjoy!Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-27264446283609316662015-08-22T21:39:00.001-04:002015-08-22T21:39:36.051-04:00DNA reverse complement program in RustAfter a very long hiatus, I am starting to write for pleasure again. In addition to posting some opinion pieces here, I am working on another blog, <a href="http://delagoya.github.io/rusty-bio/">Rusty Bioinformatics</a>, that is more focused on (re)learning bioinformatics programming.<br />
<br />
The first post of note is a writing a simple program to provide the reverse complement of a DNA sequence on the command line, written in the Rust programming language. Find it here:<br />
<br />
<a href="http://delagoya.github.io/rusty-bio/dna-reverse-complement" target="_blank">DNA reverse complement in Rust</a><br />
<br />
<br />
Enjoy!Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-62966293950565388792015-08-22T21:30:00.001-04:002015-08-22T21:30:43.685-04:00The AWS genomics ecosystem<div class="p1">
<span class="s1">This past week, my good friend and collegue at AWS, <a href="https://www.linkedin.com/pub/christopher-crosbie/61/349/814"><span class="s2">Chris Crosbie</span></a>, published <a href="https://blogs.aws.amazon.com/bigdata/post/TxB9H9MGP4JBBQ/Extending-Seven-Bridges-Genomics-with-Amazon-Redshift-and-R"><span class="s2">a blog post</span></a> about analyzing whole genome sequence data with one of our genomic analysis platform partners, <a href="https://www.sbgenomics.com/"><span class="s2">Seven Bridges Genomics</span></a>, importing the resulting VCF files to <a href="http://aws.amazon.com/redshift/"><span class="s2">Amazon Redshift</span></a>, and doing a simple analysis of the data within <a href="https://www.bioconductor.org/help/bioconductor-cloud-ami/"><span class="s2">R and Bioconductor</span></a>.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">The post is lengthy and technically dense, but it is delightful in that it showcases one of the major advantages of working with genomics data on AWS: namely the rich and diverse ecosystem of tools and platform providers that a researcher can bundle together to fit their exact needs.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">There has certainly been a failing on my part to communicate the fullness of this ecosystem, and Chris’s post has been a good reminder to me that I need to communicate better (and more often) to the broader bioinformatics community about how to effectively pair down their choices when facing some analysis challenge.</span></div>
<div class="p1">
</div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">PS: If you like Chris’s post, be sure to read the other one we put together with Matt Wood about <a href="http://tothestars.io/2015/5/controlled-access-genomics-data-in-the-cloud.html"><span class="s2">uploading dbGaP data to Amazon S3</span></a>.</span></div>
Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-23306083618878544782013-08-04T09:33:00.002-04:002013-08-04T09:33:50.943-04:00Exciting New AdventuresLast week marked a major turning point in my career, as it was my last week working for Penn. I started in September 1997, almost 16 years of service to a fantastic organization that allowed me to learn, experiment, and grow as a person, engineer, and scientist. It has made me what I am today and I will forever be grateful to the organization and everyone I have worked for and with. Leaving it was an incredibly hard choice to make, but I am confident it was the right thing to do.<br />
<br />
Monday August 5th I start at Amazon Web Services as a Sr. Solutions Architect. While I have some idea of what the role entails, which is generally to help people use cloud computing for solving scientific and high-performance computing problems, I am under no illusion that I will know what it will actually be like in practice. Or what career paths just opened up for me.<br />
<br />
I am super excited and am bouncing-off-the-walls excited for tomorrow to come.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-84528010403558941142013-06-22T11:49:00.001-04:002013-06-22T11:49:53.721-04:00The new Mac Pro, Intel Phi, and the future of BioinformaticsThere was a time not too long ago where you could get a relatively easy publication in bioinformatics simply by slapping together a current algorithm and a GPU, and benchmark it against the same data set using a single-thread for the algorithm. I tended to dismiss all of these efforts, because once you look closely at what they actually did, you could find ways to easily outperform the GPU implementation using standard threading, or Map-Reduce over a cluster, at much less cost in the short and long terms. Data staging to the GPU was generally the Acheles heel of these algorithms, and there was just no getting around that step in high-throughput genomics.<br />
<br />
The other detriment to these sorts of algorithms was that most bioinformatitians tended to stick to interpreted languages such as Perl, Ruby and Python. The programmer productivity gains were tremendous and the languages are “fast enought”. <br />
<br />
There are two major changes in the ecosystem that are having me rethink my long-standing biases. First and foremost, properly conducted next-generation sequencing experiments are quite ridiculous in size. Certainly a lot bigger that any technology before it (in biology). New algorithms, like <a href="https://code.google.com/p/rna-star/">STAR</a> for RNA-Seq alignment, show us the value of compiled languages paired with big RAM and big data pipes.<br />
<br />
Second, the introduction of faster bus speeds (Thunderbolt 2), the Intel Phi architecture, and the new Mac Pro with its pair of standard pro-class GPUs, will drastically alter the economics and complexity of staging data to GPUs for computation. <br />
<br />
In parallel to this, you have <a href="http://continuum.io/">Continuum IO</a> developing Python modules for fast scientific computation and I/O on GPUs (<a href="https://github.com/numba/numba">Numba</a> and <a href="http://blaze.pydata.org/">Blaze</a>). Last but not least, we have <a href="http://en.wikipedia.org/wiki/C%2B%2B11">C++11</a> and <a href="http://gcc.gnu.org/projects/cxx0x.html">GNU gcc 4.8</a>, with a slew of features being added to the standard library useful for bioiformatics (<em>hello regular expressions</em> <code>;)</code> ).<br />
<br />
It is my prediction that the future of NGS bioinformatics, or at least the cutting edge, will lie not with Hadoop and other “Big Data” infrastrucures, but rather with high-performance algorithms that take advantage of the new architectures coming to the pro-sumer market. Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-34984434519886495672013-06-22T09:28:00.001-04:002013-06-22T09:29:13.871-04:00Don't do thisThis past week I gave my boss and co-workers notice that I was moving on from UPENN. <b>Normally you wouldn't do this sort of thing until you actually have an offer letter in hand</b>, and for most people I would recommend the latter strategy.<br />
<br />
So why did I do it?
The main reason is that I love the group I work for. Specifically <a href="http://www.itmat.upenn.edu/faculty_fitzgerald.shtml">Garret FitzGerald</a> and <a href="http://www.med.upenn.edu/apps/faculty/index.php/g275/p8127424">John Hogenesch</a> are fantastic researchers, mentors, and all around great people. I have a broad role and I don't want them to be caught with their pants down when I do have that offer letter. Even with the four weeks notice that I will be giving them, it still won't be enough time to properly hand off my duties to other people. I specifically wanted them to have as much time to prepare and to prioritize my responsibilities.<br />
<br />
You may be thinking to yourself right about now "if you love it so much, why are you leaving?" Simply put, I currently don't see a long term future for me at UPENN. Once I realized that, I started really asking myself "what am I good at?", "what do I like to do the most?", and "what do I want to do in the future?"
To answer the first two, I like and am extremely good at talking to researchers about their problems and about how to solve those problems with informatics. I like to build prototypes and design long-term scalable architectures that address actual research questions. In short, a Solutions Architect role fits my skillset very nicely.<br />
<br />
The last question has to do with the problem domain I want to solve. Genomics, my current research field, is poised to change medicine is ways we can't yet predict. We are already seeing the value of genetic sequencing in the diagnostics field, especially with <a href="http://www.nytimes.com/2013/05/14/opinion/my-medical-choice.html">respect to cancer</a>. We are starting to see that our internal microbe populations play a larger part in our overall health than previously thought. Most of the current work studying the microbiome would not be possible without next generation sequencing. I want to be part of that sea change. I want to enable the use of high-throughput sequencing, and other 'omics technologies, in biomedical research that will have a huge and immediate impact on medicine. It should be noted that I could probably do that now in the group I am in, and I had already taken that into account when I thought long and hard about wanting to move on.<br />
<br />
So where to now? Good question. I'll have an answer for you, my captive audience, in about three weeks. Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-22497227654141700862012-01-18T15:31:00.001-05:002012-08-30T09:50:34.169-04:00Getting Promise Pegasus to actually email you in case of drive failure<p>
<b>UPDATE:</b> <a href="http://contrarysheep.com"> ContrarySheep</a> (e.g. <a href="http://www.christophergriffith.com/#p=-1&a=0&at=0"> GriffithStudio</a>) has posted an excellent update to these scripts that take advantage of launchd to run these processes. See <a href="https://gist.github.com/3490955"> this gist</a> for more information.
</p>
So the Promise Pegasus thunderbolt external storage array is a super piece of hardware for the price. One thing they list as a feature is the ability to email a user in case of a drive failure. One problem, though, is that I could not get this to work for me. <br />
<br />
A nice feature of the system, though, is that the included Promise Utility GUI that ships with the array also installs quite a full featured command line utility. For instance, you can get a quick report of the drives in the array by issuing the following:<br />
<br />
<br />
<script src="https://gist.github.com/1635346.js?file=example.out.txt"></script><br />
<br />
<b>w00t!</b><br />
<br />
<br />
I wrote a quick little Ruby script to grab the drive information and send me an email in case the status is not "OK".<br />
<br />
<script src="https://gist.github.com/1635346.js?file=pegasus_check.rb"></script> <br />
<br />
<b>DISCLAMIER:</b> This snippet is provided as-is. I am not guaranteeing this works at all, and is using the assumption that the status will change from "OK" to something else if there is an issue with a drive. Use at your own risk.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-66514859319666249852011-02-21T09:48:00.000-05:002011-02-21T09:48:59.182-05:00Compile BLAT on x86_64 Ubuntu 10.04Recently I needed to compile BLAT for x86_64 on Ubuntu 10.04 server. The BLAT pre-compiled binaries for linux are 32-bit. Here is how you do it:<br />
<br />
<br />
<br />
<ol><li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">apt-get install build-essential</span></li>
<li>Download BLAT source</li>
<li>remove the -Werror compiler flag to treat warnings as errors</li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mkdir -p ~/bin/x86_6</span>4</li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">export MACHTYPE=x86_64</span></li>
<li>make</li>
</ol><div>Step (3) above requires editing the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><blat src="">/inc/common.mk </blat></span><span class="Apple-style-span" style="font-family: inherit;">file, line 17 goes from </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><code><br />
HG_WARN_ERR = -DJK_WARN -Wall -Werror<br />
</code><br />
<div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: inherit;">to </span></div><code><br />
HG_WARN_ERR = -DJK_WARN -Wall<br />
</code><br />
Make places the executables into <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">~/bin/x86_64</span> directory. Either add this to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$PATH</span> or copy the files to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/usr/local/bin</span>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-47876961702508925292010-01-22T21:38:00.010-05:002010-01-22T22:25:03.206-05:00Ruby + AWS == Easy Map-ReduceAs read lengths grow, we have found that the current crop of short read aligners don't seem to do very well with RNA-seq data. In a recent experiment we were only able to map about 40% of the reads using <a href="http://bowtie-bio.sourceforge.net/index.shtml">Bowtie</a>. <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC187518/?tool=pubmed">BLAT</a> was able to map an additional 45% percent of the reads, but was orders of magnitude slower than Bowtie (3 hours vs 8 days.)<br /><br />I decided to try out my hand at using <a href="http://aws.amazom.com">Amazon Web Services</a> (AWS) to parallelize the BLAT search. Naturally I investigated <a href="http://hadoop.apache.org/"> Hadoop</a>, and in particular <a href="http://aws.amazon.com/elasticmapreduce/">AWS Elastic Map Reduce</a> (EMR) service, which handles the particulars of setting up a Hadoop cluster for you. While Hadoop is a great project, and lots of people are having great success with it, there is a certain level of investment that must be made to learn how to mold your particular data analysis pipeline to fit into Hadoop's assumptions. For example, Hadoop assumes that the input file is large and that it will need to spilt the input for parallel processing on the worker nodes. The default method for splitting the input is on line breaks, which is not useful for FASTA sequence files. There are ways to customize this behavior, but I had to have results in two days for <a href='http://www.med.upenn.edu/pharm/faculty/hogenesch/hogenesch.html'> John Hogenesch</a> to present at a cloud computing session for <a href="http://www.scienceonline2010.com/">Science Online 2010</a>.<br /><br />Lucky for me, I had previously investigated another Map Reduce framework, <a href="http://github.com/documentcloud/cloud-crowd">CloudCrowd</a>, which was developed by <a href="http://www.documentcloud.org/"> Document Cloud</a>, a non-profit agency that archives source documents (scan PDF's, thumbnail pages, OCR) for news articles. The system is elegantly simple, built around the notion of a central work queue and worker nodes communicating over HTTP requests. A "job" is defined by specifying a set of input documents, an action to perform on those documents, and any other options that you want to pass along. The action is a simple Ruby class. CloudCrowd takes care of distributing the input files to worker nodes, and cleans up after itself pretty nicely. One last point is that the documentation is fairly complete, easy to use, and provides clear examples that work off the bat on install.<br /><br />For our use case, the action we want to perform is the align all of the unmapped sequences using BLAT. In order for CloudCrowd to run BLAT, we must define a Ruby class that inherits from <code>CloudCrowd::Action</code> and implements a system call to BLAT:<br /><br /><script src="http://gist.github.com/284378.js?file=blat.rb"></script><br /><br /><code>CloudCrowd::Action</code> takes care of a lot of the details for staging a work unit, such as creating working directories, downloading input files, launching the subprocesses, and cleaning up after itself when the job completes.<br /><br />One difference between Hadoop and CloudCrowd is that while Hadoop will split the input and parcel out work items to nodes, CloudCrowd will send an entire input file to a single node. This means that I had to split the sequences into multiple FASTA files before sending them on to CloudCrowd. Using 100K sequences per file, we had 303 FASTA files to align. Note, that this is not lost work, as I would have had to define the split routine for Hadoop anyway. All of the input files, BLAT executable, and the search database were transferred to a S3 bucket.<br /><br /><script src="http://gist.github.com/284378.js?file=split_fasta.rb"></script><br /><br /><br />Next I provisioned an small EC2 instance to act as the central CloudCrowd master, giving it a stable IP and configuring the connection options for CloudCrowd to connect to S3 and HTTP basic authentication for the nodes (see the CloudCrowd documentation). This manual process is fine for the master, but I worker nodes to automatically configure themselves, start the CloudCrowd node process, connect to the master and get jobs. You can provide EC2 nodes with a shell script that get's executed on start in what they call "user data". Here is my script (modified to protect my credentials and the data ;) ):<br /> <br /><script src="http://gist.github.com/284378.js?file=bootstrap_node.sh"></script><br /><br />That script takes the public ubuntu 9.10 image (ami-55739e3c), adds the latest public repositories to the apt sources, updates all the packages and installs the latest Ruby and RubyGems packages. In addition it installs the CloudCrowd gem, downloads and installs BLAT, uses a pre-authenticated URL to fetch the pre-packaged CloudCrowd instance that I made (which has my AWS credentials) that actually defnes the BLAT job, and runs a CloudCrowd node process.<br /><br />Now as nodes became available, they connected to the master and started working. If the nodes go down, the system would automatically cope with the node failures by resending the work to other nodes. We could also add new nodes as we see fit to decrease the time to completion. This type of fluid set up allowed me to take advantage of EC2 "spot instances" which have a variable pricing model based on demand and tends to be cheaper than the full price (12¢ vs. 34¢ per hour), with the caveat that your instance may come up at some random time and get shut down whenever the price exceeds your request maximum. For this particular job, I bid $0.25 and the actual price fluctuated from $0.12 to $0.18 per hour.<br /><br />Finally we create a HTTP request using the following Ruby script, which designates the input files and database located in a public S3 bucket, as well as defines the parameters to use:<br /> <br /><script src="http://gist.github.com/284378.js?file=submit_blat_job.rb"></script><br /><br />The system is not perfect and could use a lot of tweeking, but not bad for a day's work.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-51442329275332425992008-10-07T12:42:00.007-04:002008-10-07T12:50:13.404-04:00Heuristically Beneficial Artifacts<div xmlns="http://www.w3.org/1999/xhtml">I've recently embarked on a study to see what effect clustering mass spectra has on the design of MRM experiments. Specifically one clusters spectra from the shotgun proteomics discovery phase of the experiment to see how it affects the selection of peptide species and the transitions that a mass spec looks for during the second targeted sequencing phase of the experiment.<br /><br />While the work is ongoing, a very interesting (at least to me) result has come out of plotting the unfiltered SEQUEST results from the non-clustered and clustered version of the mass spec data. The say a picture is worth a thousand words, so:<br /><br /><img src="http://lh3.ggpht.com/delagoya/SOuO-ZoWbsI/AAAAAAAAA4Q/2NInGLLJ-ds/%5BUNSET%5D.png" style="max-width: 800px;" height="633" width="422" /><br />The chart is showing the XCORR values of the clustered spectra plotted against the XCORR values for their respective members, independent of what peptide was identified for each. The color coding represent scores for peptides that are part of the decoy database, where:<br /><br /><ul><li>blue = a decoy hit was scored in the clustered spectra, but not the original</li><li>green = a decoy hit was scored in the original spectra, but not the clustered spectra</li><li>red = both spectra scored a decoy peptide</li></ul>What you'll note is that spectra that all clustered score a different set of peptides higher than do the original spectra, and <span style="font-style: italic;">vice </span>versa. At least for this data set, which is SILAC labeled, we note that each method is identifying a different population of peptides. This goes against all current publications, so I have to be careful in the interpretation of these results and will need a significant amount of validation experiments, but this is exciting stuff!<br /><br />Particularly, it is going to be very interesting to see whether clustering actually helps or hinders ion selection when designing MRM experiments. If it does help ion selection, I have already come up with the catchy name, Heuristically Beneficial Artifacts<small><sup> TM</sup></small>. If not, well, we will at least have looked at the real-world effects of a methodology when applied to a new type of experimental goal.<br />More on this as it develops.<br /></div>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-63116033930984306872008-10-06T14:11:00.003-04:002008-10-06T14:25:30.690-04:00Not so much about science...So in looking back over that last few entries, it seems I have strayed from my original mission, which was to cover issues regarding informatics and <span style="font-weight: bold;">how it relates to science</span>. The first part seems to have dominated the posts recently, so I am dropping all pretenses starting <a href="http://appmecha.wordpress.com/">a new blog</a>, over at Blogger competitor <a href="http://wordpress.com/">Wordpress</a>.<br /><br />Why a new platform? I figured I would give Wordpress a spin to see how it compares to blogger. Also I like the template they had for the blog ;) So far, the interface is similar, but looks a bit cleaner than Blogger's. Overall, it seems the same functionality and workflow, with the addition of one thing that I can see coming in very useful for a site like I envision, permanent pages.<br /><br />In effect, If I write something that I want to be prominently displayed at all times, I create a page. This can come in handy say, if I give a set of instructions for setting up your development environment, or for providing a listing of resources. As a regular blog post these types of items are "discoverable" by searching, but they will eventually go away as new posts drown them out.<br /><br />Another agument for the Page as opposed to a post is to combat a post from becoming stale. How many times have you thought you found an answer to a question in a blog post but it turns out said post is a bit old and the solution no longer applies? I guess <a href="http://stackoverflow.com">stackoverflow</a> seeks to remedy this, but bloggers can do their part by posting "important" and generally applicable posts with prominence as a page, thus they are always reminded to update it, since is it always visible.<br /><br />Anywho, there I go again, talking strictly about web applications. Well, I guess I'll shut up now and you can check <a href="http://appmecha.wordpress.com/">AppMecha</a> for more posts related to these issues.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-20214102450154708752008-09-15T10:10:00.001-04:002008-09-15T10:10:25.322-04:00Why Thn.gs should send a chill down evey ISV's spine<div xmlns='http://www.w3.org/1999/xhtml'>Independent software vendors (ISV's) have recently been touting some of the most useful applications that I have seen in my entire career. From 37signals to GitHub to CulturedCode, the genius of these applications are simple interfaces to complex sets of requirements. That last one, CulturedCode, has a problem though. Their <a href='http://www.culturedcode.com/things/'>flagship about to launch product</a> just got <a href='http://thn.gs/'>pwned</a> by <a href='http://shergin.com/' target='_blank'>some russians</a>. <br/><br/>While 37signals and GitHub are already web applications, with a service based revenue stream, CC's product is an OS X desktop application that works beautifully as a get-things-done task orginizer, with a perpetual licensing model. The aforementioned russians took every piece of this application and made it into what appears to be an almost complete equivilient of the desktop application. And if that we not bad enough, they integrated it into Google Gears for offline use. <br/><br/>What can CC do about this? Not much, unless they already have a huge bankroll for lawyers. And this situation should send chills down the spine of every small software shop releasing small useful tools. The very nature of these shops constrains feature creap, which in turn forces the designers and developers to squeeze the most they can out of what they have in place, which in turn makes the software simple and powerful at the same time. But it also makes these applications vulnerable to xeroxing using the web, (relatively) cheap labor pools, and robust distribution network with firewall-like immunity to legal action in the form of international borders ( "litigation-wall" ?).<br/><br/>Anyone want to take bets on who is next on the feed tray? My guess is a <a href='http://www.delicious-monster.com/'>Delicious Library</a> web clone. <br/><br/>So take home message is, design software that is not easily replicated, either by feature or connectivity, or realize that your app really is easy to xerox and have a large maerketing scheme to drown out any news of the enemy.<br/><br/>Speaking of Google, there was that <a href='http://blog.searchenginewatch.com/blog/080408-123318'>little hiccup in relations</a> related to releasing a clone 37signal's campfire with the release App Engine. And then there is Chrome+Gears, the browser-DB combination that makes web-applications even more desktop like. Not that they are alone in this, Adobe (AIR) and Microsoft (SilverLight will certainly "extend" the reach of IE) are walking the same browser-as-platform path, and trying to building their market share on the "everything should be free" web culture, and that really raises my mercury. Fucking piracy enablers.</div>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-91746215504440100492008-09-12T09:57:00.001-04:002008-09-12T09:57:48.453-04:00Licensing makes Cloud Computing ... difficult<div xmlns='http://www.w3.org/1999/xhtml'>I have thought a lot in recent months about how to best leverage cloud computing resources, or utility computing, that are increasingly becoming available to the general development community and one issue in particular makes me cringe: LICENSING. <br/><br/>It just so happens that in my field, proteomics, the open source set of tools gather a lot of press, but really most submissions to publish research are still using commercial algorithms for the initial data analysis, even as plenty of research has been show (and published) that results from commercial and open source algorithms are comparable. There is an inherent level of trust manuscript reviewers have in the commercial offerings that is hard to overcome, hence most researchers still opt to use the commercial algorithms as the gold standard. <br/><br/>Not that this is a bad thing, mind you. As someone whom supports the informatics efforts for many researchers, I find that the commercial offerings are much more stable, and are subsequently easier to support and maintain than most of the current crop of open source offerings. <br/><br/>The trouble lies with rigid licensing models of commercial offerings. Specifically you must purchase perpetual licenses for a certain number of compute cores. Such a model is just not compatible with what I would like to do as a service center, namely to provide software-as-service billing to researchers. True the high up-front licensing can be amortized over the life of the support contract, but it assumes that the computers running the algorithm are already procured and dedicated to the software (also not a bad assumption in most cases, since the hardware costs pale in comparison to the licensing). <br/><br/>In effect, there is no way to make a utility computing model, such as one offered by Amazon Web Services, work with these sorts of license restrictions. The set-up and tear down of a compute job is too high to be a viable full time solution. <br/><br/>What I would like to do is augment my current computing capacity during crunch times. Dedicated licensing prevents this. As does the way most networked algorithms work, but that's another post.<br/><br/><br/><br/></div>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-80379590640950733572008-04-14T22:21:00.010-04:002008-04-28T10:40:31.939-04:00Pharma's FuturesAlternate title: "Bet Big, lose a long long time, then win so big that it makes your losses look like pocket change. Or maybe I'll just lose my shirt. Maybe both simultaneously, who knows?"<br /><br />This post is motivated by a set of lectures I attended as part of the <a href="http://www.itmat.upenn.edu/symposium.shtml">3rd Annual ITMAT Symposium on Translational Medicine</a> (Full disclosure, ITMAT is currently my employer). Here are the first two lectures given today (for posterity in case the link above goes to the ether some day) that are the focus of this post:<br /><br />Nassim Nicholas Taleb, PhD, London Business School, “Errors in the Analyses of Market Potential for Drugs”<br /><br />Dale Nordenberg, Healthcare Industry Advisory, PricewaterhouseCoopers LLP, “Global Trends and Drug Development”<br /><br /><talk listing="">The first talk was by the author of <a href="http://www.amazon.com/Black-Swan-Impact-Highly-Improbable/dp/1400063515/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1209391101&sr=8-1">The Black Swan</a>, a very entertaining read that outlines the very large effects that highly improbable events have on the economic markets. The essence of his book, and his talk, were that in not normally distributed markets, such as the securities market, a rare event can either capture huge amounts of wealth or lead to huge amounts of losses, and the the incremental gains/losses reported over years are really just the noise in the data, and hence unimportant in a sense. Dr. Taleb backed up his claim with a few charts of trade earnings over a decade and most notably showed that two days, in which rare occurrences took place, he both gained 98% of his total revenue (and lost it as well) <span style="font-style: italic;">over the ten years he had data for.</span> Two days in ten years. Yikes.<br /><br />Taleb went on to claim that these types of events are equally applicable to big pharma. Currently the top 6 or so blockbuster drugs capture over 90% of the revenue, and that's no where near normal. Recent <a href="http://www.portfolio.com/news-markets/top-5/2008/01/30/Vioxx-Pain-for-Merck">high-profile litigation</a> also showed that rare events can hurt a companies' bottom line as well.<br /><br />The second talk by Dale Nordenberg focused on what industry pundits and analysts are calling "<a href="http://www.google.com/url?sa=t&ct=res&cd=1&url=http%3A%2F%2Fwww.pwc.com%2Fextweb%2Fpwcpublications.nsf%2Fdocid%2F91BF330647FFA402852572F2005ECC22&ei=FeAVSOXgG5XKepq-oaYC&usg=AFQjCNELHj-5soNbenip4NpqaRWb3CM4VQ&sig2=rHb5-c7229WKchS8py04Pg">Pharma 2020</a>", or the shift in the industry towards providing personalized medicine. This is pharma's equivalent to the so-called "long tail" economics model popularized by Internet sales. Selling a larger variety of drugs, each tailored for use by fewer people.<br /><br />There is no doubt in my mind that the gains in safety and efficacy of future drugs offered to the public would be enormous if pharma did indeed move in this direction, but I am a bit skeptical about the financial incentive of big pharma to follow the piper. It is essentially assuming that pharma can reach a normal earnings distribution if this model is followed, which assumes that big pharam will be insulated from the risks of non-normally distributed earnings environments.<br /><br />But if we take Taleb's lecture at face value in the context of current litigation practices in the US, then we already know that the risk model that pharma faces is one where a rare event will have a very large detrimental effect. Without the equally large and insular effects of blockbusters on revenue, how will a company survive a class action? If I where a CEO, I would probably be paying lip-service to the Pharma 2020 vision, in the hope that such actions would lead to protection from Congress, but my horse's name would still be "Blockbuster McGee".<br /></talk>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-4829222222068048012008-04-14T09:42:00.006-04:002008-04-28T10:44:26.935-04:00Google App Engine: constraints are goodThere has been a lot of rejoicing and jeering over Google's web application deployment offering, Google App Engine. Most of the gripes (that I have seen) have been about the constraints of only supporting a single language, limited database capabilities, no file or OS access of any kind. Frankly I think there is an element of FUD to all of these.<br /><br />First for language choice, AppEngine has been receiving a lot of flack over the choice of a hobbled Python over and above <span style="font-weight: bold;">all</span> other languages. Also there have been a couple of gripes overheard at the lack of real foreign key constraints in the relational layer.<br /><br />Boo frickin hooo. Stop whining and get coding and you'll come to the same conclusion all artists, composers, coders, and generally anybody that ever created anything did. Namely that constraints are sometimes the inspiration for the creation. Sometimes it is the constraints that shape the work more than anything else. Lack there of may sometimes lead to interesting experimentation, but I put forth that actively not following a system of conduct is itself a constraint. Trying to be original is hard work, and all the harder by not framing your work within something familiar.<br /><br />But I also should stop bitching and get to work. My idea has the potential to reshape the way collaboration science is conducted, but I need to deliver the tool to the audience. Seems like G-Apps would be a perfect test bed.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-44472007211043906562008-04-10T21:08:00.003-04:002008-04-10T21:14:43.556-04:00If can't beat 'em, join 'emJust as Twitter was effectively killing my (and <a href="http://blog.cbcg.net/articles/2008/04/05/twitter-and-me">others</a>) blogging fecundity, I saw that they posted a handy-dandy little link on the dash for inserting Tweets into your blog. Sweet. Now I can have the best of both worlds.<br /><br />The Twitter giveth and taketh away.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-66756554931944124992008-03-13T09:13:00.004-04:002008-03-13T09:22:57.673-04:00Twitter is killing my blog<a href="http://www.twitter.com/delagoya">Twitter</a> is killing my blog initiative. In fact this post can just be a straight copy-paste from some recent tweets (reverse chronological order):<br /><br /><table class="doing" id="timeline" cellspacing="0"><tbody><tr class="hentry" id="status_770388224"><td class="thumb vcard author"><a href="http://twitter.com/delagoya" class="url"><img alt="Angel Pizarro" class="photo fn" id="profile-image" src="http://s3.amazonaws.com/twitter_production/profile_images/31077152/avatar_normal.png" /></a> </td> <td class="content"> <strong><a href="http://twitter.com/delagoya" title="Angel Pizarro">delagoya</a></strong> <span class="entry-title entry-content"> Ah, just saw that on XP you can't see my character cleverness. Sux4U dude </span> <span class="meta entry-meta"> <a href="http://twitter.com/delagoya/statuses/770388224" class="entry-date" rel="bookmark"><abbr class="published" title="2008-03-12T14:16:31+00:00">about 23 hours</abbr> ago</a> from <a href="http://snook.ca/snitter/">Snitter</a> <span id="status_actions_770388224"> <a href="http://twitter.com/account/archive#" onclick="new Ajax.Request('/favourings/create/770388224', {asynchronous:true, evalScripts:true, onLoading:function(request){$('status_star_770388224').src='/images/icon_throbber.gif'}, parameters:'authenticity_token=' + encodeURIComponent('7e31279902c9461b7980015796afae0e04e21382')}); return false;"><img alt="Icon_star_empty" id="status_star_770388224" src="http://assets1.twitter.com/images/icon_star_empty.gif?1205356711" border="0" /></a> <a href="http://twitter.com/status/destroy/770388224" onclick="if (confirm('Sure you want to delete this update? There is NO undo!')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', '7e31279902c9461b7980015796afae0e04e21382'); f.appendChild(s);f.submit(); };return false;" title="Delete this update?"><img alt="Icon_trash" src="http://assets1.twitter.com/images/icon_trash.gif?1205356711" border="0" /></a> </span> </span> </td> </tr> <tr class="hentry" id="status_770386566"> <td class="thumb vcard author"> <a href="http://twitter.com/delagoya" class="url"><img alt="Angel Pizarro" class="photo fn" id="profile-image" src="http://s3.amazonaws.com/twitter_production/profile_images/31077152/avatar_normal.png" /></a> </td> <td class="content"> <strong><a href="http://twitter.com/delagoya" title="Angel Pizarro">delagoya</a></strong> <span class="entry-title entry-content"> suppose a hash assignment {:foo ➠ :bar } </span> <span class="meta entry-meta"> <a href="http://twitter.com/delagoya/statuses/770386566" class="entry-date" rel="bookmark"><abbr class="published" title="2008-03-12T14:13:01+00:00">about 23 hours</abbr> ago</a> from <a href="http://snook.ca/snitter/">Snitter</a> <span id="status_actions_770386566"> <a href="http://twitter.com/account/archive#" onclick="new Ajax.Request('/favourings/create/770386566', {asynchronous:true, evalScripts:true, onLoading:function(request){$('status_star_770386566').src='/images/icon_throbber.gif'}, parameters:'authenticity_token=' + encodeURIComponent('7e31279902c9461b7980015796afae0e04e21382')}); return false;"><img alt="Icon_star_empty" id="status_star_770386566" src="http://assets1.twitter.com/images/icon_star_empty.gif?1205356711" border="0" /></a> <a href="http://twitter.com/status/destroy/770386566" onclick="if (confirm('Sure you want to delete this update? There is NO undo!')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', '7e31279902c9461b7980015796afae0e04e21382'); f.appendChild(s);f.submit(); };return false;" title="Delete this update?"><img alt="Icon_trash" src="http://assets1.twitter.com/images/icon_trash.gif?1205356711" border="0" /></a> </span> </span> </td> </tr> <tr class="hentry" id="status_770385883"> <td class="thumb vcard author"> <a href="http://twitter.com/delagoya" class="url"><img alt="Angel Pizarro" class="photo fn" id="profile-image" src="http://s3.amazonaws.com/twitter_production/profile_images/31077152/avatar_normal.png" /></a> </td> <td class="content"> <strong><a href="http://twitter.com/delagoya" title="Angel Pizarro">delagoya</a></strong> <span class="entry-title entry-content"> and also wondering if I can use ruby's syntatic suger with random windings </span> <span class="meta entry-meta"> <a href="http://twitter.com/delagoya/statuses/770385883" class="entry-date" rel="bookmark"><abbr class="published" title="2008-03-12T14:11:36+00:00">about 23 hours</abbr> ago</a> from <a href="http://snook.ca/snitter/">Snitter</a> <span id="status_actions_770385883"> <a href="http://twitter.com/account/archive#" onclick="new Ajax.Request('/favourings/create/770385883', {asynchronous:true, evalScripts:true, onLoading:function(request){$('status_star_770385883').src='/images/icon_throbber.gif'}, parameters:'authenticity_token=' + encodeURIComponent('7e31279902c9461b7980015796afae0e04e21382')}); return false;"><img alt="Icon_star_empty" id="status_star_770385883" src="http://assets1.twitter.com/images/icon_star_empty.gif?1205356711" border="0" /></a> <a href="http://twitter.com/status/destroy/770385883" onclick="if (confirm('Sure you want to delete this update? There is NO undo!')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', '7e31279902c9461b7980015796afae0e04e21382'); f.appendChild(s);f.submit(); };return false;" title="Delete this update?"><img alt="Icon_trash" src="http://assets1.twitter.com/images/icon_trash.gif?1205356711" border="0" /></a> </span> </span> </td> </tr> <tr class="hentry" id="status_770385561"> <td class="thumb vcard author"> <a href="http://twitter.com/delagoya" class="url"><img alt="Angel Pizarro" class="photo fn" id="profile-image" src="http://s3.amazonaws.com/twitter_production/profile_images/31077152/avatar_normal.png" /></a> </td> <td class="content"> <strong><a href="http://twitter.com/delagoya" title="Angel Pizarro">delagoya</a></strong> <span class="entry-title entry-content"> It's like the evolution of large emails ☞ short emails ☞ IM </span> <span class="meta entry-meta"> <a href="http://twitter.com/delagoya/statuses/770385561" class="entry-date" rel="bookmark"><abbr class="published" title="2008-03-12T14:10:53+00:00">about 23 hours</abbr> ago</a> from <a href="http://snook.ca/snitter/">Snitter</a> <span id="status_actions_770385561"> <a href="http://twitter.com/account/archive#" onclick="new Ajax.Request('/favourings/create/770385561', {asynchronous:true, evalScripts:true, onLoading:function(request){$('status_star_770385561').src='/images/icon_throbber.gif'}, parameters:'authenticity_token=' + encodeURIComponent('7e31279902c9461b7980015796afae0e04e21382')}); return false;"><img alt="Icon_star_empty" id="status_star_770385561" src="http://assets1.twitter.com/images/icon_star_empty.gif?1205356711" border="0" /></a> <a href="http://twitter.com/status/destroy/770385561" onclick="if (confirm('Sure you want to delete this update? There is NO undo!')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', '7e31279902c9461b7980015796afae0e04e21382'); f.appendChild(s);f.submit(); };return false;" title="Delete this update?"><img alt="Icon_trash" src="http://assets1.twitter.com/images/icon_trash.gif?1205356711" border="0" /></a> </span> </span> </td> </tr> <tr class="hentry" id="status_770384818"> <td class="thumb vcard author"> <a href="http://twitter.com/delagoya" class="url"><img alt="Angel Pizarro" class="photo fn" id="profile-image" src="http://s3.amazonaws.com/twitter_production/profile_images/31077152/avatar_normal.png" /></a> </td> <td class="content"> <strong><a href="http://twitter.com/delagoya" title="Angel Pizarro">delagoya</a></strong> <span class="entry-title entry-content"> notice that twitter is killing my blog initiative. Why blog when a short tweet will do? </span> <span class="meta entry-meta"> <a href="http://twitter.com/delagoya/statuses/770384818" class="entry-date" rel="bookmark"><abbr class="published" title="2008-03-12T14:09:23+00:00">about 23 hours</abbr> ago</a> from <a href="http://snook.ca/snitter/">Snitter</a> <span id="status_actions_770384818"> <a href="http://twitter.com/account/archive#" onclick="new Ajax.Request('/favourings/create/770384818', {asynchronous:true, evalScripts:true, onLoading:function(request){$('status_star_770384818').src='/images/icon_throbber.gif'}, parameters:'authenticity_token=' + encodeURIComponent('7e31279902c9461b7980015796afae0e04e21382')}); return false;"><img alt="Icon_star_empty" id="status_star_770384818" src="http://assets1.twitter.com/images/icon_star_empty.gif?1205356711" border="0" /></a> <a href="http://twitter.com/status/destroy/770384818" onclick="if (confirm('Sure you want to delete this update? There is NO undo!')) { var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);var s = document.createElement('input'); s.setAttribute('type', 'hidden'); s.setAttribute('name', 'authenticity_token'); s.setAttribute('value', '7e31279902c9461b7980015796afae0e04e21382'); f.appendChild(s);f.submit(); };return false;" title="Delete this update?"><img alt="Icon_trash" src="http://assets1.twitter.com/images/icon_trash.gif?1205356711" border="0" /></a> </span> </span></td></tr></tbody></table><br /><br />Really? It's that easy? Can you beat that ease-of-brain-dump blogger.com? I think not. Some picasa functions came close for posting quick notes on photos, but there isn't really a native blogger equivalent (that I know of). The whole create post -> edit post -> post post life cycle is too long for a quick set of thoughts you want to jot down.<br /><br />I could probably research this more, but really why would I when twitter is always there, and I actually have an audience that comments back to my posts. And so back to my original point: Twitter is killing my blog.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-53345059793936782032008-02-15T12:45:00.002-05:002008-02-15T13:10:10.955-05:00Sequel's lackluster to_*Sequel is a great bare-bones ORM, but the bare-bones quality of Sequel::Model leave something to be desired. For instance the obj.to_json method just calls the default ruby object inspect method, which prints out the class name and memory space. Not helpful. Also no to_xml() for easy REST incorporation. Almost makes me want to go back to ActiveRecord, but then what's the point in using Merb?<br /><br />Anywho, it's not that hard to extend the functionality of Sequel::Model, so I have started writing some gems to make development with Merb + Sequel a little easier. Like real to_xml & to_json methods on the model instances and instance collections. More on this as it develops.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-49977512722628885432008-02-15T08:23:00.005-05:002008-03-13T09:38:53.965-04:00Merb TLS mail plugin gem<span style="font-weight: bold;">UPDATE: </span>Now available on github : <a href="http://github.com/delagoya/merb_tlsmail/tree/master">merb_tlsmail github page</a><br /><br />My <a href="http://defsci.blogspot.com/2008/02/secure-smtp-server-tls-from-merb-apps.html">previous post</a> on sending mail via a TLS SMTP server on <a href="http://merbivore.org/">merb</a> covered monkey patching Merb::Mailer.<br /><br />I took the time to code this up as a gem, using Merb's meta-programming routines to extend Merb:Mailer in a standard way (for merb that is). See <a href="http://merb.lighthouseapp.com/projects/7588/tickets/26-plugin-for-tls-smtp">this open ticket</a> in the <a href="http://merb.lighthouseapp.com/">Merb lighthouse issue tracker</a> to download the gem until it is released as a proper plugin.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-30530314390103350922008-02-12T16:35:00.000-05:002008-02-12T16:50:37.484-05:00Secure SMTP server (TLS) from merb appsIt seems that Merb's Mailer class is either using a local sendmail client or a non-TLS enabled SMTP server. This is not a unique problem to merb, but rather a deficiency in Ruby 1.8.<br /><br />I took some time to look around and found that Rails has the same problem, and it was fixed via a plugin, not a gem as is the "merb way". There was also a gem that packaged the Net::SMTP classes from Ruby 1.9, which do have TLS support. It isn't hard to guess what I did next.<br /><br />I monkey punched Merb::Mailer to overwrite the net_smtp method and added two config options into merb_init.rb. See the pastie for the code example <a href="http://pastie.caboo.se/151190">here</a>.<br /><br /><a href="http://pastie.caboo.se/151190">http://pastie.caboo.se/151190</a>Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-35829102635816973952008-02-07T10:39:00.001-05:002008-02-07T11:19:43.664-05:00Where has the Sematic web failed us?News about the <a href="http://en.wikipedia.org/wiki/Semantic_web">Semantic Web</a> has being the "next big thing" has been hitting web application developers over the head for years, like it was news about the iPhone. But where are the products? Who uses it? Expect for academic papers, a standards process that nobody pays attention to, and ontology narcissists, nobody uses RDF or OWL or any of those supposedly "next generation" tool sets and languages. OK, maybe <a href="http://www.powerset.com/">Powerset</a> will, but I'll believe that when I see it.<br /><br />Certainly the <a href="http://swoogle.umbc.edu/">swoogle</a> is no google, although it is starting to address what I see is the most overlooked part of the semantic web, <a href="http://en.wikipedia.org/wiki/Usability">usability</a>. It seems that developers and proponents of Web 3.0 thinmk regular users of the web are a lot smarter than <a href="http://en.wikipedia.org/wiki/The_Nation_of_Gods_and_Earths">they are</a>. Swoogle does actually show a pretty <a href="http://swoogle.umbc.edu/index.php?option=com_frontpage&service=digest&queryType=digest_swd&searchString=http%3A%2F%2Fmged.sourceforge.net%2Fontologies%2FMGEDOntology.owl">nicely formatted report on the metadata it has for a result</a>, if you know what you are looking at that is. Yet the main result link leads to the originating ontology, which is and RDF xml file. Yeah, that's helpful. Even if Joe Public is aware enough to click on the <span style="font-style: italic;">metadata</span> link, instead of <a href="http://www.i-am-bored.com/bored_link.cfm?link_id=9644">the big red button</a> that is the main link, I can't ever image him making heads or tails of the report, or for that matter caring.<br /><br />Why is that? Why is usability not even a concern for majority of ontology & semantic web developers? What makes this situation even more of a disaster is that tagging (and tag clouds) are so wide spread and ridiculously easy to understand. How is tagging any different than RDF annotations? A little more text, that's what. Oh and querying RDF is a bitch, so developers are also affected by the situation, making adoption of this "standard" that much more unlikely.<br /><span style="font-style: italic;"><br />PS: I am not part of, or hold any affiliation to, NG&E, but "the 85%" is one of those stereotypes that ring true to me</span>.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-87983986877552270542008-01-04T11:19:00.000-05:002008-01-07T11:28:38.443-05:00Zed.. very humorousZed Shaw's <a href="http://www.zedshaw.com/rants/rails_is_a_ghetto.html">latest rant</a> is a hoot. When I first read it, it was clearly the draft he mentioned it was. I'm glad he posted it as a draft, though, because the next iteration did give a chance for DHH to clarify, and also gave Zed a chance to frame the whole rant a bit better with his admission that he himself was the main person responsible for almost going to the poor house.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-44487817184909763322007-12-11T15:18:00.001-05:002007-12-11T15:40:43.821-05:00Ask not for what you think you need...I have been reflecting a lot recently on why researchers think they need grand solutions for relatively small problems. Specifically there is a perception among benchers that are conducting large-ish experiments that they need some sort of LIMS to manage their data. Frankly as I've stated many times (at least in conversations with other IT folk, and even with benchers) most researchers don't want a LIMS. What they really want is fancy file storage.<br /><br />Yet, the town folk keep insisting "we need LIMS, please give...". LIMS are probably the only thing that potentially fits the bill for experimental data management, so that's what benchers ask for, when most likely a digital asset management application would suffice. Heck, even all those bit torrent sites can potentially do the job that researchers need.<br /><br />If you are a coder that is continually asked to provide a LIMS to researchers, or if you are a bencher that is interested in LIMS for data management, here are a set of questions you can ask before going any further down the rough and tumbly road that is LIMS adoption:<br /><br /><ol><li>Are you in a regulated environment?</li><li>Are you willing to <span style="font-weight: bold;">mandate</span> use of the LIMS?</li><li>Do you have adequate personnel to support the LIMS locally (e.g. do you have a dedicated person that will <span style="font-weight: bold;">actively</span> promote the use of the LIMS, train folks, configure the system, do extensive follow-up, etc. Vendor support will only be of help at the start of the adoption process.)<br /></li><li>Do you have a lot of spare cash? (Think 6 figures to buy an initial bank of licenses)</li><li>Will you have a lot of spare cash for the next 3-5 years? (Think 5 figures to keep annual support and maintenance up to date.)</li></ol>If you answered "No" to any of these questions, seriously reconsider buying what is traditionally known as a LIMS. Instrument control, automatic data acquisition, yada, yada, all those marketing features used to sell a LIMS don't mean squat if no one uses it in the first place.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.comtag:blogger.com,1999:blog-7940730010022485111.post-12484862524736769622007-11-29T09:34:00.000-05:002007-11-29T09:56:52.801-05:00Python 25 & MySQL on LeopardThere was a bit of trouble when I tried to use macports to install the excellent <a href="http://trac.edgewall.org/">Trac</a> project management and issue tracking application. Specifically, the <a href="http://db.macports.org/port/show/3013">py25-mysql</a> port did not compile correctly.<br /><br />Trolling through the InterWeb, I found <a href="http://www.keningle.com/?p=11">this post</a> about a fix to compile the python module from scratch. The post, however, has more instructions than are necessary, so here is my revised procedure:<br /><br /><ol><li>Download the source from <a href="http://downloads.sourceforge.net/mysql-python/MySQL-python-1.2.2.tar.gz?modtime=1172959928&big_mirror=0">here</a>. </li><li>Unpack the archive and edit the _mysql.c file to comment out lines 37-39:<br /><code><br />// #ifndef uint<br />// #define uint unsigned int<br />// #endif<br /></code><br /></li><li>Edit the site.cfg file to set the mysql_config path. For me this was:<br /><code><br />mysql_config = /usr/local/mysql/bin/mysql_config<br /></code></li><br /><li>Compile as normal<br /><code><br />python setup.cfg config<br />python setup.cfg build<br />sudo python setup.cfg install<br /></code></li></ol><br /><br />Now if only someone would make these changes in the port file, then trac installs would be super easy, instead of just easy-ish.Angelhttp://www.blogger.com/profile/09938400316307441118noreply@blogger.com