I found a way thanks to the Biopython project, which offers a Python module to access the resources of the National Center for Biotechnology Information (NCBI) by providing an interface to their EUtils. Since the back-end was already taken care of, almost, at least, I sought to write a small Plasma applet. Which is what I’m presenting today. It’s written in Python, and uses the Python ScriptEngine to work. Currently, it searches the “Gene” database at NCBI by inputting the “Entrez Gene IDs”, that are numerical IDs that uniquely identify a gene record, and returns name, official symbol, organism, and a description if it’s present. It does not support anything else (see below).

The code lives in a git repository at github. WARNING: The code may be a complete mess (I’m not too well versed in GUI stuff, I mostly do text file manipulation) If you are so daring, you can obtain and install it in a very simple manner:

git clone git://github.com/cswegger/plasma-genesearch.git
cd plasma-genesearch
zip -r ../plasma-genesearch.plasmoid *
plasmapkg -i ../plasma-genesearch.plasmoid

After that you will see an “Entrez Gene Searcher” in your add applets dialog. Once added, it’ll look like this:

Pretty horrible, isn’t it? Well, once you get past that, you can input an ID (only IDs will work for now) in the text field (which doesn’t clear the text: see further on) and push “Go!”. The following is an example with ID 10000, which corresponds to the human gene AKT3:

“Search again” will bring you back to the search form.

Now, what has this to do with Planet KDE? Well, I’m asking for some code review from the community, if it’s possible, and suggestions to improve the horrid default look. I am especially interested in layouting, since I did not quite understand how it works, I mean, it should not work and it does….

Other things that need to be improved are:

• The Plasma.TextEdit is not cleared upon clicking. Is there a signal I can catch for that, so I can connect it to clear()?
• Proper searching. Bio.Entrez already does this: what I need is a way to display the records properly.
• A way to link the names to URLs, and have them open in Konqueror.

That should be it. I hope to work on it some more next weekend….

At least in the life sciences, it’s hard to see such a mentality. I can understand , but at the same time, ? For me, such an idea would be optimal. Once the paper is out, you can release your software (GPL would be best) and make sure someone will improve or mantain in. Of course you won’t be able to publish for each upgrade you do, but I would generally think of that as a bad policy, one made just to increase the publication count.

Does something like that happen with FOSS in other research areas?

R is not a performing language at all, it doesn’t parallelize well when using HPC (at least from the talks I’ve had with people studying the matter), and in general is a memory and resource hog. For example, it takes much more to perform RMA via R that with RMAExpress (which is a C++ application): the latter works also better with regards to memory utilization. I can understand the complexity of some statistical procedures, but what about ?

The surprising aspect is that aside by a few exceptions (like the aforementioned RMAExpress) no one has tried to write more performing implementations of certain algorithms. I for one would welcome a non-R implementation of SAM (the original implementation works in Excel… ugh) or similar algorithms. Otherwise we would be stuck with programs that are interesting, but way too memory hungry (AMDA comes to mind).

The subsequent idea was “let’s implement auto-detection of common identifiers in the class”. The problem is… is there any actual documentation on how identifiers are made? So far, using regular expressions, I’ve tracked down a few:

• RefSeq
• GenBank
• Entrez Gene
• UCSC Genome Browser
• Ensembl

However, I have no idea if I have implemented all types of these IDs. Does anyone know a place where to look these information up?

(On a related note: my thesis defense will be on January 14th, 2008, so I have to get the printing going)

Following up my recent post, I’ve been looking for alternatives to TMeV. So far I’ve found the R package pvclust and the Pycluster library, part of BioPython. The first one also performs bootstrapping (I’m not sure if it’s similar to what support trees do, but it’s still better than no resampling at all). I’ve found another Python project but it is still too basic to perform what I need.

Pvclust would be my first interest, but it only plots dendrograms and not heatmaps, and the clustering must be done twice by transposing the data (it only clusters columns). The package’s web page shows the various options and what to do with it.

Pycluster, on the other hand, can be used to generate files which can be read by the Java TreeView program, where you can view a heat map of the results and their annotations. Although there’s documentation available, it is not part of the Biopython documentation (as usual, I’d say: lack of documentation is a plague for Biopython). In any case, doing a cluster analysis is rather simple, but we need to remember that we need to do two cluster runs (one for genes, the other for experiments). Here I show an example with hierarchical clustering, but the documentation (Python part on chapter 8) has examples also with other methods such as SOMs or k-means.

from Bio.Cluster import *

#   Load data, in Cluster format
data = DataFile("somefile.txt")

#   Clustering using Pearson's correlation and average linkage
gene_clustering=data.treecluster(method="a",dist="c",transpose=0)

#   Same as above, but clustering samples
exp_clustering = data.treecluster(method="a",dist="c", transpose=1)

#   We then save the results to a series of files to view in Java TreeView
data.save("name",gene_clustering,exp_clustering)

Java TreeView is a program to view trees and heat maps. Unlike its counterpart TreeView, it’s truly cross-platform (Java) and GPLed, a nice added bonus. You can load the files directly and display the results like in this picture, taken with the sample data available on the project page.

It’s still not perfect (no data shown on the main map page, only with the detailed view) but a good start, nevertheless. I’ll investigate whether I can complement TMeV usage with these tools.

I use TMeV mostly because I didn’t find any other implementation of the hierarchical clustering algorithm with support trees. However, I’ve stumbled upon a very annoying bug in the most recent version. Normally I use average linkage clustering and as the distance metric I employ the Pearson’s correlation, and with gene and sample bootstrapping: with certain files this makes TMeV report errors at random during the iterations.

What you see on the left is the “error” that TMeV gives. As you can see, it is all but informative. Digging a bit, it shows that Java throws an ArrayIndexOutOfBoundsException, but I wonder how, since this happens with different data files that have nothing in common at all.

Since I don’t want to pass for someone who just whines, I contacted the MeV developers and offered to give also example files, but I’ve got no response at all. Luckily, the older version of MeV (3.1) is still around and works.

Now, I wonder how this was released in the first place: it’s not the only bug I found, the other being a scaling algorithm that would mistake the files for Affymetrix MAS5 expression values (when they have nothing in common with that). There is absolutely no mention of that in the TMeV page. At least some release notes would be helpful.

Why am I making a big fuss over this? Because this is not the first time that I’ve wasted my time working around bugs instead of using the software for what it was meant: for research. Instead, there seem to be little interest since the publication is already out. Didn’t someone tell those people that if you release some software (especially if it is widely used) you’re expected to provide at least little maintenance?

I did a grave mistake. First of all, Biopython‘s SOFT parser is horribly broken (doesn’t work at all) and quite undocumented: I could work around the lack of documentation (API docs) but not with the fact that it wouldn’t work. So I turned to R, which offers a GEO query module through Bioconductor.

Again that proved to be a terrible mistake. For a file containing 183 samples, the analysis is going on since four hours and with no sign of completing anytime soon (not to mention a  possible memory leak). After this, I gave up. I’m going to get the reduced data sheet and write a small parser in Python myself.

What is frustrating is the lack of quality: I could concentrate on my own work rather than reinventing the wheel for the nth time if the existing implementations worked. What’s the point in releasing non-working software? I could understand bugs, but this is one step further.

Let me first say what RMA is about: it stands for “Robust Multi-array Average” and is a model-based quantification method for Affymetrix arrays, originally developed by Irizarry et al. It has a number of advantages over the Microarray Analysis Suite 5 (MAS5) algorithm used by Affymetrix software, especially with weakly expressed transcripts. It is commonly made up of three steps: background correction, quantile normalization and median polish.

RMAExpress is a C++, GUI-based program (using wxWidgets) that performs this process. The main advantage over the various R implementations is speed, as R doesn’t really excel in this regards. You can adjust the various RMA parameters, and you can also view the model representations, to see if some areas on the array perform differently (e.g., when there are irregularities in the signal intensities).

What I liked best is that you can use custom chip definition files (CDFs). Dai et al. have already shown that old 3′ GeneChips have outdated annotations, and have proposed new CDFs to compensate. We have already tested their improvement and it gives a nice increase in the number of annotated genes. RMAExpress processes these CDFs just fine.

Finally, you can export data either in log2 format (to use in procedures like SAM) or in absolute form (which I need for my work). The program is extremely light and processes a good number of arrays fairly quickly. Windows users have a pre-built binary, while Linux ones need to build sources. The instructions on the page are overly complicated: here’s how I managed to build it on Kubuntu:

sudo aptitude install libwxtgk2.8-dev
mkdir tmp
cd tmp
tar xvzf  /path/to/RMAExpress_1.0beta3_src.tar.gz
make all

After that, just run RMAExpress from its directory.

After all, I’m quite pleased with the program and I will keep using it in the future.