dennogumi.org » Science

Multiscale bootstrap clustering with Python and R

Einar — Sun, 29 May 2011 12:11:40 +0000

While reading the statistics for my blog, I noticed that a number of searches looked for hierarchical clustering with Python, which I covered quite a while ago. Today I’d like to present an updated version which uses more robust techniques.

Defining the problem

Since Eisen’s original paper on clustering, this form of analysis has been widely used by a lot of researchers. However, as it is known, such systems may be susceptible to an ordering bias: in other words, the order of the samples and/or genes might influence the final result. That’s why popular software such as TMeV offers alternative approaches, based on bootstrapping.

In this specific form of bootstrapping, the samples and/or genes are randomly shuffled a number of times (1000 or more iterations are a good starting point) and the resulting dendrograms checked for consistency and robustness of partitioning. In other words, a p-value is calculated, our null hypothesis being that the arrangement of samples/genes is merely due by chance. Depending on the software, this value might be expressed either in form of p-value or percentage (TMeV calls it support).

In the past years, I found an interesting method developed by Hidetoshi Shimodaira: the technique, called multiscale bootstrap resampling, aims at determining more accurate p-values out of the bootstrapping. Shimodaira calls the resulting p-value an AU value, where AU stands for “approximately unbiased”, a more precise p-value than the one obtained through bootstrapping alone.

In addition to this nice algorithm, a R package was also provided, named pvclust (it’s available on your favorite CRAN mirror). And that’s exactly what we’ll use for this exercise.

Prerequisites

Some of the readers of this blog might remember my disdain of R: while I need to use it for Bioconductor, I’m often annoyed by its weird syntax, and difficult to understand error messages. Luckily, thanks to the hard work of Laurent Gautier and contributors, there’s rpy2, a nice R-to-Python bridge. All the examples here require this package, version 2.1 or newer (I’d recommend the release candidate of 2.2, it’s really nice). Unfortunately, this means that Windows users are out of luck as there’s no version of rpy2 2.1 or 2.2 available for that platform..

Also, don’t forget to have the pvclust package installed in R.

Loading and preparing the data

Let’s start first by importing the necessary bits:

import rpy2.robjects as robjects
from rpy2.robjects.packages import importr

The second line is important, because it’ll let us play with R libraries as they were packages. Case in point, we’ll get the “base” and “pvclust” libraries loaded:

base = importr("base")
pvclust = importr("pvclust")

Now we can manipulate them as if they were modules, and (most) of R’s dotted functions have been converted to underscores, as the dot is the namespace operator in Python. Example: as.data.frame becomes as_data_frame.

Next, we’ll load the data in a data.frame. rpy2 conveniently gives us the DataFrame class, which is a no-nonsense wrapper to R’s data.frames. For this exercise, we’ll load a set of normalized data from GSE4984, a microarray experiment with dendritic cells expoosed to different stimuli. It’s just a matter of downloading the data from Array Express (if you ask why from AE and not GEO: the latter doesn’t have a clearly-identified link for normalized data) and then loading it in a data.frame as:

dataframe = robjects.DataFrame.from_csvfile("GSE4984.txt", sep="\t ", row_names=1)

The resulting Python object has all the attributes of a R data.frame but with added Python goodness. We can use the colnames and rownames attributes to access the row names (if set) and column names of the object, and likewise we can use nrow and ncol to quickly glance at the rows/columns.

Since a full array has a lot of genes, we’re going to choose only the first 500 genes:

rows = robjects.IntVector(range(1,501))
subset = dataframe.rx(rows, True)

An IntVector is a rpy2 object which replicates R’s vectors of integers: there are variants for strings, floats, integers, lists (R lists, not the Python type) and factors. rx is an accessor that mimicks R’s item access: in short, it’s equivalent to

subset <- dataframe[rows, ]

rpy2 has another accessor, rx2, which mimicks the [[ ]] access in data.frames.

Clustering

Once we have the data, it’s time to do some serious clustering on it:

result = pvclust.pvclust(subset, nboot=100, method_dist="correlation", method_hclust="average")

We’re using a small number of permutations (100) because the computation times are long. You can change the distance metric and the linkage types using method_dist and method_hclust. Internally the data.frame is converted to a matrix, so ensure you have valid data (i.e. numeric) prior to proceeding.

Notice that this will just cluster the columns by default. If we want to cluster genes, we have to transpose the data.frame. In this case we have to first convert it to a matrix, then transpose it:

matrix = base.as_matrix(subset)
subset_transposed = matrix.transpose()
result_rows = pvclust(subset_tranposed, nboot=100, method_dist="correlation", method_hclust="average")

Once the computation is done, we have a pvclust object which holds information on the results. What we’re most interested in is the hclust attribute, as it holds a dendrogram object we can use for plotting (either standalone or via a heat map). We can also manipulate the object with the pvpick function, for example to color the trees of the dendrogam basing on their AU values.

To get a fast representation, we can just dump the object as it is to a dendrogram which will show AU and BP values for each element of the cluster:

graphics = importr("graphics")
graphics.plot(result)

Or we can do the same, but to a PDF:

graphics = importr("graphics")
grdevices = importr("grDevices")
grdevices.pdf("myresult.pdf", paper="a4")
graphics.plot(result)
grdevices.dev_off()

Of course, we might want a heat map (everyone wants pretty heat maps, right?). In that case we extract both dendrograms and use something like gplots’ heatmap.2 function to represent it (you will need the gplots package installed in order for the following to work):

gplots = importr("gplots")
row_dendrogram = result_rows.rx2("hclust")
column_dendrogram = result.rx2("hclust")
gplots.heatmap_2(subset, Rowv=row_dendrogram, Colv=column_dendrogram, col=gplots.greenred(255), density_info="none")

You can add the grdevices lines like above to make a PDF of the plot. If you notice we have used the rx2 accessors here, just as I wrote above, to access the hclust attribute of the pvclust object.

Moving further

pvclust as-is it’s quite slow. There’s however a parallelized version, called parPvclust, which uses snow to parallelize the clustering, either through multiple machines or using multiple cores. Setting snow properly up is beyond the scope of this tutorial, but it may be worth investing if you cluster a lot of data.

Akademy: my own BoF

Einar — Sat, 29 May 2010 19:55:37 +0000

My Akademy talk proposal was not accepted, but the organizers were kind enough to offer me the chance to hold a BoF on the same subject. Now I bet you wonder on what I’m going to discuss, and I think the title already gives you an idea:

KDE and bioinformatics: the missing link

Although in the KDE community we have our fair share of scientists (hey there, Stuart!), my BoF will focus on the adoption of KDE in the field of bioinformatics (my day job, not-so-by-chance) on the "outsiders" front and how to improve the current situation. To elaborate further, bioinformatics is a rather broad field where biological data are treated with computational methods. The oldest and most famous branch of bioinformatics is sequence analysis and related field, where sequences of DNA are analyzed, for example, to find common ancestors among several species, or to reconstruct the genetic code of an organism by comparing it to a related species. Another recent example is related to high-throughput technologies, technologies which produce huge amounts of data from a very small number of experiments ("ultramassive sequencing" and DNA microarrays are examples of such a technology).

Either way, bioinformaticians have to deal with large amounts of data all the time, and usually there’s no "shrink-wrap" solution to the problems they have to face, software-wise. That’s because we do research, so we need to find something new. So the solution is often to write algorithms, or re-implement existing ones in a form that is suited for the tasks at hand. So, bioinformaticians also write software, although they’re by no means (usually) professional coders: some have a mathematical or statistical background, others (like me) come from an experience at the lab bench. What kind of programs bioinformaticians write? Normally scripts and small stuff, but in certain cases even full blown-algorithms and applications. Some become so famous that are even trend-setters.

Which brings us to the heart of the matter: how does KDE stand in all of this? Sadly, not too well. I’ve done some research in the published literature, but there’s just one hit returned that’s proper: a KDE application for neuroscience (based on the 3.5.x Development Platform) published in 2008. I know that big research places like CERN use KDE, but to my knowledge smaller realities such as research group code in the majority of the cases for Windows or for web-based solutions. Given that at least a signficant portion of bioinformaticians uses UNIX-like operating systems, the question we need to answer is: why?

The first and foremost problem is related to market share. Research groups don’t even know that KDE exists, so it’s unlikely they develop something using the Development Platform (even now that’s becoming more cross-platform). This is where some promo efforts could help. Secondly, the problem lies in the "difficulty" (notice the quotes!) of developing using the KDE Development platform: most bioinformaticians, as I wrote, are not professional coders, and few of them know C++. The most used languages in bioinformatics are Perl and Java (with some Python and Ruby thrown into the mix). Thus, the need for proper bindings. The bindings are there, thanks to the excellent work of the kde-bindings team, but documentation is still lacking (namely in the examples department, but also in tutorials and getting started guides that aren’t aimed at C++). Some documentation is auto-generated, and while the KDE API docs are usually not too hard to read, they can still scare off newcomers. Of course this is not the fault of the kde-bindings team: namely, more help is needed.

Promo efforts and better bindings are the keys to spread KDE more in the field of the bioinformatics. This is what my BoF is about, plus an informal discussion on the use of FOSS in academia and related matters.

Interested? If you are, you can come to the BoF which will be on Tuesday, 6th July at 15.00 in the Area 2 of the main room at Demola.

I’ll also be around later till the following morning (sadly, two days is the best I can do to attend) in case you’re interested for a chat.

DataMatrix 0.8 is finally out

Einar — Sat, 13 Jun 2009 13:29:40 +0000

At last, after months of inactivity, I pushed out a new release of DataMatrix. Although the version bump is small (0.8) there are a lot of changes since last releases. The most notable include:

Ability to apply functions to elements of the matrix
Ability to filter rows by column contents
Ability to transpose rows with columns
An option to load text files produced by R (which are, by design, broken)
Removed the getter for columns, using dictionary-like syntax directly
A lot of bug fixes

The download links on the project page have been updated, along with the documentation. Also, there is another change, because from now on the official Git repository is hosted on gitorious.org, and no longer on github, because gitorious (the software) is also free, while github.com’s is not. It’s mainly a philosophical issue (the same that prompted me to switch from twitter to identi.ca).

Also, from today DataMatrix is also officially hosted on the Python Package Index (with the name “datamatrix”), meaning that you can use easy_install to quickly install it.

If you use this module, let me know what you think (including bugs, if you find them).

Gene search applet: suggestions and code review needed

Einar — Tue, 31 Mar 2009 17:33:09 +0000

In the past months I’ve always wanted to write a small Plasma applet to aid me in some boring tasks as a bioinformatician. One example (for the non-scientific crowd out there) is when I find a specific gene out of my analysis work which I want to take a look at. I am often lazy, so instead of firing up the browser to look at the online resources, I wanted to write something which could access said resources programmatically.

I found a way thanks to the Biopython project, which offers a Python module to access the resources of the National Center for Biotechnology Information (NCBI) by providing an interface to their EUtils. Since the back-end was already taken care of, almost, at least, I sought to write a small Plasma applet. Which is what I’m presenting today. It’s written in Python, and uses the Python ScriptEngine to work. Currently, it searches the “Gene” database at NCBI by inputting the “Entrez Gene IDs”, that are numerical IDs that uniquely identify a gene record, and returns name, official symbol, organism, and a description if it’s present. It does not support anything else (see below).

The code lives in a git repository at github. WARNING: The code may be a complete mess (I’m not too well versed in GUI stuff, I mostly do text file manipulation) If you are so daring, you can obtain and install it in a very simple manner:

git clone git://github.com/cswegger/plasma-genesearch.git
cd plasma-genesearch
zip -r ../plasma-genesearch.plasmoid *
plasmapkg -i ../plasma-genesearch.plasmoid

After that you will see an “Entrez Gene Searcher” in your add applets dialog. Once added, it’ll look like this:

Pretty horrible, isn’t it? Well, once you get past that, you can input an ID (only IDs will work for now) in the text field (which doesn’t clear the text: see further on) and push “Go!”. The following is an example with ID 10000, which corresponds to the human gene AKT3:

“Search again” will bring you back to the search form.

Now, what has this to do with Planet KDE? Well, I’m asking for some code review from the community, if it’s possible, and suggestions to improve the horrid default look. I am especially interested in layouting, since I did not quite understand how it works, I mean, it should not work and it does….

Other things that need to be improved are:

The Plasma.TextEdit is not cleared upon clicking. Is there a signal I can catch for that, so I can connect it to clear()?
Proper searching. Bio.Entrez already does this: what I need is a way to display the records properly.
A way to link the names to URLs, and have them open in Konqueror.

That should be it. I hope to work on it some more next weekend….

Moving on

Einar — Fri, 27 Feb 2009 16:30:57 +0000

Some say that all good things must come to an end. I’m not entirely sure that this is a universal truth, but I can say that at some point in life there are decisions that need to be taken.

In this case I made my own: today was the last day in Dr.Cristina Battaglia’s laboratory, a place where I spent my three-year Ph.D. course and one year as a post-doc research fellow.

Those four years were not bad at all. They were interesting, and provided a good learning experience. I think I owe quite a bit to that place, especially because I was able to learn and improve my skills alongside the analysis and research work. So my thanks go to my former supervisor (Dr.Cristina Battaglia) and all my colleagues. It’s been a fun ride.

Science and KDE: kile

Einar — Sun, 22 Feb 2009 20:49:20 +0000

During the course of my research work, I may obtain results that are worthy of publication in scientific journals. Since my master’s thesis I’ve been using LaTeX as my writing platform, mainly because I can concentrate on content rather than presentation (I find it useful also for writing non-scientific stuff as well). Also, I can handle bibliography (essential for a scientific publication) very well without using expensive proprietary applications (such as Endnote).

In my early days I used kLyX first, then LyX, but I found the platform to be too limited for my tastes, and also LaTeX errors were difficult to diagnose. I needed a proper editor, and that’s when I heard of kile, a KDE front-end for LaTeX. Kile is currently at version 2.0.2 and is a KDE 3 application. However, in KDE SVN work is ongoing to produce a KDE4 version (2.1) and that’s what I’ll look at in this entry.

Obtaining kile 2.1

First and foremost, a disclaimer. kile 2.1 has not been released yet in any form, and so should be considered unstable and crash-prone. That said, it runs more or less well on my platform.

The first thing to do is to grab the sources from SVN:

svn checkout svn://anonsvn.kde.org/home/kde/trunk/extragear/office/kile

That will put kile’s sources in a directory called “kile”. The next step is to compile it (as usual, you need KDE4 development packages/files installed):

cd kile mkdir build; cd build cmake -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ../ make

Followed by the usual make install as root or using sudo.

kile 2.1 at a glance

This is how kile looks when loaded on my system:

(For the inquisitive people, it’s not a scientific work, rather a sci-fi like book I’m writing).

Kile uses the katepart for editing, so that means all the goodies that come with Kate can be used, including the recently-added vim input mode. Aside from editing and LaTeX syntax highlighting, kile offers a configurable LaTeX command completion, like this screenshot shows:

From the toolbars and the menus you can insert almost every LaTeX command known to mankind. For the people less apt with LaTeX, kile offers a series of wizards in order to make the creation of figures, tables and even complete documents. The one I’m showing here is the Quick Start wizard, which enables you to select document classes, add packages, and add information like author and date. As I was saying earlier, kile 2.1 is still a work in progress, and that explains why the dialog is still a little unrefined.

Like with its KDE3 counterpart, kile offers the possibility of using “projects”, which means you can collect LaTeX documents, bib files, and so on, and associate them together. You can also set a master document, so that even if you are editing other files (included in the master document), when you build your LaTeX file the compilation runs on the master document. Even in this case, a wizard helps in creating a project and the master document.

Lastly, kile has a plethora of other options, including customizing what you can use to build LaTeX files and view them (DVI, PS, PDF…), as shown in this screenshot.

Conclusions

I have merely scratched the surface of this application, which is extremely powerful and can help anyone with their LaTeX needs. While the many options may be confusing, I think that this application is already geared towards a technically-inclined userbase and so it doesn’t matter much. kile 2.1 is still unstable but extremely promising, and I’m looking forward to its release.

Science and KDE: rkward

Einar — Sat, 07 Feb 2009 18:55:53 +0000

I try to use FOSS extensively for my scientific work. In fact, when possible, I use only FOSS tools. Among these there is the R programming language. It’s a Free implementation of the S-plus language, and it’s mainly aimed at statistics and mathematics. As the people who read my scientific posts know, I don’t like R much. But sometimes it’s the only alternative.

Well, what does R have to do with KDE? With this post I’d like to start a series (hopefully) of articles that deals with KDE programs used for scientific purposes. In this particular entry, I’ll focus on rkward, a GUI front-end for R.

Introduction

Although R is a programming language, it’s mainly used in an interactive session, started from the terminal. The standard installation can be improved by the use of add-on packages, libraries in R-speak, which can be installed from the Internet (Comprehensive R Archive Network or CRAN) or from local files. One of the most famous third party repositories is the Bioconductor project, which hosts a lot of packages used by life scientists who do bioinformatics.

The Windows version of R has a GUI (Rgui) which provides extra functionality, such as package management and loading, and other goodies. Although there were plan for a GTK+ frontend for Linux, the project is (as far as I know) stuck in a limbo.

That’s where rkward comes to the rescue. It’s a GUI front-end for R for KDE4, which aims to provide a graphical shell for many R commands and environments (and especially the publication-quality plotting figures).

Getting rkward

rkward is available from Sourceforge.net. Unfortunately, if you use a recent (>=2.8) version of R it won’t compile, due to the changes in R itself. For that, you need to directly download the sources off SVN with a command like this


svn co https://rkward.svn.sourceforge.net/viewvc/rkward/trunk/rkward/

Either way, the sources are compiled the usual, way, that is


cd rkward-xxx # Your rkward source dir
mkdir build; cd build
cmake  -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ../
make

Followed by make install as root or using sudo, depending on your distribution.

rkward at a glance

This is how rkward looks when loading it up (yes, it’s in Italian because that is my own locale). You have the R console (which I brought up) and then an output window which is used to display results. There is also another tab called “mio.dataset” (my.dataset) which keeps data, in a spreadsheet-like form. This is useful when you want to create your own datasets from scratch, or if you want to inspect one you have loaded.

So how do you start coding? You can create a new script using the “Script File” button. Like that, you can input R commands and then execute them all at once, or the current line. If you prefer interactive work, you can use the R command line (shown in the screenshot).

You can also use rkward to import data: R provides a series of functions (like read.table) to load data sets (usually comma- or tab-delimited text files). rkward provides a complete GUI to those functions, which is shown in the screenshot above. Notice that for working, it requires PHP (the line command version).

Ok, we have data loaded. Now we may want to do some operations: rkward provides front-ends to many of R’s statistical functions. In the screenshot, we can see the GUI for a two-variable t-test. Notice how it shows also the code, so the most experienced R people can view exactly what it does.

Like with statistics, R has powerful support for graphics, and even in this case rkward offers some frontends, for example histograms, boxplots, and scatter plots. You can also plot all kinds of distributions.

Lastly, rkward can manage your R packages (R package management is akin to one of a Linux distribution), and als your package sources. You can install or upgrade packages, and select where they’ll get installed to.

Conclusions

rkward is a nice frontend for the R programming language, which adds a GUI with the power of KDE to R. Unfortunately the program is still somewhat unstable (also shown by a warning when you run it) and its main developer has currently very little time to work on it. In case you may want to help, you can hop to the rkward-devel mailing list.

Published! (and it matters more)

Einar — Tue, 06 Jan 2009 17:39:39 +0000

Finally I can lift the curtain of silence and tell the reason why I’ve been very busy before Christmas: it all lies in the publication of a paper, “Using Pathway Signatures as Means of Identifying Similarities among Microarray Experiments”, which is finally out on this week’s issue of PLoS ONE. It’s different from the previous paper I mentioned (which was not my first publication, either), for two main reasons:

It’s a bioinformatics paper;
I am first author there.

The second point is very important because usually for a person doing bioinformatics is more difficult to end up as first author in a paper, since most we do is “something in the middle” like data analysis. Therefore, this paper is quite important for me. Also, it deals with an interest of mine, mainly analysis of biological networks using high-throughput platforms such as microarrays. Actually I’m also interested in network reconstruction, but I need to study far more than what I’m doing right now.

In any case, let’s hope this is the first of a (hopefully long) series!

DataMatrix 0.7 has been released

Einar — Sat, 27 Dec 2008 15:33:07 +0000

Finally a new entry! I’ve been extremely busy with other things, that is why I did not have time to write more. One of the main reason is related to an important landmark in my professional career, but I’ll write more about it after January 1st (hint: those who follow my Twitter updates may have already understood).

As a nice way to break the hiatus, I’m releasing a new version of DataMatrix, my implementation of R’s data.frame in Python. Although the version bump is small, there are loads of improvements. First of all, there is proper support for file-like objects, as well as support for appending and inserting both rows and columns. writeMatrix has been substantially improved and now writes files correctly, and I have added (experimental) support for a DataMatrix object that does not require files – EmptyMatrix. Also, there is now proper documentation. Last but not least, unit tests have been added, a good way to watch out for regressions in the code.

Finally, this version marks the entrance of dalloliogm as contributor to the code. He gave quite a number of helpful hints, especially with regards to unit tests.

I’m quite satisfied on how DataMatrix behaves – as a matter of fact I use it extensively on a number of internal projects.

You can grab DataMatrix 0.7 as a source package or as a Windows installer. Comments are welcome.

The plague of cross-database annotations

Einar — Sun, 02 Nov 2008 14:15:20 +0000

Recently I had to annotate a large (10,000+) number of genes identified by Entrez Gene IDs. My goal was to avoid “annotation files” (basically CSV files) that a part of wet lab group likes, because I wanted to stay up-to-date without having to remember to update them. So the obvious solution was to use a service available on the web, and in an automated way. For reference, I just tried to attach gene symbol, gene name, chromosome and cytoband.
I tried many services:

UCSC Genome Browser: it has a MySQL server but it’s rather slow and I did not want to clog it up. Using their tables and .sql files I managed to get a first shot at annotation, but about 2,000 genes were without annotation!
NCBI’s own Entrez Gene: This needs EUtils, and in Biopython there is not a parser for Entrez Gene XML entries. I had to scrap the idea because I did not have time.
Ensembl: I decided to use the Biomart service, through Rpy. There were missing genes, and sometimes the IDs were “converted” in something else (I had no time to figure out what was happening). Also some perfectly valid genes (in Entrez Gene) were not present in Ensembl.

In the end I just grabbed Bioconductor’s “org.Hs.eg.db” package and used its sqlite gene database (from Entrez Gene) to annotate the list, with only 97 missing IDs (mostly genes that had changed identifiers). However, this effort revealed a problem:the annotations are not consistent between databases. This is a real pain when doing microarray-based analysis, because you often have large number of genes and perceived lack of annotation might get lead to a number of them getting discarded.

I thought the situation was better than this. If I annotate genes in different databases with the same ID, I expect to get identical results. I mean, it’s not like Gene or Ensembl have little resources… or am I wrong?