• UCSC Genome Browser: it has a MySQL server but it’s rather slow and I did not want to clog it up. Using their tables and .sql files I managed to get a first shot at annotation, but about 2,000 genes were without annotation!
• NCBI’s own Entrez Gene: This needs EUtils, and in Biopython there is not a parser for Entrez Gene XML entries. I had to scrap the idea because I did not have time.
• Ensembl: I decided to use the Biomart service, through Rpy. There were missing genes, and sometimes the IDs were “converted” in something else (I had no time to figure out what was happening). Also some perfectly valid genes (in Entrez Gene) were not present in Ensembl.

In the end I just grabbed Bioconductor’s “org.Hs.eg.db” package and used its sqlite gene database (from Entrez Gene) to annotate the list, with only 97 missing IDs (mostly genes that had changed identifiers). However, this effort revealed a problem:the annotations are not consistent between databases. This is a real pain when doing microarray-based analysis, because you often have large number of genes and perceived lack of annotation might get lead to a number of them getting discarded.

I thought the situation was better than this. If I annotate genes in different databases with the same ID, I expect to get identical results. I mean, it’s not like Gene or Ensembl have little resources… or am I wrong?

R is not a performing language at all, it doesn’t parallelize well when using HPC (at least from the talks I’ve had with people studying the matter), and in general is a memory and resource hog. For example, it takes much more to perform RMA via R that with RMAExpress (which is a C++ application): the latter works also better with regards to memory utilization. I can understand the complexity of some statistical procedures, but what about ?

The surprising aspect is that aside by a few exceptions (like the aforementioned RMAExpress) no one has tried to write more performing implementations of certain algorithms. I for one would welcome a non-R implementation of SAM (the original implementation works in Excel… ugh) or similar algorithms. Otherwise we would be stuck with programs that are interesting, but way too memory hungry (AMDA comes to mind).

It turns out that it’s a fault of both limma and the data sets, because apparently the raw data found in the Stanford Microarray Database have different length, gene-wise (a result of not all spots on the array being good?) and limma itself does need equal length tables to form a single object (I stumbled upon the same problem when doing my thesis, but I used a hack to work around it), and does not perform any checking.

According to the documentation, the “merge” command should be used to deal with these cases, but here’s what I get:

>> RG1 = read.maimages(file="file1.txt",source="smd")
Read file1.txt
>> RG2 = read.maimages(file="file2.txt",source="smd")
Read file2.txt
>> merge(RG1,RG2)
Error in merge(RG1,RG2): Need row names to align on
>> rownames(RG1)
NULL
>> rownames(RG2)
NULL 

I’m going to ask the Bioconductor ML and see what they tell me.

The problem sometimes lies in the data, sometimes lies in the analysis software and sometimes in a combination of factors. When doing work on a public data set (Zhao et al., 2005), I had to start analysis from raw data. Now, I tried using both the limma and marray Bioconductor packages, but both of them bail out with cryptic error messages. From what I’ve learnt by googling around, it seems that R doesn’t like batch loading of tables of different length.

I have 177 samples and I have to normalize them all together. Apparently this is a quirk of marray and limma (or worse, R itself) which is preventing me to work properly. And this is not the first time it happens, either: in the past year I’ve lost a lot of time dealing with software issues rather than performing real analsis. The problem has been posted already on some R mailing lists (and on BioC, too), but judging from the responses I doubt I’ll see a solution.

I guess I’ll have to work around this somehow (and of course, this doesn’t improve the idea I have of R…).

The subsequent idea was “let’s implement auto-detection of common identifiers in the class”. The problem is… is there any actual documentation on how identifiers are made? So far, using regular expressions, I’ve tracked down a few:

• RefSeq
• GenBank
• Entrez Gene
• UCSC Genome Browser
• Ensembl

However, I have no idea if I have implemented all types of these IDs. Does anyone know a place where to look these information up?

(On a related note: my thesis defense will be on January 14th, 2008, so I have to get the printing going)

Let me first say what RMA is about: it stands for “Robust Multi-array Average” and is a model-based quantification method for Affymetrix arrays, originally developed by Irizarry et al. It has a number of advantages over the Microarray Analysis Suite 5 (MAS5) algorithm used by Affymetrix software, especially with weakly expressed transcripts. It is commonly made up of three steps: background correction, quantile normalization and median polish.

RMAExpress is a C++, GUI-based program (using wxWidgets) that performs this process. The main advantage over the various R implementations is speed, as R doesn’t really excel in this regards. You can adjust the various RMA parameters, and you can also view the model representations, to see if some areas on the array perform differently (e.g., when there are irregularities in the signal intensities).

What I liked best is that you can use custom chip definition files (CDFs). Dai et al. have already shown that old 3′ GeneChips have outdated annotations, and have proposed new CDFs to compensate. We have already tested their improvement and it gives a nice increase in the number of annotated genes. RMAExpress processes these CDFs just fine.

Finally, you can export data either in log2 format (to use in procedures like SAM) or in absolute form (which I need for my work). The program is extremely light and processes a good number of arrays fairly quickly. Windows users have a pre-built binary, while Linux ones need to build sources. The instructions on the page are overly complicated: here’s how I managed to build it on Kubuntu:

sudo aptitude install libwxtgk2.8-dev
mkdir tmp
cd tmp
tar xvzf  /path/to/RMAExpress_1.0beta3_src.tar.gz
make all

After that, just run RMAExpress from its directory.

After all, I’m quite pleased with the program and I will keep using it in the future.