• Ability to apply functions to elements of the matrix
• Ability to filter rows by column contents
• Ability to transpose rows with columns
• An option to load text files produced by R (which are, by design, broken)
• Removed the getter for columns, using dictionary-like syntax directly
• A lot of bug fixes

The download links on the project page have been updated, along with the documentation. Also, there is another change, because from now on the official Git repository is hosted on gitorious.org, and no longer on github, because gitorious (the software) is also free, while github.com’s is not. It’s mainly a philosophical issue (the same that prompted me to switch from twitter to identi.ca).

Also, from today DataMatrix is also officially hosted on the Python Package Index (with the name “datamatrix”), meaning that you can use easy_install to quickly install it.

If you use this module, let me know what you think (including bugs, if you find them).

I found a way thanks to the Biopython project, which offers a Python module to access the resources of the National Center for Biotechnology Information (NCBI) by providing an interface to their EUtils. Since the back-end was already taken care of, almost, at least, I sought to write a small Plasma applet. Which is what I’m presenting today. It’s written in Python, and uses the Python ScriptEngine to work. Currently, it searches the “Gene” database at NCBI by inputting the “Entrez Gene IDs”, that are numerical IDs that uniquely identify a gene record, and returns name, official symbol, organism, and a description if it’s present. It does not support anything else (see below).

The code lives in a git repository at github. WARNING: The code may be a complete mess (I’m not too well versed in GUI stuff, I mostly do text file manipulation) If you are so daring, you can obtain and install it in a very simple manner:

git clone git://github.com/cswegger/plasma-genesearch.git
cd plasma-genesearch
zip -r ../plasma-genesearch.plasmoid *
plasmapkg -i ../plasma-genesearch.plasmoid

After that you will see an “Entrez Gene Searcher” in your add applets dialog. Once added, it’ll look like this:

Pretty horrible, isn’t it? Well, once you get past that, you can input an ID (only IDs will work for now) in the text field (which doesn’t clear the text: see further on) and push “Go!”. The following is an example with ID 10000, which corresponds to the human gene AKT3:

“Search again” will bring you back to the search form.

Now, what has this to do with Planet KDE? Well, I’m asking for some code review from the community, if it’s possible, and suggestions to improve the horrid default look. I am especially interested in layouting, since I did not quite understand how it works, I mean, it should not work and it does….

Other things that need to be improved are:

• The Plasma.TextEdit is not cleared upon clicking. Is there a signal I can catch for that, so I can connect it to clear()?
• Proper searching. Bio.Entrez already does this: what I need is a way to display the records properly.
• A way to link the names to URLs, and have them open in Konqueror.

That should be it. I hope to work on it some more next weekend….

In this case I made my own: today was the last day in Dr.Cristina Battaglia’s laboratory, a place where I spent my three-year Ph.D. course and one year as a post-doc research fellow.

Those four years were not bad at all. They were interesting, and provided a good learning experience. I think I owe quite a bit to that place, especially because I was able to learn and improve my skills alongside the analysis and research work. So my thanks go to my former supervisor (Dr.Cristina Battaglia) and all my colleagues. It’s been a fun ride.

In my early days I used kLyX first, then LyX, but I found the platform to be too limited for my tastes, and also LaTeX errors were difficult to diagnose. I needed a proper editor, and that’s when I heard of kile, a KDE front-end for LaTeX. Kile is currently at version 2.0.2 and is a KDE 3 application. However, in KDE SVN work is ongoing to produce a KDE4 version (2.1) and that’s what I’ll look at in this entry.

Obtaining kile 2.1

First and foremost, a disclaimer. kile 2.1 has not been released yet in any form, and so should be considered unstable and crash-prone. That said, it runs more or less well on my platform.

The first thing to do is to grab the sources from SVN:

svn checkout svn://anonsvn.kde.org/home/kde/trunk/extragear/office/kile

That will put kile’s sources in a directory called “kile”. The next step is to compile it (as usual, you need KDE4 development packages/files installed):

cd kile
mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ../
make

Followed by the usual make install as root or using sudo.

kile 2.1 at a glance

This is how kile looks when loaded on my system:

(For the inquisitive people, it’s not a scientific work, rather a sci-fi like book I’m writing).

Kile uses the katepart for editing, so that means all the goodies that come with Kate can be used, including the recently-added vim input mode. Aside from editing and LaTeX syntax highlighting, kile offers a configurable LaTeX command completion, like this screenshot shows:

From the toolbars and the menus you can insert almost every LaTeX command known to mankind. For the people less apt with LaTeX, kile offers a series of wizards in order to make the creation of figures, tables and even complete documents. The one I’m showing here is the Quick Start wizard, which enables you to select document classes, add packages, and add information like author and date. As I was saying earlier, kile 2.1 is still a work in progress, and that explains why the dialog is still a little unrefined.

Like with its KDE3 counterpart, kile offers the possibility of using “projects”, which means you can collect LaTeX documents, bib files, and so on, and associate them together. You can also set a master document, so that even if you are editing other files (included in the master document), when you build your LaTeX file the compilation runs on the master document. Even in this case, a wizard helps in creating a project and the master document.

Lastly, kile has a plethora of other options, including customizing what you can use to build LaTeX files and view them (DVI, PS, PDF…), as shown in this screenshot.

Conclusions

I have merely scratched the surface of this application, which is extremely powerful and can help anyone with their LaTeX needs. While the many options may be confusing, I think that this application is already geared towards a technically-inclined userbase and so it doesn’t matter much. kile 2.1 is still unstable but extremely promising, and I’m looking forward to its release.

]]> http://www.dennogumi.org/2009/02/science-and-kde-kile/feed 14 http://www.dennogumi.org/2009/02/science-and-kde-rkward http://www.dennogumi.org/2009/02/science-and-kde-rkward#comments Sat, 07 Feb 2009 18:55:53 +0000 Einar http://www.dennogumi.org/?p=533 I try to use FOSS extensively for my scientific work. In fact, when possible, I use only FOSS tools. Among these there is the R programming language. It’s a Free implementation of the S-plus language, and it’s mainly aimed at statistics and mathematics. As the people who read my scientific posts know, I don’t like R much. But sometimes it’s the only alternative.

Well, what does R have to do with KDE? With this post I’d like to start a series (hopefully) of articles that deals with KDE programs used for scientific purposes. In this particular entry, I’ll focus on rkward, a GUI front-end for R.

Introduction

Although R is a programming language, it’s mainly used in an interactive session, started from the terminal. The standard installation can be improved by the use of add-on packages, libraries in R-speak, which can be installed from the Internet (Comprehensive R Archive Network or CRAN) or from local files. One of the most famous third party repositories is the Bioconductor project, which hosts a lot of packages used by life scientists who do bioinformatics.

The Windows version of R has a GUI (Rgui) which provides extra functionality, such as package management and loading, and other goodies. Although there were plan for a GTK+ frontend for Linux, the project is (as far as I know) stuck in a limbo.

That’s where rkward comes to the rescue. It’s a GUI front-end for R for KDE4, which aims to provide a graphical shell for many R commands and environments (and especially the publication-quality plotting figures).

Getting rkward

rkward is available from Sourceforge.net. Unfortunately, if you use a recent (>=2.8) version of R  it won’t compile, due to the changes in R itself. For that, you need to directly download the sources off SVN with a command like this

svn co https://rkward.svn.sourceforge.net/viewvc/rkward/trunk/rkward/

Either way, the sources are compiled the usual, way, that is

cd rkward-xxx # Your rkward source dir
mkdir build; cd build
cmake  -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ../
make

Followed by make install as root or using sudo, depending on your distribution.

rkward at a glance

This is how rkward looks when loading it up (yes, it’s in Italian because that is my own locale). You have the R console (which I brought up) and then an output window which is used to display results. There is also another tab called “mio.dataset” (my.dataset) which keeps data, in a spreadsheet-like form. This is useful when you want to create your own datasets from scratch, or if you want to inspect one you have loaded.

So how do you start coding? You can create a new script using the “Script File” button. Like that, you can input R commands and then execute them all at once, or the current line. If you prefer interactive work, you can use the R command line (shown in the screenshot).

You can also use rkward to import data: R provides a series of functions (like read.table) to load data sets (usually comma- or tab-delimited text files). rkward provides a complete GUI to those functions, which is shown in the screenshot above. Notice that for working, it requires PHP (the line command version).

Ok, we have data loaded. Now we may want to do some operations: rkward provides front-ends to many of R’s statistical functions. In the screenshot, we can see the GUI for a two-variable t-test. Notice how it shows also the code, so the most experienced R people can view exactly what it does.

Like with statistics, R has powerful support for graphics, and even in this case rkward offers some frontends, for example histograms, boxplots, and scatter plots. You can also plot all kinds of distributions.

Lastly, rkward can manage your R packages (R package management is akin to one of a Linux distribution), and als your package sources. You can install or upgrade packages, and select where they’ll get installed to.

Conclusions

rkward is a nice frontend for the R programming language, which adds a GUI with the power of KDE to R. Unfortunately the program is still somewhat unstable (also shown by a warning when you run it) and its main developer has currently very little time to work on it. In case you may want to help, you can hop to the rkward-devel mailing list.

• It’s a bioinformatics paper;
• I am first author there.

The second point is very important because usually for a person doing bioinformatics is more difficult to end up as first author in a paper, since most we do is “something in the middle” like data analysis. Therefore, this paper is quite important for me. Also, it deals with an interest of mine, mainly analysis of biological networks using high-throughput platforms such as microarrays. Actually I’m also interested in network reconstruction, but I need to study far more than what I’m doing right now.

In any case, let’s hope this is the first of a (hopefully long) series!

As a nice way to break the hiatus, I’m releasing a new version of DataMatrix, my implementation of R’s data.frame in Python. Although the version bump is small, there are loads of improvements. First of all, there is proper support for file-like objects, as well as support for appending and inserting both rows and columns. writeMatrix has been substantially improved and now writes files correctly, and I have added (experimental) support for a DataMatrix object that does not require files – EmptyMatrix. Also, there is now proper documentation. Last but not least, unit tests have been added, a good way to watch out for regressions in the code.

Finally, this version marks the entrance of dalloliogm as contributor to the code. He gave quite a number of helpful hints, especially with regards to unit tests.

I’m quite satisfied on how DataMatrix behaves – as a matter of fact I use it extensively on a number of internal projects.

You can grab DataMatrix 0.7 as a source package or as a Windows installer. Comments are welcome.

• UCSC Genome Browser: it has a MySQL server but it’s rather slow and I did not want to clog it up. Using their tables and .sql files I managed to get a first shot at annotation, but about 2,000 genes were without annotation!
• NCBI’s own Entrez Gene: This needs EUtils, and in Biopython there is not a parser for Entrez Gene XML entries. I had to scrap the idea because I did not have time.
• Ensembl: I decided to use the Biomart service, through Rpy. There were missing genes, and sometimes the IDs were “converted” in something else (I had no time to figure out what was happening). Also some perfectly valid genes (in Entrez Gene) were not present in Ensembl.

In the end I just grabbed Bioconductor’s “org.Hs.eg.db” package and used its sqlite gene database (from Entrez Gene) to annotate the list, with only 97 missing IDs (mostly genes that had changed identifiers). However, this effort revealed a problem:the annotations are not consistent between databases. This is a real pain when doing microarray-based analysis, because you often have large number of genes and perceived lack of annotation might get lead to a number of them getting discarded.

I thought the situation was better than this. If I annotate genes in different databases with the same ID, I expect to get identical results. I mean, it’s not like Gene or Ensembl have little resources… or am I wrong?

In this new version there have been a few improvements and correction to a couple bugs (for example saveMatrix did not really save) and the start (only a stub at the moment) of an append function to add more columns (I’ll also think about a function to add rows).

DataMatrix is licensed under the GNU GPL, version 2 only. You can download the installer (Windows) or the source distribution (Linux and other *nixes). The only requirement is Python 2.5 or later installed on your system.

The README currently is a stub, but you can browse the pydoc generated documentation, which details how to instantiate and use DataMatrix objects (or you can turn to my older post).

Also, since git is the new “cool feature of the day”, DataMatrix is is hosted on github’s repository, and you can grab the source with

git clone git://github.com/cswegger/datamatrix.git

Comments and suggestions are welcome. I’ll be putting a static page on DataMatrix tomorrow, if time permits.

The idea is to store the values of the text file as a dictionary of columns which includes then a list of (row name, row value) tuples. Like this, you can access the columns by their name (I need to see if it’s workable to also use numbers), or you can view specific rows, including all or a subset of the columns. It’s decently faster and it allows for non-sequential access, which you can’t do when reading a file (or a file-like structure).

Requirements

I have tested this on Python 2.5.1. Older versions may or may not work. All modules called by this one should be shipped with Python itself.

Download and installation

Download the py file directly. Currently there is no installation mechanism, so copy it wherever Python can find it.  There’s some API documentation generated with pydoc.

This module is licensed under the GNU General Public License, version 2.

Usage

First of all, import the module

import datamatrix

Then open a file and instantiate a DataMatrix object

fh = open("somefile.txt")
data = datamatrix.DataMatrix(fh)

By default no column with row names is specified, so if you have one, you have to specify it:

data = datamatrix.DataMatrix(fh, row_names=1)

More options are in the documentation.

Once the DataMatrix is initialized, you can view how many columns are there and also view rows with the getRow method:

>> data.columns
["GeneID","Great_Exp1","Great_Exp2"]

>> data["Great_Exp1"]
[("Gene1",56.34),
...
]

>> data.getRow(5)
["NOT_EXISTENT","56.545","4.56"]

Sometimes you’d want to get only the column without the row identifier, and that’s where getColumn comes in:

>> data.getColumn("Great_Exp1")
[56.34,2.55.....]

Should you want to save a DataMatrix instance, you can use the writeMatrix function:

datamatrix.writeMatrix(data,fname="/path/to/somewhere/file.txt")

That’s all. Questions and suggestions, especially on coding and improvements, are very welcome.