Fortunately, the Nepomuk developers have come up with a high level API to query already stored metadata, and today’s post will deal with querying tags in Nepomuk. As per the past tutorials, the full source code is available in the kdeexamples module.

Let’s start off with the basic imports:

import sys

import PyQt4.QtCore as QtCore

import PyKDE4.kdecore as kdecore
import PyKDE4.kdeui as kdeui
from PyKDE4.kio import KIO
from PyKDE4.nepomuk import Nepomuk
from PyKDE4.soprano import Soprano

Then let’s create a simple class that wil be used for the rest of this exercise:

class NepomukTagQueryExample(QtCore.QObject):

    def __init__(self, parent=None):

        super(NepomukTagQueryExample, self).__init__(parent)

__init__ is just used to construct the instance, nothing more. The bulk of the work is in the query_tag() function, which we’ll take a look at in parts.

    def query_tag(self, tag):

        """Query for a specific tag."""

        tag = Nepomuk.Tag(tag)

First of all we convert the tag we want to query into a proper Nepomuk.Tag() instance. Of course we should use an already existing tag: even if Nepomuk.Tag() automatically creates new tags, it makes little sense to query for a newly created tag, doesn’t it?

For our job, we need to use properties which define the terms of our query. As we’re looking for tags, we’ll use Soprano.Vocabulary.NAO.hasTag():

        soprano_term_uri = Soprano.Vocabulary.NAO.hasTag()
        nepomuk_property = Nepomuk.Types.Property(soprano_term_uri)

The first call generates an URI pointing to a specific RDF resource for this specific term, which is then wrapped as a Nepomuk.Types.Property in the second call. While the C++ API docs don’t show this, I found it to be necessary, or the Python interpreter would raise a TypeError. Notice that this is not the only term we can use: aside for tags, there are a lot of other URIs we can use for querying, listed in the Soprano API docs.

Once we have our property set up, it’s time to define which kind of query we’re going to use. In this case, since we want to check for the presence of tags, we use a Nepomuk.Query.ComparisonTerm, which is a query term used to match values of specific properties (in our case, tags):

        comparison_term = Nepomuk.Query.ComparisonTerm(nepomuk_property,
                Nepomuk.Query.ResourceTerm(tag))

Our tag is wrapped in a ResourceTerm, which is used exactly for the purpose. Now we make the proper query: in this specific case, we want to look up files tagged, so we use a FileQuery. We could also get other items, such as mails (in Akonadi): in that case we could use a a Nepomuk.Query.Query():

        query = Nepomuk.Query.FileQuery(comparison_term)

Lastly, we want to get some results out of this query. There are different methods, but for this tutorial we’ll use the tried-and-tested KIO technology:

        search_url = query.toSearchUrl()
        search_job = KIO.listDir(kdecore.KUrl(search_url))
        search_job.entries.connect(self.search_slot)
        search_job.result.connect(search_job.entries.disconnect)

First we convert the query to a nepomuksearch:// url, which then we pass to KIO.listDir, to list the entries. Unlike my previous post on KIO, this job emits entries() every time one is found, so we connect the signal to our search_slot method. We also connect the job’s result() signal in a way that it will disconnect the job once it’s over.

Finally, let’s take a look at the search_slot function:

    def search_slot(self, job, data):

        # We may get invalid entries, so skip those
        if not data:
            return

        for item in data:
            print item.stringValue(KIO.UDSEntry.UDS_DISPLAY_NAME)

Entries are emitted as UDSEntries: to get something at least understandable, we turn them into the file name, which is obtained by the stringValue() call using KIO.UDSEntry.UDS_DISPLAY_NAME.

That’s it. As you can see, it was pretty easy. Of course there’s more than that. For further reading, take a look at Nepomuk’s Query API docs, and Query Examples. Bear in mind however that to the best of my knowledge, the “fancy operators” mentioned there will not work with Python.

Happy Nepomuk querying!

Defining the problem

Since Eisen’s original paper on clustering, this form of analysis has been widely used by a lot of researchers. However, as it is known, such systems may be susceptible to an ordering bias: in other words, the order of the samples and/or genes might influence the final result. That’s why popular software such as TMeV offers alternative approaches, based on bootstrapping. 

In this specific form of bootstrapping, the samples and/or genes are randomly shuffled a number of times (1000 or more iterations are a good starting point) and the resulting dendrograms checked for consistency and robustness of partitioning. In other words, a p-value is calculated, our null hypothesis being that the arrangement of samples/genes is merely due by chance. Depending on the software, this value might be expressed either in form of p-value or percentage (TMeV calls it support). 

In the past years, I found an interesting method developed by Hidetoshi Shimodaira: the technique, called multiscale bootstrap resampling, aims at determining more accurate p-values out of the bootstrapping. Shimodaira calls the resulting p-value an AU value, where AU stands for “approximately unbiased”, a more precise p-value than the one obtained through bootstrapping alone.

In addition to this nice algorithm, a R package was also provided, named pvclust (it’s available on your favorite CRAN mirror). And that’s exactly what we’ll use for this exercise.

Prerequisites

Some of the readers of this blog might remember my disdain of R: while I need to use it for Bioconductor, I’m often annoyed by its weird syntax, and difficult to understand error messages. Luckily, thanks to the hard work of Laurent Gautier and contributors, there’s rpy2, a nice R-to-Python bridge. All the examples here require this package, version 2.1 or newer (I’d recommend the release candidate of 2.2, it’s really nice). Unfortunately, this means that Windows users are out of luck as there’s no version of rpy2 2.1 or 2.2 available for that platform..

Also, don’t forget to have the pvclust package installed in R.

Loading and preparing the data

import rpy2.robjects as robjects
from rpy2.robjects.packages import importr

base = importr("base")
pvclust = importr("pvclust")

dataframe = robjects.DataFrame.from_csvfile("GSE4984.txt", sep="\t ", row_names=1)

rows = robjects.IntVector(range(1,501))
subset = dataframe.rx(rows, True)

subset <- dataframe[rows, ]

result = pvclust.pvclust(subset, nboot=100, method_dist="correlation", method_hclust="average")

matrix = base.as_matrix(subset)
subset_transposed = matrix.transpose()
result_rows = pvclust(subset_tranposed, nboot=100, method_dist="correlation", method_hclust="average")

graphics = importr("graphics")
graphics.plot(result)

graphics = importr("graphics")
grdevices = importr("grDevices")
grdevices.pdf("myresult.pdf", paper="a4")
graphics.plot(result)
grdevices.dev_off()

gplots = importr("gplots")
row_dendrogram = result_rows.rx2("hclust")
column_dendrogram = result.rx2("hclust")
gplots.heatmap_2(subset, Rowv=row_dendrogram, Colv=column_dendrogram, col=gplots.greenred(255), density_info="none")

How do I get it?

git clone http://git.gitorious.org/kde-vlc-snapper/kde-vlc-snapper.git

sudo python setup.py install

kdevlcsnapper

Before starting, let me say that creating this tutorial was only possible thanks to the help of Sebastian Trueg, who helped me by pointing out some mistakes I was doing.

The example here is not showing the extra methods to set up a KApplication, etc.: the full code for this tutorial is available in the kdeexamples module.

Let’s start with the basics.

import sys
from PyQt4 import QtCore
from PyKDE4 import kdecore
from PyKDE4 import kdeui
from PyKDE4.nepomuk import Nepomuk

This will import all the bits needed to test our experiment. As a second step, we’ll create a dummy empty file.

dummy_file = open("dummy.txt", "w")
dummy_file.write("Some text\n")
dummy_file.close()

Or, if we have Python 2.6+ (as pointed out in the comments):

with open("dummy.txt", "w") as handle:
    handle.write("Some text\n")

Now that we have our file, it’s time to do something productive with it. But first and foremost, we have to ensure that Nepomuk is running. To do so, we make a simple check (EDIT: fixed the syntax):

result = Nepomuk.ResourceManager.instance().init()
if result != 0:
     return

Neomuk.instance().init() must return 0 if Nepomuk is properly set up. Once this is taken care of, we can manipulate the semantic information of our file. Thus, Nepomuk needs to be made aware of it: this is done by creating a resource that points to the actual file:

file_info = QtCore.QFileInfo("dummy.txt")
absolute_path = file_info.absoluteFilePath()
resource = Nepomuk.Resource(kdecore.KUrl(absolute_path)

Notice that we must use an absolute file path, or the resource will be not created properly and although no errors will happen when tagging, changes will not be made. Let’s now create a tag, which is done by simply constructing a Nepomuk.Tag instance:

tag = Nepomuk.Tag("test_example")
tag.setLabel("test_example")

In the first line we create the tag, then we associate it with a label, so that it will be displayed in applications such as Dolphin. The nice thing is that if the Tag already exists, it will be recycled: no duplicates will occur. A simple call to addTag to the resource we created earlier will now tag it:

resource.addTag(tag)

We can also add comments that can show up in Dolphin as well by using the setDescription method:

resource.setDescription("This is an example comment.")

What if we want to remove tags and descriptions? To wipe them all, we can use the remove() method of the Resource, otherwise we can strip elements by using removeProperty along with the tagUri() or descriptionUri() methods of the resource:

resource.remove() # strip everything
resource.removeProperty(resource.descriptionUri()) # remove comment
resource.removeProperty(resource.tagUri()) # remove tags

That’s it. As you can see, adding semantic information from PyKDE4 isn’t that hard. Sooner or later I’ll try my hand at queries and report back my findings.

The rest is up to you to figure out.

First of all, now there isn’t one, but two Python modules:

• ocslib, a pure Python module that can be used to interface with OCS-based forum systems;
• ocslibkde, a PyKDE4 based module that can be used to interface with OCS-based forum system in KDE applications.

Currently ocslib supports reading and posting, while ocslibkde only reading (as of now). Both can be retrieved from the kde-forum-mods repository under the ocs-client subdirectory. The Python lib needs unit-testing, then I’ll be able to push a tarball soon for people to test (but you can always check out the Git repository). With regards to the PyKDE4 library, I plan on making a proof-of-concept plasmoid soon that shows how to use the API.

Speaking of API, here are some examples using ocslib:

>>> from ocslib import service# Connect to OCS
>>> ocs_service = ocslib.service.OCService("http://www.example.com")
#Retrieve all forums
>>> forums = ocs_service.list_forums()
# Elements have attributes for name, posts, etc.
>>> print forums[0].name
"Test forum"
#Retrieve threads for forum 15
>>> threads = ocs_service.list_forum_threads(forum_id=15)
# Retrieve thread 8945 from forum 15
>>> messages = ocs_service.show_thread(forum_id=15, topic_id=8945)
>>> print messages[0].text
"Hello world!
#Post to a forum - requires authentication
>>> ocs_service = service.OCService("http://www.example.com", username="foo", password="bar")
>>> message = "Hello, KDE people!"
>>> subject = "Test message"
>>> ocs_service.post(forum_id=15, subject=subject, message=message)
True # Return code of operation

Feedback (especially on the API) welcome!

A dedicated application would be usually much better than a browser, because you can work around the intrinsic limitations of the browser itself. The problem is that you can’t really access a forum with anything else than a browser. That is, it used to be like this, but now things are changing.

In the past months fellow administrator bcooksley has been working quite hard implementing the Open Collaboration Services (OCS) specification in the KDE Community Forums. For the uninformed, it’s the same API that powers OpenDesktop.org and related web pages. This means that you could access the forum contents programmatically using a REST API and parsing the XML that is returned by the service.

Unfortunately, bcooksley had no time to implement a client that would make use of this newly-made service.

That’s where I stepped in. This morning I committed in the kde-forum-mods repository the first implementation of a backend to access the forums’ OCS service. Currently it’s extremely basic – just a few classes to wrap the XML responses into decent data representation, and a basic class to perform reading requests: that means that technically it is possible to request forum listings, thread listing, and posts. I’m still working on the ability of replying and posting messages.

Being a Pythonista, the backend is written entirely in Python: currently it uses the standard library plus dateutil and lxml to do its bidding, but the next steps would be to turn it into a PyKDE4 library to access all the KDE related goodness (hello, KIO!). Bear in mind that currently there is no application using this: I merely completed (part of) the backend.

If you’re interested, the code can be found on gitorious.org, in the ocs-client directory, branch experimental, inside the kde-forum-mods repository.

from PyQt4 import QtCore
from PyQt4 import QtGui
from PyKDE4 import kdeui

class MyGUI(QtGui.QWidget):

    def __init__(self, parent=None):
        super(MyGUI, self).__init__(parent)
        self.pushbutton = kdeui.KPushButton()
        self.pushbutton.setText("Push me!")

        QObject.connect(self.pushbutton, QtCore.SIGNAL("clicked()"),
                               self.button_pushed)

    def button_pushed(self):
        print "Button clicked"

The main advantage of this syntax is that it’s very close to the C++ equivalent, and so you can translate easily from C++ to Python. Unfortunately the advantages of this syntax end here. The disadvantages, at least from a Python coding perspective, outweigh the advantages:

• It’s extremely error-prone: make a typo, and not only your signal won’t be connected, but you won’t even get a warning, your program will simply do nothing;
• In case you have overloaded signals, you have to type the exact signature, going back to the first problem;
• It’s not Pythonic at all.

So, in recent PyQt versions (and thus also in PyKDE4) a new style approach was introduced (although the old style is always present should it be the need to). Using the new style, the signals become a property of the object that emits them. and then you use the connect function of that property. Here’s the example using the new style-signals:

from PyQt4 import QtCore
from PyQt4 import QtGui
from PyKDE4 import kdeui

class MyGUI(QtGui.QWidget):

    def __init__(self, parent=None):
        super(MyGUI, self).__init__(parent)
        self.pushbutton = kdeui.KPushButton()
        self.pushbutton.setText("Push me!")
        # New style
        self.pushbutton.clicked.connect(self.button_pushed)

    def button_pushed(self):
        print "Button clicked"

As you can see it’s much clearer, and much more Pythonic. Also, typos will trigger an AttributeError, which means you’ll be able to track where the problem is.

What about overloaded signals? Normally the first defined is the default, but you can use a dictionary-like syntax to access other overloads (signal names are completely made up here):

# One signal is without arguments, the other has a bool

# Signal without arguments
self.my_widget.connected.connect(self.handle_errors)
# Signal with a book
self.my_widget.connected[bool].connect(self.handle_errors)

Signals are emitted with the emit() function and disconnected with the disconnect() function:

# Emit a signal
self.pushbutton.clicked.emit()
# Emit a signal with a value (an int)
self.my_widget.valueChanged.emit(int)
# Disconnect another
self.my_tabwidget.currentIndexChanged.disconnect()

To define new signals, you can use the pyqtSignal function, specifying which values will the signal take (if any): just define that as a class constant (like in the example) and then you can access them like the wrapped ones:

class MyWidget(QWidget):

    # Signal with no arguments
    operationPerformed = QtCore.pyqtSignal()

    # Signal that takes arguments
    valueChanged = QtCore.pyqtSignal(int)

I merely scratched the surface with this. For more information, check out PyQt’s reference manual, which also covers other cases.

I knew already that the various ioslaves can store metadata, consisting of key-value pairs which are specific on the slave used. Normally you can get the whole map by accessing the metaData function of the job you have used, in the slot connected from the result signal. For some reason, however, in PyKDE4 calling metaData() triggers an assert in SIP, which ends in a crash (at least in my application; I stil need to debug further). KIO jobs have also the queryMetaData function, which returns the value of the key you have queried. Unfortunately, there was no way I could find the name.

Thus began my search for the right key. Googling didn’t help, and on IRC I got the first answers I needed but not enough to reach the goal. Until I saw a commit by David Faure in trunk/kdelibs/kio/ which touched a file called DESIGN.metadata (link is for the branch version). After checking with webSVN, that was exactly the thing I was looking for! It lists all the keys for the metadata, indicating also to which ioslave they begin. After that, the solution was easy.

Of course I’m not leaving you hanging there and now I’ll show you how, in PyKDE4, you can quickly check for the server response:

from PyKDE4.kio import KIO
from PyQt4.QtCore import SIGNAL
[...]

class my_widget(QWidget):
[...]

    def check_address(self, url):

        # You can add optional flags such as KIO.HideProgressInfo
        job = KIO.get(KUrl(url))
        self.connect(job, SIGNAL("result (KJob *)"), self.slot_result)

    def slot_result(self, job):

        if job.error():
            # Bail out if there's an error
            return    

        # Get the HTTP response through queryMetaData
        http_response = job.queryMetaData("responsecode")
        print "Got response: %s" % unicode(http_response)

This snippet does a few things. Firstly, it gets the specified URL, using KIO.get (KIO.stat doesn’t set the required metadata). Notice that the call is not wrapped in the new-style PyQt API because result (KJob *) isn’t wrapped like that (there’s a bug open for that). In any case, the signal passes to the connecting slot (slot_result) where we first check if there’s an error (perhaps the address didn’t exist?) and then we use queryMetaData("responsecode") to get the actual response code.

If you want to do error checking basing on the result, bear in mind that KIO operates asynchronously, so you should use a signal to tell your application that the result is what it expected or not.

I wonder if this should be documented in Techbase…