I am considering trying to get cdat_lite working with numpy as people
here assure me the transition shouldn’t be too hard.

People from the scipy vendor Enthought are here in force. They have a
framework for interactive visualisation called Chaco which looks
great, although not really relevant for web-based development. They
havn’t tried it with basemap yet. It was also reassuring to see
everyone agrees matplotlib installation is a problem.

I think I’m going to start using IPython again after I saw a demo.
This is an interactive python shell with enhanced introspection,
system access, debugging etc. It works particularly well with
matplotlib as a visualisation environment.

The lighting talks this evening were both entertaining and
informative. What about ZjengoGears — the 7th generation
web-framework that knows how you want to design your website? More
seriously, tgwebservices looked like a pain-free way of supporting SOAP
+ WSGI (can it really be that simple?) and there was an impassioned
plea to add numpy to the standard library.

Oh and Eggs seems one of the buzzwords of the conference — lampooned
as such at times. I felt I might be preaching to the converted but the content went well. There were some embarrasing technical difficulties brought on by my S5 presentation system. I think I’ll go with the safety of ooimpress next time. The material is available here and here, although I hope to fix a few typos and get out better versions soon.

Here I am at PyCon2007 in Dallas, seeing face to face many of the
people who’s names line up in my email browser each morning. I’ve
just come out of one on Python web frameworks and now feal slightly
better qualified to evaluate the superfluity of options.

In the lineup were Django, Pyjamas, Zope, Spyce, Turbogears, Pylons,
CherryPy, Twisted. Quite a diverse selection.

There’s some good notes on the session at
http://www.b-list.org/weblog/2007/02/23/pycon-2007-web-frameworks-panel

One jucy comment was “the plethora of web frameworks seems Perlish and
not Pythonic”. This was very well handled by pointing out web
frameworks are a lot less mature than Python and there is still a lot
of experimenting going on. That’s a good answer for framework
developers but doesn’t make choosing between the options any easier.
Still, I think I should give django a try sometime.

Other interesting talks: a mind-blowingly technical talk on iterators — still exciting though,
Heartening talk about teaching python to school children and a
rallying call to improve Python advocacy.

The good news is that every single sponsor in the lighting talks were
recruiting. The bad news is that Python development has a supply-side
problem unless it gets the word out about it’s power.

I hope more is to come but I have a Python for Science BoF to go to.

The BADC has found CDAT’s data management layer (CDMS) invaluable in developing server-side analysis tools. It handles the sorts of calendars only found in numerical modelling, abstracts away NetCDF coordinate variables on a veriety of grids and allows aggregation of huge multi-file datasets into a logical dataset. We like it so much that we developed an input layer for the UK Met. Office PP format.

However CDAT aspires to be much more than this. It is a comprehensive data analysis environment with a GUI and visualisation components and as such has grown into rather a heafty package. It’s a 160Mb download from sourceforge including, among other things, it’s own Python tarball. This can be rather inconvenient if you have your own personalised Python environment. Non-default installation can be tricky and time consuming — not too much of a problem when setting up a single workstation but more arduous at a place such as the BADC where we have quite a heterogeneous network of Linux systems. Add to this an evolving codebase, as the cdunifpp component has matured, and you have a recipe for multiple installations, most of which are out of date.

cdat_lite tries to fix this by taking out the bits of CDAT we find most useful — the libcdms I/O layer and the core python packages cdms, cdutil, cdtime, unidata, genutil, regrid and xmgrace

The first cut of cdat_lite was relatively easy to create. To eggify the CDAT packages only required a new setup.py script and a single patch to unidata to load a datafile with the pkg_resources API. Similarly libcdms had it’s own configure script and Makefile that could be called from setup.py. I could build binary eggs for x86_64 and i686 from within my sandbox that seemed to work once deployed. The fun came when I became fussier about how easy it should be to install. I wanted the tarball to work with easy_install as well as the pre-built eggs. I wanted this to work on as many machines at the BADC as possible. In particular RedHat Enterprise, SUSE 10 i686 and x86_84 and kubuntu i686. This opened up several windy roads.

Dependencies 1: Numeric

CDAT needs Numeric. It needs the Numeric header files to compile libcdms. Numeric is quite often present on the target python installation but not always. When it is installed it may or may not include the Numeric headers and probably doesn’t have an EGG-INFO directory (therefore it isn’t detected by setuptools). Numeric isn’t on the cheeseshop and isn’t easily downloadable from sourceforge because it a legacy package.

At first I just said you needed Numeric before you begin but this didn’t seem a very good advert for the ease of using eggs. If Joe user had a Numeric installation without the header files he’d be stuck. In the end I decided to mirror the Numeric tarball on the cdat_lite site and include it as a dedendency. This means Numeric will be downloaded and built automatically be easy_install. You might end up installing Numeric twice (one egg, one not) but you’ll always have the headers.

Sorting this out was a big learning experience for me in how to use setuptools or, more precisely, how to extend distutils in a setuptools compatible way. The location of the Numeric headers can be determined by:

>>> from Numeric_headers import get_numeric_include
>>> get_numeric_include()
'/usr/lib/python2.4/site-packages/Numeric-24.2-py2.4-linux-i686.egg/Numeric_headers'

If Numeric isn’t installed you can tell setuptools to install it with the setup_requires keyword. The problem is that in this case Numeric will be installed from within setup. Therefore, you can’t import Numeric_headers in setup.py because it might not be installed yet.The solution is to subclass setuptools.command.build_ext, adding get_numeric_include() to self.include_dirs. This way you can still add your own include_dirs on the command line or in a configuration file. It’s a pity it’s so difficult to work out how to do this without reading the distutils source.

Dependencies 2: NetCDF

Again netcdf is usually installed but not always. An added complication is that on x86_64 the library must be compiled with the -fPIC options for it to be usable within a DSO. I discovered that on some of our x86_64 machines we had a netcdf installation that seemed fine (libnetcdf.a, ncdump and ncgen present) but it was useless for building python extension modules.

Clearly there was a case for cdat_lite building it’s own libnetcdf.a. The current version includes the netcdf tarball although it could download it from unidata in the future. When compiled, the egg includes it’s own copy of the netcdf libraries and headers. This will allow future eggified CDAT components (vcs_lite is in the works) to compile and link against a consistent netcdf installation.

x86_64 compatability

There is a cautionary tale about the virtue of unit tests here. I was quite pleased with how easy it was to compile a x86_64 egg. The modules imported fine and I tested reading a PP file. I publicised cdat_lite at the BADC and moved on. When I returned to polish the code I built a hand full of trivial unit tests, including reading a NetCDF file. I then discovered that it couldn’t read NetCDF on x86_64!

It turns out that CDAT 4.1 isn’t compatible with x86_64. There is an important patch in the CDAT SVN which was obviously going to fix this so I merged in the SVN trunk. cdat_lite is now based on a particular revision of the CDAT SVN. This isn’t ideal but is better than not having NetCDF.

Future

I hope vcs_lite will be ready for registering at the cheeseshop soon. It will provide the vcs canvas and hardcopy output without any Tk widgets or VCDAT. This is all we need to building web applications based on CDAT.

It would be fun to try and write a pydap plugin on top of of cdat_lite. It looks easy enough (famous last words …)

Those visitors from the PyCon website might like to pick up a recent presentation on Eggs I gave to my department at the Rutherford Appleton Laboratory: see this wiki page. My audience was a broad mixture of scientific software developers (some in Python, some not) and members of the BADC who need to know what direction NDG is going. For PyCon I’ll have to refactor this considerably. I had originally designed the two talks to be part of a series of 3 but it now looks like there isn’t an egg-specific advanced talk (although #46 seems to include some eggs material).

Also see the tarball of the demo I gave at the end. With ez_setup.py and a few lines of shell script I was able to create an OpeNDAP server with WMS and KML output (mainly thanks to pydap). Ending a presentation with Google Earth is always a good bet.

Browsing through the accepted PyCon talks I can see it’s going to be well worth the trip. For a start Roberto De Almeida will be giving a talk on pydap and Dr. Travis E Oliphant will be talking about NumPy so there’s instant overlap with Geophysical applications. I’m also looking forward to hearing about IronPython, WSGI and Zope 3. Finally who could miss Guido’s key note?

Ah well! I guess there’s no escaping becoming a fully fledged Python geek groopie now.

We are dealing with a mixture of symbolic information (chemical structure and reactions), numeric or algebraic information (rate coeeficients, temperature dependent rate expressions, branching ratios) and textual information (references, explanation of experimental details, etc.). For the chemistry specific symbolic stuff there is, in my view, only one game in town: CML.

I’ve been keeping half an eye on CML for some time but I am a long way from knowing everything about it, so the following opinions come with the usual caveat of my fallibility. CML was born well before the world and his aunt jumped on the XML bandwagon. In keeping with the original vision of XML it is of the “XML as document” rather than “XML as data type” school, aiming to enable more semantically rich electronic publishing. In fact CML offers much more than just a representation of species and reactions. Through it’s various modular extensions STTML, CMLReact, CMLComp and CMLCM — to mention just the most mature ones — CML can capture a lot of the structure of a typical chemistry publication.

This is great news for us, particularly IUPAC where they currently publish data as a set of PDF files. If desired I expect it would be relatively straightforward to translate all the IUPAC datasheets, in their entirety, into a mixture of CMLCore, CMLReact and STTML.

CML can achieve this by being unashamedly liberal about semantics. The schema (CMLCore, STMML, CMLReact) are sprinkled with elements documented as “deliberately very general” (cml:substance), “content model is deliberately lax” (cml:identifier) and “no controled semantics” (cml:observation). This might be worrying if one had to write software to handle all possible constructions but that isn’t our requirement and there is a clear mechanism for being more restrictive with many elements having a convention attribute (again no controlled vocabulary ).

It’s worth noting here that there is a difference between the communication format and the underlying model. The MCM and IUPAC needn’t agree on how they store and serve the information provided the format and communication medium is agreed (What software engineers would call the interface). CML would be a good fit for communicating information from IUPAC to the MCM. It could also be used as the underlying data model of the IUPAC database and it would have the advantage of being similar to what they have now. However, I believe CML alone would be quite a restrictive model for the MCM, for here we have an example of where document-centric information falls down.

One could describe the overall structure of the MCM as a forrest of trees starting from a relatively small set of root species (those species thought to be representative of primary VOC emmisions), branching rapidly on each reaction but also overlapping as intermediate species are formed from different pathways. Finally everything ends up as CO₂ + water.

There is, therefore, only two alternatives if you want to systematically divide the MCM into discreet documents (well, 3 if you include 1 document per reaction). Either put the whole 4500 species and 12600 reactions in one document or have 4500 documents, one for each reactant. Although the former is technically feasible, in practice people want to browse and extract subsets of the mechanism. A relational model is well suited to this as it doesn’t force document boundaries on the data. This is what is done at the moment with the MCM website’s MySQL backend.

However, at the BADC there is a consensus that dataset specific databases are bad news for data curation. Unlike a document or file you can’t pass a database around (without SQL compatability headaches), they pose subtle citation challenges and each instance has a different structure, requiring it’s own expert to maintain it.

Therefore, for deposition at the BAD at least, we will need a document-centric format for the entire MCM and the challenge is to try and keep all the nice features you get with a database. Maybe this could be achieved with one big CML document and XQuery? I know next to nothing about the technology. I would favour taking the 4500 CML documents and annotating them with RDF.

I have tried this before and have found the information on pythonpaste.org rather mystifying. I get the idea of a wsgi toolbox but there must be more to justfy the hype. So this time I looked a little further a field to this post (google hit no. 3, to be precise). This looked more interesting. The paster script appeared to be similar to some of the things Eclipse can do in the Java world: create skeleton projects and deploy web applications.

I new that Paste was heavily into eggs so I decided to start hacking

$ easy_install paste
$ paster create --list-templates
bash paster: command not found

Not a good start, and quite the reverse of what I would expect from an egg-enabled package. Interogating the Paste egg reveals no sign of paster. I must have the wrong package. I dig deeper into pythonpaste.org; there are components called python.deploy and python.script. It could be either of these since paster is a script that deploys stuff. I install both and we are back on track.Now I’ve been forced to look closer at pythonpaste.org I begin to realise the front page is a red herring. The real exciting stuff goes on in Paste Deploy/Script and these are the packages that people usually mean when they evangelise about Paste.

Paste Deploy is a tool for configuring stacks of WSGI components. With it you can plug WSGI applications, middleware and servers together using configuration files and Egg entry points. The Python Deploy docs have an excellent example of how this can work to serve multiple apps from the same server and include HTTP authentication into the mix. This is easily the most impressive use of entry points I have seen.

Paste Script (a.k.a. paster) works closely with Deploy. It allows you to startup a WSGI servers on the command line, including Deploy configurations. Another neat feature is that Paste Script Templates can layout a skeleton project for you, a la Eclipse Projects. Templates are also activated with entry points, therefore framework developers can provide a paste.paster_create_template entry point in their eggs and paster create will create the boilerplate for you.

What I like especially about all this is that it’s just glue and isn’t dependent on any one server or framework. This is, after all, the point of WSGI. You can deploy your app on Paste’s wsgi server or use flup’s FastCGI implementation. Frameworks such as TurboGears and Django are becoming compatible with paste create (by the looks of things there are some issues relating to thread control still to be resolved).

Also the idea of deploying web apps as eggs is very exciting. It invokes a vision of deploying all BADC web services through a common configuration framework with authentication, etc. handled transparently. Thus the app developer needn’t write a single line of authentication code. Paste definitely seems to bring python closer to that vision.

Stop Press: And I now discover the SVN version of Paste has a wsgi filter for OpenId!

I should have had an inkling of the reason from an experience I had a year or so ago whilst developing an application for the Climateprediction.net project. I chose to use mod_python but their site used the mod_php. It transpired that when using the distro.’s packages these two modules didn’t sit well together. They both tried to link to different versions of libxml and uggly core dumps followed. I fixed it by recompiling mod_php but it was hardly a solution one would want to repeat.

Bryan Lawrence has brought together some insightful wisdom on the matter here and here. I’m embarassed to admit it’s taken me so long to read them. Also the Wikipedia entry for FastCGI explain’s it’s resugrence. In a world of a million and one web frameworks it isn’t feasible to re-implement a webserver for each one, let alone squeeze them all into one server process with mod_*.

It all helps to reassure me that my instincts were right when I began to go cold on Zope. It is a great platform to work with once you know how but it’s great weakness is that it reinvents everything at once: object publisher, templates, authentication, permissions, the lot. It makes for an enormous learning curve and then people build more complexity on top with Plone and the like. It’s just not very agile.

Casting about for promising OS projects I was lead to the Blue Obelisk website — a gathering of chemical informaticians working on various projects two of which seem to stand out from the crowd: OpenBabel and the Chemistry Development Kit.

These two projects clearly have a thriving developer community and a lot going for them. In my experimenting I had cause to contact developers from both projects and was delighted to get quick constructive feedback (more on this later).

OpenBabel is written in C++ with SWIG based bindings to Python, Perl and Ruby. As it’s name suggests it concentrates on translating between chemical file formats, however the API provides a good general cheminformatics toolbox. CDK by contrast aims to provide a toolkit to underpin interactive tools such as JChemPaint and Jmol and is written in Java. The range of features covered by CDK are much broader than OpenBabel but quite a lot of them appear to be work in progress.

The common field between the MCM and IUPAC is atmospheric chemical kinetics and therefore central to our requirements is the representation of radical species. Perhaps not surprisingly in a field dominated by pharmaceuticals and biochemistry, cheminformatics tools have tended to do a poor job of representing radicals. As a starting point I wanted to see how OpenBabel and CDK handled radical input, particularly in SMILES and MDL Molfile formats since the MCM already uses these formats.

SMILES is a particular problem here because it has never officially supported radicals. The MCM has used the Accord Excel & Access plugins which use an extension where “[C.]” signifies a carbon radical centre. So how did the two toolkits match up?

OpenBabel

Browsing the OpenBabel wiki turned up this page showing that the OpenBabel developers are very aware of the problem. Although OpenBabel supports “[C.]” etc. as input, it always generates SMILES radicals using explicit hydrogens. E.g.

$ echo "CC[C.]" | babel -ismi -osmi
CC[CH2]
$ echo "CC[CH]C" | babel -ismi -osmi
CC[CH]C

This form is unambiguous and was also recognised by a couple of non-opensource toolkits with accademic licences (Marvin and CACTVS). OpenBabel handled the “RAD” property of MDL molfiles successfuly so I thought it had everything we needed. There were a couple of hitches yet to uncover but before I go into them let’s turn to CDK.

CDK

I’m not a Java developer but with the languages’ dominance in many areas I’m always happy to find an oportunity to learn the language properly. For my test I used JPype as a familiar python shell to play with the CDK API. Even though I’d already taken a liking to OpenBabel I wanted to give CDK a good test because there was one feature CDK has that OpenBabel doesn’t: molecule depiction, that is generating 2D coordinates and a resulting image for chemical structures.

Unfortunately CDK’s radical support is less complete. It rejected the “[C.]” SMILES extension and added hydrogens rather than radicals when fed hydrogen deficient SMILES atoms. It did recognise radicals in MDL molfiles the CML spinMultiplicity attribute, producing a reasonably rendered image of a cyclic radical.

Since SMILES radical support is so important to us I submitted a bug report on sourceforge and got a quick response pointing out that SMILES doesn’t support radicals at all. This exposes’ SMILES’ great weakness.  SMILES is a proprietary format of the Daylight Corporation and to my knowledge has never had a published standard, therefore different implementers have extended SMILES to support radicals in different ways. You can’t blame CDK for concentrating on compliance with the daylight toolkit but I hope opensource tools can converge on this issue. To my mind OpenBabel’s approach of using hydrogen-deficiency is the way forward.

It looks like we would be able to use CDK to depict species by passing molfiles or CML to it. In this way we could use OpenBabel and CDK together to do what we want.

Returning to OpenBabel

I was having a lot of success using the Python bindings to OpenBabel until I tried a peroxy radical. It’s easiest to illustrate the problem on the command-line:

$ echo "CCO[O]" | babel -ismi -osmi
CCOO

Here OpenBabel doesn’t appear to convert the hydrogen deficient oxygen atom into a radical. This problem is exposed further by feeding babel output back to babel:

$ echo "CCO[O.]" | babel -ismi -osmi | babel -ismi -osmi
CCOO

Disaster! babel forgets the radical centre on successive conversions. However, there is a happy ending. I have learned never to criticise an opensource project until you have tested the bleeding edge version, so I checked out the development release from the sourceforge SVN. With relief I discovered the bug had been fixed:

$ echo "CCO[O.]" | babel -ismi -osmi | babel -ismi -osmi
CCO[O]

The sting in the tail

Openbabel was to deal me one last surprise. The Python bindings on the SVN version failed to compile. I sent a message to openbabel-scripting and got a response within minutes (some of the developers must be in Europe). This is a very good sign for anyone thinking of starting critical development work with an opensource codebase. A small patch and everything was fine. By the time you read this it may well be in SVN.

One part of the jigsaw might be openbabel, a cheminformatics toolkit with a Python wrapper, but I have always drawn a blank in finding expert system tools that integrate with Python — until now.

PyCLIPS is an interface between python and the venerable CLIPS expert system shell. For a relatively young opensource project it looks very impressive. The documentation is very detailed and it’s distributed as an Egg therefore I was able to test it out very quickly.

I can see an oportunity to rekindle my interest in expert systems. Maybe one day there will even be a Mechen2?

As a test I decided to see how quick and easy it would be to install these components using easy_install:

1. numpy (note: NOT Numeric)
2. matplotlib. For visualisation
3. basemap. A toolkit package for matplotlib which adds map projection support.
4. pycdf. A libnetcdf.a wrapper.

None of these projects distribute eggs and they all require C extension compilation so I wasn’t expecting it to work out of the box. Although I was right about that, easy_install got surprisingly far.

My test system was a linux-i686 Suse10.0 machine. Using the standard Python I checked distutils was installed, downloaded ez_setup.py and created a local setuptools installation directory on my PYTHONPATH.

$ easy_install numpy just worked as did $ easy_install matplotlib. The link from the Python cheeseshop to pycdf was broken so I had to download the tarball by hand. Once this was done $ easy_install -f . pycdf also worked thanks to libnetcdf.a being in a standard place. This was truely better than I had expected.

Unfortunately basemap wasn’t so helpful. Compilation went fine but I couldn’t import the module. Since this package contains data files I wasn’t surprised that it needed installing with $ easy_install --always-unzip and the BASEMAP_DATA_PATH environment variable needed setting but there was something else wrong.

Here my emersion in the setuptools system came to my rescue. The basemap package is designed to sit inside matplolib’s heirarchy. When two components are sitting in different eggs want to appear as a merged package tree they need to use a feature of setuptools called namespace_packages.

I dived into the basemap code and discovered it had been written with optional support for setuptools. However, getting namespace_packages working requires a very precise code layout and basemap had got it wrong. Once the problem was identified fixing it was straightforward:

$ pushd /matplotlib
$ cp toolkits/__init__.py .
$ popd

Now everything worked and I was able to create one of the standard basemap examples.

It’s a pity basemap falls down on this small point and I hope it is fixed in the future. I know installing matplotlib can be a bit daunting and using easy_install definitely improved matters. It makes the prospect of deploying this software as part of a application server much more appealing.