The problem

Today I needed to keep an eye on PostgreSQL logs. Luckily, I decided upon installation to log everything using the “csvlog” format. But there’s a small catch, depending how you read that log. This catch is newline characters in database queries.

This has nothing to do with PostgreSQL directly. In fact, it does the right thing, in that it quotes all required fields. Now, a quoted field can contain a newline character. But if you read the file on a line-by-line basis (using methods like file_handle.readline, this will case problems. No matter what programming language you use, if you call readline, it will read up to the next newline character and return that. So, let’s say you have the following CSV record:

If you read this naïvely with “readline” calls, you will get the following:

Now, this is really annoying if you want to parse the file properly.

The solution

Read the file byte-by-byte, and feed a line to the CSV parser only if you hit a newline outside of quoted text. Obviously you should consider the newline style (\n, \r or \r\n) and the proper quote and escape characters when doing this.

What about Python?

It turns out, Python’s csv module suffers from this problem. The builtin CSV module reads files line-by-line. However, it is possible to override the default behavior.

For my own purpose, I wrote a simple script, reading from the postgres log until interrupted.

You are free to use this for your own purpose, modify or extend it as you like.

You can find it here: exhuma/postgresql-logmon

I do a lot of work on the go. Offline. Sometimes it takes a long time to push changes to a remote repository. As always, Murphy’s law applies, and the one repo that explodes into my face is the one with ten days worth of work in it.

While working, suddenly my laptop hang. Music looping. No mouse movement. Nothing. The only possible solution was to do a cold-reboot. I was not worried. Everything was saved, and I only changed a few lines and can easily recover if something went awry. So I rebooted.

Once back in the system, I immediately wanted to do a git status and git diff. Git spat back the following error message:

My Laptop had shown some strange and erratic behaviour over the last few days already. I already left a memtest running for about 24 hours earlier this week without errors. The only possible explanation left was the hard-disk.

Fun times ahead! 10 days of work at risk… 10 days of important changes! Sweat building up my forehead. Bloody sweat!

I trust my tools to keep my code safe. I trust git. I trust vim. I do microscopic commits, and I knew my current uncommitted changes only involved a few lines. So maybe only the last commit got corrupted? Let’s see…

Read the rest of this entry »

docutils supports adding a wide range of unicode glyphs into documents, while still keeping the source document readable and plain ASCII.

The character mappings are available in include files, and have to be included into the document before being useful.

For example, adding a horizontal arrow is done by including 'isoamsa.txt' which provides (amongst others) the replacement |hArr|. An example document might be:

Hav a look at the extensions available in the official docutils distribution for more information.

When using Doctrine 2 with a MySQL Database which has tables with ENUM datatypes, you might run into the following error message:

‘Unknown database type enum requested, Doctrine\DBAL\Platforms\MySqlPlatform may not support it.’

This is because Doctrine 2 doesn’t support the ENUM DataType natively as you can read here: Doctrine Cookbook.

In their Doctrine Cookbook they give a solution how you can resolve this by mapping the ENUM to a STRING datatype. But if you’re using Zend Framework 2 you’ll probably run into the same question as I have: where do I put this stuff for it to work?

This might not be the perfect solution, but it worked for me.

Adding the required code to the onBootstrap() method of the Module.php file of the “default” module did the job.

While cleaning up my build process using ‘closure’, I stumbled across plovr. After using it for only 20 minutes, I am convinced that this should be in the toolbox of *everyone* developing with the closure library!

While the documentation is still a bit sparse, you can have it set up in no time! There’s no need to regurgitate a basic setup example in this post. Everything necessary is readily availble over at http://plovr.com/!

One of the really useful things in python are the setup scripts. When doing “serious” business, you really should look into them. Setup scripts are amazingly powerful. But they don’t necessarily need to be complex. But because of this flexibility, the documentation around them seems like a lot to read. Additionally, the state of packaging has changed quite a bit over the years. So you now have a couple of packages (distutils, setuptools, distribute) which all seem to try to solve the same problem. See the current state of packaging for more details on this.

This post attempts to summarize the important bits using a “Hello World” project and steeping through the process of creating the setup.py file:

• Creating a package distribution
• Automatic generation of executables
• Version numbers
• Dependency management
• Publishing your package (thus also making it available for automatic dependency resolution)
• Some more food for thought.

Read the rest of this entry »

Here’s a small bash function to provide TAB-completion for fabric tasks.

Simply add the following to your ~/.bashrc

You may already have a block like if [ -f /etc/bash_completion ]... in that case, simply add the extra line into that block.

Have fun

I recently switched work, and now after a lot of JavaEE development for the past 5 years, I am finally back home: Web-development. During my Java time, I did some small web bits here and there. Mainly to keep up with evolution, and followed the massive move towards more JavaScript heavy development. During my free time I took baby steps with a couple of JavaScript libraries, starting with Prototype, then script.aculo.us, MochiKit and MooTools to get some visual effects done. Without doubt, I preferred MochiKit because of it’s similarity to Python. I also dipped into Dojo because of it’s nice HTML Form widgets, fooled around with backbone.js. But eventually I ended up with jQuery, simply because it currently has the backing of The Community. All I’ve done in all of these libraries can only be summarized as “wetting my feet”. At some point I consumed all the Crockford material I could find, and boy was that eye-opening. I realized I was not aware of the most important aspects of JavaScript (especially: “Everything is global! Use Namespaces!”)!

The next thing that caught my eye, was CoffeeScript. It allowed me to write pythonesque code, and compile right down to proper JavaScript.

So what has this to do with the closure library? Well, recently (as stated above) I started web-development again. And some parts of the project made it obvious, that a good dash of JavaScript would help the project a lot. So I needed to take a decision on what library I would base my work on. My first reflex was jQuery. But out of sheer coincidence I stumbled across a blog post from someone discussing jQuery and closure. Unfortunately, I don’t have the link anymore The TL;DR of that blog post was however (paraphrasing):

If I had known about closure before, I would have used it instead of jQuery, because it makes it easier to structure the source code.

This made me more that curious. So the next thing I did, was read the closure tutorials, and also read the API top to bottom. I wanted to know what was in there before deeming it useful. And boy did it look interesting! The most interesting feature, as the author of said blog post pointed out, it the ability to easily structure your code. Using goog.provides and goog.require you can very easily split your JS files into multiple, well structured files, and the closure compiler will then re-combine them into one file, all properly ordered. To me, this feels a lot like writing Java or Python, which both allow you to structure your code very well. This also allows you to easily write modules which can in turn be re-used with ease in other projects (or, you could publish them).

For now, I am not missing anything from jQuery except the easy DOM querying from it. But I can happily live without it.

With my current experience, I can see the following benefits:

• Makes it easy to structure code
• The optional linter forces you to write proper/clean code. Especially with the --strict flag. Using it right from the start forces you to write in a uniform coding style, which makes code more readable!
• The compiler solves dependency chains with the help of goog.provide and goog.require, and can optionally minify your code.
• The library feels very professional. It’s even got a Logger loosely inspired by java.util.logger, and let’s you do exactly what you expect it to! This is awesome for debugging!
• The library source code is very readable

And some disadvantages:

• The community is much smaller than the one around jQuery. But so far I could figure out all I needed by reading the code.
• It does not seem to have versioned downloads. You have to checkout from SVN trunk. I would prefer to be able to say: “I am basing my work on version 1.0″ instead of “trunk”
• Instead of linking one CSS file (as in jQuery), you have to link multiple styles and figure out which ones by yourself. But given the filenames are self-explanatory, that’s not too difficult.
• There are no predefined CSS themes. Something jQuery’s theme builder would be nice. But, I assume that the closure-style-sheet project should help writing CSS files easily. I did not yet look into that one!

As I progress with the library I’ll post my findings…

Here are two related links I found while looking for the mentioned blog post:

• http://stackoverflow.com/questions/1690197/what-does-google-closure-library-offer-over-jquery
• http://sayspy.blogspot.com/2010/10/lessons-learned-porting-from-jquery-to.html

Soon, we will need to send out notifications as soon something bad happens with a scheduled task on Windows. The following JScript file runs natively on Windows and is capable of just that. It uses the command line tool “schtasks” to query the information and wraps the result into a list of usable object instances.

It’s possible to use this list to react to important events in the job executions. For example, you could loop through the list and send emails to the appropriate people if the variable “lastResult” is non-zero.

Simple script… still, I thought I’d share…