As of Python 3.2, the “ipaddress” module has been integrated into the stdlib. Personally, I find it a bit premature, as the library code does not look to be very PEP8 compliant. Still, it fills a huge gap in the stdlib.

In the last days, I needed to find a way to collapse consecutive IP networks into supernets whenever possible. Turns out, there’s a function for that: ipaddress.collapse_addresses. Unfortunately, I was unable to use it directly as-is because I don’t have a collection of networks, but rather object instances which have “network” as a member variable. And it would be impossible to extract the networks, collapse them and correlate the results back to the original instances.

So I decided to dive into the stdlib source code and get some “inspiration” to accomplish this task. To me personally, the code was fairly difficult to follow. About 60 lines comprised of two functions where one calls the other one recursively.

I thought I could do better. And preliminary tests are promising. It’s no longer recursive (it’s shift-reduceish if you will) and about 30 lines shorter. Now, the original code does some type checking which I might decide to add later on, increasing the number of lines a bit, and maybe even hit performance. I’m still confident.

A run with 30k IPv6 networks took 93 seconds with the new algorithm using up 490MB of memory. The old, stdlib code took 230 seconds to finish with a peak memory usage of 550MB. All in all, good results.

Note that in both cases, the 30k addresses had to be loaded into memory, so they will take up a considerable amount as well, but that size is the same in both runs.

I still have an idea in mind to improve the memory usage. I’ll give that a try.

Here are a few stats:

With the new algorithm:

and with the old algorithm:

I’ll add more details as I go… I’m too “into it” and keep forgetting time and to post fun stuff on-line… stay tuned.

I’ve seen my fair share of code fragments colourising console output. Especially when using logging. Sometimes the colour codes are directly embedded into the format string, which makes it really hairy to deal with different colours for different levels. Sometimes even the log message is wrapped in a colour string along the lines: LOG.info("{YELLOW}Message{NORMAL}") or something equally atrocious.

Most logging frameworks support this use-case with “Formatters”. Use them! Here’s a quick example of how to do it “the right way™”:

Clicky clicky → https://gist.github.com/exhuma/8147910

According to the Python documentation about __repr__, a call to repr() should give you a valid Python expression if possible. This is a very useful guideline. And it is also something I I like to implement in my Python projects as much as possible.

Now, for mapped database entities, you might argue that it makes sense to have a default constructor as long as it accepts the primary key columns.

By default, it is possible to create new instances by specifying column values in SQLAlchemy. For example:

It should be possible to create such “repr” values automatically for primary keys. All the required meta info is available. Digging through the SA docs, I found that it is possible the customize Base in order to add behaviour to all mapped entities!

Here’s the result:

With this in place, all representations of DB entities will finally make sense and be copy/pasteable directly into your code.

Of course, by nature of ORMs, the new instances created this way will be detached from the session and need to be merged before you can do any DB related operations on them! A simple example:

More than once I needed to create files on the staging/production box which I had no need of on the local development box (For example complex logging configuration).

This fragment contains a simple function which tries to do this in a safe manner, and also ensuring proper cleanup in case of failure.

The problem

Today I needed to keep an eye on PostgreSQL logs. Luckily, I decided upon installation to log everything using the “csvlog” format. But there’s a small catch, depending how you read that log. This catch is newline characters in database queries.

This has nothing to do with PostgreSQL directly. In fact, it does the right thing, in that it quotes all required fields. Now, a quoted field can contain a newline character. But if you read the file on a line-by-line basis (using methods like file_handle.readline, this will case problems. No matter what programming language you use, if you call readline, it will read up to the next newline character and return that. So, let’s say you have the following CSV record:

If you read this naïvely with “readline” calls, you will get the following:

Now, this is really annoying if you want to parse the file properly.

The solution

Read the file byte-by-byte, and feed a line to the CSV parser only if you hit a newline outside of quoted text. Obviously you should consider the newline style (\n, \r or \r\n) and the proper quote and escape characters when doing this.

What about Python?

It turns out, Python’s csv module suffers from this problem. The builtin CSV module reads files line-by-line. However, it is possible to override the default behavior.

For my own purpose, I wrote a simple script, reading from the postgres log until interrupted.

You are free to use this for your own purpose, modify or extend it as you like.

You can find it here: exhuma/postgresql-logmon

One of the really useful things in python are the setup scripts. When doing “serious” business, you really should look into them. Setup scripts are amazingly powerful. But they don’t necessarily need to be complex. But because of this flexibility, the documentation around them seems like a lot to read. Additionally, the state of packaging has changed quite a bit over the years. So you now have a couple of packages (distutils, setuptools, distribute) which all seem to try to solve the same problem. See the current state of packaging for more details on this.

This post attempts to summarize the important bits using a “Hello World” project and steeping through the process of creating the setup.py file:

• Creating a package distribution
• Automatic generation of executables
• Version numbers
• Dependency management
• Publishing your package (thus also making it available for automatic dependency resolution)
• Some more food for thought.

Read the rest of this entry »

Problem:

Solution:

This most likely indicates that you are missing the “libpq” headers:

should solve the problem

If you want command completion and a history in your python shell, export the PYTHONSTARTUP env var (export PYTHONSTARTUP=$HOME/.pystartup) in your bashrc and create a file ~/.pystartup with the following contents:

Somethin that I need quite often is to create custom accessors and mutators for class-attributes. For example convert this:

into this:

This particular example allows an easy tracking if a class contains changes. Without the need of calling myclass.get_some_attribute() or myclass.set_some_attribute(foo). You can simply do myclass.some_attribute = foo and the has_changes attribut will change accordingly.

If your class has many attributes, writing custom accessors and mutators can be tedious. So here’s a small Vim-mapping that get’s you started. Sure, you may still need to fine-tune some generated code, but the bulk is there.

Put this into your vimrc, or (like I do) into the ~/.vim/ftplugin/python.vim file so it get’s only loaded for python files. Then you only need to write the attribute name of the class, put your cursor on that line, be sure to be in normal mode (hit a few time <esc>) 😉 and hit F6

If you want to change the shortcut, simply change the first parameter to this mapping line.

This function uses the Haversine formula to calculate the distance which takes into account the spherical nature of the earth.
As the earth is not a perfect sphere, this function approximates this by using the average radius.