Django lxml encode error

Python script using lxml library that work fine on console suddenly throwing out error when importing it from django views module.

File "lxml.etree.pyx", line 123, in init lxml.etree (src/lxml/lxml.etree.c:160385) TypeError: encode() argument 1 must be string without null bytes, not unicode"

It unlikely problem with the encoding of the content I want to parse because it just importing the module and I'm not calling any function that do the parsing yet. Almost giving up in my search until I found this answer[1] on Stackoverflow. It turn out on console I'm using Python 2.6 while mod_wsgi, which run the django app is compiled against python 2.7.
[1]:http://stackoverflow.com/questions/9465248/runtimewarning-compiletime-version-2-6-of-module-lxml-etree-does-not-match-ru

PostgreSQL dump list file

The error first started with something like 'INVALID COMMAND \N ..'. This actually not the real error. Use psql command line switch --set ON_ERROR_STOP=1 to stop immediately on error so we can see the actual error.

When restoring my postgresql dump to the new hosting, got error that saying plpgsql language already exists. This probably because my hosting already setup plpgsql in all databases by default while dump consist a line to create the language extension. Googling around, I found out that pg_restore provide what they call list file that list out what kind of object should be restored. So the first step is to generate the list file out of your database dump:-

pg_restore -l db.dump > db.list

db.dump is your database dump file and when specifying -l option, the output would be list of object to restore. We save that list in db.list file. Now we can open up db.list with any text editor and comment out the line that mention the creation of plpgsql. The list file should look something like:-

3178; 1262 1525521 DATABASE - cc_live myname
6; 2615 1313721 SCHEMA - audit myname
7; 2615 1313722 SCHEMA - cct myname
3; 2615 2200 SCHEMA - public postgres
3179; 0 0 COMMENT - SCHEMA public postgres
3180; 0 0 ACL - public postgres
;1007; 2612 1313725 PROCEDURAL LANGUAGE - plpgsql myname
639; 1247 1313728 TYPE cct daily_sale myname
641; 1247 1313729 DOMAIN public bigint_unsigned myname

So we comment out (by putting semicolon at the beginning of the line) plpgsql object. Then we restoring the dump, we specify the list file to pg_restore:-

pg_restore -x -O -L db.list db.dump | psql new_db 

This is useful if the dump file very large or using the archive format which mean you can't edit it directly with editor. The list file supposed to be editable.

Test sending email through telnet

This the minimal telnet session to test sending to email server.

$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 example.com ESMTP
helo
250 example.com
mail from:me@gmail.com
250 ok
rcpt to:x9tayay@example.com
250 ok
data
354 go ahead
Subject: OTA

ota ajaj.
.
250 ok 1326364891 qp 1790
quit
221 example.com

The mail server in this case running qmail. If the email is bounced, the returned email may end up in your spam folder as the sender domain does not match, we send this from localhost instead of from gmail.com.

Using IRC as Time Tracker

I have tried all sort of time tracker, todo lists and what not but still don't really get  what I want out of that tools. The requirement is very basic and simple, a tool that allow me to track how much time I have spent on certain things throughout the day. While there are lot of great tools with a degree of simplicity, it still having too much friction for me to effectively use them.

Emacs orgmode  I think pretty much close to what I want. That just a guess since I just gleaned over it's features and what people are saying about it. But to learn emacs just to use this is a bit too much. I took note in vim in outline mode - that's how my mental mode work. It would be nice if vim can record the time whenever I add new outline. Most of the todo lists or time tracker out there only allow you to define simple title and explanation of the task. If that was web based app, don't ever think of having some outlining support.

Then I noticed that it quite easy for me to rebuild my mental state on what I have done throughout the day by looking at my svn commit log or the skype chat log with my colleagues. So I stop looking  for the tools and just used the history browser of our svn to compose my daily report. It work well except that not all tasks would have a commit log - or in other word something that has svn repository. I might be doing some server config  or work in some web interface and all these not related to any of our svn repo.

The skype chat log gave me an idea that I maybe can use it to track my time. Each post got a time stamp so it perfect. But to whom should I chat with ? Setup new fake account ? Possible but I think there must be a better way. Enter IRC. I used irssi as my irc client and since it console based app, that mean almost no context switch in order to use it. It just few SHIFT+Arrow key away from the current console. To use public irc server for this definitely no-no. Setting up the whole irc server seem a bit over kill although is much easier nowadays thanks to apt-get.

Searching around, I found hircd - python code (in 400 lines) implementing basic IRC server. Simply run the python code and fire up irssi to connect to localhost. It just work. Automate some tasks such as automatically joining a channel, autolog on and I'm ready to go. If you work on multiple project and want to split the log, we can just create new channel but for now I stay in single channel as it easy to keep track.

Todo:-

1. Put the chat log ($HOME/irclogs) into spideroak synced directory.

2. Integrate with pormodoro technique ? 

Read text file on non JSR-75 phone

So my Samsung GT-E2230M does not support JSR-75 which is Java API on mobile devices that allow access to the device's file system such as the SD card. This mean any midp application in the phone can't  simply open arbitrary text files and also mean I can't use Opera Mini save webpage feature.

Searching around, I found EbookMe application that allow me to package a text files into a complete midp application to be installed on the phone just like any java apps on the phone. It a little bit cumbersome but at least now I have something to read while in waiting and away from my laptop and Internet connection.

Using it quite simple, provided you have Java properly setup on your system.

sh ebookme.sh -Dname="Whatever Prog" -Dout=whatever whatever.txt

The unzipped tarball provide a wrapper script ebookme.sh that you can run by simply passing the text file name as parameter. The -Dname and -Dout is optional to customize the name of generated midp app. The command would produce 2 files - whatever.jad, whatever.jar. You then transfer whatever.jar to your mobile phone and install it like a normal midp application.

Another limitation of the phone is it can't open anything more than 200KB. This mean for large text such as the free book The Art of Community by Jono Bacon I had to convert the pdf into text first and then use unix split to split it into multiple files.

Recently, I'd also found html2text python lib that allow you to convert any HTML page into text file formatted in markdown. This make reading the text on the small phone screen much pleasurable.

Bluetooth modem in Ubuntu 10.04

Finally work with my Samsung GT-E2230M. Although I'll not be using this since there're abundant of much cheaper and faster 3G connections, who know I'll need it some day so it's good to know that it could. The blueman package in the standard repositories so far provided all the bluetooth functionalities that I need so far.

Connecting up to PPP with the handphone as a modem is as simple as right click the respective phone in the bluetooth manager and navigate to "Serial Ports" => "Dial Up Networking". First time trying this it managed to established the connection but subsequent tries greet me with socket error. Then I have a good luck by clicking the setup assistant (the gear icon) and go through the 2 step wizard. Once connected, I can see the ppp0 interface is up with IP address assigned but I can't ping any public IP address. Checking the route, the routing table is empty. Try adding some default gateway "route -a [ppp0 ip addres] gw default" but still can't ping outside address.

Finally found what I need in this blog post. The ppp0 must be set as a default connection.

# ip route change default via [ppp0 ip address] dev ppp0.

Voila ! I'm connected to the Net now.

Formatting floats in Python

Not really straightforward since there are number of issues. In this SO question, Alex Martelli simply suggested using the rstrip() string method to remove the zero. Others suggested the "g" flag, such as "%.1g". One issue with "g" flag is if the number to be formatted is larger than the format width specify, it would display it in scientific notation. It can be fixed by using large enough width such as "%.99g".

Django has floatformat template filter and it seem to use different approach in django.core.template.defaultfilters

def floatformat(text):
    """
    Displays a floating point number as 34.2 (with one decimal place) -- but
    only if there's a point to be displayed
    """
    try:
        f = float(text)
    except ValueError:
        return ''
    m = f - int(f)
    if m:
        return '%.1f' % f
    else:
        return '%d' % int(f)