Marius Gedminas wrote about a useful technique for formatting paragraphs — registering a view for “*” that does the processing for you.

While developing SchoolTool we used a bit more complicated, but slightly more secure technique — adapters implementing IPathAdapter.

To get the same effect that was described in Marius blog post you do this:

import cgi

from zope.interface import implements
from zope.traversing.interfaces import ITraversable
from zope.traversing.interfaces import IPathAdapter
from zope.component import adapts


class TalFiltersPathAdapter(object):
    """Collection of filters to be used in views for text processing."""

    adapts(None)
    implements(IPathAdapter, ITraversable)

    def __init__(self, context):
        self.context = context

    def traverse(self, name, furtherPath=()):
        handler = getattr(self, name)
        return handler(furtherPath)

    def paragraphs(self, furtherPath=()):
        if self.context is None:
            return ''
        paras = filter(None, [s.strip() for s in self.context.splitlines()])
        return "".join('<p>%s</p>\n' % cgi.escape(p)
                        for p in paras)

Register it like this:

   <zope:adapter
       for="*"
       name="filter"
       provides="zope.traversing.interfaces.IPathAdapter"
       factory=".TalFiltersPathAdapter"
       />

And use it like this:

<p tal:replace="structure object/attribute/filter:paragraphs" />

The benefits of this technique – you don’t have a zope.Public view registered on all your objects, it will not “hog” ‘paragraphs’ view name (view names unless they are registered for a specific skin are a shared resource) and you can only access this functionality from TAL templates. The downside — it is slightly more complicated and more TAL specific, so if you are not using TAL templates you can’t really use this.

Zope3

Tests are good! writing tests is good! Sometimes they break and you can’t understand what’s wrong. Pylons is using webtest.TestApp for functional test and TestResponse has a showbrowser method, a method that displays the response in your browser, but most of the time it’s just not enough.

When I started using Pylons, I missed the ability to start a server in the middle of the test so much (I had implemented this functionality for Zope3 functional tests before) that I just had to add it to my Pylons testing environment. As Pylons is a WSGI application, it was even easier than I expected:

import pylons
import webbrowser
import sys
import urllib2

from webtest import TestApp
from wsgiref.simple_server import make_server


def addPortToURL(url, port):
    """Add a port number to the url.

        >>> addPortToURL('http://localhost/foo/bar/baz.html', 3000)
        'http://localhost:3000/foo/bar/baz.html'
        >>> addPortToURL('http://foo.bar.com/index.html?param=some-value', 555)
        'http://foo.bar.com:555/index.html?param=some-value'

        >>> addPortToURL('http://localhost:666/index.html', 555)
        'http://localhost:555/index.html'

    """
    (scheme, netloc, url, query, fragment) = urllib2.urlparse.urlsplit(url)
    netloc = netloc.split(':')[0]
    netloc = "%s:%s" % (netloc, port)
    url = urllib2.urlparse.urlunsplit((scheme, netloc, url, query, fragment))
    return url


class ServingTestApp(TestApp):

    request = None

    def do_request(self, req, status, expect_errors):
        self.request = req
        return super(ServingTestApp, self).do_request(req, status, expect_errors)

    def serve(self, page_url=None):
        try:
            if page_url is None:
                page_url = getattr(self.request, 'url', 'http://localhost/')
            # XXX we rely on browser being slower than our server
            webbrowser.open(addPortToURL(page_url, 5001))
            print >> sys.stderr, 'Starting HTTP server...'
            srv = make_server('localhost', 5001, pylons.test.pylonsapp)
            srv.serve_forever()
        except KeyboardInterrupt:
            print >> sys.stderr, 'Stopped HTTP server.'

Now instead of the usual TestApp you just use ServingTestApp

class TestController(TestCase):

    def __init__(self, *args, **kwargs):
        if pylons.test.pylonsapp:
            wsgiapp = pylons.test.pylonsapp
        else:
            wsgiapp = loadapp('config:%s' % config['__file__'])
        # Replace:
        # self.app = TestApp(wsgiapp)
        # With
        self.app = ServingTestApp(wsgiapp)
        url._push_object(URLGenerator(config['routes.map'], environ))
        TestCase.__init__(self, *args, **kwargs)

And in your code you can do this:

from sample.tests import *

class TestHelloController(TestController):

    def test_index(self):
        response = self.app.get(url(controller='hello', action='index'))
        # Test response...
        self.app.serve()

This is immensely useful when you want to find out what to click next, while writing a test for a part of the application that you haven’t worked with for a while. Or browser stress-testing, because now you can just use your testing framework to generate thousands of throw away objects in a database that will be gone when a test is over, and will be there again when you run the test for the second time.

I use it a lot when I want to test the UI of deleting objects, or when I work on things that have a uni-directional workflow. Testing the step number 7 in your browser requires you completing the first 6 steps, doing that every time you want to look at the transition from step 7 to step 8 is just too much work.

Pylons, testing

The problem

We have released the Polish version of Ututi a week ago, and that taught me a couple of lessons:

1. I18n in Pylons support is lacking
2. You must have tests for your translations just as you test your code

There are some flaws in Pylons I18n that got me longing for Zope3 I18n.

• There is no ‘default’ translation. When I used Zope3 I used to be able to say _('ok-button-text', default='OK'). With Pylons I have to have an English to English translation, which means that translators cannot see the default text, which leads to mistakes like 'ok-mygtuko-tekstas' instead of 'Gerai' or 'OK'
• Using Python formatting directives leads to tracebacks if there are bugs in the translations. If someone translates 'Hi %(fullname)s!' into 'Labas %(fullname)', all the pages that try showing this message will end up as error pages, because of the missing 's'.
• Mako templates are very very very translation unfriendly. Simple translatable texts look like: ${_('Hi!')} and more complex texts in our templates end up looking like:

  ${_('A new file %(link_to_file)s was uploaded for the subject %(subject_title)s') % dict(
          filename=h.link_to(c.filename, c.file_url),
          subject_title=c.subject_title)}
  

  It looks like perl already, I am not even talking about what happens when you have something like an email, with multiple lines of text and multiple embedded links, to translate. Though Zope’ish

  <span i18n:translate="">A new file
  <a tal:attributes="href view/file_url" i18n:name="link_to_file" tal:content="view/filename" />
  was uploaded for subject
  <tal:block i18n:name="subject_title" tal:content="view/subject_title" />
  </span>
  

  is just as ugly for short pieces of text I would surely prefer it for something that is more than 2 lines of text. An Emacs macro that wraps any selected text in ${_('<text here>')} helps to reduce the strain, but I’d prefer dedicated markers for translatable text, ones that would be easier on the Shift-pressing hand than what we have now.
• Babel has problems extracting some strings, some times, from Mako templates. The workaround -
  • run tests (yay almost 100% coverage),
  • copy template cache into your ’src’,
  • extract the translations,
  • remove the copy.
  • And then remove all the fuzzy markers from plural strings that are marked as # ,python-format by babel, which as you can guess is all the plural strings. We don’t want to guess the position of the hours in a sentence like 'uploaded %(hours)s ago'
  I still haven’t managed to find the time to reduce the problematic templates to something that I can fit into a bug report. (Update: Seems like the bug that is causing some (if not all) of the problems was reported 8 months ago in the mako bug tracker)

The solution

Now that we’re done explaining what’s wrong, let’s talk about something more constructive – making sure that tracebacks do not happen, because of typos in translations. First – we need a nice translation tester. Candidates:

1. potest
2. gettext-lint
3. pofilter from translate-toolkit

potest

Last commit 78 weeks ago. Can’t parse plural forms. Verdict — unusable.

gettext-lint

Seems to be written in Python, but packaging uses autoconf !? which generates a Makefile that does nothing. Seems cumbersome to use, can’t check html tags (some translators get this idea of translating <strong> into the target language), does not handle %(foo)s syntax.

pofilter

The tarball has all parts that are needed to package this tool as an egg, but it is not easy_installable. So what do we do? The same thing we do every night,

• extract,
• python setup.py sdist,
• scp dist/translate-toolkit.tar.gz pow.lt:~/www/eggs/

Now we just make a new virtualenv and easy_install translate-toolkit in it:

virtualenv translations
cd translations
bin/easy_install translate-toolkit --find-links=http://pow.lt/eggs

And test it on one of the projects in my src that has a lot of translations — SchoolTool:

bin/pofilter ../trunk/schooltool/src/schooltool/locales \
               -o ./out -t printf -t xmltags -t variables --openoffice

(I pass the --openoffice parameter, so that it would recognize Zope3 translation markers, like ${calendar_title}, as variables)

This results in a bunch of PO files in the ./out, each file containing the errors for the corresponding translation file.

# (pofilter) variables: do not translate: ${event_title}
#: /src/schooltool/app/browser/templates/recevent_delete.pt:4
#: /src/schooltool/app/browser/templates/recevent_delete.pt:14
msgid "Deleting a repeating event (${event_title})"
msgstr "Šalinamas pasikartojantis įvykis (${event title}"

Pretty cool, eh? pofilter has most of the functions I need, I just have to integrate it into my sandbox and extend it a little bit. So first I add:

[test_translations]
find-links = http://pow.lt/eggs/
recipe = zc.recipe.egg
eggs = translate-toolkit
       lxml
entry-points = pofilter=translate.filters.pofilter:main

to my buildout.cfg.

(I added an entry point to the [test_translations] section, because all the translate-toolkit scripts seem to be defined as plain scripts and not registered as console_script entry points)

Customizing pofilter is slightly difficult. I could not find any defined hooks that would allow me to customize the functionality. And “xmltags” seems to be picking up all the translated “title” attributes on links, which is annoying. So after reporting this as a bug I just find the “main” function in translate.filters.pofilter, copy it and produce – this:

from translate.filters.pofilter import cmdlineparser
from translate.filters.checks import StandardChecker

from translate.filters.checks import CheckerConfig

ututiconfig = CheckerConfig(
    canchangetags = [("a", "title", None)]
    )

class UtutiChecker(StandardChecker):

    def __init__(self, **kwargs):
        checkerconfig = kwargs.get("checkerconfig", None)
        if checkerconfig is None:
            checkerconfig = CheckerConfig()
            kwargs["checkerconfig"] = checkerconfig
        checkerconfig.update(ututiconfig)
        StandardChecker.__init__(self, **kwargs)


def main():
    parser = cmdlineparser()
    parser.add_option("", "--ututi", dest="filterclass",
        action="store_const", default=None, const=UtutiChecker,
        help="use the standard checks for Ututi translations")

    parser.run()

Then registered this new function as an entry point instead of the old one:

[test_translations]
find-links = http://pow.lt/eggs/
recipe = zc.recipe.egg
eggs = ututi
entry-points = pofilter=ututi.tests.translations:main

Now if I will pass “–ututi” to pofilter it will not raise warnings for title attributes anymore.

Icing on the cake

Tests are pretty useless if they are not run, and we want to run our tests after every modification to the code, and after every commit to our git server. As I am using make as my tool to run everything, I just added these two targets to the Makefile.

.PHONY: test_translations
test_translations: bin/pofilter
	bin/pofilter --progress=none -t xmltags -t printf --ututi src/ututi/i18n/ -o parts/test_translations/
	diff -r -u src/ututi/tests/expected_i18n_errors/ parts/test_translations/

.PHONY: update_expected_translations
update_expected_translations: bin/pofilter
	bin/pofilter --progress=none -t xmltags -t printf --ututi src/ututi/i18n/ -o parts/test_translations/
	rm -rf src/ututi/tests/expected_i18n_errors/
	mv parts/test_translations/ src/ututi/tests/expected_i18n_errors/

Even after changes pofilter is still reporting 3-4 false positives, that I will have to resolve with our translators, so instead of expecting absolutely no output, I am just asking for the output to be identical to the old one. If it is a known/accepted failure – we let it be.

And of course – made our dear Hudson run this after every commit, for when I forget to do it myself.

Fin!

i18n, Pylons, zc.buildout

Seeing the Cyclomatic complexity in VIM video got me slightly envious. As I am a long time GNU Emacs user I have decided to get back at those VIM people. Emacs can do everything that VIM can, only better after all So I present you pycomplexity — an Emacs mode that can do the same thing that vim-complexity does: At the moment the mode is “Works on my machine” certified. But it’s open source, so contributions are welcome. Big thanks go to Markus Triska the author of the linum.el and Gary Bernhardt for writing the complexity code calculator. Oh as for the more part: Coming up next – documenting how to set up pyflakes-enabled flymake: Competition spurs innovation after all

Emacs, Python

Sometimes when writing tests or copying production database into testing servers you need to get your database back to a clean state, and you just don’t want to “DROP DATABASE“. I can’t even do “DROP DATABASE” during my test run, because my testing user does not have the rights to do DROP/CREATE databases. So I am abusing^W taking advantage of the fact that by default all the tables, procedures, triggers and pretty much everything you add to the database are added to the default schema. As I am not adding any new schemas, I can just “DROP SCHEMA PUBLIC CASCADE” which gets rid of the default schema, and everything that is related to it (don’t forget to add it back with “CREATE SCHEMA PUBLIC” so you could add tables to the database). I am not sure whether this operation can have any dangerous side effects, but for a quick and 100% reliable teardown – it works perfectly. And I am not really worrying about our testing database… I tried to use the “-c” flag which you can pass to pg_restore. Running:

pg_dump production -Fc -O | pg_restore -d testing -c

works fine most of the time. The command will delete all the tables and all the triggers that were present in your production database from your testing database, and “DROP SCHEMA PUBLIC” (notice the missing CASCADE). But it will fail if you had any other tables in your testing database. As I am going to be testing my evolution scripts on the testing database a lot, it will have additional tables and triggers most of the time and I really don’t want that interfering with my tests.

postgresql