User Generated Forms with WTForms

18 Apr 2017

As part of my past work with the Office for National Statistics (ONS), I worked with the survey runner team to add additional features to their Electronic Questionnaire which enabled the business to take regular surveys electronically via the web. During my final months there, I took it upon myself to tackle what was seen to be one of the projects major pieces of technical debt, its use of a custom form renderer. This was identified early on as a feature that shared a lot in common with existing form libraries, but our current renderer wasn't able to be easily isolated from the rest of the system and was therefore pushed onto a debt backlog after a possible approach had been prototyped.

Due to the hard work of the entire team, we'd made some strategic changes to the system that meant this was possible in my later months. WTForms was the library we wanted to use, but we needed a number of features which weren't documented or supported out of the box. Chief among these was the ability to create forms dynamically, informed by a JSON description.

Dynamic form definition

WTForms generally expects metaclasses that describe form definitions, as per the following example from their docs:

class MyForm(Form):
  first_name = StringField(u'First Name', validators=[validators.input_required()])
  last_name  = StringField(u'Last Name', validators=[validators.optional()])

I found it easier to think as these definitions as 'schemas' for form classes. Typically you'd use them to describe a class which doesn't change, so WTForms use of them for describing our dynamic forms led to some problems.

When the instantiated form is passed to jinja and rendered as part of a template helper call, appropriate inputs will be rendered for each field definition, with names corresponding to the attribute names (first_name, last_name). We can then take the post request form and pass as an argument to our form class to validate during a submission.

def register(request):
  form = MyForm(request.POST)
  if request.method == 'POST' and form.validate():
      user = MyForm()
      user.first_name =
      user.last_name =
  return render_response('register.html', form=form)

The approach taken for the ONS was to read in a json description from file and set field attributes dynamically within a method upon what was essentially an empty form class. I've simplified the actual project code below to remove detail outside the example.

def get_answer_fields(question, data):
    answer_fields = {}
    for answer in question['answers']:
        name = answer.get('label') or question.get('title')
        answer_fields[answer['id']] = get_field(answer, name)
    return answer_fields

def generate_form(json_for_page, data):

    class QuestionnaireForm(Form):
        answer_fields = {}

        for question in SchemaHelper.get_questions(json_for_page):

            answer_fields.update(get_answer_fields(question, data))

            for answer_id, field in answer_fields.items():
                setattr(QuestionnaireForm, answer_id, field)

        if data:
            form = QuestionnaireForm(MultiDict(data))
            form = QuestionnaireForm()

        return form

Here you see that we retrieve details of the json structure for a survey page, and maintain a map of answer fields to be set on the class. The answer ids within the page structure will always be unique and is used as the key for the mapping to each field type. In this way, the resultant form is a collection of fields able to be driven dynamically from our json description.

Validating Dynamic Forms

As well as the fields on our forms being customisable, the validators used upon them were too. This meant for instance, that some fields were optional or required and the messages used for validation needed to be informed by the survey's json description.

Below you can see an example of the message for a DateRequired validator being updated based on the message from the schema. You can also see an example of modifying the behaviour of the validator, based on whether or not the date validator includes a day attribute as part of the data sent. This allows for the same validator to be used for different form representations of dates.

class DateRequired(object):
     def __init__(self, message=None):
         if not message:
             message = error_messages['MANDATORY']
         self.message = message

     def __call__(self, form, field):
         if hasattr(form, 'day'):
             if not and not and not
                 raise validators.StopValidation(self.message)
             if not and not
                 raise validators.StopValidation(self.message)

In more complex situations, it was necessary to pass custom data to our validators to later check against. You can see below, when validating custom date ranges, we pass the 'to' date and initialise the validator with it, before the call to the validator is made with the form (in this case a subform for just a date).

class DateRangeCheck(object):
    def __init__(self, to_field_data=None, messages=None):
        self.to_field_data = to_field_data
        if not messages:
            messages = error_messages
        self.messages = messages

    def __call__(self, form, from_field):

        if and form.month and form.year and self.to_field_data:
            to_date_str = "{:02d}/{:02d}/{}".format(int(self.to_field_data['day'] or 0), int(self.to_field_data['month'] or 0),
                                                    self.to_field_data['year'] or '')
            from_date_str = "{:02d}/{:02d}/{}".format(int( or 0), int( or 0),
                                             or '')

            from_date = datetime.strptime(from_date_str, "%d/%m/%Y")
            to_date = datetime.strptime(to_date_str, "%d/%m/%Y")

            date_diff = to_date - from_date

            if date_diff.total_seconds() == 0:
                raise validators.ValidationError(self.messages['INVALID_DATE_RANGE_TO_FROM_SAME'])
            elif date_diff.total_seconds() < 0:
                raise validators.ValidationError(self.messages['INVALID_DATE_RANGE_TO_BEFORE_FROM'])

We actually found it necessary to go one step further in that we needed to halt validation for forms in surveys which were considered optional. The solution was an OptionalForm, allowing empty forms to be considered valid if they had no content.

class OptionalForm(object):
     Allows completely empty form and stops the validation chain from continuing.
     Will not stop the validation chain if any one of the fields is populated.
     field_flags = ('optional',)

     def __call__(self, form, field):
         empty_form = True

         for formfield in form:
             has_raw_data = hasattr(formfield, 'raw_data')

             is_empty = has_raw_data and len(formfield.raw_data) == 0
             is_blank = has_raw_data and len(formfield.raw_data) >= 1 \
                 and isinstance(formfield.raw_data[0], string_types) and not formfield.raw_data[0]

             # By default we'll receive empty arrays for values not posted, so need to allow empty lists
             empty_field = True if is_empty else is_blank

             empty_form &= empty_field

         if empty_form:
             raise validators.StopValidation()

Final Thoughts

One of the main pain points with WTForms adoption was the lack of form-level validation supported out of the box. We had for instance a number of custom forms, which fell outside the generic examples above, where it was necessary to override the validate method on the form itself. This added complexity to the form that I felt belonged in a validator, similar to the field level ones WTForms supports defined in separate classes. Adding to option of form validators would be great and cover most of the bases and seems like an oft-requested feature.

I felt that the use of WTForms really helped create a far more structured approach to form design, creation and validation within our project. Getting to the point where we could isolate and shift it over to a common library was a real achievement.

Tagged python, wtforms, | Leave a comment

There's No Such Thing as a Side Project

20 Jan 2016

As a freelancer who has worked with clients using non disclosure agreements in the past, it's often difficult to showcase or build a portfolio of my work. I'm not alone on this, I know. Sarah Parmenter for one has posted about this problem as a designer.

As a developer, I sometimes feel it's impossible to easily show a concise representation of my work in a public arena anyway. How can I convey my role in what will usually be a complex process involving many other parties? How can I show what's involved in selecting and iterating on a simple idea to reach a final build? How can I wrap up all the knowledge I've learnt about a domain if the reader knows nothing? How can I do all this when the final product isn't available to interact with anywhere? The short answer is: I can't.

Somehow, despite all these problems of showing off my past work, I'm still able to find clients. In fact, often I'm lucky enough to have clients find me. I put this down in part to twitter, where people can find and follow me and learn more about my skills and character. The six degrees of separation really helps me out…

Using By-products to Sell Yourself

But there's another weapon any designer or developer has in their arsenal that can be used to demonstrate why we should be hired and that's our own projects. I'm pretty sure that without showcasing any projects of my own, I'd never have got my first job in the industry as I had no client work to put in my portfolio.

You may wonder why I don't call such endeavours 'side-projects'? After all that seems to be the convention. Well, in the case of my portfolio, they serve as much of an example of skills as any of my paid work. It seems a disservice to belittle them by classing them in such a way. Everything I've done is a project in it's own right.

Jason Fried of 37signals has written in the past about selling your by-products. Well, any project I create could be considered a by-product of working within our industry. I could choose to sell any one of those, but equally I can use them to sell myself.

There are some caveats that go along with using our own projects in this type of way. Firstly, we need to make it abundantly clear when listed that these are our own. We don't want to confuse potential clients about the nature of any project. On my own site, I list the two underneath one another, titled 'work' and 'projects' respectively.

Secondly, we need to treat them with the same level of professionalism as we would do for any client based work. This can be difficult given we've not faced any restrictions on how to approach a task. As developers we're constantly told to 'ship early, ship often', but we I'd balance that with 'don't ship too early'. There's no point in showcasing any projects that you yourself aren't proud of. I find a top tip here is to track my projects as I would do for any client. Knowing I'm officially on the clock whilst pursuing my own endeavours makes me think differently about them.

In thinking about my entire body of work as a whole, I've gained new perspective the value of everything I create. In fact, I'd argue the projects I spend my own time on better illustrates where my abilities and interests lie. In turn, this will hopefully attract clients who share the same interests. There's also something to be said for keeping our brains active and solving problems, even when they're our own. I much prefer this than incessantly looking to plug gaps in in my calendar.

I can already hear a bunch of you clambering from the rooftops, reeling at the thought of doing work for nothing and that actually, you'd prefer to be paid for what you're good at. In an odd sort of way, isn't this spec work? Really, working hard on your own projects should be treated as an investment - the payoff being future work you will be offered as a result of demonstrating you're capable. If your portfolio is bare, then working on something to showcase is of use to get you to the next level. If not, then working on your own projects allows you to, as Jessica Hische so eloquently puts it: 'exercise parts of my brain that go unused during client time'. I personally find that having the freedom to be able to bounce my ideas around and get feedback from friends one thing that motivates me to push onward with them.

Projects as Businesses

There's also the opportunity for your projects to become your sole business. Dribbble, Gumroad and TeuxDeux are all great examples of successful self initiated projects. Simon Willison and Natalie Downe sold their fantastic startup Lanyrd to Eventbrite. The story of how they created this whilst travelling the world is really quite a read. Lanyrd started out as a small project and eventually ended up receiving funding from Y Combinator. I'm not saying you should create projects with a view to them being acquired, but if they're good ideas, hey who knows what may follow?

If your own projects enjoy moderate success, there's also scope for making a little extra income over your client work. Given they've been approached in a professional manner, it could be possible to wrap them up as commercial products. If you work with open source, you might consider selling a commercial licence for your handiwork. That may not mean you can retire on the profits or move away from client work altogether, but with enough of these projects, you may be in a position to adopt a more comfortable working situation. Certain members of our industry have a knack for pulling this off well. Drew Wilson and Sam Soffes are two sickeningly prolific projectors to take note of, who both have built up a stack of projects which together earn them enough not to worry about client work.

Value It All

So let's not worry about the value of building side-projects, or as I prefer, 'projects'. Don't feel like that your work is 'just' a side-project, or you'll end up treating it as such. There's going to be value in any project you embark on, either directly or indirectly, short or long term, through skills or financially. Just get out there, make great stuff and show it to people.

Tagged projects, business, | Leave a comment

Blogging with Metalsmith

30 Jun 2015

I'm switching blogging platforms once more. Moving from Wordpress to Jekyll enabled me to no longer deal with the multitude of updates and protected me hacks I could be susceptible to, but Jekyll being written in Ruby (a language I'm not overly familiar with) has been a problem. I've been able to cobble together scripts to deal with the custom parts of my blog build, but it's never felt like home.

I've decided to start using Metalsmith - a static site generator written in Javascript. It's existed before my switch to Jekyll and now seems to have enough of a plugin ecosystem in order to provide a collection of plugins to provide behaviour I need to make use of.


Metalsmith operates much like a task runner like gulp, where a set of functions manipulate a number of files, one after another in order to eventually build a static site to your liking. A simple build script could be as follows:

var Metalsmith = require('metalsmith'),
    markdown   = require('metalsmith-markdown'),
    templates  = require('metalsmith-templates');


This is just standard node script with dependencies installed via npm. Metalsmith assumes defaults of src and build for the source of markdown and destination of html. An example of markdown containing some useful front-matter might be as follows:

title: Blogging with Metalsmith
author: Ian
date: 2015-06-30
year: 2015
month: 2015/06
  - Javascript
  - metalsmith
  - javascript
I'm switching blogging platforms once more...

It also assumes a default template directory under templates, containing a default post template 'post.html' - which in the example is written in a swig template as below.

    <div class="post">
    <h1><a href="/{{ path }}">{{ title }}</a></h1>
    <div class="post_date">{{ date | date('d M Y') }}</div>
    {{ contents | safe }}

I've managed to reduce my build time (though I'm not currently compiling sass and have added a new sitemap), but am much more familiar with the toolchain in question. Hopefully that'll lead to further, more frequent updates not just in content, but also the design of my company site and this blog.

I'll be adding future blogs about my metalsmith toolchain and hurdles with that I've overcome.

Tagged metalsmith, blogging, javascript, | Leave a comment

Processing Camera RAWs with OpenImageIO and Python

15 Sep 2014

I recently discovered the library OpenImageIO, an awesome tool for reading and writing image files. What makes this of particular interest is the sheer variety of image files supported (BMP, Cineon, JPG, JPG-2000, GIF, DPX, OpenEXR, Targa, TIFF) (as well as variety of camera raw formats) and the fact it can perform image transformations upon them very easily. Given it's designed for use in media and VFX environments it sounds like it will be useful for the type of work I've previously been involved with in stop motion.

Additionally, it comes with Python bindings, so you can do all of this without having to learn C++ if, like me, it's been 10 years since you last used it. It also means not having to resort to use of the commandline tools I've often used in the past for similar transformations.

Mac Installation

I'm on a mac and as such, brew is my weapon of choice for installing libraries. Unfortunately, openimageio is only available through a tap and an old version at that. I managed to compile outside of brew, but found that I couldn't get python to read the bindings correctly. I wanted to install a 1.5 version of the library, so found the best solution was to download the formula, modify it and tap homebrew/science to get the libraries it depends on. Your local openimageio formula takes precedence over the tapped version, so it won't get overwritten.

wget -O /usr/local/Library/Formula/openimageio.rb
brew tap homebrew/science

You can edit the formula here if required, using more recent version.

brew install openimageio

If all goes to plan, brew should install openimageio along with all dependencies. Additionally, I've edited my $DYLD_LIBRARY_PATH to include the path to the installation (/usr/local/Cellar/openimageio/1.5.3dev/lib).

Reading and Processing RAWs

OpenImageIO uses the ImageBuf class to handle representation and manipulation of images in memory and functions of the ImageBufAlgo class to transform them. It's best to demonstrate through some examples.

import OpenImageIO as oiio

# Read a camera raw, crop and write out to a tiff
buf = oiio.ImageBuf("Dino_001_01_X1_0066.cr2")
cropped = oiio.ImageBuf()
oiio.ImageBufAlgo.crop(cropped, buf, oiio.ROI(1208, 4901, 814, 2385))

# Create a new larger buffer and paste the crop into it, vertically centred
extended = oiio.ImageBuf(oiio.ImageSpec (3693, 2077, 3, oiio.FLOAT))
oiio.ImageBufAlgo.paste(extended, 0, 253, 0, 0, cropped)

# Create a new buffer, resize the extended image to 1920x1080 and add some text
resized = oiio.ImageBuf(oiio.ImageSpec (1920, 1080, 3, oiio.FLOAT))
oiio.ImageBufAlgo.resize(resized, extended)
oiio.ImageBufAlgo.render_text(resized, 1300, 1030, "00066.cr2", 50, "Arial")
oiio.ImageBufAlgo.render_text(resized, 1600, 1030, "00:00:02:18", 50, "Arial")

THat ultimately takes the following lovely shot of a dino from RAW form to that with extended borders and burnt in information.

What's great about this is that we've used a single library to perform this series of transformations and we've not had to break out to use commandline operations to do so, instead manipulating them from the comfort of python.

Chaining commands

From using various javascript libraries (jQuery, Underscore), I've become used to the simplicity of being able to quickly chain a whole bunch of commands together. For that reason, I wrapped a number of the OpenImageIO ImageBufAlgo methods up into a single class OiioChain so that it's possible to perform the same set of above transformations using a somewhat more concise syntax. You can find it on github.

from oiio_chain import OiioChain

chain = OiioChain("Dino_001_01_X1_0066.cr2")

chain.crop(1208, 4901, 814, 2385)\
    .extend(3693, 2077).resize(1920, 1080)\
    .text(1300, 1030, "00066.cr2").text(1600, 1030, "00:00:02:18")\

Displaying Images with Image Viewer

OpenImageIO even comes with it's own built in image viewer, to handle quick display of images or to incorporate viewing capabilities into your own software. You can access this from the terminal by using the executable "iv".

I'm really enjoying experimenting with OpenImageIO and am currently experimenting putting it into a service which performs on demand storage and transformation of media which I'll write about further at a later date. Currently, there's no means of being able to process stills into videos, for which further tools would be required, but I've heard word that work is ongoing to implement libav capabilities.

There's such a huge amount that's possible with the library which I've not covered and it should definitely be at the top of your list if you're looking at handling RAW media for your own projects.

Tagged openimageio, python, | Leave a comment

Extracting Shot Thumbnails from a video using FFMPEG

31 Jul 2014

Recently, Andy Davies asked the following question on twitter:

I found this pretty interesting, especially given I'd used various open source tools for assembling storyboards in the past. I knew it was possible to extract images at desired intervals and left it as that.

After a bit of digging last night, I discovered it's actually possible to extract images based on changes in the video. We can use this to extract a rough approximation to individual shots.

I've run the following command on the Big Buck Bunny in order to extract 138 individual changes from the 10 min short. I'm using ffmpeg version 2.3 (I had tried using ffmbc 0.7, but it is unable to parse some of the options).

ffmpeg -ss 3 -i big_buck_bunny_1080p_surround.avi -vf "select=gt(scene\,0.2)" -vsync vfr -s vga -f image2 out%02d.jpg

The real gold here is the '-vf "select=gt(scene\,0.2)"', which grabs scene changes based on a difference in the frame of 20% or more. There are certain discrepancies in a couple of shots, but in the main this method works well and quickly. I specify the output format (-f) and size (-s) so that I can get small thumbs rather than the 1920x1080 the short is in.

You can see how well this performs through the selections made for the first 10 changes shown below.

In order to assemble the thumbs into an individual image I've used the "montage" command from imagemajick.

montage *.jpg -geometry 320x180+2+2 -tile 3x4 sample.jpg

This assembles the images in the current directory in a 3x4 grid and outputs them to sample.jpg. If there are more images that will fit in a 3x4 grid, multiple output images will be created to represent the strip.

Animation is a really good exemplar for this technique, probably due to the obvious changes between each shot. However I can imagine it would fare less well with say a live action feature, where scene changes may be more gradual and/or lighting is more subtle.

Tagged imagemajick, ffmpeg, | Leave a comment