Blogging with Metalsmith

30 Jun 2015

I'm switching blogging platforms once more. Moving from Wordpress to Jekyll enabled me to no longer deal with the multitude of updates and protected me hacks I could be susceptible to, but Jekyll being written in Ruby (a language I'm not overly familiar with) has been a problem. I've been able to cobble together scripts to deal with the custom parts of my blog build, but it's never felt like home.

I've decided to start using Metalsmith - a static site generator written in Javascript. It's existed before my switch to Jekyll and now seems to have enough of a plugin ecosystem in order to provide a collection of plugins to provide behaviour I need to make use of.

Metalsmith

Metalsmith operates much like a task runner like gulp, where a set of functions manipulate a number of files, one after another in order to eventually build a static site to your liking. A simple build script could be as follows:

var Metalsmith = require('metalsmith'),
    markdown   = require('metalsmith-markdown'),
    templates  = require('metalsmith-templates');

Metalsmith(__dirname)
  .use(markdown())
  .use(templates('swig'))
  .build();

This is just standard node script with dependencies installed via npm. Metalsmith assumes defaults of src and build for the source of markdown and destination of html. An example of markdown containing some useful front-matter might be as follows:

---
title: Blogging with Metalsmith
author: Ian
date: 2015-06-30
year: 2015
month: 2015/06
categories:
  - Javascript
tags:
  - metalsmith
  - javascript
---
I'm switching blogging platforms once more...

It also assumes a default template directory under templates, containing a default post template 'post.html' - which in the example is written in a swig template as below.

<html>
  <body>
    <div class="post">
    <h1><a href="/{{ path }}">{{ title }}</a></h1>
    <div class="post_date">{{ date | date('d M Y') }}</div>
    {{ contents | safe }}
    </div>
  </body>
</div>

I've managed to reduce my build time (though I'm not currently compiling sass and have added a new sitemap), but am much more familiar with the toolchain in question. Hopefully that'll lead to further, more frequent updates not just in content, but also the design of my company site and this blog.

I'll be adding future blogs about my metalsmith toolchain and hurdles with that I've overcome.

Tagged metalsmith, blogging, javascript, | Leave a comment

Processing Camera RAWs with OpenImageIO and Python

15 Sep 2014

I recently discovered the library OpenImageIO, an awesome tool for reading and writing image files. What makes this of particular interest is the sheer variety of image files supported (BMP, Cineon, JPG, JPG-2000, GIF, DPX, OpenEXR, Targa, TIFF) (as well as variety of camera raw formats) and the fact it can perform image transformations upon them very easily. Given it's designed for use in media and VFX environments it sounds like it will be useful for the type of work I've previously been involved with in stop motion.

Additionally, it comes with Python bindings, so you can do all of this without having to learn C++ if, like me, it's been 10 years since you last used it. It also means not having to resort to use of the commandline tools I've often used in the past for similar transformations.

Mac Installation

I'm on a mac and as such, brew is my weapon of choice for installing libraries. Unfortunately, openimageio is only available through a tap and an old version at that. I managed to compile outside of brew, but found that I couldn't get python to read the bindings correctly. I wanted to install a 1.5 version of the library, so found the best solution was to download the formula, modify it and tap homebrew/science to get the libraries it depends on. Your local openimageio formula takes precedence over the tapped version, so it won't get overwritten.

wget -O /usr/local/Library/Formula/openimageio.rb https://raw.githubusercontent.com/Homebrew/homebrew-science/master/openimageio.rb
brew tap homebrew/science

You can edit the formula here if required, using more recent version.

brew install openimageio

If all goes to plan, brew should install openimageio along with all dependencies. Additionally, I've edited my $DYLD_LIBRARY_PATH to include the path to the installation (/usr/local/Cellar/openimageio/1.5.3dev/lib).

Reading and Processing RAWs

OpenImageIO uses the ImageBuf class to handle representation and manipulation of images in memory and functions of the ImageBufAlgo class to transform them. It's best to demonstrate through some examples.

import OpenImageIO as oiio

# Read a camera raw, crop and write out to a tiff
buf = oiio.ImageBuf("Dino_001_01_X1_0066.cr2")
cropped = oiio.ImageBuf()
oiio.ImageBufAlgo.crop(cropped, buf, oiio.ROI(1208, 4901, 814, 2385))
cropped.write("cropped.tiff")

# Create a new larger buffer and paste the crop into it, vertically centred
extended = oiio.ImageBuf(oiio.ImageSpec (3693, 2077, 3, oiio.FLOAT))
oiio.ImageBufAlgo.paste(extended, 0, 253, 0, 0, cropped)

# Create a new buffer, resize the extended image to 1920x1080 and add some text
resized = oiio.ImageBuf(oiio.ImageSpec (1920, 1080, 3, oiio.FLOAT))
oiio.ImageBufAlgo.resize(resized, extended)
oiio.ImageBufAlgo.render_text(resized, 1300, 1030, "00066.cr2", 50, "Arial")
oiio.ImageBufAlgo.render_text(resized, 1600, 1030, "00:00:02:18", 50, "Arial")
resized.write("final.jpg")

THat ultimately takes the following lovely shot of a dino from RAW form to that with extended borders and burnt in information.

What's great about this is that we've used a single library to perform this series of transformations and we've not had to break out to use commandline operations to do so, instead manipulating them from the comfort of python.

Chaining commands

From using various javascript libraries (jQuery, Underscore), I've become used to the simplicity of being able to quickly chain a whole bunch of commands together. For that reason, I wrapped a number of the OpenImageIO ImageBufAlgo methods up into a single class OiioChain so that it's possible to perform the same set of above transformations using a somewhat more concise syntax. You can find it on github.

from oiio_chain import OiioChain

chain = OiioChain("Dino_001_01_X1_0066.cr2")

chain.crop(1208, 4901, 814, 2385)\
    .extend(3693, 2077).resize(1920, 1080)\
    .text(1300, 1030, "00066.cr2").text(1600, 1030, "00:00:02:18")\
    .write("final.jpg")

Displaying Images with Image Viewer

OpenImageIO even comes with it's own built in image viewer, to handle quick display of images or to incorporate viewing capabilities into your own software. You can access this from the terminal by using the executable "iv".

I'm really enjoying experimenting with OpenImageIO and am currently experimenting putting it into a service which performs on demand storage and transformation of media which I'll write about further at a later date. Currently, there's no means of being able to process stills into videos, for which further tools would be required, but I've heard word that work is ongoing to implement libav capabilities.

There's such a huge amount that's possible with the library which I've not covered and it should definitely be at the top of your list if you're looking at handling RAW media for your own projects.

Tagged openimageio, python, | Leave a comment

Extracting Shot Thumbnails from a video using FFMPEG

31 Jul 2014

Recently, Andy Davies asked the following question on twitter:

I found this pretty interesting, especially given I'd used various open source tools for assembling storyboards in the past. I knew it was possible to extract images at desired intervals and left it as that.

After a bit of digging last night, I discovered it's actually possible to extract images based on changes in the video. We can use this to extract a rough approximation to individual shots.

I've run the following command on the Big Buck Bunny in order to extract 138 individual changes from the 10 min short. I'm using ffmpeg version 2.3 (I had tried using ffmbc 0.7, but it is unable to parse some of the options).

ffmpeg -ss 3 -i big_buck_bunny_1080p_surround.avi -vf "select=gt(scene\,0.2)" -vsync vfr -s vga -f image2 out%02d.jpg

The real gold here is the '-vf "select=gt(scene\,0.2)"', which grabs scene changes based on a difference in the frame of 20% or more. There are certain discrepancies in a couple of shots, but in the main this method works well and quickly. I specify the output format (-f) and size (-s) so that I can get small thumbs rather than the 1920x1080 the short is in.

You can see how well this performs through the selections made for the first 10 changes shown below.

In order to assemble the thumbs into an individual image I've used the "montage" command from imagemajick.

montage *.jpg -geometry 320x180+2+2 -tile 3x4 sample.jpg

This assembles the images in the current directory in a 3x4 grid and outputs them to sample.jpg. If there are more images that will fit in a 3x4 grid, multiple output images will be created to represent the strip.

Animation is a really good exemplar for this technique, probably due to the obvious changes between each shot. However I can imagine it would fare less well with say a live action feature, where scene changes may be more gradual and/or lighting is more subtle.

Tagged imagemajick, ffmpeg, | Leave a comment

Splitting a date range in Python

01 Jul 2014

Dates are one of those annoying things that shouldn't be, but are regularly difficult in web apps. I used the following two methods in a recent page to neatly break a date range into distinct segments as part of a analytics app I'm currently working on. Hopefully someone else will find them helpful.

import datetime, calendar

# Find the delta between two dates based on a desired number of ranges
def datedelta(startdate, enddate, no_of_ranges):
    start_epoch = calendar.timegm(startdate.timetuple())
    end_epoch = calendar.timegm(enddate.timetuple())

    date_diff = end_epoch - start_epoch

    step = date_diff / no_of_ranges

    return datetime.timedelta(seconds=step)

date_delta allows me to create the timedeltas between two dates based on a desired number of segments.

# Edit 18/07/2014 - I realised dates needed the hrs, mins an secs correctly
# adjusted to beginning / end of day
def datespan(startdate, enddate, delta=datetime.timedelta(days=1)):
    currentdate = startdate

    while currentdate + delta < enddate:
        todate = (currentdate + delta).replace(hour=23, minute=59,second=59)

        yield currentdate, todate

        currentdate += delta
        currentdate.replace(hour=0, minute=0,second=0)

I can then pass the delta into datespan above, which returns an iterable I can then use.

# Get timedeltas based on splitting the range by 10
delta = date_delta(startdate, enddate, 10)

for from_datetime, to_datetime in datespan(startdate, enddate, delta):
    print from_datetime, to_datetime

The result forms part of the following d3 chart:

Tagged coding, python, | Leave a comment

Tips for being a more Productive Freelancer

16 Jun 2014

Having heard the question raised on @workingoutshow, I thought I'd take some time to share the things that I've found have really helped me be productive in my work. These may not work for everyone, but hopefully some points will help for others.

Find a Good Environment

The single biggest change I've found which has had a major impact on my work is changing my environment. We're lucky enough to have a small garden at our house and with the arrival of my daughter back in October, I was quick to realise that working in the open plan lounge in our house just wasn't a viable option any more. Here in Cardiff, we've been lucky enough to have several co-working spaces open locally, but for me I prefer being able to switch off background noise and clear my head when I really need to knuckle down and focus. Around the same time, we made some major changes to the back of our house and garden which meant that there was space enough for a desk in our sunroom which overlooks it. I now spend 90% of my working time looking out over our little green space, which makes for a welcome break from staring at my screen and gives me a self-contained and more importantly, quiet environment in which to think. I also get to enjoy the space more overlooking the various wildlife that visits during the day. It's funny how good a little bird taking a dust bath every now and again will make you feel.

Organise Your Tasks

I use trello for organising tasks I want to get done, with a board for each project. These are organised in descending priority, using trellos card defaults of "To Do", "Done" and "Doing" to track what I've achieved for a project. If whilst during the course of one task, I realise something needs doing over and above the scope of the task I'm currently working on, I'll add it to my todo list and work on it when all other tasks with a higher priority have been completed. This strategy works well for me in terms of being able to blast through a whole heap of development tasks without having to think too much about what I need to do next.

Plan Your Next Day Before Leaving Your Desk

I like to end my day at a distinct break between tasks. If my day ends at a point where something is incomplete, I'll be sure to keep it in my to-do list for the following day, where I can come back to it. Before I leave my desk each day I'll generally try and plan out what makes sense to cover (given the time I have) in the following working day. This allows me to hit the ground running each day, without any getting up to speed time first thing in the morning.

Track Your Time

Tracking my time is important not only for my client work, but also my own projects. As well as giving me exact totals for invoicing, it allows me to compare estimates against the reality of a project and all the unforeseen problems that may arise during the course of it. My time tracking is a simple project based on/off approach, where I use a custom tool I developed myself.

Eliminate Distractions

Although obvious, trying to cut down on distractions pays dividends. Switch off tweetdeck, close skype and either disable notifications on your phone or (my favourite) put it out of your sight completely. I don't tend to pay any thought to my phone if I can't see it, and it's like a nervous twitch to pick it up and check whether anything has happened in the last 5 mins. When not aware it's an option, I probably get twice as much work done. It's easy to kid yourself that checking devices like this is part of your work, but when at the detriment of your day's schedule you need to switch off and focus on the bigger picture.

These are just a few measures I take to be more productive in my day - I'd welcome any tips other freelancers have for their own situations.

Tagged productivity, | Leave a comment