Scaling a Linear Domain to an Ordinal Range with d3

07 Mar 2014

I’ve had the opportunity of using d3 quite a lot over the past few months for a number of clients. It offers some amazing flexibility for chart generation and much more.

Anyway, I thought I’d share a quick tip I developed for mapping a linear set of values onto an ordinal scale. For those who’re seasoned pros at d3, this probably seems trivial, but had me stumped for some time today.

I’d picked out a colour palette I wanted to use for a particular graph, as per below:

var colours = ["#B8D0DE", "#9FC2D6", "#86B4CF", "#73A2BD", "#6792AB"];

The only examples I’ve seen similar to this are where it is assumed you want to vary darkness of colours based on value or vary the domain based on the number of colours you want. Not a good fit.

I wanted to pick one of my values based on a linear value from my data set. My first thought was to make use of the ordinal scale function provided by d3. Something like this:

var colour = d3.scale.ordinal()

In doing this, I got something that *looked* a bit like it was working, but not the way I expected. In fact, the way an ordinal scale works is that it provides a 1-to-1 mapping of domain values to the range, rather than any kind of interpolation between them. In this case, it was expecting only 5 distinct data values (to match up against the colours) and for everything over and above that, it wrapped them round to the beginning of the domain again. The solution then is fairly simple once you’ve got your head around that.

What I did next, was to create a scale that gave the index of the colour we were going to be mapping to. This works well, because the indices are linear and d3 has the ability to do the dirty work in that respect.

var colourIndex = d3.scale.linear()
  .domain([0, d3.max(data)])
  .range([0, colours.length - 1]);

  .attr("fill", function(d){
    return colours[Math.ceil(colourIndex(d))];

Here we end up with an index ranging across all the indices of the colour array, and a colour appropriately selected from the palette as expected. You can see the resultant effect in the graph linked to below:

Tagged charting, d3, | Leave a comment

Shipping in March

03 Mar 2014

This month, I’ve decided to take some time out to ship projects.

I have a huge list of projects accumulating I’d like to push out of the door, but as a father and freelancer I have lack of a certain commodity in able to achieve them. That would be time.

Currently I don’t have any work booked in for a while, so along with Viv, we’ve decided I should try and get some of these projects released.

That sounds easy, but I’m well aware from past experience it isn’t. I don’t want to work outside of working hours (9-5.30pm) as I know the impact it has on my family. My hope is by setting myself some achievable goals I can at least get some way to ticking off some checkboxes on this growing todo list.

Some of these projects are products I’ve not yet completed, projects I’ve released but need work or ideas that haven’t made it to the build stage yet. As a freelancer, I hope that some of them may even help out other people in a similar situation to me.

Over the coming month, I’m going to attempt to ship at least 4 projects out of the door. 1 a week for 4 weeks. I’ll be blogging about my progress here, and posting summaries of how it went at the end of each week. The end date of the 28th is also my birthday, so it would be great present to myself to have come some way to meeting my goal.

If you’re interested in receiving progress updates via email, signup here:

Tagged projects, | Leave a comment

Setting Up an .is Domain with ISNIC

27 Oct 2013

I’ve purchased a couple of .is straight from the Icelandic Domain Registry (ISNIC) in the past year with a view to using them with my own vps. The process of setting up nameservers isn’t great, given their somewhat confusing management interface. That’s summed up in a recent tweet below.

Given the potential savings compared to a ‘normal’ registrar, it’s likely you’ll want to make use of ISNIC themselves – who are much like an Icelandic version of Nominet.

Here’s the process I went through to configure my domain to make use of the nameservers provided by my (non-icelandic) host. The information provided here is a little misleading as it’s not necessary to use a “pre-approved” registrar according to the rules outlined here. To be clear, there’s no need to use something like, and you can point ISNIC directly to your hosts nameservers.

Once you’ve gone through the process of checking your domain is available and registering it, you’ll need to ensure your hosting provider has appropriate dns entries for your domain. If you’re using, cPanel or something that basically means adding it there. If you’re managing your own dns entries, add the appropriate entries within your manager.

Here’s the key bit. Now wait…. Seriously, for about 6 hours or so do something else. It will take some time for these dns changes to propagate and ISNIC to accept them. Basically these details need to be bounced up through the hierarchy of dns servers to the root dns servers that will respond to a request for your new domain, which takes time. In the olden days, I remember waiting days for this to happen.

Following that wait, head to the “redelegate” option within your new “My Page” area at ISNIC. Choose the “Not Registered” when asked for your ISP.

Go ahead and enter the nameservers you want to use. If the dns changes haven’t yet propagated, you’ll probably get the error “The nameserver ‘’ has not been registered”, as shown below:

Misleading Error

This is a pretty misleading error, given what it actually means is “your domain doesn’t seem to be registered yet on those nameservers”. So wait a bit more for propagation to occur…

Be aware of the rules for ISNIC nameservers and check your host complies with them, otherwise you may find you encounter other errors. When the changes are picked up, you should finally be able to add the entries without getting an array of red boxes, much like below.

Successful nameserver entry

Tagged dns, hosting, iceland, nameservers, | Leave a comment

A Decade on the World Wide Web

14 Aug 2013

10 years ago today, I started this blog. That’s just under a 3rd of my life. In other words, that’s quite a long time ago.

I started out using Movable type and very quickly (3 days later) ditched it in favour of the b2 “weblog tool”, which WordPress was later spawned from.

Back then, I blogged frequently about what was going on in my life, which tended to centre around University coursework rather than technical posts. These days, a lot has changed – I have a wife, son, house, business and as a result, blog a lot less frequently – which honestly, upsets me a little.

It’s also crazy to look at the progress of WordPress over that time and how now, ten years on I’m considering moving away to a simpler, static tool.

Given it’s now my industry, it would be easy for me to ponder what might have been over my time on the web (what if this, that..) but I’m left thinking that for all the facets of my life I do make visible on my blog, it’s the moments I don’t share that are most important to me.

I wonder what the next decade will bring.

Tagged Web, | Leave a comment

Second Rate Britizens

06 Jul 2013

I decided to attempt a simple app this week, first using GoCardless, then later the beta of Stripe. It requires the payment of funds between two individuals, neither of which might be a business. GoCardless suggests signing up to use their partner programme, which involves registering a user as a merchant. This isn’t entirely suitable given the individual will be asked many queries on the nature of their business (which they don’t have). Stripe have a similar offering, known as “Connect” which asks similar questions. It’s at this point users would probably start turning away.

The recently announced “Payouts” from Stripe, seems like the most appropriate solution, but it doesn’t yet exist in the UK (and it’ll probably be a while before it does). Bigger players like Amazon’s fps require that you have a US card and haven’t ever made noises that they may be expanding into European markets.

As developers in the UK, we get a bit of a bum deal. Software products are often first targeted towards the biggest (US) market and we fine folks have to sit around and wait until the smaller pieces of the puzzle are picked up and put in place. Sucks to be us.

I’ve blogged in the past about the many payment providers cropping up and their services being unavailable in the UK. I can understand why this happens, but it’s frustrating not least because it hinders any momentum that might be able to be built at an earlier date.

I doubt I’ll ever have the same momentum as I did with my idea this week. It’s unlikely therefore, it will ever see the light of day. A US counterpart could easily offer an international version of my idea today.

Tagged rant business, | Leave a comment

The Developer Dream

27 Jun 2013

I suffer from idea overload. It seems that every day I have an idea for something. As a developer, I’m lucky in that to be able to take an idea to inception very quickly. Best of all, I can show it at inception stage to a huge audience (if I had one), garner feedback and quickly iterate on that idea. To date, I’ve mainly created these by myself – a casualty of doing paid work in the main like this too.

Most of these ideas in the have been simple hacks – solutions to things that bug me. One of them has been a proper business too. I’m able to do something and make it happen, to potentially tap a huge audience. Get them to that first step. The problem I have is, not a single one of these so much as turns a profit, though they serve as wonderful educational devices for me. I would hazard a guess that some of them have indirectly got me work.

Some of my ideas never get past the domain registration phase before being knocked on the head. A friend many years ago gave me the great quote that ‘My domains are my todo list’. This totally hits the nail on the head for me – if I’ve felt like an idea was good enough to make it to the registration phase, there’s something there worth pursuing. Unfortunately, I have quite a long list to work through.

I heard it said that “The Developer Dream” is that we (developers) “build a product/service which we’re able to earn a living from”. But the majority of developers I know, don’t even begin to pursue the ideas they have. This surprises me.

There seems to be such an abundance of opportunities that I can’t imagine not building my own ideas, however bad they might seem to others.

Why hold back? Time? Yes, we have other commitments – family, work and maybe even a hobby or two. I’m not suggesting we make drastic sacrifices or sell our kids in order to create stuff. I’d would however, like to think that the payoff for building our ideas might make our initial holdups seem insignificant longer term. I’ve said it before and I can’t speak from as the voice of successful experience, but what do I get out of never trying?

Tagged hacking, ideas, webdev, | Leave a comment

The Problem with Full Stack JS Applications

03 Apr 2013

Whilst node.js heralded an era of being able to use a single language for both server and client side development, nobody mentioned the confusion such an approach could cause. It’s inevitable that there’s going to be similarities in code that constructs data on the server and that which presents it on the client. The fact that the two are written in the same language and the structure of them is so tightly coupled can make it easy to lose track of where you are and what you’re doing when knee deep in code.

Having primarily been using a full js stack (express on the server, backbone on the client) for side projects over the last year or so, I’ve certainly found myself tied up in this kind of knot. For example, if I’m looking at event.js, which represents an event on a calendar it could be the server event representation, used as an accessor for a db or a client side backbone model presenting the events attributes as part of a view. Usually (if you’re using a sensible class naming scheme) it is enough to know the file (.php or .js) within which you’re working and if that’s not enough, then the language syntax usually is a dead giveaway for where you are in your codebase. But this isn’t the case when working in one language. My guess is you’re going to have to take a second glance in order to figure out where you are, which isn’t a big issue, but is frustrating.

Currently my apps typically are structured like this:

server.js (Main node/express file)
-lib (Classes for DB Calls)
-node_modules (Node requires)
-public (Client backbone app)
        -lib (Backbone and other dependencies)
-routes (Contains the main express routes)

But I actually think the following is a better approach:

-client (Renamed from public)

At least with the second approach, it is easier to have a clear idea of where to find classes on first opening them. Unfortunately, this doesn’t however solve my problem of moving between server side and client logic and pausing for a moment before realising I am where I need to work. This is a first world problem for which I don’t believe there is a solution.

Tagged javascript, web development, | Leave a comment

So You Want to Run a Fairtrade Business?

28 Mar 2013

This is a brief retrospective of what I’ve learned in the 6 years I’ve been running Fair & Bare, a Fairtrade t-shirt company along with my wife.

If you’re a new follower of mine, you heard right – I run a Fairtrade t-shirt company. I’ve not actively been marketing it more recently for a number of reasons, but I still have a strong belief in our original vision. That vision was that Fairtrade shirts at the time weren’t being well received by the world, despite being available. We believed they could be, if we were only to find some that had good designs (and were possibly less ‘preachy’).

So that’s what we set out to do. From 2007-2011 we were running as an online design competition, inspired mainly by Threadless and Emptees, and helped via a great community of designers through a site I built entirely myself (whilst doing a PhD, getting married and buying a house). I’ve since learned that I don’t believe the competition sourced design is the best fit for what we are trying to do, but it constantly excited me that we *could* do it. I still hope some day to be proved wrong though.

Lessons Learned

The Ethical market pales in comparison to the cheaper alternatives. That probably won’t come as news to you. Most people – like it or not, will buy cheap over quality if they’re perceived to be comparable. This has certainly been my experience so far, having touted our wares at student markets, Fairtrade stalls and online. Be prepared to experience it first hand, and to repeatedly go over your story to get people to part with their cash. That is, after all, your unique selling point.

While we’re at it, if you’re selling online, be prepared to experience the same. When so much hard work has gone into creating your goods, you’ll find it heartbreaking to have a couple of people take up an offer from a newsletter of thousands. My advice would be not to get your hopes up….

But don’t entirely despair. When you make those few sales you’ll feel such a strong connection with your customer, even though you may never meet them. Repeat customers will give you a warm glow inside and smile all day long.

The Fairtrade supply chain is slow(er) and (more) fallible. We constantly had trouble with every supplier we used for our shirts over how quickly we could obtain stock and sometimes with the stock itself. Obviously, once our community had selected a design, we wanted it printed and available for sale. We knew we would have to wait for designs to be printed, but what we hadn’t counted on was how long it could take to get hold of shirts we’d ordered. Other businesses might be able to cope better with such delays. 2 months to go from selected to online is not ideal for our business model that relies on constant momentum, especially when we were dealing with a a single product at a time.

Depending on your supplier, you may also experience variations in sizes and colour. Be wary, you’re going to have to deal with all the resulting issues that arise from these problems with your customers or be lumbered with stock you’re unable to shift.

Times are hard for everyone. Since having started, 2 of the 3 suppliers we used have shut up shop and the third has changed it’s supplier. This has drastically reduced the options for us to the point of being unviable, which is one of the reasons we haven’t put out a new design since 2011. (If anyone is able to point me in the direction of other wholesale ethical brands, I’d be keen to hear about them).

Fairtrade cotton, Fairtrade or fair trade? Organic? Soil association certified? You need to be aware of the differences, because competitors will use similar terms. Even though they mean different things to you, most customers don’t know the difference and lump them all in the same good for the producer pot.

First impressions count. Originally, we used to send out our t-shirts in recycled packaging we sourced from envelopes we had kicking around our house. I have to cringe at this now – the idea that old crumply packets were being received with sticky-taped-on labels. That was our first physical impression on our customer. *Sigh*. Our heart was in the right place though, we didn’t (and still don’t) want to be creating more waste. We’ve since moved on to shipping goods out in recycled packets.

Marketing is key. When you’re not marketing your goods, you’re not seen to be active. Just by talking about your business, you’ll get interest in your product and make sales. Our buzz came from regular competitions and launches of new shirts. With every new shirt we’d let our mailing list, community and followers know of its arrival.

Finally, a word on branding. I constantly wrestle with wether we would have been better off branding ourselves first and foremost a t-shirt design company, rather than a Fairtrade one. I think the answer is yes. Our whole ethos was originally based around the lack of well designed Fairtrade shirts. Ultimately, if a design looks good, a customer is going to be happy paying a little more if it’s what they want. The ethical aspect of the cotton is, in their mind, a bonus. But if they’re not even visiting your site or stall because they see you first and foremost as an ethical brand, and that doesn’t entirely float their boat, then you’ve lost a sale.

The whole experience of running Fair & Bare has been a bit of a roller coaster of highs and lows. I’ve learned so much from running it and hope to continue to do so.

It wouldn’t be right of me to post about Fair & Bare here without letting my readers in on a special offer. For this Easter weekend, when you buy two shirts, you’ll get the third free on using the code “EASTEREGG” when checking out.

Tagged fairtrade, tshirts, | Leave a comment

Breaking up Relationships with CouchDB

01 Mar 2013

[NB: This is an unpublished post I wrote in 2010 on getting started with CouchDB. Therefore, despite all the code and examples being relevant, it may be considered somewhat ‘belated’]

Beware – there’s a bunch of home wreckers out there intent on removing the love of your life and replacing it with a wicked mistress.

For me, my first experiments working with databases were performed with Oracle, the staple of our computer science course at the time. We were taught how to identify common structures in what we wanted to store (or had already been stored) and how to represent the relationships which existed between them. There was an entire series of lectures dedicated to this fine art, much of which I now can’t remember.

There has been an uptake in the number of developers working with NoSQL, or document oriented databases. These alternatives do not require decisions on subdividing documents into multiple record structures to be made at all, instead allowing the entire document to be recorded as a series of simple variable types. The contents of each document can vary from one document to the next. For the majority of developers who work with relational databases, this might come as a bit of a shock.

There are a number of varieties of NoSQL flavours currently available: CouchDB, Cassandra and MongoDB are a few of the hot ones right now. I’m not going to discuss the pros and cons of each right here (for that I’d refer you to The Changelog’s NoSQL smackdown podcasts), but rather give you a whistle stop tour of CouchDB, which I’ve been working with for a few years now.

If you want to follow along, you’ll need to head to the CouchDB site and read up on installation for your platform. I’m using the CouchDB server app, which is nice and self contained.

Document oriented storage gives me the ability not to have to worry about the structure of my documents prior to storage. From my own point of view this is a huge timesaver as my apps often tend to focus on a single type of document. Think of a blog post stored in a mysql database, we have a table for the post along with all its metadata, a table for the comments and maybe another for recording pingbacks. The same blog post in couch could be represented like so:

   "_id": "83ab09b88836ab714f592293d4e02845",
   "title": "My Blog Post",
   "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
   "comments": [{"ian": "You're a wally"},{"Some user": "This post sucks"}]

A document based db may well not be suitable for the nature of your application. However, I’ve found in lots of cases, it is.

Rest Style Interface

CouchDB works across a REST style interface. To write documents back to couch, you call a HTTP PUT or POST with your JSON structure. To read documents, you call a GET and to delete them you call DELETE. For example, to add a document with “_id=someid” to the database “blog”, you would call the following:

curl -X PUT -d '{"Title":"Another Blog Post", "Content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit"}'

Similarly you can HTTP GET this uri to return the document just added. Saving the document to a couch store will add another field, “_rev” which represents the particular revision of the document you’re storing. Every time you update this document, it will be stored as a new document, along with a new revision number. You have access to all these revisions within couch, or you can choose to cleanup the database by compacting and removing all but the most recent revision of documents. Another blog entry document without any comments would be perfectly valid stored in the same database, as well as a document featuring just an array of pingback items.

You’ll notice that the structures are JSON objects. This is how all objects are exchanged between couch and other languages and over couch’s REST interface. Heading to where we stored the above blog document “” in your browser yields a collection of metadata expressed in a JSON structure about that particular database. Appending “/83ab09b88836ab714f592293d4e02845″ (the id of the document) gives us the original blog document.

This, in turn works wonderfully well with certain frontend frameworks – such as Backbone. If you configure your backbone models to construct URLs the way CouchDB expects them, you can effectively have a frontend app driven by just a CouchDB store. As CouchDB returns JSON objects, it’s able to correctly parse and load them into your backbone app.


To support the extraction of substructures and sorting within these documents, couch has the concept of map/reduce built straight into its core. Futon, couch’s built in administration application, makes creation of these a very simple process. Maps and Reductions are described in Javascript, which is great given most of us are making more and more use of it these days. Head over to and you’ll see your collection of databases right away. Navigate to the blog database and under the view dropdown, select temporary view. Couch automatically populates this with a default temporary view:

function(doc) {
  emit(null, doc);

This basically reads as: for each document emit an object with a null key and the document as the value which is what you’ll see if you go ahead and run it. Not particularly exciting really, but by changing the null key to doc.title, we can emit all those blog posts sorted by title. To do something a little more complicated, such as determining the number of comments for each blog post we can make use of the reduce function too.


function(doc) {
  for(var i=0; i < doc.comments.length; i++){
   emit(doc.title, 1);


function (keys, values){
   return sum(values);

Here, we cycle each of the comments for every blog post and emit a single value 1 for every comment. Our reduce function takes both the entire key and value set which are emitted from the map function for each document and calculates the sum of values sent to it. With these two very basic functions we can support all the queries we might want to make of the database. You can permanently store this entire map/reduce query as a view within couch to be called later. When a document is inserted which effects the result of the query, the map/reduce will be re-evaluated on the next call. It turns out that storing and querying documents in this way can be extremely performant.

I very much enjoy having the ability to write my queries in a language I’m familiar with to retrieve results, rather than having to dip into MySQL voodoo.

So, there you have it, CouchDB – I may well follow this up at a later date with a comparison with Mongo, given I’m now a certified developer.

Tagged backbone, couchdb, databases, nosql, | Leave a comment

Growing a Business with Developer APIs

20 Feb 2013

Those of your you who follow me on twitter will might be aware of a single day hack I attempted last week, which takes output from Lovefilm’s customer history and formats it so it can easily be imported into the lovely Letterboxd. It’s named Loveletterd and it seemed entirely appropriate to release it on Valentines day.

After hearing that Letterboxd had created a CSV import tool for it’s community to use and as I’m a Lovefilm developer, I felt like it would be a great use of my time to get to work on Loveletterd.

Originally, I’d thought I’d use Lovefilm’s OAuth sign-on, grab past rentals for a user and create a CSV from it formatting to Letterboxd’s import structure. I’d completely forgotten about a bug I raised on the developer forums; almost 3 years earlier – highlighting that actually this was impossible due to a bug which didn’t read a supplied argument, subsequently limiting the number of rental titles returned as the history. What’s more infuriating is that no one from Lovefilm had replied, even though the problem was reiterated by others in the same and other threads. There’s many different threads which highlight numerous other issues, all of which receive no or very little response from the Lovefilm developer team.

This is an awful way to treat a community excited about building tools with your company data. If you don’t want to cushion the cost of supporting providing such an API yourself, don’t create one, or charge for access to it. Similar sentiment has been said about twitter recently on changing their API rendering it unusable for apps that have based their business models on it. Such is the feeling of discontent that it’s led to a niche for rivals.

Lovefilm’s API “gallery” of developer apps features the message

“We will be placing details of the latest and greatest applications using our API here. Be sure to check this page regularly as new ideas are developed.”

Which is also the same as 3 years ago, shortly after the API’s first release. Good work Lovefilm.

Remembering this, I walked away thinking, “Oh, well”, before going back to my workday. Explaining it to my wife, I realised there was another route to achieve what I wanted, which is what you see on the web now. Ultimately, this is nothing more than a bunch of regular expressions.

Admittedly, I’m aware there’s changes at Lovefilm – they’ve been entirely bought out by Amazon and I can imagine the new owners would want to integrate the company with their brand better. It seems as if they’ve already switched off new developer registrations, so I’d imagine the whole API will be turned off before long.

On the flip, take a look at where I want to take my data to – Letterboxd. A site focused on film lovers and it shows. They’ve taken the route of paid pro accounts, which provide access to the import tools I mentioned earlier. The majority of users won’t make use of this though; they just want to support a site providing a great service which is open and honest. They’re also planning on providing a full API, which I’ll be interested in playing with on release. As a result, I’m sure Letterboxd users will get a bunch of new toys they can use their accounts with that external developers decide to create.

The best thing though? They’re real people who talk to me. After letting them know about my little hack, I got thanks that same day from Matthew of the Letterboxd crew, with a ticket closure shortly thereafter. That beats 3 years of waiting.

Tagged films, hacking, | Leave a comment