Blogging with Metalsmith

30 Jun 2015

I'm switching blogging platforms once more. Moving from Wordpress to Jekyll enabled me to no longer deal with the multitude of updates and protected me hacks I could be susceptible to, but Jekyll being written in Ruby (a language I'm not overly familiar with) has been a problem. I've been able to cobble together scripts to deal with the custom parts of my blog build, but it's never felt like home.

I've decided to start using Metalsmith - a static site generator written in Javascript. It's existed before my switch to Jekyll and now seems to have enough of a plugin ecosystem in order to provide a collection of plugins to provide behaviour I need to make use of.

Metalsmith

Metalsmith operates much like a task runner like gulp, where a set of functions manipulate a number of files, one after another in order to eventually build a static site to your liking. A simple build script could be as follows:

var Metalsmith = require('metalsmith'),
    markdown   = require('metalsmith-markdown'),
    templates  = require('metalsmith-templates');

Metalsmith(__dirname)
  .use(markdown())
  .use(templates('swig'))
  .build();

This is just standard node script with dependencies installed via npm. Metalsmith assumes defaults of src and build for the source of markdown and destination of html. An example of markdown containing some useful front-matter might be as follows:

---
title: Blogging with Metalsmith
author: Ian
date: 2015-06-30
year: 2015
month: 2015/06
categories:
  - Javascript
tags:
  - metalsmith
  - javascript
---
I'm switching blogging platforms once more...

It also assumes a default template directory under templates, containing a default post template 'post.html' - which in the example is written in a swig template as below.

<html>
  <body>
    <div class="post">
    <h1><a href="/{{ path }}">{{ title }}</a></h1>
    <div class="post_date">{{ date | date('d M Y') }}</div>
    {{ contents | safe }}
    </div>
  </body>
</div>

I've managed to reduce my build time (though I'm not currently compiling sass and have added a new sitemap), but am much more familiar with the toolchain in question. Hopefully that'll lead to further, more frequent updates not just in content, but also the design of my company site and this blog.

I'll be adding future blogs about my metalsmith toolchain and hurdles with that I've overcome.

Tagged metalsmith, blogging, javascript, | Leave a comment

Scaling a Linear Domain to an Ordinal Range with d3

07 Mar 2014

I’ve had the opportunity of using d3 quite a lot over the past few months for a number of clients. It offers some amazing flexibility for chart generation and much more.

Anyway, I thought I’d share a quick tip I developed for mapping a linear set of values onto an ordinal scale. For those who’re seasoned pros at d3, this probably seems trivial, but had me stumped for some time today.

I’d picked out a colour palette I wanted to use for a particular graph, as per below:

var colours = ["#B8D0DE", "#9FC2D6", "#86B4CF", "#73A2BD", "#6792AB"];

The only examples I’ve seen similar to this are where it is assumed you want to vary darkness of colours based on value or vary the domain based on the number of colours you want. Not a good fit.

I wanted to pick one of my values based on a linear value from my data set. My first thought was to make use of the ordinal scale function provided by d3. Something like this:

var colour = d3.scale.ordinal()
  .domain(mySortedDataValues)
  .range(colours);

In doing this, I got something that *looked* a bit like it was working, but not the way I expected. In fact, the way an ordinal scale works is that it provides a 1-to-1 mapping of domain values to the range, rather than any kind of interpolation between them. In this case, it was expecting only 5 distinct data values (to match up against the colours) and for everything over and above that, it wrapped them round to the beginning of the domain again. The solution then is fairly simple once you’ve got your head around that.

What I did next, was to create a scale that gave the index of the colour we were going to be mapping to. This works well, because the indices are linear and d3 has the ability to do the dirty work in that respect.

var colourIndex = d3.scale.linear()
  .domain([0, d3.max(data)])
  .range([0, colours.length - 1]);

svg.selectAll("circle")
  .data(data)
  .enter()
  .append("circle")
  .attr("fill", function(d){
    return colours[Math.ceil(colourIndex(d))];
   });

Here we end up with an index ranging across all the indices of the colour array, and a colour appropriately selected from the palette as expected. You can see the resultant effect in the graph linked to below:

Tagged charting, d3, | Leave a comment

The Problem with Full Stack JS Applications

03 Apr 2013

Whilst node.js heralded an era of being able to use a single language for both server and client side development, nobody mentioned the confusion such an approach could cause. It’s inevitable that there’s going to be similarities in code that constructs data on the server and that which presents it on the client. The fact that the two are written in the same language and the structure of them is so tightly coupled can make it easy to lose track of where you are and what you’re doing when knee deep in code.

Having primarily been using a full js stack (express on the server, backbone on the client) for side projects over the last year or so, I’ve certainly found myself tied up in this kind of knot. For example, if I’m looking at event.js, which represents an event on a calendar it could be the server event representation, used as an accessor for a db or a client side backbone model presenting the events attributes as part of a view. Usually (if you’re using a sensible class naming scheme) it is enough to know the file (.php or .js) within which you’re working and if that’s not enough, then the language syntax usually is a dead giveaway for where you are in your codebase. But this isn’t the case when working in one language. My guess is you’re going to have to take a second glance in order to figure out where you are, which isn’t a big issue, but is frustrating.

Currently my apps typically are structured like this:

server.js (Main node/express file)
-lib (Classes for DB Calls)
-node_modules (Node requires)
-public (Client backbone app)
    -css
    -images
    -js
        -lib (Backbone and other dependencies)
        -collections
        -models
        -templates
        -views
-routes (Contains the main express routes)

But I actually think the following is a better approach:

-server
    server.js
    -lib
    -node_modules
    -routes
-client (Renamed from public)
    -css
    -images
    -js
        -lib
        -collections
        -models
        -templates
        -views

At least with the second approach, it is easier to have a clear idea of where to find classes on first opening them. Unfortunately, this doesn’t however solve my problem of moving between server side and client logic and pausing for a moment before realising I am where I need to work. This is a first world problem for which I don’t believe there is a solution.

Tagged javascript, web development, | Leave a comment

Breaking up Relationships with CouchDB

01 Mar 2013

[NB: This is an unpublished post I wrote in 2010 on getting started with CouchDB. Therefore, despite all the code and examples being relevant, it may be considered somewhat 'belated']

Beware – there’s a bunch of home wreckers out there intent on removing the love of your life and replacing it with a wicked mistress.

For me, my first experiments working with databases were performed with Oracle, the staple of our computer science course at the time. We were taught how to identify common structures in what we wanted to store (or had already been stored) and how to represent the relationships which existed between them. There was an entire series of lectures dedicated to this fine art, much of which I now can’t remember.

There has been an uptake in the number of developers working with NoSQL, or document oriented databases. These alternatives do not require decisions on subdividing documents into multiple record structures to be made at all, instead allowing the entire document to be recorded as a series of simple variable types. The contents of each document can vary from one document to the next. For the majority of developers who work with relational databases, this might come as a bit of a shock.

There are a number of varieties of NoSQL flavours currently available: CouchDB, Cassandra and MongoDB are a few of the hot ones right now. I’m not going to discuss the pros and cons of each right here (for that I’d refer you to The Changelog’s NoSQL smackdown podcasts), but rather give you a whistle stop tour of CouchDB, which I’ve been working with for a few years now.

If you want to follow along, you’ll need to head to the CouchDB site and read up on installation for your platform. I’m using the CouchDB server app, which is nice and self contained.

Document oriented storage gives me the ability not to have to worry about the structure of my documents prior to storage. From my own point of view this is a huge timesaver as my apps often tend to focus on a single type of document. Think of a blog post stored in a mysql database, we have a table for the post along with all its metadata, a table for the comments and maybe another for recording pingbacks. The same blog post in couch could be represented like so:

{
   "_id": "83ab09b88836ab714f592293d4e02845",
   "title": "My Blog Post",
   "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
   "comments": [{"ian": "You're a wally"},{"Some user": "This post sucks"}]
}

A document based db may well not be suitable for the nature of your application. However, I’ve found in lots of cases, it is.

Rest Style Interface

CouchDB works across a REST style interface. To write documents back to couch, you call a HTTP PUT or POST with your JSON structure. To read documents, you call a GET and to delete them you call DELETE. For example, to add a document with “_id=someid” to the database “blog”, you would call the following:

curl http://127.0.0.1:5984/blog/someid -X PUT -d '{"Title":"Another Blog Post", "Content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit"}'

Similarly you can HTTP GET this uri to return the document just added. Saving the document to a couch store will add another field, “_rev” which represents the particular revision of the document you’re storing. Every time you update this document, it will be stored as a new document, along with a new revision number. You have access to all these revisions within couch, or you can choose to cleanup the database by compacting and removing all but the most recent revision of documents. Another blog entry document without any comments would be perfectly valid stored in the same database, as well as a document featuring just an array of pingback items.

You’ll notice that the structures are JSON objects. This is how all objects are exchanged between couch and other languages and over couch’s REST interface. Heading to where we stored the above blog document “http://127.0.0.1:5984/blog&#8221; in your browser yields a collection of metadata expressed in a JSON structure about that particular database. Appending “/83ab09b88836ab714f592293d4e02845″ (the id of the document) gives us the original blog document.

This, in turn works wonderfully well with certain frontend frameworks – such as Backbone. If you configure your backbone models to construct URLs the way CouchDB expects them, you can effectively have a frontend app driven by just a CouchDB store. As CouchDB returns JSON objects, it’s able to correctly parse and load them into your backbone app.

Map/Reduce

To support the extraction of substructures and sorting within these documents, couch has the concept of map/reduce built straight into its core. Futon, couch’s built in administration application, makes creation of these a very simple process. Maps and Reductions are described in Javascript, which is great given most of us are making more and more use of it these days. Head over to http://127.0.0.1:5984/_utils and you’ll see your collection of databases right away. Navigate to the blog database and under the view dropdown, select temporary view. Couch automatically populates this with a default temporary view:

function(doc) {
  emit(null, doc);
}

This basically reads as: for each document emit an object with a null key and the document as the value which is what you’ll see if you go ahead and run it. Not particularly exciting really, but by changing the null key to doc.title, we can emit all those blog posts sorted by title. To do something a little more complicated, such as determining the number of comments for each blog post we can make use of the reduce function too.

Map:

function(doc) {
  for(var i=0; i &lt; doc.comments.length; i++){
   emit(doc.title, 1);
  }
}

Reduce:

function (keys, values){
   return sum(values);
}

Here, we cycle each of the comments for every blog post and emit a single value 1 for every comment. Our reduce function takes both the entire key and value set which are emitted from the map function for each document and calculates the sum of values sent to it. With these two very basic functions we can support all the queries we might want to make of the database. You can permanently store this entire map/reduce query as a view within couch to be called later. When a document is inserted which effects the result of the query, the map/reduce will be re-evaluated on the next call. It turns out that storing and querying documents in this way can be extremely performant.

I very much enjoy having the ability to write my queries in a language I’m familiar with to retrieve results, rather than having to dip into MySQL voodoo.

So, there you have it, CouchDB – I may well follow this up at a later date with a comparison with Mongo, given I’m now a certified developer.

Tagged backbone, couchdb, databases, nosql, | Leave a comment

Why I Choose Popular Frameworks

04 Jan 2013

I feel compelled to write this post so late in the day after clocking Tyler Renelle’s rant on github about his problems with the popularity of certain JS frameworks (in the main Backbone.js). After having written apps in a variety of JS frameworks (Backbone, Spine and Meteor), I’ve learned that the feature set of a framework is not the only thing I hold dear when making a choice between them.

Documentation, support, learning curve and most importantly community are what also guide me when building anything. If I need to make an app quickly, I’ll of course develop it in my favourite tool (Backbone), because I’ve been using it for years and therefore doesn’t hold a cost. If I have a problem, I know there’s a ton of hackers who’ve probably already worked on solutions already and mechanisms are in place for them to offer advice. There may be better documented more capable tools, but I can’t foot the expense in the short term that might come with investigating something new and run the risk of hitting problems I can’t solve. In the long term, as a framework grows in popularity and garners these other factors then I may make that jump.

The real reason these frameworks are popular right now is in part because they are already popular.

Tagged backbone, js, popularity, | Leave a comment