Splitting a date range in Python

01 Jul 2014

Dates are one of those annoying things that shouldn’t be, but are regularly difficult in web apps. I used the following two methods in a recent page to neatly break a date range into distinct segments as part of a analytics app I’m currently working on. Hopefully someone else will find them helpful.

import datetime, calendar

# Find the delta between two dates based on a desired number of ranges
def datedelta(startdate, enddate, no_of_ranges):
    start_epoch = calendar.timegm(startdate.timetuple())
    end_epoch = calendar.timegm(enddate.timetuple())

    date_diff = end_epoch - start_epoch

    step = date_diff / no_of_ranges

    return datetime.timedelta(seconds=step)

date_delta allows me to create the timedeltas between two dates based on a desired number of segments.

# Edit 18/07/2014 - I realised dates needed the hrs, mins an secs correctly
# adjusted to beginning / end of day
def datespan(startdate, enddate, delta=datetime.timedelta(days=1)):
    currentdate = startdate

    while currentdate + delta < enddate:
        todate = (currentdate + delta).replace(hour=23, minute=59,second=59)

        yield currentdate, todate

        currentdate += delta
        currentdate.replace(hour=0, minute=0,second=0)

I can then pass the delta into datespan above, which returns an iterable I can then use.

# Get timedeltas based on splitting the range by 10
delta = date_delta(startdate, enddate, 10)

for from_datetime, to_datetime in datespan(startdate, enddate, delta):
    print from_datetime, to_datetime

The result forms part of the following d3 chart:

Tagged coding, python, | Leave a comment

Tips for being a more Productive Freelancer

16 Jun 2014

Having heard the question raised on @workingoutshow, I thought I’d take some time to share the things that I’ve found have really helped me be productive in my work. These may not work for everyone, but hopefully some points will help for others.

Find a Good Environment

The single biggest change I’ve found which has had a major impact on my work is changing my environment. We’re lucky enough to have a small garden at our house and with the arrival of my daughter back in October, I was quick to realise that working in the open plan lounge in our house just wasn’t a viable option any more. Here in Cardiff, we’ve been lucky enough to have several co-working spaces open locally, but for me I prefer being able to switch off background noise and clear my head when I really need to knuckle down and focus. Around the same time, we made some major changes to the back of our house and garden which meant that there was space enough for a desk in our sunroom which overlooks it. I now spend 90% of my working time looking out over our little green space, which makes for a welcome break from staring at my screen and gives me a self-contained and more importantly, quiet environment in which to think. I also get to enjoy the space more overlooking the various wildlife that visits during the day. It’s funny how good a little bird taking a dust bath every now and again will make you feel.

Organise Your Tasks

I use trello for organising tasks I want to get done, with a board for each project. These are organised in descending priority, using trellos card defaults of “To Do”, “Done” and “Doing” to track what I’ve achieved for a project. If whilst during the course of one task, I realise something needs doing over and above the scope of the task I’m currently working on, I’ll add it to my todo list and work on it when all other tasks with a higher priority have been completed. This strategy works well for me in terms of being able to blast through a whole heap of development tasks without having to think too much about what I need to do next.

Plan Your Next Day Before Leaving Your Desk

I like to end my day at a distinct break between tasks. If my day ends at a point where something is incomplete, I’ll be sure to keep it in my to-do list for the following day, where I can come back to it. Before I leave my desk each day I’ll generally try and plan out what makes sense to cover (given the time I have) in the following working day. This allows me to hit the ground running each day, without any getting up to speed time first thing in the morning.

Track Your Time

Tracking my time is important not only for my client work, but also my own projects. As well as giving me exact totals for invoicing, it allows me to compare estimates against the reality of a project and all the unforeseen problems that may arise during the course of it. My time tracking is a simple project based on/off approach, where I use a custom tool I developed myself.

Eliminate Distractions

Although obvious, trying to cut down on distractions pays dividends. Switch off tweetdeck, close skype and either disable notifications on your phone or (my favourite) put it out of your sight completely. I don’t tend to pay any thought to my phone if I can’t see it, and it’s like a nervous twitch to pick it up and check whether anything has happened in the last 5 mins. When not aware it’s an option, I probably get twice as much work done. It’s easy to kid yourself that checking devices like this is part of your work, but when at the detriment of your day’s schedule you need to switch off and focus on the bigger picture.

These are just a few measures I take to be more productive in my day - I’d welcome any tips other freelancers have for their own situations.

Tagged productivity, | Leave a comment

Porting a Wordpress blog to Jekyll: Part 2

02 Jun 2014

In the second of my posts in porting a wordpress blog over to jekyll, I outline how I’ve replicated wordpress’s default permalink structure on my own blog.

Creating Category and Tag Pages

Jekyll doesn’t create category or tag index pages for those categories listed in your _posts source. Instead, you’ll need to use a plugin to recreate them. I used the tutorial here as the basis for my category/tag generator, with my plugins/generate_cats_and_tags.rb source below:

module Jekyll
    class CatsAndTags < Generator
        def generate(site)
          site.categories.each do |category|
            build_subpages(site, "category", category)
          end

          site.tags.each do |tag|
            build_subpages(site, "tag", tag)
          end
        end

        # Do the actual generation.
        def build_subpages(site, type, posts)
          posts[1] = posts[1].sort_by { |p| -p.date.to_f }
          atomize(site, type, posts)
          paginate(site, type, posts)
        end

        def atomize(site, type, posts)
          path = "/#{type}/#{posts[0]}"
          atom = AtomPage.new(site, site.source, path, type, posts[0], posts[1])
          site.pages << atom
        end

        def paginate(site, type, posts)
          pages = Pager.calculate_pages(posts[1], site.config['paginate'].to_i)
          (1..pages).each do |num_page|
            pager = Pager.new(site, num_page, posts[1], pages)
            slug = posts[0].gsub(' ', '-')
            base_path = "/#{type}/#{slug}"
            path = base_path
            if num_page > 1
              path += "/page/#{num_page}"
            end
            newpage = GroupSubPage.new(site, site.source, path, base_path, type, posts[0])
            newpage.pager = pager
            site.pages << newpage

          end
        end
    end

    class GroupSubPage < Page
        def initialize(site, base, dir, base_path, type, val)
          @site = site
          @base = base
          @dir = dir
          @name = 'index.html'

          self.process(@name)
          self.read_yaml(File.join(base, '_layouts'), "group_index.html")
          self.data[type] = val
          self.data['base'] = base_path
        end
    end

    class AtomPage < Page
        def initialize(site, base, dir, type, val, posts)
            @site = site
            @base = base
            @dir = dir
            @name = 'atom.xml'

            self.process(@name)
            self.read_yaml(File.join(base, '_layouts'), "group_atom.xml")
            self.data[type] = val
            self.data["grouptype"] = type
            self.data["posts"] = posts[0..9]
        end
    end
end

I chose to pass in an extra parameter to the Pager, defining the “base_path” for which an index page has been generated. This enables absolute links to pages and prevents situations like /category/javascript/category/javascript/page/1, which occur when relative pagination is generated when on a category page.

A view making use of it would look like the following:

<div class="pagination">
        {% if paginator.previous_page %}
          <a href="{{ page.base }}/page/{{ paginator.previous_page }}" class="previous">Previous</a>
        {% else %}
          <span class="previous">Previous</span>
        {% endif %}
        <span class="page_number ">Page: {{ paginator.page }} of {{ paginator.total_pages }}</span>
        {% if paginator.next_page %}
          <a href="{{ page.base }}/page/{{ paginator.next_page }}" class="next">Next</a>
        {% else %}
          <span class="next ">Next</span>
        {% endif %}
</div>

Creating Archive Pages

In a similar fashion, Jekyll doesn’t create pages or pagination for previous years or months. Even if they are not oft used by others, I find it useful myself to hit up my archives for previous years on occasion and laugh at the random work I did for my undergraduate.

I created the following plugin to generate indexes at /{year} /{year}/{month} and /{year}/page/{pageNo} and /{year}/{month}/page/{pageNo} if necessary.

module Jekyll

  class ArchiveGenerator < Generator
    safe true
    def generate(site)
      if site.layouts.key? 'archive_index'
        site.posts.group_by{ |c| {"month" => c.date.month, "year" => c.date.year} }.each do |period, posts|
          posts = posts.sort_by { |p| -p.date.to_f }
          archive_dir = File.join(period["year"].to_s(), "%02d" % period["month"].to_s())
          paginate(site, archive_dir, posts)
        end
        site.posts.group_by{ |c| {"year" => c.date.year} }.each do |period, posts|
          posts = posts.sort_by { |p| -p.date.to_f }
          archive_dir = period["year"].to_s()
          paginate(site, archive_dir, posts)
        end
      end
    end

    def paginate(site, dir, posts)
      pages = Pager.calculate_pages(posts, site.config['paginate'].to_i)
      (1..pages).each do |num_page|
        pager = Pager.new(site, num_page, posts, pages)
        archive_path = "/" + dir
        path = archive_path
        if num_page > 1
          path += "/page/#{num_page}"
        end
        newpage = ArchiveIndex.new(site, site.source, path, archive_path, posts)
        newpage.pager = pager
        site.pages << newpage

      end
    end
  end

  class ArchiveIndex < Page
    def initialize(site, base, dir, base_path, posts)
      @site = site
      @base = base
      @dir = dir
      @name = 'index.html'

      self.process(@name)
      self.read_yaml(File.join(base, '_layouts'), 'archive_index.html')
      self.data['base'] = base_path
    end
  end
end

The layout of the paginator for the archives is the same as that for categories and tags.

That’s it for my jekyll roundup, I other long time wordpress users find this information useful.

Tagged jekyll, static sites, wordpress, | Leave a comment

Porting a Wordpress blog to Jekyll: Part 1

20 May 2014

As my last post indicated, I recently made the decision to leave wordpress for jekyll for my blogging software of choice. There were a number of hurdles I had to overcome in order to get to a state I was happy with, emulating my previous blog structure. This first post details a few of the setup problems and how I’ve overcome them.

Using Compass for a Jekyll theme

The first problem I hit was my use of compass to create parts of the theme I intended to use for my blog. When attempting to use jekyll-compass to enable that, I hit the following error:

Unable to activate compass-0.12.6, because sass-3.3.7 conflicts with sass (~> 3.2.19) (Gem::LoadError)

After searching for a bit I came across the following issue which detailed a solution - but unfortunately it didn’t work for me. Being somewhat of a newbie to Ruby, I didn’t realise that after creating a Gem as instructed in the fix, I actually needed to “bundle install” it. After a little more digging and tweaking against Mark Turners advice, I ended up with a Gem file like so:

source "https://rubygems.org"

gem 'sass', "~> 3.2.19"
gem 'jekyll-compass', "1.0.6"

Which has finally given me what I wanted.

Using Custom Pygments Themes for Syntax highlighting

Jekyll uses pygments for fancy pants syntax highlighting. Although an example syntax.css is linked to in the jekyll documentation, I prefer a darker theme myself. Jekyll doesn’t tell you it’s possible to generate any one of the numerous pygments styles like this:

pygmentize -f html -S monokai -a .highlight > monokai.css

I’ve used the monokai theme for my own highlighting. There are demos of each of the themes on the pygments site - here is an example. You can then use that as the basis for your own custom highlighting. I’ve created each one of the 19 possible themes and put them up on github.

Moving to Disqus Commenting

Given I now had a static site, I needed a means of continuing supporting comments. Disqus does just that by embedding comments using javascript into post pages. Adding disqus proved to be the biggest hurdle for me. I had a large number of comments from people on my blog over the years and therefore I didn’t want to lose them. After exporting my site as xml and uploading them to disqus, I finally set jekyll loose on my site along with disqus embeds, but I couldn’t see them. The problem I found after a frustrating hour or two of uploading new versions of my site over ftp, was jekyll used a trailing slash (as the post is an index within a directory) to denote links to posts. It’s possible however to do away with this using .htaccess if you’re using Apache.

Options -Multiviews +FollowSymLinks
RewriteEngine On
RewriteBase /
DirectorySlash Off

# remove the trailing slash
RewriteRule ^(.*)\/(\?.*)?$ $1$2 [R=301,L]

# rewrite /dir/file to /dir/file/index.html
RewriteRule ^([\w\/-]+)(\?.*)?$ $1/index.html$2 [L,T=application/x-httpd-html]

However, when I began using github pages, this fix didn’t work as github only serves static files and doesn’t allow use of htaccess. Instead, I had to resign to the fact that I would have to have trailing slashes (apparently they’re quicker anyway). This then meant I had to use the Redirect Crawler option of disqus’s “Migrate Threads” wizard (seen below) to crawl my new blog on github and determine where the posts were now located.

In my next post I’ll go through how I use plugins to support archive, category and tag pages in jekyll.

Tagged jekyll, static sites, wordpress, disqus, pygments, | Leave a comment

Bye Bye Wordpress

18 May 2014

I’ve finally made the move to jekyll for building this site. It is now hosted on github pages, comments are handled by disqus and my post source is written in markdown. My entire site is visible on github and revision control for post entries is handled there too.

What finally convinced me with the release of jekyll 2 was the introduction of native sass support. Originally, the prose theme I’m using for my site was using scss, which I later ported to a wordpress theme in order to blog with it. I had the scss for the original design available and therefore thought the jump to using jekyll wouldn’t be that great (or so I thought - overlooking all the features jekyll doesn’t support compared to wordpress). I’m going to blog individually about the pieces of the puzzle in other posts as there are number of things that tripped me up along the way.

But for now, I’d like to bid a fond farewell to wordpress - my go to blogging software for the last 10 years. You once were diminutive, got out of my way and let my voice be heard. That’s no longer the case. These days the only time we speak is when you want me to update you. I’ve got better things to do. Theres a ton of features the length of my arm which I’ve never used and the majority of features which I use rarely. I long for those simpler days once again. You’ve served me well, but I’ve moved on.

Tagged jekyll, static sites, wordpress, | Leave a comment

Copyright Law Finally Aligns with Me

07 Apr 2014

On June 1st 2014, some much needed changes are being made to exceptions to copyright law which amongst other things, will allow conversion of personal media between file formats.

Finally, we’ll able to make personal conversions of digital media (at least they will be now counted as legal copies). I’ve long made copies of my cd’s to listen to whilst coding (iTunes allows me to do so simply) – but it is not currently legal to move these files to another storage medium, such as an mp3 player. The fact is that most people have been ripping cds and copying them to iPods since it was technically possible, but music companies have never prosecuted anyone as the implications would stem far and wide and would the costs would too prohibitive to be able to enforce them.

It’s also not currently legal to rip movies to another format. So copying a dvd to a mkv file to play elsewhere is construed illegal. The fact movie studios put protection in place on their dvds to prevent you from doing so puts most people off doing so anyway. From June 1st this also changes so it will also be legal (but probably will be as difficult due to the copy protection in place). The guidance doesn’t specifically mention blu-rays by name, but it does say “The exception will apply to any copies you have bought, other than computer programs” – which I see to include blu-rays. However, it does also mention that copy protection may still be in place on those formats. Which you can raise a complaint to the secretary of state about if you think it’s too restrictive (good luck getting a response there!).

I’m a big fan of said changes, mainly due to the fact I own a huge amount of discs which aren’t currently allowed to be any format other than that they were distributed in. DVD’s, CD’s and Vinyl(!). I’ve found this frustrating, as my own stance is I should be able to manipulate the media to my hearts content within my own 4 walls. In fact, in order to listen to myself practicing when I used to dj, I need to make recordings which change their format. The fact that I’ll be able to do this within the law makes me a very happy chap indeed. Up until now, the law has been fairly grey on the matter, but hasn’t prosecuted anyone for format conversion (as far as I’m aware).

The new guidance is quick to point out that it’ll still be illegal to distribute the media you create – You have to “own” the media you’re converting. Therefore duplicating a friends mp3 collection at work from their hard drive will still be illegal. If you sell the original format from which you made copies, then your copies will again be illegal. Again this aligns with how I feel things should be done. I feel like I should be paying someone for the entertainment I enjoy from said media. The artist/record labels decide a price and that’s what I should pay – but I don’t feel like they can make me pay multiple times for the same thing, especially if I can create said formats from media I already own. If you hold a copy but haven’t paid for it, I feel it fair for it to be deemed illegal, however possible it is technically to achieve.

In summary, I feel these changes are great (but long overdue).

You can read a full summary of changes to consumers issued by the Intellectual Property Office here. Amusingly, the document explains you may be affected if you “read books, watch films or listen to music” – or my favourite “use electronic devices”. I guess that covers anyone reading this then.

Tagged copyright, law, Music, | Leave a comment

Shipping in March: Weeks 2 & 3

23 Mar 2014

So, anybody paying close attention may have noticed I’ve skipped a week on updating my progress on shipping in March. There wasn’t much to report in week 2 aside from I was busily updating my Nifty Digits site. This hasn’t yet gone live, mainly due to some restrictions on current client projects and publishing about my previous work. I hope to be able to publish it before the end of the month. I spent approximately 3 days on this, 1 on client work and 1 on working on Hourglass.

This past week however I’ve been working on a revamp for Fair & Bare and breaking out the shopping software I built for it into a new product. The reasoning for this is two-fold – we have stock for Fair & Bare we want to rave about and sell and I spent a long time building a shopping cart system that’s only ever been used by me! It makes sense (at least to me) that making this byproduct available (as per the 37signals mantra). Fair & Bare has existed for 7 years now and it’s backend has barely changed in that time and it’s still happily taking orders. This week has been an opportunity to rebuild it in a more robust manner, with more features and without everything that annoys me about current checkout systems.

Granted, it’s currently small and will probably won’t compare in terms of the number of features to cart provider’s that already exist – but what I hope it will provide is a simple, easy to manage way of selling stuff online. I’ve had a huge amount of thoughts on what would be the best way to package it, but have settled on a SaaS based system, rather than self hosted. I’ve been able to use stripe to simplify the way payments are taken and I’m planning on integrating a number of other tools to make life easier for developers.

Again, it’s incomplete and not ready to release – but it is in a place that I’m able to use it as an admin. I’m planning on getting F&B using it and then releasing it on it’s own. If you’d like to see some features then please comment below and let me know.

The one thing I have realised about working on my own projects is the freedom I have to talk about it. No NDAs, no worry of treading on people’s toes, just (hopefully) helpful info. Several blog post ideas have been found over the course of the work I’ve been doing so far this month, one of which has already been published. I’m hoping I’ll finally have something to show you all before the end of this week.

Tagged ecommerce, shopping, time tracking, | Leave a comment

Shipping in March: Week 1

10 Mar 2014

I promised a progress update on how my attempt to get 4 projects shipped this month was going. On my first week, I decided to focus on my time tracking tool, hourglass. I’ve had this in the works for sometime and unfortunately, it’s fallen by the wayside to client work.

I’ve failed on my first week in getting this released to the world, but I’m far happier with how it’s working now and the value it might give to any potential user than previously. I’d had a working app and a shiny new design for it prior to last week, so first on my agenda was to bring that in and start from there. Beyond that, I bug fixed a number of known issues, followed by building out a number of reporting features which I feel are critical to have before release. It’s also allowed me to better think about the decisions I’ve made regarding implementation and whether or not it makes sense to change them at any stage soon.

Unfortunately, over this time I hit a major hurdle of my internet connection being killed due to scheduled maintenance. I had been sent a letter on this, but hadn’t clocked the date. I was lucky that in Cardiff we have a number of excellent co-working facilities that allow me to turn up unannounced and fled home for the sanctuary of Foundershub. In the end I lost probably a couple of hours to this problem and I’m glad I made the decision early on to move before I’d lost the best part of the day.

I’m going to be moving on to my next project this week as planned and move back to hourglass if time allows. I feel that experiencing the variety of these projects will hopefully allow me to continue some kind of momentum. Out of all the ones I have planned, I feel that hourglass is probably one of the larger ones too.

Tagged | Leave a comment

Scaling a Linear Domain to an Ordinal Range with d3

07 Mar 2014

I’ve had the opportunity of using d3 quite a lot over the past few months for a number of clients. It offers some amazing flexibility for chart generation and much more.

Anyway, I thought I’d share a quick tip I developed for mapping a linear set of values onto an ordinal scale. For those who’re seasoned pros at d3, this probably seems trivial, but had me stumped for some time today.

I’d picked out a colour palette I wanted to use for a particular graph, as per below:

var colours = ["#B8D0DE", "#9FC2D6", "#86B4CF", "#73A2BD", "#6792AB"];

The only examples I’ve seen similar to this are where it is assumed you want to vary darkness of colours based on value or vary the domain based on the number of colours you want. Not a good fit.

I wanted to pick one of my values based on a linear value from my data set. My first thought was to make use of the ordinal scale function provided by d3. Something like this:

var colour = d3.scale.ordinal()
  .domain(mySortedDataValues)
  .range(colours);

In doing this, I got something that *looked* a bit like it was working, but not the way I expected. In fact, the way an ordinal scale works is that it provides a 1-to-1 mapping of domain values to the range, rather than any kind of interpolation between them. In this case, it was expecting only 5 distinct data values (to match up against the colours) and for everything over and above that, it wrapped them round to the beginning of the domain again. The solution then is fairly simple once you’ve got your head around that.

What I did next, was to create a scale that gave the index of the colour we were going to be mapping to. This works well, because the indices are linear and d3 has the ability to do the dirty work in that respect.

var colourIndex = d3.scale.linear()
  .domain([0, d3.max(data)])
  .range([0, colours.length - 1]);

svg.selectAll("circle")
  .data(data)
  .enter()
  .append("circle")
  .attr("fill", function(d){
    return colours[Math.ceil(colourIndex(d))];
   });

Here we end up with an index ranging across all the indices of the colour array, and a colour appropriately selected from the palette as expected. You can see the resultant effect in the graph linked to below:

Tagged charting, d3, | Leave a comment

Shipping in March

03 Mar 2014

This month, I’ve decided to take some time out to ship projects.

I have a huge list of projects accumulating I’d like to push out of the door, but as a father and freelancer I have lack of a certain commodity in able to achieve them. That would be time.

Currently I don’t have any work booked in for a while, so along with Viv, we’ve decided I should try and get some of these projects released.

That sounds easy, but I’m well aware from past experience it isn’t. I don’t want to work outside of working hours (9-5.30pm) as I know the impact it has on my family. My hope is by setting myself some achievable goals I can at least get some way to ticking off some checkboxes on this growing todo list.

Some of these projects are products I’ve not yet completed, projects I’ve released but need work or ideas that haven’t made it to the build stage yet. As a freelancer, I hope that some of them may even help out other people in a similar situation to me.

Over the coming month, I’m going to attempt to ship at least 4 projects out of the door. 1 a week for 4 weeks. I’ll be blogging about my progress here, and posting summaries of how it went at the end of each week. The end date of the 28th is also my birthday, so it would be great present to myself to have come some way to meeting my goal.

If you’re interested in receiving progress updates via email, signup here: http://eepurl.com/PxGe9

Tagged projects, | Leave a comment