Splitting a date range in Python

01 Jul 2014

Dates are one of those annoying things that shouldn't be, but are regularly difficult in web apps. I used the following two methods in a recent page to neatly break a date range into distinct segments as part of a analytics app I'm currently working on. Hopefully someone else will find them helpful.

import datetime, calendar

# Find the delta between two dates based on a desired number of ranges
def datedelta(startdate, enddate, no_of_ranges):
    start_epoch = calendar.timegm(startdate.timetuple())
    end_epoch = calendar.timegm(enddate.timetuple())

    date_diff = end_epoch - start_epoch

    step = date_diff / no_of_ranges

    return datetime.timedelta(seconds=step)

date_delta allows me to create the timedeltas between two dates based on a desired number of segments.

# Edit 18/07/2014 - I realised dates needed the hrs, mins an secs correctly
# adjusted to beginning / end of day
def datespan(startdate, enddate, delta=datetime.timedelta(days=1)):
    currentdate = startdate

    while currentdate + delta < enddate:
        todate = (currentdate + delta).replace(hour=23, minute=59,second=59)

        yield currentdate, todate

        currentdate += delta
        currentdate.replace(hour=0, minute=0,second=0)

I can then pass the delta into datespan above, which returns an iterable I can then use.

# Get timedeltas based on splitting the range by 10
delta = date_delta(startdate, enddate, 10)

for from_datetime, to_datetime in datespan(startdate, enddate, delta):
    print from_datetime, to_datetime

The result forms part of the following d3 chart:

Tagged coding, python, | Leave a comment

Tips for being a more Productive Freelancer

16 Jun 2014

Having heard the question raised on @workingoutshow, I thought I'd take some time to share the things that I've found have really helped me be productive in my work. These may not work for everyone, but hopefully some points will help for others.

Find a Good Environment

The single biggest change I've found which has had a major impact on my work is changing my environment. We're lucky enough to have a small garden at our house and with the arrival of my daughter back in October, I was quick to realise that working in the open plan lounge in our house just wasn't a viable option any more. Here in Cardiff, we've been lucky enough to have several co-working spaces open locally, but for me I prefer being able to switch off background noise and clear my head when I really need to knuckle down and focus. Around the same time, we made some major changes to the back of our house and garden which meant that there was space enough for a desk in our sunroom which overlooks it. I now spend 90% of my working time looking out over our little green space, which makes for a welcome break from staring at my screen and gives me a self-contained and more importantly, quiet environment in which to think. I also get to enjoy the space more overlooking the various wildlife that visits during the day. It's funny how good a little bird taking a dust bath every now and again will make you feel.

Organise Your Tasks

I use trello for organising tasks I want to get done, with a board for each project. These are organised in descending priority, using trellos card defaults of "To Do", "Done" and "Doing" to track what I've achieved for a project. If whilst during the course of one task, I realise something needs doing over and above the scope of the task I'm currently working on, I'll add it to my todo list and work on it when all other tasks with a higher priority have been completed. This strategy works well for me in terms of being able to blast through a whole heap of development tasks without having to think too much about what I need to do next.

Plan Your Next Day Before Leaving Your Desk

I like to end my day at a distinct break between tasks. If my day ends at a point where something is incomplete, I'll be sure to keep it in my to-do list for the following day, where I can come back to it. Before I leave my desk each day I'll generally try and plan out what makes sense to cover (given the time I have) in the following working day. This allows me to hit the ground running each day, without any getting up to speed time first thing in the morning.

Track Your Time

Tracking my time is important not only for my client work, but also my own projects. As well as giving me exact totals for invoicing, it allows me to compare estimates against the reality of a project and all the unforeseen problems that may arise during the course of it. My time tracking is a simple project based on/off approach, where I use a custom tool I developed myself.

Eliminate Distractions

Although obvious, trying to cut down on distractions pays dividends. Switch off tweetdeck, close skype and either disable notifications on your phone or (my favourite) put it out of your sight completely. I don't tend to pay any thought to my phone if I can't see it, and it's like a nervous twitch to pick it up and check whether anything has happened in the last 5 mins. When not aware it's an option, I probably get twice as much work done. It's easy to kid yourself that checking devices like this is part of your work, but when at the detriment of your day's schedule you need to switch off and focus on the bigger picture.

These are just a few measures I take to be more productive in my day - I'd welcome any tips other freelancers have for their own situations.

Tagged productivity, | Leave a comment

Bye Bye Wordpress

18 May 2014

I've finally made the move to jekyll for building this site. It is now hosted on github pages, comments are handled by disqus and my post source is written in markdown. My entire site is visible on github and revision control for post entries is handled there too.

What finally convinced me with the release of jekyll 2 was the introduction of native sass support. Originally, the prose theme I'm using for my site was using scss, which I later ported to a wordpress theme in order to blog with it. I had the scss for the original design available and therefore thought the jump to using jekyll wouldn't be that great (or so I thought - overlooking all the features jekyll doesn't support compared to wordpress). I'm going to blog individually about the pieces of the puzzle in other posts as there are number of things that tripped me up along the way.

But for now, I'd like to bid a fond farewell to wordpress - my go to blogging software for the last 10 years. You once were diminutive, got out of my way and let my voice be heard. That's no longer the case. These days the only time we speak is when you want me to update you. I've got better things to do. Theres a ton of features the length of my arm which I've never used and the majority of features which I use rarely. I long for those simpler days once again. You've served me well, but I've moved on.

Tagged jekyll, static sites, wordpress, | Leave a comment

Copyright Law Finally Aligns with Me

07 Apr 2014

On June 1st 2014, some much needed changes are being made to exceptions to copyright law which amongst other things, will allow conversion of personal media between file formats.

Finally, we’ll able to make personal conversions of digital media (at least they will be now counted as legal copies). I’ve long made copies of my cd’s to listen to whilst coding (iTunes allows me to do so simply) – but it is not currently legal to move these files to another storage medium, such as an mp3 player. The fact is that most people have been ripping cds and copying them to iPods since it was technically possible, but music companies have never prosecuted anyone as the implications would stem far and wide and would the costs would too prohibitive to be able to enforce them.

It’s also not currently legal to rip movies to another format. So copying a dvd to a mkv file to play elsewhere is construed illegal. The fact movie studios put protection in place on their dvds to prevent you from doing so puts most people off doing so anyway. From June 1st this also changes so it will also be legal (but probably will be as difficult due to the copy protection in place). The guidance doesn’t specifically mention blu-rays by name, but it does say “The exception will apply to any copies you have bought, other than computer programs” – which I see to include blu-rays. However, it does also mention that copy protection may still be in place on those formats. Which you can raise a complaint to the secretary of state about if you think it’s too restrictive (good luck getting a response there!).

I’m a big fan of said changes, mainly due to the fact I own a huge amount of discs which aren’t currently allowed to be any format other than that they were distributed in. DVD’s, CD’s and Vinyl(!). I’ve found this frustrating, as my own stance is I should be able to manipulate the media to my hearts content within my own 4 walls. In fact, in order to listen to myself practicing when I used to dj, I need to make recordings which change their format. The fact that I’ll be able to do this within the law makes me a very happy chap indeed. Up until now, the law has been fairly grey on the matter, but hasn’t prosecuted anyone for format conversion (as far as I’m aware).

The new guidance is quick to point out that it’ll still be illegal to distribute the media you create – You have to “own” the media you’re converting. Therefore duplicating a friends mp3 collection at work from their hard drive will still be illegal. If you sell the original format from which you made copies, then your copies will again be illegal. Again this aligns with how I feel things should be done. I feel like I should be paying someone for the entertainment I enjoy from said media. The artist/record labels decide a price and that’s what I should pay – but I don’t feel like they can make me pay multiple times for the same thing, especially if I can create said formats from media I already own. If you hold a copy but haven’t paid for it, I feel it fair for it to be deemed illegal, however possible it is technically to achieve.

In summary, I feel these changes are great (but long overdue).

You can read a full summary of changes to consumers issued by the Intellectual Property Office here. Amusingly, the document explains you may be affected if you “read books, watch films or listen to music” – or my favourite “use electronic devices”. I guess that covers anyone reading this then.

Tagged copyright, law, Music, | Leave a comment

Scaling a Linear Domain to an Ordinal Range with d3

07 Mar 2014

I’ve had the opportunity of using d3 quite a lot over the past few months for a number of clients. It offers some amazing flexibility for chart generation and much more.

Anyway, I thought I’d share a quick tip I developed for mapping a linear set of values onto an ordinal scale. For those who’re seasoned pros at d3, this probably seems trivial, but had me stumped for some time today.

I’d picked out a colour palette I wanted to use for a particular graph, as per below:

var colours = ["#B8D0DE", "#9FC2D6", "#86B4CF", "#73A2BD", "#6792AB"];

The only examples I’ve seen similar to this are where it is assumed you want to vary darkness of colours based on value or vary the domain based on the number of colours you want. Not a good fit.

I wanted to pick one of my values based on a linear value from my data set. My first thought was to make use of the ordinal scale function provided by d3. Something like this:

var colour = d3.scale.ordinal()

In doing this, I got something that *looked* a bit like it was working, but not the way I expected. In fact, the way an ordinal scale works is that it provides a 1-to-1 mapping of domain values to the range, rather than any kind of interpolation between them. In this case, it was expecting only 5 distinct data values (to match up against the colours) and for everything over and above that, it wrapped them round to the beginning of the domain again. The solution then is fairly simple once you’ve got your head around that.

What I did next, was to create a scale that gave the index of the colour we were going to be mapping to. This works well, because the indices are linear and d3 has the ability to do the dirty work in that respect.

var colourIndex = d3.scale.linear()
  .domain([0, d3.max(data)])
  .range([0, colours.length - 1]);

  .attr("fill", function(d){
    return colours[Math.ceil(colourIndex(d))];

Here we end up with an index ranging across all the indices of the colour array, and a colour appropriately selected from the palette as expected. You can see the resultant effect in the graph linked to below:

Tagged charting, d3, | Leave a comment