Splitting a date range in Python

01 Jul 2014

Dates are one of those annoying things that shouldn't be, but are regularly difficult in web apps. I used the following two methods in a recent page to neatly break a date range into distinct segments as part of a analytics app I'm currently working on. Hopefully someone else will find them helpful.

import datetime, calendar

# Find the delta between two dates based on a desired number of ranges
def datedelta(startdate, enddate, no_of_ranges):
    start_epoch = calendar.timegm(startdate.timetuple())
    end_epoch = calendar.timegm(enddate.timetuple())

    date_diff = end_epoch - start_epoch

    step = date_diff / no_of_ranges

    return datetime.timedelta(seconds=step)

date_delta allows me to create the timedeltas between two dates based on a desired number of segments.

# Edit 18/07/2014 - I realised dates needed the hrs, mins an secs correctly
# adjusted to beginning / end of day
def datespan(startdate, enddate, delta=datetime.timedelta(days=1)):
    currentdate = startdate

    while currentdate + delta < enddate:
        todate = (currentdate + delta).replace(hour=23, minute=59,second=59)

        yield currentdate, todate

        currentdate += delta
        currentdate.replace(hour=0, minute=0,second=0)

I can then pass the delta into datespan above, which returns an iterable I can then use.

# Get timedeltas based on splitting the range by 10
delta = date_delta(startdate, enddate, 10)

for from_datetime, to_datetime in datespan(startdate, enddate, delta):
    print from_datetime, to_datetime

The result forms part of the following d3 chart:

Tagged coding, python, | Leave a comment

The Developer Dream

27 Jun 2013

I suffer from idea overload. It seems that every day I have an idea for something. As a developer, I’m lucky in that to be able to take an idea to inception very quickly. Best of all, I can show it at inception stage to a huge audience (if I had one), garner feedback and quickly iterate on that idea. To date, I’ve mainly created these by myself – a casualty of doing paid work in the main like this too.

Most of these ideas in the have been simple hacks – solutions to things that bug me. One of them has been a proper business too. I’m able to do something and make it happen, to potentially tap a huge audience. Get them to that first step. The problem I have is, not a single one of these so much as turns a profit, though they serve as wonderful educational devices for me. I would hazard a guess that some of them have indirectly got me work.

Some of my ideas never get past the domain registration phase before being knocked on the head. A friend many years ago gave me the great quote that ‘My domains are my todo list’. This totally hits the nail on the head for me – if I’ve felt like an idea was good enough to make it to the registration phase, there’s something there worth pursuing. Unfortunately, I have quite a long list to work through.

I heard it said that “The Developer Dream” is that we (developers) “build a product/service which we’re able to earn a living from”. But the majority of developers I know, don’t even begin to pursue the ideas they have. This surprises me.

There seems to be such an abundance of opportunities that I can’t imagine not building my own ideas, however bad they might seem to others.

Why hold back? Time? Yes, we have other commitments – family, work and maybe even a hobby or two. I’m not suggesting we make drastic sacrifices or sell our kids in order to create stuff. I’d would however, like to think that the payoff for building our ideas might make our initial holdups seem insignificant longer term. I’ve said it before and I can’t speak from as the voice of successful experience, but what do I get out of never trying?

Tagged hacking, ideas, webdev, | Leave a comment

Growing a Business with Developer APIs

20 Feb 2013

Those of your you who follow me on twitter will might be aware of a single day hack I attempted last week, which takes output from Lovefilm’s customer history and formats it so it can easily be imported into the lovely Letterboxd. It’s named Loveletterd and it seemed entirely appropriate to release it on Valentines day.

After hearing that Letterboxd had created a CSV import tool for it’s community to use and as I’m a Lovefilm developer, I felt like it would be a great use of my time to get to work on Loveletterd.

Originally, I’d thought I’d use Lovefilm’s OAuth sign-on, grab past rentals for a user and create a CSV from it formatting to Letterboxd’s import structure. I’d completely forgotten about a bug I raised on the developer forums; almost 3 years earlier – highlighting that actually this was impossible due to a bug which didn’t read a supplied argument, subsequently limiting the number of rental titles returned as the history. What’s more infuriating is that no one from Lovefilm had replied, even though the problem was reiterated by others in the same and other threads. There’s many different threads which highlight numerous other issues, all of which receive no or very little response from the Lovefilm developer team.

This is an awful way to treat a community excited about building tools with your company data. If you don’t want to cushion the cost of supporting providing such an API yourself, don’t create one, or charge for access to it. Similar sentiment has been said about twitter recently on changing their API rendering it unusable for apps that have based their business models on it. Such is the feeling of discontent that it’s led to a niche for rivals.

Lovefilm’s API “gallery” of developer apps features the message

“We will be placing details of the latest and greatest applications using our API here. Be sure to check this page regularly as new ideas are developed.”

Which is also the same as 3 years ago, shortly after the API’s first release. Good work Lovefilm.

Remembering this, I walked away thinking, “Oh, well”, before going back to my workday. Explaining it to my wife, I realised there was another route to achieve what I wanted, which is what you see on the web now. Ultimately, this is nothing more than a bunch of regular expressions.

Admittedly, I’m aware there’s changes at Lovefilm – they’ve been entirely bought out by Amazon and I can imagine the new owners would want to integrate the company with their brand better. It seems as if they’ve already switched off new developer registrations, so I’d imagine the whole API will be turned off before long.

On the flip, take a look at where I want to take my data to – Letterboxd. A site focused on film lovers and it shows. They’ve taken the route of paid pro accounts, which provide access to the import tools I mentioned earlier. The majority of users won’t make use of this though; they just want to support a site providing a great service which is open and honest. They’re also planning on providing a full API, which I’ll be interested in playing with on release. As a result, I’m sure Letterboxd users will get a bunch of new toys they can use their accounts with that external developers decide to create.

The best thing though? They’re real people who talk to me. After letting them know about my little hack, I got thanks that same day from Matthew of the Letterboxd crew, with a ticket closure shortly thereafter. That beats 3 years of waiting.

Tagged films, hacking, | Leave a comment

The Move Towards Static Websites

22 Jan 2013

There’s some discussion on new podcast “The Back to Front Show” on why of late we’re seeing so many static websites cropping up. It seem like now more than ever, we’re seeing both developers and designers jump ship to this way of working, having long been using database backed solutions (such as WordPress/Drupal) to manage their publishing.

When I talk about a static site, I think of it of raw HTML sat on a server and nothing else. However, it seems as if the industry has adopted the term to refer to sites created from tools which are capable of creating/emulating this HTML, whether it be a local, dropbox or a hosted/self-hosted solution. Typically both types of tool are powered through conversion of flat files into the markup which is served publicly. Through using them you can achieve something akin to a typical blog, without the need for a database (or possibly even a server-side language to power it).

Pro’s

Simplicity – Most of the draw toward toward such setups seems to be on the simplicity over flexibility. I can power a site with a minimal setup, but still regularly and simply update and maintain it, using a partial based environment. With a dropbox based solution, I just need to drop my flat files into a folder and everything is updated transparently. Mark Boulton has previously written about his requirements for writing on the web and why he went with Statamic:

“I wanted a way of simply publishing words, rather than getting all wrapped up in php, theming files and database problems.”

Simplicity it seems, rules.

I previously wrote an article on how you could publish using Octopress and Amazon’s S3 service. For anyone who has a relatively low traffic site, such a setup could also work out cheaper than a monthly hosting package.

Development Time – If you’re one for creating custom themes, a static site is also a much simpler solution than wrestling with styling the innerds of a full fledged cms. Typically posts are stored in markdown and can be styled along with assets using a partial templating system (eg. based on YAML or Liquid). Having recently developed a wordpress theme I can testify to the complexity of having to dig through documentation on function usage and to determine which template file should be modified to achieve what I want within a page.

Source Control – As Rick quite rightly points out below in the comments, having all your posts available in flat files makes them perfect for managing/backing up through your revision control tools of choice.

Cons

Customisation – For me, the big loss is the lack of flexibility. I hear everyone cry, “Well, duh – it’s static”. Without a server side language handling requests, I’m unable to schedule posts, handle comments, create forms or trigger email. If I intend on continuing to offer them, I have to find third party services to deal with each of these aspects. I’m well aware there’s services like disqus and wufoo that may solve some of these problems, but it seems like a major hassle to integrate each time something like this is required. It may be that this is just me, you may be perfectly happy using these solutions, or may not require them.

Remote publishing – When using a local based solution (e.g. Jekyll), you don’t have the option of blogging remotely, due to the lack of tools to create and push out your content. Gone are the days of being able to log into a site from your phone and add a small update. For me, this is less of an issue as I can’t recall if I’ve ever written a post in this way and try to be more methodical, less spontaneous about what I’m writing from the comfort of my desktop.

Example Systems

If you’re interested in running a static site, then I’ve listed a few that seem pretty popular at the time of writing, along with a brief list of their features.

Jekyll – Locally run, File based, Open source, Powers Github pages.
Octopress – Locally run, Jekyll/File based, but with preconfigured HTML templates, CSS, Javascript, Open Source
Statamic – Self Hosted, File Based, License Required
Kirby – Self Hosted, File based, jQuery inspired API, License Required
Stacey – Self hosted, File based, Open source
Scriptogram – Dropbox/File based

Tagged blogging, blogs, | Leave a comment

If You Build It

03 May 2012

If my years in academia have taught me anything, it’s that insight obvious to you isn’t often obvious to others.

As a result, I’ve pretty much adopted a gung-ho hacker attitude towards getting my own ideas built whereby I’ll just often work to get an iteration of an site out as fast as possible. The only factor here is my own time and I’m in the fortunate position to have some of that available now and again. There’s a variety of reasons why I might attempt a project, ranging from it may later evolve into something able to draw an income further down the line, that I’m excited by learning from the tools I choose to use or that I feel moved enough by a problem to do something to try and prompt change.

I had one of these ideas last weekend, whereby I was wondering what the specifics of the fees of kickstarter projects broke down to. The information on what fees are taken is published on their site, so I set about a simple phantomjs script (available on github) that could scrape the information from kickstarter. After doing so, the idea moved on to graphing said information via an app to a simple raphaeljs and again further once I’d realised some other stats that could be gathered from the info I was grabbing. The end result is Starter Stats – which is, (I think) a neat little site for showing stats on any kickstarter project.

It took a little longer than I’d hoped, but I’m really pleased with the final site as it stands. You should also be able to just replace the beginning of any kickstarter project page (http://kickstarter.com) with “http://starterstats.com&#8221; and you’ll be sent to the relevant stats page on the app. Pages are generated on-demand every 15 minutes. Go on, take a look and let me know what you think.

I could have not spent my time on this, instead kicking back and waiting for someone else to build such a tool. But I couldn’t see that happening. No, better for me to build, learn whilst doing so and move on to my next project. What do you get by not building?

Tagged kickstarter, phantomjs, raphaeljs, | Leave a comment