If You Build It

Posted on 3/5/2012 by Ian

If my years in academia have taught me anything, it’s that insight obvious to you isn’t often obvious to others.

As a result, I’ve pretty much adopted a gung-ho hacker attitude towards getting my own ideas built whereby I’ll just often work to get an iteration of an site out as fast as possible. The only factor here is my own time and I’m in the fortunate position to have some of that available now and again. There’s a variety of reasons why I might attempt a project, ranging from it may later evolve into something able to draw an income further down the line, that I’m excited by learning from the tools I choose to use or that I feel moved enough by a problem to do something to try and prompt change.

I had one of these ideas last weekend, whereby I was wondering what the specifics of the fees of kickstarter projects broke down to. The information on what fees are taken is published on their site, so I set about a simple phantomjs script (available on github) that could scrape the information from kickstarter. After doing so, the idea moved on to graphing said information via an app to a simple raphaeljs and again further once I’d realised some other stats that could be gathered from the info I was grabbing. The end result is Starter Stats – which is, (I think) a neat little site for showing stats on any kickstarter project.

It took a little longer than I’d hoped, but I’m really pleased with the final site as it stands. You should also be able to just replace the beginning of any kickstarter project page (http://kickstarter.com) with “http://starterstats.com” and you’ll be sent to the relevant stats page on the app. Pages are generated on-demand every 15 minutes. Go on, take a look and let me know what you think.

I could have not spent my time on this, instead kicking back and waiting for someone else to build such a tool. But I couldn’t see that happening. No, better for me to build, learn whilst doing so and move on to my next project. What do you get by not building?

Posted in Web Apps | Tagged , , | Leave a comment

How to Switch Your VPS in 2 Hours

Posted on 1/3/2012 by Ian

Friday last week I woke up to a barrage of emails coming into my inbox. All of these were coming from notification of spam comments upon this very blog. By the time I’d woken up I’d had something like 75 emails. This was pretty strange given that Akismet was there guarding my blog ever since it’s been available and I’d never had a email about a dodgy comment since I’d set it up (something like 5 years ago). They continued to flood in during my breakfast as fast as I could delete them.

The problem stemmed from the fact that network connectivity from my slicehost account had been knocked out of action – which had never happened before. I presumed this was down to the migration of slicehost over to Rackspace infrastructure, which I’d been informed of, but the network dying was probably unintended.

Either way, it was frustrating and I instantly decided to finally migrate away from slicehost and setup a second linode account. The funny thing is, that most of this was aided by slicehosts own documentation on configuring your server. If you’re doing little more than hosting a number of mysql / php driven sites, it is in fact really simple to switch. Here’s how I went about it, my sites all experience fairly low traffic, so I’m aware this solution might not be right for everyone.

Backup All Your Important Data

First of all, you’re going to need to backup everything you have in your sites directory. For me, that’s everything under /var/www.

tar -zcvf backup-feb-2012.tar.gz /var/www

Next, you’ll want to do the same for your Apache sites:

tar -zcvf backup-apache-feb-2012.tar.gz /etc/apache2/sites-available

Finally, I took a mysqldump of my entire mysql setup:

mysqldump —-all-databases -u root -p password | gzip >dbbackup-feb-2012.sql.gz

Setup Your New VPS

Having done that, you have pretty much everything you need to bring your server back to life as it is now. Go and grab a new vps from linode or somewhere else and set it up using the details starting from here: http://articles.slicehost.com/2010/10/18/ubuntu-maverick-setup-part-1 I configured my vps to have exactly the same accounts and passwords as my original vps, to make life easier for me. Once SSH and your firewall has been setup and you’re all updated, you’ll be able to transfer all the files you just created onto your fresh vps setup, using filezilla, or a direct connection between the two machines. You’ll then need to install Apache: http://articles.slicehost.com/2010/5/19/installing-apache-on-ubuntu, PHP: http://articles.slicehost.com/2010/5/19/installing-php-on-ubuntu and MySQL: http://articles.slicehost.com/2011/3/10/installing-mysql-server-on-ubuntu

On your new machine, you’ll need to restore the files you backed up to /var/www.

tar -xzf backup-feb-2012.tar.gz
sudo cp -r var/www/* /var/www/
sudo chown -R www-data:www-data /var/www/*
rm -rf ~/var

Then the apache site files:

tar -xzf backup-apache-feb-2012.tar.gz
sudo cp etc/apache2/sites-available/* /etc/apache2/sites-available/

For every file that exists in the sites directory and is to be enabled, we need to enable it. You can use a2ensite for this if you like.

cd /etc/apache2/sites-enabled
sudo ln -s ../sites-available/ianwootten.co.uk.conf ianwootten.co.uk.conf (for each site file)

Install any apache modules you might require and restart apache. If you’re using wordpress or a php framework like me, you’ll most likely need mod_rewrite.

sudo a2enmod rewrite
sudo /etc/init.d/apache2 restart

You’ll then need to repopulate your new mysql install with all your old data.

gunzip -c databasebackup.sql.gz | mysql -u root -p

Configure Email

I use msmtp to handle all my email on my server, which sends through a single gmail automailer address for my domain. If you want to send email that way do the following:

sudo apt-get install msmtp ca-certificates
sudo nano /etc/msmtprc

Paste the following setup in the file:

defaults
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt

account default
host smtp.gmail.com
port 587
auto_from off
auth on
user automailer@somedomain.com
password mymailpass
from automailer@somedomain.com
maildomain somedomain.com
logfile /var/log/msmtp.log

sudo chmod 0644 /etc/msmtprc

You can test whether it’s working using a the command as:

echo -e "Subject: A Test Email\r\n\r\nThis is a test" |msmtp --debug --from=default -t mymail@gmail.com

Configure DNS

You need to configure how your vps is going to handle particular requests for various domains. You’ve probably had to do this already with your current vps, so just copy the entries through whatever management system your new host provides. For sites that have a fairly simple setup, your host may provide a simple set of default entries to use, or you can clone entries that have the same setup as each other.

We’re now at the point where we have a complete copy of our vps up and running, and the vps knows how to handle requests for domains, however currently our domain registrar points to our old vps, so the requests will never reach our new server. Head to your domain registrar and change the entries for a single domain, to point to your new vps (ns1.linode.com, ns2.linode.com etc). Now sit and wait patiently….Once that domain has propagated correctly (ping it to see if has a new or old ip), and you’re happy your site is working as expected, go ahead and change all of your domains dns entries to the ip address of your new vps.

Relax…

All in all, I don’t think this took me more than a couple of hours working with the terminal open. I certainly didn’t worry about migrating in the way I have done previously. As I opted for a uk datacentre with linode, a side effect of the change is all my sites are much much faster – and a better setup with linode is in fact cheaper than slicehost. I left my slicehost account running for an entire week prior to shutting it down yesterday – just in case…

Posted in Tutorials | Tagged , , , , , | Leave a comment

Where’s my Full Stack Payment Service?

Posted on 16/2/2012 by Ian

Unless you’ve been living under a rock, you’ll realise that there’s an uprising against paypal amongst developers. Most of this stems from the multitude of crazy requirements they put upon sellers and the locks they can put upon your account without notice.

This in part has led to a number of payment alternatives cropping up. Here in the UK for instance, you have Google checkout and checkout by Amazon pitching up against paypal. None of these solutions are ideal though if you’re looking to make say, a service that bills on a monthly basis.

Enter the subscription billing tools – there’s recurly, Chargify and Saasy, to name 3. All of these go some way to ease the management of customer subscriptions. This is great if you have an established banking history as a business and are able to get hold of a merchant account. Not so great for me – an independent developer with neither of those.

Enter Stripe, a full-stack payment processor for developers which has no requirement on merchant accounts or gateways. Integration with your own website is as simple as pasting in some javascript. Sounds great…..Except I need to be in the US to use it. *Facepalm*. I even had beta access to stripe in it’s infancy and just had to smile at my invitation code.

Recently here in the UK, Go Cardless has appeared with an interesting model, based around the use of direct debits to set up subscriptions and make payments. That strikes me as a little weird from a customer point of view. I’m ok paying my gas by direct debit, but would I be ok paying for a piece of software? Given it’s unusual for a website to bill this way, I’d argue most people wouldn’t be.

Unfortunately for us independent developers in the UK, Go Cardless really is our best option until Stripe or some other full stack offering lands here. I know Stripe are supposedly working hard on doing so, but payments are at an incredibly frustrating point here right now.

Posted in Software, Web | Tagged , , , | Leave a comment

New Adventures 2012 – Self Help for Web Designers

Posted on 23/1/2012 by Ian

Last week, like many others working on the web, I took the trip to Nottingham for New Adventures 2012. I’d seen the wonderful praise dished out for it last year and felt like I wanted to see what it was all about first hand.

I’ve learnt from experience that the most important part of any conference is not the talks or speakers involved, but the community around it and therefore I headed up the night before to ensure I didn’t miss a thing.

Colly and his team had done a lovely job of organising enough to keep us busy what with bowling, screenprinting, fr00tball, the after party and of course the main event.

Despite not actually branding myself as a web designer, it was great to feel welcomed and able to chat one on one with others about our day to day work. Dan Mall’s talked helped somewhat here, forcing introductions with everyone around me. I’m pretty much work in solitary confinement as a developer and to discuss the thoughts and ideas I have with other like minded creatives is always welcome.

During a dinner following the conference, another dev commented that the most of the presentations weren’t specific to web design. They were in fact so abstract, that it was more a “Self Help Class for Web Designers”. I pretty much agree with this. But that’s with a caveat – what else do you attend these presentations for? NA Conf is there to share, inspire and enable designers to be better designers. I want to leave giddy eyed with possibilities. Remember there are workshops too for specifics…

After the whole conference, I feel like there’s a whole lot more I could do to better myself at my craft. Hopefully over the course of the year I’ll do something about it. Most of the notes I made are to this effect too and I’m sure I’ll take away something different each time I look at them.

There’s only so much that can be covered in 45 mins, and for me I want to see a speaker paint me the big picture. After NAConf, I’ve got a whole bunch of big pictures to get through.

Posted in Design, Web Apps | Tagged , , , | Leave a comment

Goodbye 1 & 1

Posted on 12/1/2012 by Ian

I just sent the following email to 1 & 1′s billing department. I’ve been with them for coming up to 7 years now and they were, when I first used them, the only registraar I knew about and I was really happy.

As the number of domains I own has grown, their account admin has become really totally useless to manage them all. I recieved a payment request out of the blue today and it tipped me over the edge.

> We contacted you regarding a declined payment or an expired credit/debit card and have not yet heard back from you.
> Our records indicate that there is an unpaid balance in your 1&1 customer account no.: K####### as listed below.

Firstly – No you didn’t.

Secondly – It would be far more useful telling me which domain name in my account a payment demand is in reference to rather than my account number, as I have more than 1 domain with you.

Thirdly – The steps you’ve listed here to find my payment details miss steps and references links that don’t exist.

Fourthly – Threatening action when you haven’t actually sent a primary payment request email is rude, forceful and obnoxious.

> Please be aware that if this matter is not resolved, you may be sent to a debt collection agency. If the account is sent to a debt collection agency a £15.00 > charge will be added to the unpaid balance.

I’ve decided to take all of my business elsewhere at the earliest opportunity. I’d hope you’d take heed of these comments and pay your other longstanding customers the respect they deserve.

Posted in Computers and Internet, General | Tagged , | 4 Comments

Pagination in CouchDB Apps

Posted on 2/11/2011 by Ian

I’ve been working on some fun little node.js / couchdb projects of late. Given the fact I don’t use either as part of my work, I’ve spent some downtime experimenting and slowly iterating my approaches as I learn best practice.

I hit what I consider to be a fairly frustrating hurdle that couchdb threw up that I’ve been blissfully unaware of through all my couchdb dev. When it came to doing pagination it turns out I’ve always been doing it the “bad” way. Oh, well that’s upsetting.

The Wrong Way

My “slow” approach has always been to take the page no as a argument in the url, generating “skip” and “limit” variables to be used as parameters to my store. So for example, if I wanted to have the 2nd page of my app showing 10 items: 

var skip = (pageno==1) ? (0) : ((pageno-1) * 10);
curl -X GET http://127.0.0.1:5984/stuff/_design/stuff/_view/by-name?skip=10&limit=10

It turns out, that although you might think you’re starting at a particular result, CouchDB still starts at the first result, due to the way the view is created from the b-tree index, couchDB just surpresses the results you skip. This isn’t good news when you’re trying to skip say, 10000 results.

The Suggested Way

The suggested solution is to perform requests and instead of using a “skip” parameter, keep track of the startkey at which the next page begins. This is possible, by requesting a page 1 item longer than that of the number of items on a page and using the key of the result in any requests. So now, for a first page my query is: 

curl -X GET http://127.0.0.1:5984/stuff/_design/stuff/_view/by-name?limit=11

Returning something like: 

{"total_rows":17,"offset":0,"rows":[
  {"id":"8177bf155b952652129836a5d354b30e","key":"Ian Wootten","value":null},
  {"id":"bae2c490c70480aec7096d79e1e3bfc3","key":"Isambard Kingdom Brunel","value":null},
  {"id":"eaae74cfbe5cd13ea6b50dfd090827ca","key":"Christopher Columbus","value":null},
  {"id":"491e68b08d73256f060ebf4b8e063e1c","key":"Elizabeth Fry","value":null},
  {"id":"b45d8a7b9edee9ca66ac0860196f4504","key":"Edward Jenner","value":null},
  {"id":"8a4d3f46885701ffcc7532aeac7a5ae9","key":"Florence Nightingale","value":null},
  {"id":"71e6534c17429eca2cd9450cfc95c6bb","key":"Samuel Pepys","value":null},
  {"id":"6cbad847f0ae959b281b471a72d60587","key":"Pocahontas","value":null},
  {"id":"e5b026ec5c92c20f1575a2901defe14e","key":"Mary Seacole","value":null},
  {"id":"84a371a7b8414237fad1b6aaf68cd16a","key":"George Stephenson","value":null},
  {"id":"321aeb36e20d62660eb0d03c9fcd27b2","key":"Joe Bloggs","value":null}
]}

From the 11th returned result, I have the key “Joe Bloggs” – which can be used as a startkey arg to couch to obtain my second page. If we have duplicate keys, it is also neccessary to keep tabs on the last document’s id and supply as a startkey_docid arg in order to correctly page through everything.

What personally I dislike about the suggested approach, is the inability to create simple requests to arbritrary pages, even with low numbers. We always need to follow a path of links from the first page in order to view particular results. CouchDB’s response is “Not even Google is doing that!”, which is kind of weak to me. I want nice clean urls ala myapp.com/page/2 or myapp.com?page=2.

In fact, such a suggested approach only really allows us to have a single “more” type link in order to fetch results. Passing a startkey as part of a url param eg /page/321aeb36e20d62660eb0d03c9fcd27b2 just sounds (and looks) plain nasty and isn’t very good from a UX point of view for any users we may have.

At the moment, clean tangible page urls (the right way) are only possible using custom middleware. I’ve yet to find anything suitable for node.js. I intend to investigate how to cache document keys for low numbered pages as a separate db in order to produce a solution for my current project and I hope to write a later post detailing how I’ve got on.

Posted in Code, Web, Web Apps | Tagged , , | Leave a comment

Hosting an Octopress Blog on Amazon S3

Posted on 9/9/2011 by Ian

Octopress is a framework for blogging based on the static site generator jekyll. In short jekyll takes markdown and turns it into blog style html ready to be served straight away with Apache whilst Octopress dresses it up nicely with HTML5, responsive layout goodness and gives you a bunch of options for code formatting and the like. A perfect little blogging setup for hackers for those of us who typically won’t need all the bells and whistles on our blog I’m sure you’ll agree.

Using Octopress, you’d typically write your posts in markdown and process them locally. You’d then have a number of HTML pages to upload to your web server, which would appear no different than any other blog in terms of functionality.

The most interesting thing that coincincides with the creation of jekyll is that it is now possible to serve static websites straight from Amazon S3, meaning that it isn’t neccessary to have a web server to serve a Octopress blog. I’ve been playing a little and have been able to serve my own octopress blog which you can see over at www.ianwootten.com. Here’s how to do it:

Configure an S3 Bucket to Act as a Website

NB: If you want to use your own domain, then you’re going to need to create an S3 bucket from the AWS Console named the same as your domain (so for me I created a bucket called “www.ianwootten.com”.

To configure your bucket as a website, from the AWS Console, select the new bucket and click “Properties”. Choose the “Website” tab from the box that appears and check “enabled” and set the Index document as “index.html”. Click “Save”. Now move to the “permissions” tab and click “Add bucket policy”. Enter a policy as below, with “www.ianwootten.com” replaced with the name of the bucket you’re using.


{
"Version": "2008-10-17",
"Id": "",
"Statement": [
{
"Sid": "PublicReadForGetBucketObjects",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::www.ianwootten.com/*"
}
]
}

If you’ve opted not to use your own domain, then that’s it for S3. Your new bucket website will be available at a combination of the bucket name and the S3 storage location. In my case, this is http://www.ianwootten.com.s3-website-us-east-1.amazonaws.com/. There’s a list of endpoints on the “website endpoints” page in Amazons S3 website docs.

Configure DNS

If you’re wanting to use your own domain, then you’ll need to visit your domain registraar and edit some DNS entries. In my case, I added a CNAME entry “www” pointing to www.ianwootten.com.s3-website-us-east-1.amazonaws.com. You’ll need to use a tool like wwwizer in order to forward from the root of your domain to the www subdomain. e.g. “ianwootten.com” to “www.ianwootten.com”. To get that working, you’ll also need to modify the @ A record and point it to 174.129.25.170. Read this serverfault post if you’re interested to know why.

For more info on CNAME configuration and S3 see Amazons S3 website docs (Specifically the notes on virtual hosting).

Download Octopress and write some Markdown

Having successfully configured Amazon S3 to host your blog, you need to tackle the age old problem of writing some content for it. This was particularly hard for me, given my lack of knowledge about markdown! You may like to take a look at John Grubers explanation of markdown if you’re suffering like me. The Octopress blogging basics gives a good overview of how your new local blogging workflow will work. Essentially new posts are written in the ./_source/posts folder and when rake generate is executed, your entire websites content is output to the ./public folder. You can also run rake preview in order to preview on a local server.

TO publish your blog, it’s merely a matter of transferring your /public content to the S3 bucket you’ve generated.

Having done all that you’ve now no need to run your own server. You’re still dependent on those that operate at S3, wwwizer and your domain registraar, but you’ve removed your own from the equation. You may be interested in checking out Jerome Bernard’s tip on easily deploying Octopress blogs with s3cmd.

Posted in Computers and Internet, Geek | Tagged , , , , , , | 6 Comments

Lovefilm Filmstream Plugin v0.2

Posted on 20/7/2011 by Ian

Just a quick note to say I’ve updated the plugin that generates my filmstream pages. It’s become obvious that the Lovefilm API only exposes the last 10 features that you’ve rented in their “at home” list. This isn’t that great if you’ve been renting more than a few months. This is the same as the information you’d see in your account when logged in.

I’ve made a simple modification to use the actual ids that Lovefilm issue for films, rather than a autoincrement value. This way, there’s no need to empty what’s currently stored when updating and we capture the users history of rentals…

You can grab it over here

Posted in Code | Tagged , , | Leave a comment

Participate, Don’t Preach

Posted on 14/7/2011 by Ian

Last week, tickets went on sale for dConstruct 2011 and sold out within an impressive 7 hours. Having attended in 2010 and 2008 and judging it to be filled with the highest calibre web celebs from across the globe, I coughed up the entrance fee as part of the excited gold-rush. The cost of £150 includes access to the pre and after parties and the conference itself.

These type of events are stuffed to the brim with amazing, intelligent web professionals, many of whom I would consider to be colleagues. I’ve followed their progress, listened to them speak, bought their publications and hacked on code with them at many other previous events.

It’s often quoted within tech circles that the best parts of these events are not the talks themselves, but the networking parties that surround them. I really struggle in these environments, nervously trying to find a crowd I feel comfortable chatting with, introducing myself and trying to strike up some cutting edge debate on the topics of the day. I’m a developer who usually lurks behind a computer screen for 8 hours a day, so why wouldn’t I find it awkward? It feels like uncharted territory every time I find myself there. Although meeting some great, interesting people, I have ultimately never forged any new lasting relations in this way, where the interaction is somewhat forced and artificial. Maybe it’s just me, but given the com mon nature of everyones work, you’d think it might be easier. As a friend once observed – it’s like everyone turns up to sell their product, but there are no buyers.

It’s also great to be able to chat within the inner circle of the celebrities who attend. You’ve heard from them all year round and now have a brief opportunity to chat. I guess really I’d love to bounce questions and thoughts off of these guys all evening, but often I get the feeling I’m being evaluated against the rest of the attendees in the room in terms of worthiness. Until you’re edging
into such circles yourself, I’m not entirely sure what they would get from these conversations either.

In fact, what I have found is that my most fruitful and long lasting relations have been founded at events when in a participatory nature. Most of these have been Hack Days (where you’re invited to quickly hack together software as an exemplar of new apis from a number of vendors) and some of them have been at Bar Camps (where all attendees have to speak on a subject close to their heart). I feel closer to these people than I could ever do with someone I spoke with briefly in a noisy bar. I’ll also remember faces and names of people even if I never had the chance to interact with them.

This year, a new variety of geek meet has arisen, which flip these more traditional events on their heads and instead focus on providing environments that focus on “the best bits”. They allow everyone to participate on level pegging rather than follow a traditional conference structure. Earlier this month there was Geek Karting, The Insites Tour is kicking off next week and later this year there’s Activate. I really like the idea of these as a way of sparking interesting relationships and I’m hopefully going to head to Insites next week to see what Elliot and Keir have been able to cook up. I’d hope to see more of these unusual types of event and ultimately participate in more of them.

I still feel that conferences have their place, otherwise I wouldn’t have coughed up the fee for dConstruct. I enjoy hearing what these illustrious speakers have to say. That part of the event is still important for sparking debate and learning from for months and years afterward. However, I just don’t find surrounding parties as valuable to me as I first would have thought.

Posted in Events | Tagged , , , | 1 Comment

Maintaining References to Sockets with Express and Socket.io

Posted on 4/7/2011 by Ian

I hit a frustrating problem when trying to use expressjs alongside socket.io recently. The solution may seem somewhat trivial, but I struggled with it for a while, until finally asking for help on the socket.io irc channel. I’m not sure if a better solution exists for my case, but this is what I have for now.

In the main route of my express app, I needed to iterate across an array, performing a REST call using each element as a parameter and spit the results back for socket.io to send out to the browser. To get some idea of what that would look like, imagine the following.

app.get('/', function(req, res){
for(var i=0; i < someArray.length; i++){
 var someVariable = doSomething(function(err, results){
   socket.emit('msg', results);
});
}
});

My problem then became, how do I emit in this way, given my use case? All examples I could find on the socket.io homepages demonstrate emitting msgs on connection, like this:

io.sockets.on('connection', function (socket) {
  // send custom events to browser socket
  socket.emit('hi', { 'i can send': 'json!' });
});

But I wanted to emit messages as part of a loop, within an app route. How would I do that? I could wrap the loop with the connection logic:

app.get('/', function(req, res){
io.sockets.on('connection', function (socket) {
for(var i=0; i < someArray.length; i++){
var someVariable = doSomething(function(err, results){
   socket.emit('msg', results);
});
}
});
});

But if I do this, then my connection listener would be added on every refresh of the route. This would mean I'd end up with duplicate binding of event listeners and a load more results messages would be emitted than I'd expect. This would result in crazy behavior for any user who might choose to refresh your app.

Actually, the solution is rather simple although not entirely obvious given the nature of using node. If you maintain a reference to the socket as a global variable and then later emit upon that same socket, like so:

var global_socket;

io.sockets.on('connection', function (socket) {
  global_socket = socket;
});

app.get('/', function(req, res){
for(var i=0; i < someArray.length; i++){
 var someVariable = doSomething(function(err, results){
   global_socket.emit('msg', results);
});
}
});

This way, a connection will be established and maintained and we won't get duplicate bindings when refreshing the route of the app. Simple when you know how. I had a rather interesting discussion following this on irc about whether the use of express and socket.io together is warranted, given that socket.io supports use of variable passing and ultimately could entirely replace an express app.

Posted in General | Tagged , , , , | 7 Comments