Meta
Nouse on the Mobile
Phew… production days are busy! Thanks to everyone for keeping the discussion lively here on the web, and keep it up!
At time of writing I’ve still got a few things to finish off, like adding author photos for our beautiful writers, but am pleasantly interrupted from all this by an email from Nouse Web Developer Andrew Fairbairn.
It’s done. Culminating months of hard work and overcoming of technical hurdles by Andrew, I’m incredibly excited to introduce to you Nouse’s very own mobile website!
The new site can be found at m.nouse.co.uk and will work on both computer and mobile – though obviously it’s been designed and optimised for the mobile.
We hope you like it and find it useful, but if you happen to cast a critique-al eye at it and find something amiss, we have surely obviated the need for a frown to cross your face.
In other words, please send your feedback to mobile@nouse.co.uk 
Although we could automate feedback by writing programs to send ourselves mail, we’ve tried it and it’s not quite the same, so we gladly await yours instead.
As you can tell from the paragraph(s) above, I’ve gradually slipped back into nerd mode, and so I’ll take that as my cue to stop talking.
Feel free to end here with a massive grin on your face and, of course, go on the mobile site!
Passing on the baton
It’s all over. 4 years at York, and so much has changed in that 4 years, and so much will continue to change. As I was leaving the YSTV studio on Friday for the last time, after signing out my last ever key to the Nouse office, the workmen were putting up a sign in the Goodricke porters’ lodge: “Department of Mathematics. James College”. When Hes East is complete, the Uni will be unrecognisable to me if I ever come up and visit, my department won’t be next to the library any more, and Nouse might have moved into bigger offices in Langwith along with YUSU (here’s hoping). And yet, I will look back fondly on those things I remember which even people now don’t – living in Alcuin when it was a building site, the fountain between Vanbrugh and Physics, being able to quickly hop between Alcuin and Langwith without having to run down a massive hill and cross a road.
But I digress. I’m very nostalgic and sad about leaving, but I know I’ve got an awesome team to take my place. With the difficulties of holding by-elections over the summer, and with me technically no longer a student and no longer able to hold an officership in Nouse, Ali “Fantasy Football” Clark is Acting Technical Director until next term’s elections – and with the huge success of Fantasy Football and winning the technical achievement YUMA, Nouse.co.uk will surely continue to grow on to bigger and better things.
I’m not completely walking away just yet though – I have plenty of time this summer, and a few ideas I’d like to see in motion (including one suggested by a very drunk deputy politics editor in Ziggy’s back in autumn term) will come to fruition by next year.
Looking back over the year it’s staggering to see how far we’ve come: collaborating with other media societies, a redesigned website with a minisite for every major event, and according to Google Analytics an increase of 79% in visits to the site.
Now, this may be “journaljism”, blowing my own trumpet, but I’m pretty pleased with my achievements, but of course I couldn’t have done it by myself. Henry, for understanding the importance of the website and making sure it gets the right amount of emphasis, Jim, for putting up with my constant “This might be an interesting story” phone calls and e-mails, Mike Tomasello, Ali Clark and Alex Muller for turning up to and putting up with my 12 hour hackathons to get those new minisites up and running, Jenny O’Mahony for teaching my how the Nouse site worked when I took over, Tom Brearley for helping rebuild the Nouse network, and for everyone in the office who suffered when the part I tried to do without Tom’s help collapsed, Anna for being incredibly patient with all of those videos, and Matthew for those 2am discussions on the landing. And of course, Lida, Beth and Emily for editing all those pieces that people sent to go online, and to be there every Tuesday when everyone else was relaxing for producing the paper whilst we had our own copy, paste and resize image frenzy to get everything online for people to comment on. All of the writers, many of whom I never met, for actually getting the important stuff, content, and of course, all of our commenters. Some comments made me laugh (one of my favourite ones was left by someone who used the e-mail address “youknowwhothisis@northwood.com” in the comment field), some made me cry, but most were good and important discussion. I like to think that some of the issues brought up by our articles and the subsequent discussion actually led to real change.
Now that’s over, I can cover the thing I actually wanted to write about: Every term I sent out a “top 10″ to the Nouse mailing list covering what the most read articles of the term are. Mainly because I find these kind of stats interesting, but I know some of you do too.
So here’s the top 10 of this entire year:
10. JSoc accuse University staff member of anti-Semitism – I personally was surprised that this was the only article where Nouse was accused of being irresponsible to appear in the top 10, but our first article in this countdown was a highly controversial article from February following accusations made by JSoc against a staff member causing an absolute flood of comments and 2700 views.
9. Friday’s coverage of Roses – the smallest day of Roses takes quite a low position in the top 10 with only 2730 views.
8. The sad news of Ron Weir passing away – it’s no surprise such a moving story as this appears so high, with both current students, staff and alumni adding their consolations to their article.
7. Police search student accommodation – one of the biggest news stories of the year, and it happened outside of term time too, meaning Nouse.co.uk was the only site to cover it, bringing us 2830 views.
6. York’s sex trade – probably slightly surprising to see a feature from 2007 this high, but further investigation reveals the reason. Nouse has a very strong Google presence (incoming links from national papers, other blogs, etc all make it), which actually makes this article second on Google for “sex in York”. What can we say, sex sells – 2910 views this year in fact.
5. Sunday at Roses – no surprise to see the second biggest day of Roses take number 5 with 2930 views.
4. The gender-neutral toilets UGM – covering a very controversial topic and an absolutely staggering number of comments, this is also the most commented piece on the entire website, 173 comments and 3180 views.
3. The liveblog of Roses Saturday – another unsurprising one here, the biggest day of Roses takes it place at number 3 with 3950 views.
2. The sports team’s interview with John Motson – largely helped by a huge amount of incoming links from major forums and the “blogosphere”, this tallied up nearly 4780 views.
1. YUSU elections live blog – I guess this is no real surprise, it’s the biggest single night of the year for campus media, but there it is – at over 9000 views.
And that’s it. Bye.
Downtime
As any webmaster will know, downtime is evil. It’s to be avoided at all costs. When it happens, you look bad, you lose visitors, and if you’re a commercial entity, advertising revenue.
The Nouse site was down for a few hours around midday today, and although as much I hope to be able to say “it’s not my fault” and blame external factors, it was. Problems I did not foresee during what should have been a lightweight procedure essentially crippled our MySQL server, until I figured out the problem and solved it.
Last night I started an overhaul of our databases. Due to a legacy issue with WordPress, our database stored UTF-8 text, but it was identified as Latin-1. This caused a huge number of internal problems – backups are double encoded UTF-8 (which is horrible, every quote is replaced with three characters) and the databases actually held incorrect data, which made doing anything outside of WordPress complex – MySQL would convert any UTF-8 data incoming to Latin-1, despite us actually wanting UTF-8.
The solution was deceptively simple, and thanks to blogger Alex King, I solved this last night.
The motivation to solve this is that this character encoding issue was blocking me from introducing something I wanted to do, and mentioned a few entries ago – migrating the original Nouse site over to WordPress.
It’s this migration that took the site down today. Our central “wp_posts” table hits 213 MB in size, with the vast majority of that being fulltext search indexes for the search function and the related posts plugin. It’s these indexes that started to kill the site. An INSERT was taking up to 2 minutes, which was fine on the wp_posts table in our testing site because nothing else was trying to read that at the same time, but what I did not realise was when I ran the operation live on the server, it would start blocking normal wp_post reads, tying up Apache processes and quickly hitting the connection limit.
It took me quite a long time to realise the cause of the long inserts. Once I did, a simple ALTER TABLE wp_posts DISABLE KEYS solved it, re-enabling at the end to enable the fulltext searching once again. This change was dramatic, instead of running for an hour and only getting a sixth of the way through the conversion, by the time I had switched back to the other terminal, the job had completed.
So, I apologise for the downtime earlier – I had a few people e-mail me to complain about the site being down, but think of the silver lining on the cloud – we now have an additional 800 or so articles in our archive, stretching back to 2003.
The articles back then are written in a different style, less “broadsheet”, more of an “us against them” mentality (Nouse wasn’t redesigned into it’s current form until 2005 – look at the old PDFs for comparison!), but for people like me who like seeing the way things were and with posterity, they’re fascinating.
Lies, damn lies and statistics
Okay, so it’s been quite some time since my last blog, but I’ve got a very good excuse – I’m a finalist, and essay deadlines and looming exams have got in the way of doing Nouse things.
Nothing too exciting has happened technically with the website since Roses, with the exception of the EU Elections tab, which was quite a simple job with HTML and then abstracting some of the lightbox code from the YUSU Elections minisite to make it reusable.
One thing I’ve talked about in the past with non-Nouse people is how we monitor things, both the Nouse server, and the website itself.
The website itself has nothing particularly fancy, Google Analytics is the workhorse and what we use as our official statistics, but we also have AWStats running which analyses log files hourly and makes the result available, instead of the lag that GA has.
One tool I’ve seen that does look particularly interesting is Woopra (Tim Ngwena tweeted about this a while back), which offers live tracking and analysis. This would be particularly useful for watching the statistics on live events such as Roses or Elections, but I’m unsure if putting yet more JavaScript in the site is such a good idea, and Google Analytics is far too good a tool to let go, so at the moment I’m not implementing it.
So far, so normal, however as Nouse operates from Linode as opposed to a typical shared server, there is a lot more that we can monitor, and it is normally these tools that interest other people.
I’ve been using MRTG now for quite some time on my own servers and graphs are quite nifty.
MRTG is designed to map the traffic of a router, but can easily be extended by a script that provides two numbers (representing “in” and “out” bits per second, but the labels can easily be changed), a hostname and an uptime. There’s a number of fairly normal things I track, such as CPU usage, RAM usage, HDD usage, etc, but there are a few things in particular that are more important for Nouse:
The first is the number of Apache requests being handled. This is crucially important, especially considering how heavyweight WordPress is in memory use (just under 60 MB per request) and keeping this below the point at which we start swapping and performance degrades past a useful point is very important.
The second thing I track is MySQL activity – both read (SELECT) and write (UPDATE, INSERT, DELETE, etc), and is the best indicator of server load. Looking at the year graph also shows us a few interesting events:
The blue line represents SELECT statements, and the green line UPDATE, INSERT, etc. The interesting events you can see on this graphs is when I configured WP Super Cache which made a massive difference to server load, and also when I deployed the new site, which contained an annoying bug resulting in far too many INSERT statements (the Popular This Week plugin used to count a view as when a post is displayed, but WordPress thinks a post is displayed whenever it’s headline is shown, resulting in an INSERT for every headline, rather than one per pageload). Spotting once this bug was fixed is also quite easy on that graph. Another interesting point is the increased amount of queries since May, which I hope means that the momentum we gathered during Roses has continued through the rest of the term.
The final thing I track was developed for the live blog and shows how many people are currently connected. Here’s the graph from Friday:
The little blip the day before is a result of testing, and at first it might seem that the numbers here are quite low, especially as Google Analytics recorded over 2000 views for the live blog on this day alone, but I suspect that means that user behaviour is that for something like Roses, most people just popped in to check, and then moved on, coming back later, rather than leaving it to run all the time.
Of course, statistics are useless without anything to compare it against. The Yorker no longer publish their web stats on their advertising page (we do, and I try to keep them up-to-date, especially considering the massive growth we’ve seen since the new site launched in February), but if they’ve not grown since they last published them (which seems unlikely, of course), our statistics compare favourably.
(Also, in an unrelated note, I will be filming some of the Apprentice York stuff for YSTV tonight, and one of the remaining teams include Nouse Comment Editor Charlotte Hogarth-Jones, and News Correspondent Holly Hyde, so I feel obliged to recommend that you all pop down to Derwent tonight to support Mr York 2009, where Nouse editor Henry James Foy will be competing against a few people, including regular Nouse commenter, Dan Taylor).
The Nouse custom plugins
Just a short blog post this time.
Before, I’ve promised to release some of the custom plugins we use here at Nouse, and I’ve now gone and got the plugins listed on WordPress’s Extend directory for people to get.
We have 7 custom plugins and 13 standard plugins, and now 5 of those plugins will be available for you to use on your own blog if you want to. The other 2 custom plugins are very unlikely to see the light of the day, but are probably uninteresting (not that all the ones that are being released are interesting). They’re a ‘Media Charter Compliance’ plugin that sends YUSU an e-mail whenever a new news, sport or comment article is posted, and the second is our YSTV integration plugin.
However, the other 5 are available now from WordPress.org:
Our plugin to have images attached to articles and dynamically resize them on the backend.
A plugin that exports RSS feeds in a JSON form.
The Popular This Week widget.
A plugin that lets you have images associated with tags.
Our live blogging plugin. I made a screencast to demonstrate how to use this too.
So, short and sweet, but this is the giving back to the open-source community I talked about back in my first blog post.
Reflecting on Roses
It’s been a while since my last post on this blog, but as you can imagine, it’s all been go for team Nouse tech.
The week leading up to Roses was a big hard job for me and the rest of the development team, but one we accomplished well, matching the amazing work put in by the sports team, the photo team, Anna Bucks and the YSTV team, and everyone else who helped out, who as well as running the coverage and live blog were putting together a newspaper at the same time.
Everything went very smoothly technically, which was my main concern, especially with launching such a big new feature as the streaming live blog. I’m fairly sure I scared everyone in the Nouse office on Thursday night as I was shouting “fuck you” at my laptop repeatedly, normally whilst debugging in IE6 (approximately 15% of our readers still read in IE6, presumably due to computing services not upgrading the machines on campus), but it all seemed to work pretty well at the end of the day.
The drive behind launching the streaming live blog was two fold, firstly to reduce the load on Apache over the weekend and secondly to get the news out faster than anyone else. During testing, the news would slide in on the connected clients before the “Post successful” screen even loaded on our back end.
If you missed it, we also launched an image slideshow, more dynamic than the simple “picture of the day” we used for Elections, that hooked into the wonderful Flickr API to get the photos the photographers uploaded immediately. Mike Tomasello was responsible for that one.
So, back to live blogging, the crown in the jewel as it were. Now I’m going to get a bit technical, but there’s diagrams so hopefully not too technical (a rough version of this was in the office whiteboard, and Henry seemed to understand it with copious amounts of hand waving).
The bit of technology that made it all possible was Meteor Server. Basically, it serves a web page that appears to take ages to load – but web browsers will deal with web pages as they come in, not wait for it to load, so when you connect to the live blog, this web page opens in a hidden part of the page and then whenever Meteor receives a new message from us, it then sends it to all the connected browsers as a little bit more of the page. And it’s even more clever, because if this doesn’t work in your browser, or behind your connection, it “polls”, which means instead of having something open constantly, every 5 seconds it reconnects and asks if there’s any more data to receive. So at worst you’ll have a 5 second delay.
This is the software Twitterfall use to power their stuff, so big thanks to them for pointing me in the right direction and giving me some helpful tidbits.
Setting up Meteor was a bit challenging, least of all because it’s documentation isn’t great, but there also were bugs with it – chiefly the security settings it tried to set are invalid for .co.uk domains (like Nouse), but the bug didn’t prove to be too hard to track down and fix, and being good open source consumers, we gave the patch back to the community.
So, I wrote a WordPress plugin to link this all together (as well as pushing to Twitter), which works roughly as the diagram shows:
Once the message (actually an encoded JSON string) is received by the browser, the Meteor client library passes it to our event handler, written by Ali ‘Fantasy Football’ Clark, which does the magic animation stuff.
Then, of course, Richard Mitchell suggested we should have comment streaming. Another late night later, liveblogging.php catches comments as they’re posted (if they’re from an authenticated user, or when they get approved by a moderator otherwise) and then streams them out using Meteor (but on a different channel) and the event handler posts them in the appropriate place.
It’s not perfect, there’s still a couple of bugs to work out (the ‘Your comment is awaiting moderation’ message will stay on your screen even after moderation – but that’s not too difficult to solve), and sending a comment still requires a POST request – I’d prefer to have it AJAX’d, but the core of it is there.
So that’s it! That’s our secret behind live blogging. The plugin will be released as open source as soon as I’ve abstracted some Nouse specific stuff away.
Roses was a fantastic experience, it’s a shame I couldn’t get out and see any of it (I was the one responsible for updating the official score on roses2009.com, whose API fed the Nouse site, as well as being fed in part by the Nouse site itself), but as score central was YSTV, I did get to see whatever they caught on camera. I also got to see the reactions of LU Tube (Lancaster’s TV station) who were sharing the studio as our points just went up, and up, and up.
A Window to the Past
Tom Scott once said that campus has a three-year memory. And he’s right. There’s no-one at Nouse who’s been at York longer than 4 years and as this generation graduates in 10 weeks, gone will be the last few Nouse-ites who don’t pay top-up fees, those who remember every bar being open every night of the week.
Of course, it’s our job to report it here and now, but what about for future generations? Our Deputy Editor wrote a piece for the elections supplement about the history of YUSU, but there’s wasn’t really much to go on in the way of history, except for old copies of Nouse (which I’ve been informed by the library are available 24/7 in the microfilm room!) and the folklore page, which I’m informed isn’t the most accurate and unbiased account in the world.
Marie Thouaille of The Yorker recently wrote a comment piece on the Claudia Lawrence case, and she seems to have fallen into the same trap.
People disappear in films all the time, but in real life – or rather, in our comfortable uni-bubble, the reality of which we may question – we don’t expect such brutal realities to hit
It does happen, and it does happen with a worrying regularity, whether it’s unfortunate deaths, or something more serious. The OU murder of the 90s is just as shocking, but after that campus returned to normality as Marie suggested, and I’m sure it will do again. But, we should learn from the past to avoid it repeating itself.
Foy once called me a bastion of posterity (after I shouted at him for uploading an election photo of the day in a way that deleted old ones), and that’s something I’ve tried to implement into the Nouse site. The ‘Related Posts’ plugin tries to show similar stories, or stories in the same form from the past, and the ‘From The Archives’ section on the News page, but they only go back as far as early 2005, there’s still 41 years of Nouse history buried away…
The original Nouse site was launched in 2003, but for whatever reason the content on it was never moved when the site moved to WordPress in 2005. I’ve recently found a backup of the original site, and eventually will write a converter to move it to the current WordPress site, but reading back some of it is particularly interesting, especially in light of Summer Ball. Our front page from June 2003 discussed moving the Gradball (as it was called then) from Elvington Airfield to the Racecourse, and some resulting controversy.
And not just the content is interesting, PDF archives are too showing the design of the paper over the years, the images and look of the time. One of the biggest criticisms I received internally about the new site was the lack of a PDF archive, so it’s been brought back and up-to-date, but without the tedious uploading process by predecessors had – opening the PDF that went to the printers into Photoshop, resizing and thumbnailing, and then manually uploading all the pages and images.
Apparently a couple of hours of work, all replaced by a Python script. (Linked here, for the curious).
So our archives from here on forward will be on the web for all to see, but there’s still some work to do with the past. Talking to the Borthwick Archives proved enlightening, but they brought up a good point which may prove the archives to be impossible – copyright. It’s unlikely the original authors of the time would have consented their work to go on the web – especially as the web didn’t exist for a lot of the time back then.
Of course some people are glad that campus only has a 3 year memory. That their deeds will go forgotten and they can perhaps leave a youth with mistakes behind them. But the Internet – Facebook, Flickr, Twitter – is changing all that and making one big collective memory. One I want to contribute to. Of course, that means my colleagues who actually do the journalism have a greater responsibility than ever.
Looking inside the magic box
Running a website the size of Nouse.co.uk presents a number of challenges, least of all in infrastructure – providing not only the main website and the features that support it, but staging areas, development areas, SVN version control, development tools, issue tracking, wikis…
Added features bring unexpected consequences, and even unexpected uses, as the York students behind Twitterfall, Nouse Computing Officer Tom Brearley and David Somers, discovered following yesterday’s problems on the Telegraph website. The problems we experience at Nouse aren’t on the same scale, but they, and the solutions to them, are interesting. Tweets like JTownend’s show that people like to know what’s going on behind the scenes, and Brearley’s reply of “magic” leaves a lot to be desired.
I’ve an advocate of free and open-source software, where people can see the work of others (the “source code” – the recipes that make up the software) and modify and improve them as they see fit, and hopefully allowing others to do the same, resulting in a great group-think resulting in an excellent overall product.
Blogs are intended to be highly personal. They’re not “print quality”, they don’t go through an editor first, and they don’t even get sub-edited (too much). They’re aimed to be a direct tap to the heart of the blogger, and collectively the blogs coming on Nouse over the following weeks will cover a much wider subject matter than articles have covered before. And by restricting one into a particular blog, you just need to pick and choose which ones you want.
In this blog I am to cover the interesting problems that come up with running Nouse.co.uk, and demystify some of the behind-the-scenes “magic” – putting this knowledge and my solutions into the so-called “blogosphere” for other people to benefit from. Eventually (when the code is relevant and mature enough) I also aim to release some of the custom code that’s gone into Nouse.co.uk as WordPress plugins. Approximately half of the plugins installed into our WordPress were written internally, with the rest being ones from WordPress.org.
I should probably finish off by saying who I am, as my name normally doesn’t grace the side of an article. As it says up there, I’m the Technical Director of Nouse. I’m in charge of leading a team of 3 amazing developers, pretty much everything to do with the website short of the actual words on the page, and I moderate most of the comments that come through. I’m also pretty outspoken, have an opinion on everything and try to get involved with campus politics – voting, attending hustings, former JCRC officer and even had a couple of UGM motions passed. If you know me, it’s probably because my name appears in the comments section under too many of the articles on the site.
This blog is aimed to be a bit technical, so I hope that’s not geeked anyone out too much!