FreeSide On The Web

So, FreeSide is nice and clean (Thanks to M Bell for sorting cleaners and such :))

Most of the servers are back online, but Eye has a new home

Hidden server is hidden

Pretty much the same state as it was i last visited, but we do have a wasp (albeit mainly dead) problem

Loads all over the floor, and on the sides. A few alive ones about

And there’s a vending machine where we had our pile of computers

1
2
3
4
5
6

As many of you will probably know, the Department and the Robert Blackburn building are undergoing refurbishment, and therefore access is restricted

Some electrical work needed to be done, therefore power was lost to FreeSide on friday afternoon. I was the first avaliable admin to be able to get in today to get it all back online

I was greeted by a pigeon inside today (see pictures), and then a very different looking FreeSide from that of when i last saw it.

Some servers were already online, but the key server for our external access - M25 wasnt (probably a job for later, is to get this to automatically reboot if possible)

Bit of fiddling and poking, and the majority are online again. Notable exceptions are eye and bladeswitch, which had been unplugged and moved. Chances are they’ll get moved again, and being none crucial, i left them.

I couldnt actually find Gaia, it was at the other side of the room, and had been unplugged and moved, meaning our logging server is now offline for a bit. Not to us - MOAR LABELLING. It was one of the old fenner beige boxes, i think i knew which it was, but access to power cables and such, it wouldnt have been easy to get it online again

We’ve got some huge-ass cables not going over a gantry in the roof, and it would seem, some new lighting, which is nice

Get back home again and the website are down. Quick look seems that the webserver twins must’ve booted before mnemosyne had finished and the WWW directories hadnt been mounted. Quick restart, and still no websites. Buggar. Cause was Switchblade - Perlbal wasnt running on it

Part of Reboot! 2008 was to get some form of server monitoring installed and online, so we can work out whats what, rather than just poking in the dark.

So nearly, 2 months later, this is really only just starting to be properly implemented.

Phoenix made a start a couple of weeks ago, but then didnt have chance to finish it.

Putting a new server on nagios is certainly a treck round the houses (unfortunately, we cant just do a basic apt-get install nrpe nagiosplugins :()

So, after phoenix finally giving me some basic instructions on how to setup a box for monitoring, and also a link to the proper manual, i gave it a go

And it took ages! Loads of commands to enter, 2 guides to partially follow - Not Fun!

The logical answer, was therefore, to write a basic bash install script (to at least semi-automate the process). A bit of fiddling, and combining the 2 guides, has brought about a shell script that works

It is listed at http://wiki.freeside.co.uk/index.php/Admin:Nagios_Plugin_Install

So, copy and paste that into a script on the server, make it executable, and run it

Currently, unless something goes wrong, you’ll only need to enter 2 bits of detail, 1, being Y to accept the extra apt-get package installs, and the nagios user password (enter it twice)

It will then do all the client side configuration (to a basic form). Any other services to be monitored, must be manually added (see wiki for more details)

Then, when thats done, you’ve just got to hop onto gaia, using a copy of the basic client monitoring config, change the few needed parameters, paste into the config (as root), check it, and restart nagios

Give it a few minutes and you’ll have the server listed as monitored in Nagios

Jobs a good ‘un

Reedy

It seems that Reboot! 2008 has been and gone, leaving in it’s wake a trail of destruction and broken limbs … well, not really, but ‘we moved some shelves and tidied up’ doesn’t sound as exciting now, does it?

We all started to trickle in from around 9:30AM, some of us stopping off to fuel ourselves (see the pics at the end of the post).

A few of us were dispatched to assemble a rack, which was quickly dubbed ‘FailRack’ due to our inability to get it to actually resemble a rack … having completely ballsed that task up we moved onto noting down specs of the current servers and powering them down (which I’m pleased to report we managed to do correctly), moving them into the lecture theatre next door … most of this stuff worked, honest.

It was at this point we all decided we’d had enough and torched the lot broke for lunch … this may have been a somewhat extended lunch break, but hey, we totally deserved it!

After lunch we set about the easy task (I say easy because I was the one filming it) of moving the shelving units about. How these managed to withstand Hull’s recent earthquake we have no idea. With the the wobbly shelves in their new position to the left of the room, we started to move all of the machines back in and hook it all up, except for a few of the big-iron servers which Jonty and Wes seemed to be hitting with hammers (but not beans) in the lecture theatre.

After this came the networking cabinet (or was it before? I was jumped up on caffeine by this point), perhaps the most professional-looking part of the setup at the moment … which is saying something.

At the end of the day pretty much everything was working, we had managed to install a few new servers and the room looked a hell of a lot better … there was still the huge pile of broken machines around the corner, but that horror deserves an entry all of it’s own.

Huge thanks to those that came and gave up their weekend to make FreeSide rock (more) and … hey, anyone feel like doing it again next year? …

 

FreeSide have recently acquired a grand total of 27 P4 2.4GHz PC which have been “saved” from being disposed of. Many thanks to the department and all involved in making this happen.

With 27 machines (Even without memory) we can keep FreeSide running for a long time to come. However, with new machines come new ideas. It has already been mentioned by Jonty to create a Hadoop cluster (http://hadoop.apache.org/) a scalable distributed system. Sound cool, but we’ll see :).

From the new machines, we have put 2 more into use and finished installing a couple of others. These include gaia, the replacement for dagger, terra, the new FreeSide console machine, yxorp, which will soon be our new proxy and finally, bladeswitch has finally been completed as out redundent web load balancer.

First off, not quite a new addition, but the version of WPMU that FreeSide uses for its blogging system how now bee updated to version 1.5.1 which has quite a lot of new features. For those of you who have used it before, the most notable change is the admin panel which has had a lot of work done to it.

If you find any issues / bugs etc. with the update, please tell us in the #freeside IRC channel.

Main new addition is the installation of Outcast. A new webserver which is being prepared for ASP.NET application requests. This server is still in the installation phases as we try to work out what the best way to get it to work is and how to properly roll it out to all of our users. This may take some time, however, we hope that it will be sooner rather than later.

Watch this space!

Over the last couple of years,  FreeSide has grown substantially both in production machines as well as the amount of consumed space. Also, by adding and removing servers and terminals over that time, the network (and the room) has become a bit of a mess both physically and systems wise.

As such, Reboot!, the total FreeSide rebuild “event” will be taking place in the next couple of weeks. Unfortunately, this will cause some major downtime over the space of a few days (probably a weekend), while we rebuild each server system and tweak server hardware to be more efficient, add new (and some improved) services and sort the general infrastructure of the room using the new server/network racks.

Anybody who is part of FreeSide is welcome to help out (http://www.freeside.co.uk/content/join), however, keep in mind that there is limited space in the room as it is, let alone with  lot of people shifting things around, so if you would like to help out, pass us an email to admins[-at-]freeside.co.uk to say, and we will see what can be arranged.

The day should be quite fun, but there is a lot of work to be done. This will be good experience for anybody who enjoys (or has an interest in) Linux servers, network infrastructureand generally how things fit together.

See http://wiki.freeside.co.uk/index.php/Reboot!_2008 for more information on what specific changes we will be making to FreeSide.

 Reboot! is now taking place this weekend (weekend starting 11th April), we will be meeting in the Sun Cafe down Cottingham Road at 10am, have a coffee and head in to get started.

From my last post, you will know we gained some SpiderPort Atoms which we intend to roll out as soon as possible. However these spider ports run on 10BaseT network connections, which is to slow to use them as part of the network backbone. So, we’ve decided to add a second network card in each server and use that for the SpiderPort.

Unfortunately, our spare network cards are not always that reliable and as such, it may take time to get each server back up and running. Service will therefore be somewhat sporadic, to keep up with what servers are up and which are down, check out our Twitter feed at http://www.twitter.com/freeside.

Many thanks to Radius Solutions (http://www.radiussolutions.com) for their donation and thanks to John for setting it up. Unfortunately, (in the loosest way) we couldn’t fit everything in my car and will require a second trip. Out of the haul, items to note:

• HP tower server
• HP UPS
• 9 Ethernet Hubs
• 3 SpiderPorts
• 3 Other PC
• 1 laptop

These will all come in very handy in the near future. Once again. Thanks.