FreeSide On The Web

So, FreeSide is nice and clean (Thanks to M Bell for sorting cleaners and such :))

Most of the servers are back online, but Eye has a new home

Hidden server is hidden

Pretty much the same state as it was i last visited, but we do have a wasp (albeit mainly dead) problem

Loads all over the floor, and on the sides. A few alive ones about

And there’s a vending machine where we had our pile of computers

1
2
3
4
5
6

As many of you will probably know, the Department and the Robert Blackburn building are undergoing refurbishment, and therefore access is restricted

Some electrical work needed to be done, therefore power was lost to FreeSide on friday afternoon. I was the first avaliable admin to be able to get in today to get it all back online

I was greeted by a pigeon inside today (see pictures), and then a very different looking FreeSide from that of when i last saw it.

Some servers were already online, but the key server for our external access - M25 wasnt (probably a job for later, is to get this to automatically reboot if possible)

Bit of fiddling and poking, and the majority are online again. Notable exceptions are eye and bladeswitch, which had been unplugged and moved. Chances are they’ll get moved again, and being none crucial, i left them.

I couldnt actually find Gaia, it was at the other side of the room, and had been unplugged and moved, meaning our logging server is now offline for a bit. Not to us - MOAR LABELLING. It was one of the old fenner beige boxes, i think i knew which it was, but access to power cables and such, it wouldnt have been easy to get it online again

We’ve got some huge-ass cables not going over a gantry in the roof, and it would seem, some new lighting, which is nice

Get back home again and the website are down. Quick look seems that the webserver twins must’ve booted before mnemosyne had finished and the WWW directories hadnt been mounted. Quick restart, and still no websites. Buggar. Cause was Switchblade - Perlbal wasnt running on it

Part of Reboot! 2008 was to get some form of server monitoring installed and online, so we can work out whats what, rather than just poking in the dark.

So nearly, 2 months later, this is really only just starting to be properly implemented.

Phoenix made a start a couple of weeks ago, but then didnt have chance to finish it.

Putting a new server on nagios is certainly a treck round the houses (unfortunately, we cant just do a basic apt-get install nrpe nagiosplugins :()

So, after phoenix finally giving me some basic instructions on how to setup a box for monitoring, and also a link to the proper manual, i gave it a go

And it took ages! Loads of commands to enter, 2 guides to partially follow - Not Fun!

The logical answer, was therefore, to write a basic bash install script (to at least semi-automate the process). A bit of fiddling, and combining the 2 guides, has brought about a shell script that works

It is listed at http://wiki.freeside.co.uk/index.php/Admin:Nagios_Plugin_Install

So, copy and paste that into a script on the server, make it executable, and run it

Currently, unless something goes wrong, you’ll only need to enter 2 bits of detail, 1, being Y to accept the extra apt-get package installs, and the nagios user password (enter it twice)

It will then do all the client side configuration (to a basic form). Any other services to be monitored, must be manually added (see wiki for more details)

Then, when thats done, you’ve just got to hop onto gaia, using a copy of the basic client monitoring config, change the few needed parameters, paste into the config (as root), check it, and restart nagios

Give it a few minutes and you’ll have the server listed as monitored in Nagios

Jobs a good ‘un

Reedy

It seems that Reboot! 2008 has been and gone, leaving in it’s wake a trail of destruction and broken limbs … well, not really, but ‘we moved some shelves and tidied up’ doesn’t sound as exciting now, does it?

We all started to trickle in from around 9:30AM, some of us stopping off to fuel ourselves (see the pics at the end of the post).

A few of us were dispatched to assemble a rack, which was quickly dubbed ‘FailRack’ due to our inability to get it to actually resemble a rack … having completely ballsed that task up we moved onto noting down specs of the current servers and powering them down (which I’m pleased to report we managed to do correctly), moving them into the lecture theatre next door … most of this stuff worked, honest.

It was at this point we all decided we’d had enough and torched the lot broke for lunch … this may have been a somewhat extended lunch break, but hey, we totally deserved it!

After lunch we set about the easy task (I say easy because I was the one filming it) of moving the shelving units about. How these managed to withstand Hull’s recent earthquake we have no idea. With the the wobbly shelves in their new position to the left of the room, we started to move all of the machines back in and hook it all up, except for a few of the big-iron servers which Jonty and Wes seemed to be hitting with hammers (but not beans) in the lecture theatre.

After this came the networking cabinet (or was it before? I was jumped up on caffeine by this point), perhaps the most professional-looking part of the setup at the moment … which is saying something.

At the end of the day pretty much everything was working, we had managed to install a few new servers and the room looked a hell of a lot better … there was still the huge pile of broken machines around the corner, but that horror deserves an entry all of it’s own.

Huge thanks to those that came and gave up their weekend to make FreeSide rock (more) and … hey, anyone feel like doing it again next year? …

 

Over the last couple of years,  FreeSide has grown substantially both in production machines as well as the amount of consumed space. Also, by adding and removing servers and terminals over that time, the network (and the room) has become a bit of a mess both physically and systems wise.

As such, Reboot!, the total FreeSide rebuild “event” will be taking place in the next couple of weeks. Unfortunately, this will cause some major downtime over the space of a few days (probably a weekend), while we rebuild each server system and tweak server hardware to be more efficient, add new (and some improved) services and sort the general infrastructure of the room using the new server/network racks.

Anybody who is part of FreeSide is welcome to help out (http://www.freeside.co.uk/content/join), however, keep in mind that there is limited space in the room as it is, let alone with  lot of people shifting things around, so if you would like to help out, pass us an email to admins[-at-]freeside.co.uk to say, and we will see what can be arranged.

The day should be quite fun, but there is a lot of work to be done. This will be good experience for anybody who enjoys (or has an interest in) Linux servers, network infrastructureand generally how things fit together.

See http://wiki.freeside.co.uk/index.php/Reboot!_2008 for more information on what specific changes we will be making to FreeSide.

 Reboot! is now taking place this weekend (weekend starting 11th April), we will be meeting in the Sun Cafe down Cottingham Road at 10am, have a coffee and head in to get started.

Many thanks to Radius Solutions (http://www.radiussolutions.com) for their donation and thanks to John for setting it up. Unfortunately, (in the loosest way) we couldn’t fit everything in my car and will require a second trip. Out of the haul, items to note:

• HP tower server
• HP UPS
• 9 Ethernet Hubs
• 3 SpiderPorts
• 3 Other PC
• 1 laptop

These will all come in very handy in the near future. Once again. Thanks.

A lot of time this year (as well as a fair bit of money) has gone on system redundancy and data backups. Setting up FS based data was easy thanks to a couple of new hard drives and a bit of RAID goodness. MySQL however, has always caused a bit of an issue.

MySQL AB, came up with 2 forms of data backup and redundancy. The first one we tried was MySQL cluster. Even though this is now included in the main distributions of MySQL server, Cluster itself seems to be years off having a usable product(especially for Not For Profit projects such as FreeSide), however, it has an excellent base to move on from and has many advantages. Unfortunately it requires too much hardware and cannot support some of the packages we use as they are not compatible with the NDB engine at the time of writing.

The second is replication, basically a copy of all databases from one Master server to a slave server. This may sound no better that simple copy and paste, however, there is a lot more depth than that which is beyond the scope of this post. As an added incentive (as with cluster) it will be possible to use this secondary system as service redundency for MySQL so downtime at any one MySQL failure should be minimal and recoverable.

Winters nearly over, spring is nearly here and it was about time that FreeSide abided by the time honoured tradition of Spring Cleaning.

It took all of the Christmas holidays to arrange the disposal of all the useless equipment that FreeSide has accumulated over the last year or so and now we have finally managed to move most of the junk out. So on Tuesday, the task was for me to move 20 monitors of varying size from the Robert Blackburn building all the way to the Computing Services store room at the back of “Applied Science 3″. No easy task and took around 2 hours. Next came the difficult bit. With all the monitors out of the way, FreeSide had a bit more space outside where the monitors were to put all the dead PCs, which at a last count came to about 35-40 PCs of varying size and remaining components.

So, now FreeSide is back to having enough space to fill up with donated computers for the next year or so. Also, while bored doing all that, I decided to have a look at the spectrum and the commodore that had been donated a while ago. The comodore works perfectly which was a suprise and the spectrum….well it outputs some video although its still borked.

Pictures to come later.