Planned maintenance

[Nexus]

Hi everyone. I mentioned a month or so ago that I may be relocating the datacentre that houses ozmpsclub - the time has come. I've signed a lease today on a property northwest of Melbourne. It has fibre optic internet which is a huge upgrade from the FTTN vDSL connection currently in use. Keys will be picked up next week and then it will be all hands on desk dismantling and rebuilding the studio and datacentre.
To keep ozmps running during the move, I will be migrating it from Victoria to Queensland datacentre. There will be some outages of up to an hour or so at a time to make this happen.

[Nexus]

OzMPS has been migrated to Toowoomba QLD datacentre. Checking everything now to confirm all is healthy, looks fine so far. Performance may be slightly reduced.
Picking up keys to new premesis on Wednesday, ozMPS will be migrated back to VIC once datacentre has been relocated. Likely to be at least a couple weeks.

[Nexus]

Victoria datacentre is being reassembled at new premesis this week. You may be noticing some performance issues - long page load times - on and off.

This happens when your browser tries to contact ozmps via Victoria, when it happens your browser tries Qld after Vic fails

Issue will go away as Vic is brought online again


Sent from my iPhone using Tapatalk

[Nexus]

Vic datacentre is coming back online. Took a bit more time that I estimated to pull it down, transport everything and start reassembling. Once VIC datacentre is fully operational the performance issues will go away. After than there's be a couple of hours overnight outage to transfer the ozmpsclub operating system back to victoria.

[Nexus]

OzMPS is being migrated back to Victoria datacenter. Just had a brief outage to backup, and then found a filesystem error to repair. Will remain running in Queensland for today, a second brief outage will occur to complete migration to Victoria, most likely tonight (Tue).

[Nexus]

ozMPS has been migrated back to Victoria. We should see some significantly improved performance.

[Nexus]

There will be some significant outages in the near future for major maintenance on server hardware.

(1) Storage migration - a 6TB storage array will be cloned offline in order to transfer the data from an Adaptec RAID5 array to a Hewlett Packard RAID6 smart array. Generally I use HP array controllers as preference. The HP controller will not accept the adaptec array directly, so a new array will be created and the data copied across. This is likely to be at least three hours maintenance once the process is started. The cloned data does not include ozmps, but ozmps is hosted on the same server, which will be offline for the duration of maintenance. I may run this as an overnight process.

(2) Server mainboard, CPU, memory upgrades. Both primary servers are getting an update treatment from 12 core / 24 thead each to 20 core / 40 thread each. Server RAM will be boosted from 192Gb to 256Gb on server1 and 96Gb to 128Gb on server2. This is a major work and difficult to gauge exactly how long it might take. Hardware will be prepped in advance of swap, when the swap takes place this will be hours of downtime per server. ozmps will be impacted by maintenance on server2 and unaffected by maintenance on server1.

(3) Spares pulled from running systems will be shipped to QLD and recycled for use there. That will involve an outage in QLD however this won't take ozmps offline, but will likely make it slower to respond.

(1) Will be done very soon, (2) in coming weeks, (3) in due course.

[Nexus]

Storage array migration completed for server2 where ozmps is housed. Was offline from late last night through to 10:30AM.
Adaptec controller failed to read 8Mb of data from the 5,767,168Mb source array. That really shouldn't be possible for a optimal status RAID5 but that's why I use HP controllers and RAID6 as preference.
Spent a bit longer than intended trying to coax it into reading the failed 8Mb of data and then threw the towel in and just checked the filesystems when the clone was mounted.
Was a 1:1 copy including data in void space (empty filesystem space), appears the 8Mb of errors weren't in space that was actually storing data.

[Nexus]

Will be having a brief outage this afternoon for mainboard hardware maintenance on server1 which hosts ozmps. Not upgrading mainboard just yet, although that's coming soonish. Installing an IPMI management interface.
This will include flashing some firmware and a bit of stuffing around so may be an hour or two.

[Nexus]

...and back online... IPMI management just gives me more information about how the system is performing - temperatures and cooling fan status, voltages, etc etc.

[Nexus]

Lengthy outage yesterday afternoon through till midnight due to attempted hardware upgrade (mainboard)
Unfortunately the upgrade attempt was a fail and system had to be returned to previous configuration.
This was a hardware tear down and rebuild, which meant it took about twice as long as expected.
There will be another attempt at this service in due course.

[Nexus]

Hi everyone, we're about to have an outage while I attempt to transfer the system ozmps runs on to an upgraded mainboard. I'm anticipating an hour, maybe two. This is take#2 on the upgrade attempted a couple of weeks ago.

[Nexus]

Back online after migrating to an upgraded mainboard. Took about twice as long as expected as I needed to address physical fitment issues - as in bits being a very tight fit with small clearances. This is the same problem that had me abort the upgrade a couple of weeks ago. This time using a case that's 50mm longer and better design with more board clearance around large 12"x13" mainboards.

Long story short is we're back online with newer CPU/memory/mainboard hardware.

[Nexus]

Outage soon to upgrade power supply on server housing ozMPS.

[Nexus]

Well...that took longer than expected...... Todays maintenance reminded me of fitting the first 2XS shortest equal length turbo manifold prototype. It fit. Only just, with some jiggery pokery.

Ever noticed how IT maintenance always takes longer than expected? That's not always IT, often that's the boss/owner cheapskating. Cheapskating just complicates things and in IT, complicating things = time wastage.
I'm cheapskating in the sense that I make it work from the resources I have and minimise spend because private datacentre doesn't earn $ yet.

On this occasion I couldn't fit the redundant power I intended to because the dual supply which is nearly the same size as a single standard one is a few mm too large. Oh well, it'll have to do without until I've a more expensive case. Note this is a new case.
Well, since I've got it out anyway, I might as well fit the slide rails, make everything neat, yeah?

At which point I discovered the mounting rails needed about 0.5mm more clearance...or it would jam instead of sliding in.

Long story short - power supply install was a fail because cheapskated on case, and after an annoying delay while I worked out how to give the rails just a smidgen more clearance... done.

maintenance1.jpgmaintenance2.jpg