User Tag List

Page 5 of 5 FirstFirst ... 345
Results 81 to 95 of 95

Thread: Planned maintenance

  1. #81
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Hi everyone. I mentioned a month or so ago that I may be relocating the datacentre that houses ozmpsclub - the time has come. I've signed a lease today on a property northwest of Melbourne. It has fibre optic internet which is a huge upgrade from the FTTN vDSL connection currently in use. Keys will be picked up next week and then it will be all hands on desk dismantling and rebuilding the studio and datacentre.
    To keep ozmps running during the move, I will be migrating it from Victoria to Queensland datacentre. There will be some outages of up to an hour or so at a time to make this happen.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  2. #82
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    OzMPS has been migrated to Toowoomba QLD datacentre. Checking everything now to confirm all is healthy, looks fine so far. Performance may be slightly reduced.
    Picking up keys to new premesis on Wednesday, ozMPS will be migrated back to VIC once datacentre has been relocated. Likely to be at least a couple weeks.

  3. #83
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Victoria datacentre is being reassembled at new premesis this week. You may be noticing some performance issues - long page load times - on and off.

    This happens when your browser tries to contact ozmps via Victoria, when it happens your browser tries Qld after Vic fails

    Issue will go away as Vic is brought online again


    Sent from my iPhone using Tapatalk

  4. #84
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Vic datacentre is coming back online. Took a bit more time that I estimated to pull it down, transport everything and start reassembling. Once VIC datacentre is fully operational the performance issues will go away. After than there's be a couple of hours overnight outage to transfer the ozmpsclub operating system back to victoria.

  5. #85
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    OzMPS is being migrated back to Victoria datacenter. Just had a brief outage to backup, and then found a filesystem error to repair. Will remain running in Queensland for today, a second brief outage will occur to complete migration to Victoria, most likely tonight (Tue).

  6. #86
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    ozMPS has been migrated back to Victoria. We should see some significantly improved performance.

  7. #87
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    There will be some significant outages in the near future for major maintenance on server hardware.

    (1) Storage migration - a 6TB storage array will be cloned offline in order to transfer the data from an Adaptec RAID5 array to a Hewlett Packard RAID6 smart array. Generally I use HP array controllers as preference. The HP controller will not accept the adaptec array directly, so a new array will be created and the data copied across. This is likely to be at least three hours maintenance once the process is started. The cloned data does not include ozmps, but ozmps is hosted on the same server, which will be offline for the duration of maintenance. I may run this as an overnight process.

    (2) Server mainboard, CPU, memory upgrades. Both primary servers are getting an update treatment from 12 core / 24 thead each to 20 core / 40 thread each. Server RAM will be boosted from 192Gb to 256Gb on server1 and 96Gb to 128Gb on server2. This is a major work and difficult to gauge exactly how long it might take. Hardware will be prepped in advance of swap, when the swap takes place this will be hours of downtime per server. ozmps will be impacted by maintenance on server2 and unaffected by maintenance on server1.

    (3) Spares pulled from running systems will be shipped to QLD and recycled for use there. That will involve an outage in QLD however this won't take ozmps offline, but will likely make it slower to respond.

    (1) Will be done very soon, (2) in coming weeks, (3) in due course.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  8. #88
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Storage array migration completed for server2 where ozmps is housed. Was offline from late last night through to 10:30AM.
    Adaptec controller failed to read 8Mb of data from the 5,767,168Mb source array. That really shouldn't be possible for a optimal status RAID5 but that's why I use HP controllers and RAID6 as preference.
    Spent a bit longer than intended trying to coax it into reading the failed 8Mb of data and then threw the towel in and just checked the filesystems when the clone was mounted.
    Was a 1:1 copy including data in void space (empty filesystem space), appears the 8Mb of errors weren't in space that was actually storing data.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  9. #89
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Will be having a brief outage this afternoon for mainboard hardware maintenance on server1 which hosts ozmps. Not upgrading mainboard just yet, although that's coming soonish. Installing an IPMI management interface.
    This will include flashing some firmware and a bit of stuffing around so may be an hour or two.

  10. #90
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    ...and back online... IPMI management just gives me more information about how the system is performing - temperatures and cooling fan status, voltages, etc etc.

  11. #91
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Lengthy outage yesterday afternoon through till midnight due to attempted hardware upgrade (mainboard)
    Unfortunately the upgrade attempt was a fail and system had to be returned to previous configuration.
    This was a hardware tear down and rebuild, which meant it took about twice as long as expected.
    There will be another attempt at this service in due course.

  12. #92
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Hi everyone, we're about to have an outage while I attempt to transfer the system ozmps runs on to an upgraded mainboard. I'm anticipating an hour, maybe two. This is take#2 on the upgrade attempted a couple of weeks ago.

  13. #93
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Back online after migrating to an upgraded mainboard. Took about twice as long as expected as I needed to address physical fitment issues - as in bits being a very tight fit with small clearances. This is the same problem that had me abort the upgrade a couple of weeks ago. This time using a case that's 50mm longer and better design with more board clearance around large 12"x13" mainboards.

    Long story short is we're back online with newer CPU/memory/mainboard hardware.

  14. #94
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Outage soon to upgrade power supply on server housing ozMPS.

  15. #95
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    48
    Posts
    1,614
    Blog Entries
    1

    Default

    Well...that took longer than expected...... Todays maintenance reminded me of fitting the first 2XS shortest equal length turbo manifold prototype. It fit. Only just, with some jiggery pokery.

    Ever noticed how IT maintenance always takes longer than expected? That's not always IT, often that's the boss/owner cheapskating. Cheapskating just complicates things and in IT, complicating things = time wastage.
    I'm cheapskating in the sense that I make it work from the resources I have and minimise spend because private datacentre doesn't earn $ yet.

    On this occasion I couldn't fit the redundant power I intended to because the dual supply which is nearly the same size as a single standard one is a few mm too large. Oh well, it'll have to do without until I've a more expensive case. Note this is a new case.
    Well, since I've got it out anyway, I might as well fit the slide rails, make everything neat, yeah?

    At which point I discovered the mounting rails needed about 0.5mm more clearance...or it would jam instead of sliding in.

    Long story short - power supply install was a fail because cheapskated on case, and after an annoying delay while I worked out how to give the rails just a smidgen more clearance... done.

    maintenance1.jpgmaintenance2.jpg
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

Page 5 of 5 FirstFirst ... 345

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •