VIP Member

User Tag List

Results 1 to 9 of 9

Thread: Planned maintenance

  1. #1
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default Planned maintenance

    Hi everyone, I've decided to stick a couple of threads where I'll notify of planned and unplanned outages of the web service.
    This will be planned outage thread. I'll add to the list and notify of times in advance of taking the system down for maintenance.

    Next planned services are:
    1. hardware maintenance of a DVD-R drive (loose cable)
    2. Installation of pci-express breakouts to convert two x4 slots into eight x1 slots
    3. Installation of two pci-e to PCI bridge risers.
    4. Installation of PCI and pci-e devices to these slots and risers.

    This hardware is for other operating systems that run under the same hypervisor server as ozmps.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  2. #2
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Add UPS Batteries to that list. That'll probably come first.

  3. #3
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Two very brief outages just took place - upgrading router kernels in QLD and VIC - approx 30 seconds each.
    VIC outage affected 100% of users, QLD outage affected 50% of users.
    Outages were short enough users may not have noticed it taking place.

  4. #4
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Not really outages, but works notification and expected impact:

    (1) Large data transfer between from qld to vic in coming week or so. In the order of 500Gb, which will take awhile and possibly impact performance - in particular connections coming in via QLD site might seem slower than usual - QLD connections tend to be slower due to your client connections hitting the QLD reverse proxy which then refers to a server in VIC which serves the page. This will take a day and a bit.

    (2) I'll be going through logs to determine how much incoming traffic is dodgy connection attempts (probably quite a bit), and also see what I can do to grease the wheels so to speak. Likely service actions are a revision of QoS traffic priorities and testing the website performance with the QLD gateway excluded from service.

    (3) A PCI-ex bridge and accompanying PCI device is likely to be installed within 2 weeks - hopefully this weekend. This does require a powerdown - likely 30minutes.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  5. #5
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Likely outage later today - some of the service work noted previously, in addition:

    - Replacement uninterruptible power supply. Beefy 1600VA unit should harden us against power fluctuations and will take the unit that has shown stability issues out of current service. It will be re-tasked to very light loads only.
    - PCI-ex to PCI bridge has landed and I will almost certainly install this and the associated PCI card while the system is down for UPS replacement.
    - SSD Array expansion - the SSD array hosting ozmpsclub.com will be upgraded adding 240GB which will increase bandwidth by about 700Mb/sec also. Not expected to be noticable from the website while this is happening, but before this can happen a backup battery that protects the array cache needs to be replaced. Again - this will be investigated while system is down.

    Not sure what time, likely late-ish tonight
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  6. #6
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Last night outage didn't go ahead, but will either tonight or coming evenings.
    Adding to the list of to-do's - the lateralcognition.net DNS servers are getting hammered and this is resulting in slow response times. Once the nameservice lookup is complete the website performance is good, but the initial DNS lookups have gotten quite slow.
    The fix is likely to be to host DNS services externally. Will update as that goes ahead. I'm not expecting this change to require any outages.

  7. #7
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    You should be seeing better response times from the website today - DNS load has been reduced by increasing the TTL (time to live) value of the records, reducing the frequency with which other servers attempt to update their copy of the records. Seems to be much better today - change was made last night. The downside of this change is that it will take longer for future changes to DNS records to propogate and become effective. Upside - reduced DNS load, improved DNS response for clients.

  8. #8
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Going down shortly to install the UPS and PCI-ex bridge noted earlier in the week. Definately happening today.

  9. #9
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    43
    Posts
    1,294

    Default

    Back online; took closer to three hours thanks to new PCI device buggering up device <--> VM associations which all needed to be reassigned with several hypervisor reboots in between. Now running a 1600VA UPS which I'm expecting will be better behaved when the power gets iffy.
    Next service interruption may happen when a new network switch is brought online however I may be able to bring it online without interrupting service.

    Quote Originally Posted by Nexus View Post
    Going down shortly to install the UPS and PCI-ex bridge noted earlier in the week. Definately happening today.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •