VIP Member

User Tag List

Results 1 to 20 of 20

Thread: Planned maintenance

  1. #1
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default Planned maintenance

    Hi everyone, I've decided to stick a couple of threads where I'll notify of planned and unplanned outages of the web service.
    This will be planned outage thread. I'll add to the list and notify of times in advance of taking the system down for maintenance.

    Next planned services are:
    1. hardware maintenance of a DVD-R drive (loose cable)
    2. Installation of pci-express breakouts to convert two x4 slots into eight x1 slots
    3. Installation of two pci-e to PCI bridge risers.
    4. Installation of PCI and pci-e devices to these slots and risers.

    This hardware is for other operating systems that run under the same hypervisor server as ozmps.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  2. #2
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Add UPS Batteries to that list. That'll probably come first.

  3. #3
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Two very brief outages just took place - upgrading router kernels in QLD and VIC - approx 30 seconds each.
    VIC outage affected 100% of users, QLD outage affected 50% of users.
    Outages were short enough users may not have noticed it taking place.

  4. #4
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Not really outages, but works notification and expected impact:

    (1) Large data transfer between from qld to vic in coming week or so. In the order of 500Gb, which will take awhile and possibly impact performance - in particular connections coming in via QLD site might seem slower than usual - QLD connections tend to be slower due to your client connections hitting the QLD reverse proxy which then refers to a server in VIC which serves the page. This will take a day and a bit.

    (2) I'll be going through logs to determine how much incoming traffic is dodgy connection attempts (probably quite a bit), and also see what I can do to grease the wheels so to speak. Likely service actions are a revision of QoS traffic priorities and testing the website performance with the QLD gateway excluded from service.

    (3) A PCI-ex bridge and accompanying PCI device is likely to be installed within 2 weeks - hopefully this weekend. This does require a powerdown - likely 30minutes.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  5. #5
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Likely outage later today - some of the service work noted previously, in addition:

    - Replacement uninterruptible power supply. Beefy 1600VA unit should harden us against power fluctuations and will take the unit that has shown stability issues out of current service. It will be re-tasked to very light loads only.
    - PCI-ex to PCI bridge has landed and I will almost certainly install this and the associated PCI card while the system is down for UPS replacement.
    - SSD Array expansion - the SSD array hosting ozmpsclub.com will be upgraded adding 240GB which will increase bandwidth by about 700Mb/sec also. Not expected to be noticable from the website while this is happening, but before this can happen a backup battery that protects the array cache needs to be replaced. Again - this will be investigated while system is down.

    Not sure what time, likely late-ish tonight
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  6. #6
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Last night outage didn't go ahead, but will either tonight or coming evenings.
    Adding to the list of to-do's - the lateralcognition.net DNS servers are getting hammered and this is resulting in slow response times. Once the nameservice lookup is complete the website performance is good, but the initial DNS lookups have gotten quite slow.
    The fix is likely to be to host DNS services externally. Will update as that goes ahead. I'm not expecting this change to require any outages.

  7. #7
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    You should be seeing better response times from the website today - DNS load has been reduced by increasing the TTL (time to live) value of the records, reducing the frequency with which other servers attempt to update their copy of the records. Seems to be much better today - change was made last night. The downside of this change is that it will take longer for future changes to DNS records to propogate and become effective. Upside - reduced DNS load, improved DNS response for clients.

  8. #8
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Going down shortly to install the UPS and PCI-ex bridge noted earlier in the week. Definately happening today.

  9. #9
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Back online; took closer to three hours thanks to new PCI device buggering up device <--> VM associations which all needed to be reassigned with several hypervisor reboots in between. Now running a 1600VA UPS which I'm expecting will be better behaved when the power gets iffy.
    Next service interruption may happen when a new network switch is brought online however I may be able to bring it online without interrupting service.

    Quote Originally Posted by Nexus View Post
    Going down shortly to install the UPS and PCI-ex bridge noted earlier in the week. Definately happening today.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  10. #10
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Hi folks, likely to have a short outage soon as I am having maintenance done to building power.

  11. #11
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Back online after a mains power maintenance outage : Nearly got through running on batteries without shutting down but a mistake with earthing changes caused safeties to trip and I decided then to shut everything down while the work was completed.
    All back online now. Was down for about ~30min

  12. #12
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Had a brief outage while I stuffed around with power cables : I am replacing the core network switches. Replacing two separate switches and physical networks with VLANs, adding one switch -hence power shuffling. to create a spanning tree ring of three switches.

  13. #13
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Within a month google chrome will begin alerting users browsing over unencrypted connections (such as ozmpsclub.com) by throwing an alert page, highlighting the unecrypted status in the top address bar and so on:
    https://developers.google.com/web/up...ot-secure-warn

    Firefox already does some of these things, and if our chrome users are affected you will still be able to access the site, but may need to accept some security warnings.

    To avoid this, I need to implement https:// SSL encryption. ozmps needs to purchase a certificate from a certification authority and it needs to be implemented in the webservers hosting ecosystem.
    This was always going to be one of the first significant changes to be made. Un-encrypted internet "in the clear" communications are becoming strongly discouraged and there are good reasons for that; avoiding security alerts is not the only reason to make this change.

    Towards that end I have two choices :

    Leave the current web service alone and encrypt the internet traffic only. Web access is currently handled by reverse proxies which forward authorized connections to the actual servers. This allows numerous different web services and computers to live behind the proxies and be accessible from a single IP address. This will be tested. In this configuration the encryption is handled by the reverse proxy servers and the ozmps webserver configuration is not changed. Data is unencrypted while traveling over the internal server networks and encrypted over internet connections.

    Option two is to have the SSL encryption handled by the ozmps webserver itself. Data is encrypted over internal and external networks.

    Regardless of which implementation is applied the hosting ecosystem - the reverse proxies - require reconfiguration in order for them to handle https:// traffic similarly to how they handle http:// traffic.
    The final solution will depend on the flexibility/agility vs limitations of the reverse proxy software.
    I will try to have this sorted before the alerts begin to spoil the party for chrome users


    notprivate.PNG
    Last edited by Nexus; 11-09-2017 at 08:47 PM.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  14. #14
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Quick update on the SSL encryption planning. In order for this to work at all, a feature called SNI (server name indicator) needs to be functional. This means the oldest web browser devices may not support browsing the SSL encrypted https website. For this reason http will remain available for the forseeable future and once functional, I will probably add some security alerts and links on the page so you can switch between encrypted and unencrypted browsing easilt. I won't be forcing users to browse over https because that may prevent some users accessing the site.

    Our existing reverse proxy servers, that accept all internet traffic are currently using squid software.

    It doesn't support SNI when used as a reverse proxy, at least not in any of the versions I'll be running, and if it is available it's a bleeding edge feature which is likely to come with all the problems that bleeding edge software versions tend to have, so the existing reverse proxies will not be used for SSL encrypted browsing.

    Rather than replace a perfectly good of stable and reliable software I am considering running different reverse proxy software for http and https traffic, at least in the interim.

    nginx is currently likely to become the reverse proxy software used for handling encrypted web browsing as it supports reverse proxy SNI SSL certificate choice for encrypted connections. In the long term it will likely replace squid unless squid's SNI reverse proxy support matures and enters mainstream use with well documented configuration.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  15. #15
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    Further info on the SSL encryption architecture - SNI = server name indicator

    https://en.wikipedia.org/wiki/Server...cation#Support

    This has a list of software with compatibility information. The really problematic devices that will choke on the SNI based reverse proxy implementation are generally more than 5 years old. Surprisingly PHP didn't support it until 2014. I will need to double check the feature set of the supported PHP version currently in use. If this wasn't included in the build we are currently using, the ozmps webserver will need not just an update but an upgrade. I expect we won't need to upgrade the server, but I may do it anyway while I'm at it.

    I expect the technology is mature enough now that there will be only a small number of users who won't be able to use the SSL encrypted website access.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  16. #16
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    HAProxy has been chosen to implement https reverse proxy in an SSL / TLS encryption termination configuration, that meaning encyption between web services and public internet clients/browsers will be handled by the haproxy servers and traffic may or may not be encrypted across internal server networks depending on the configuration of the web service being accessed.

    http://www.haproxy.org/
    Last edited by Nexus; 01-10-2017 at 11:32 AM.

  17. #17
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    haproxy has been installed. Looking at the configuration over the next week. I'll be testing ozmps with a self generated encryption key at first and once everything is known to be working reliably I'll purchase a public certificate.

  18. #18
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    I am currently constructing a website for 'www.ozmps.club' - this has previously just been an alias domain for this forum site. Anyone using it would have seen errors attempting to login - the same applies to using 'ozmpsclub.com' - logins require you to use the domain 'www.ozmpsclub.com

    I will be launching this as soon as possible with an article on the airbag recall. Not the most glamorous of subjects, like a new MPS model at Tokyo show, but an important community service. Mazda have advised me that the closure rate for MPS' needing airbag recall servicing is 51.3%, so there's quite a few out there that haven't had the recall service.

    I am not expecting to see any change in behaviour on the forum site from this.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  19. #19
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    In the next day or so I'll be turning on a new website on 'www.ozmps.club'
    We had some issues crop up when this was added to the hosting configuration last week, and it isn't clear what the problem was. It may or may not have been an issue with our config.
    If you are trying to browse this forum and you see the new website, an ubuntu default web page, or a newscoop content management error or anything other than this forum then something is awry!
    You should only see the new website browsing to 'www.ozmps.club'
    If you find something is preventing you browsing the forum, please post a fault report to ozmpsclub on facebook

    The new website is currently just a skeleton but if you do decide to pop in early you'll get a preview and can watch it as the construction moves along. It will take a magazine format. Your accounts on ozmpsclub.com will not allow login to the new site - yet. For the time being user accounts on the site will be for people publishing articles, advertising vendors will likely also be given permission to construct web pages to promote their business and their products.

    I'm still fleshing it out - if you have ideas, want to submit an article or add your vehicle to the website concours, please let me know!
    Last edited by Nexus; Yesterday at 09:57 PM.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x7.5+48 Enkei RPF1 - 225/45R18 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 225/40R18 Federal RS-RR - CPE TMIC - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  20. #20
    Nexus's Avatar
    Nexus is online now Administrator
    Join Date
    Nov 2008
    Location
    Toowoomba
    Age
    44
    Posts
    1,351

    Default

    'www.ozmps.club' is now online - it will be up and down under construction for the next few days.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •