Tunehouse

User Tag List

Page 3 of 3 FirstFirst 123
Results 41 to 52 of 52

Thread: Unplanned outage

  1. #41
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Discovered this afternoon that one of the two gateways that your browser reaches the website through was playing up (VIC gateway), and doing so in such a way that your browsers were not falling back to the gateway that was working (QLD)
    As a result up to 50% of users trying to browse the website may have seen timeout or connection reset messages. Not sure how long this problem had been occurring - I suspect a day or two - as I hadn't seen the symptoms myself. Issue identified and reported through Facebook group. This is first time we've seen this issue that I can recall. Normally if one gateway is down, you should still be able to browse the website. In this case the gateway wasn't down in a way that caused browsers to fail over to the alternative gateway.Thanks Adrian for reporting the problem.

    VIC gateway has been restarted and is behaving normally again.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  2. #42
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    internal network link failure between QLD and VIC caused similar faults to the gateway issue identified yesterday. Back online. Outage for 50% of users was from approx 9:20AM to 10:15AM AEDT today

  3. #43
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Repeat gateway issue this afternoon. Am suspecting that this may be due to an attack on the gateway, OR some of the small changes to the website in preparation of forum update. If it's related to website adjustments, it's only affecting one gateway (which are not actually part of the website, they just direct your incoming browser requests to the server from one of two public IP addresses), so this seems unlikely as the gateways are near identical.

    Outage affecting 50% of users from about 1:15PM to 2:35PM AEST.

    Will keep an eye on, I will probably attempt the forum update again tonight. After the update the forum config can be returned to prior to update. There are options that need to be turned on for the update to process successfully.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Federal RS-RR - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - 2XS turbo manifold - Hypertech tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - PC based GPS and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift.

  4. #44
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Having some wet weather related issues with the copper internet line in VIC. Short lived outages - if you have difficulty accessing the site, please retry in a few minutes time.
    If you find the site appears down for longer than 15 minutes, please report issue to ozmpsclub facebook page

  5. #45
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Server restarted due to software fault this morning ~11:20AM. Cause unclear but suspected to be an attack on the web service.

  6. #46
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Have had a day or two of slow website performance thanks to an update to the reverse proxy gateways inadvertently breaking it's internet route, effectively taking it offline - we've been operating on the QLD gateway only while this fault was present.

  7. #47
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Website down due to full storage from sometime after 1:00AM 11/08/17 through to 12:30PM 11/08/17 AEST
    Logs had blown out and filled the storage. Resolved by archiving the logs and deleting from the webserver host.

  8. #48
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default Registration fault

    Hi everyone, registration of new users appears to be failing at present - it's unclear why or if it effects all registration tools or just manual registration (without using social media account) I'll be investigating tonight. There was what appeared to be an attack on the site a week or so ago, or it may be more benign such as a bug with the social media registration tools. Will update when I have discovered more about the problem.

  9. #49
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default Registration page restored

    The registration page was failing due to faulty human verification. Recapcha was enabled, which is an external service that asks you to select all images with roads in them and various similar checks to verify you are a human and not a bot. As this was failing, I have disabled it and replaced the feature with a simpler text-image check i.e. type what you see in this image - to verify you are human.

    Registration is expected to be fully functional again.

  10. #50
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Outage this morning due to power and internet outages at VIC datacenter.

  11. #51
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Short outage around 7:30PM for 15minutes due a server software fault. Was a significant fault that locked the hypervisor up. I am intending in the near future to apply an update to the hypervisor in the hope that these rare lockup events will be avoided altogether. We've had a small number of them this year.

  12. #52
    Nexus's Avatar
    Nexus is offline Administrator
    Join Date
    Nov 2008
    Location
    Geelong
    Age
    45
    Posts
    1,515
    Blog Entries
    1

    Default

    Significant outage this morning from early hours through to around 8:30 due to server stall. I am considering moving ozmps to a different server as the primary server has shown less than perfect stability with an uptick in events like this. a couple of years ago it was once every 6 months, which is tolerable given the highly complex device configuration. but currently it's as much as once per month. That may mean removing devices until the culprit is identified; it seems unlikely to be a major component such as mainboard memory cpu or pxi express switch.

Page 3 of 3 FirstFirst 123

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •