User Tag List

Page 3 of 4 FirstFirst 1234 LastLast
Results 41 to 60 of 69

Thread: Unplanned outage

  1. #41
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Discovered this afternoon that one of the two gateways that your browser reaches the website through was playing up (VIC gateway), and doing so in such a way that your browsers were not falling back to the gateway that was working (QLD)
    As a result up to 50% of users trying to browse the website may have seen timeout or connection reset messages. Not sure how long this problem had been occurring - I suspect a day or two - as I hadn't seen the symptoms myself. Issue identified and reported through Facebook group. This is first time we've seen this issue that I can recall. Normally if one gateway is down, you should still be able to browse the website. In this case the gateway wasn't down in a way that caused browsers to fail over to the alternative gateway.Thanks Adrian for reporting the problem.

    VIC gateway has been restarted and is behaving normally again.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Michelin PS5 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - Corksport turbo manifold - HT 98 octane tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - Bespoke Raspberry Pi Android based GPS/Carplay and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift - 2XS turbo manifold.

  2. #42
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    internal network link failure between QLD and VIC caused similar faults to the gateway issue identified yesterday. Back online. Outage for 50% of users was from approx 9:20AM to 10:15AM AEDT today

  3. #43
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Repeat gateway issue this afternoon. Am suspecting that this may be due to an attack on the gateway, OR some of the small changes to the website in preparation of forum update. If it's related to website adjustments, it's only affecting one gateway (which are not actually part of the website, they just direct your incoming browser requests to the server from one of two public IP addresses), so this seems unlikely as the gateways are near identical.

    Outage affecting 50% of users from about 1:15PM to 2:35PM AEST.

    Will keep an eye on, I will probably attempt the forum update again tonight. After the update the forum config can be returned to prior to update. There are options that need to be turned on for the update to process successfully.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Michelin PS5 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - Corksport turbo manifold - HT 98 octane tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - Bespoke Raspberry Pi Android based GPS/Carplay and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift - 2XS turbo manifold.

  4. #44
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Having some wet weather related issues with the copper internet line in VIC. Short lived outages - if you have difficulty accessing the site, please retry in a few minutes time.
    If you find the site appears down for longer than 15 minutes, please report issue to ozmpsclub facebook page

  5. #45
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Server restarted due to software fault this morning ~11:20AM. Cause unclear but suspected to be an attack on the web service.

  6. #46
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Have had a day or two of slow website performance thanks to an update to the reverse proxy gateways inadvertently breaking it's internet route, effectively taking it offline - we've been operating on the QLD gateway only while this fault was present.

  7. #47
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Website down due to full storage from sometime after 1:00AM 11/08/17 through to 12:30PM 11/08/17 AEST
    Logs had blown out and filled the storage. Resolved by archiving the logs and deleting from the webserver host.

  8. #48
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default Registration fault

    Hi everyone, registration of new users appears to be failing at present - it's unclear why or if it effects all registration tools or just manual registration (without using social media account) I'll be investigating tonight. There was what appeared to be an attack on the site a week or so ago, or it may be more benign such as a bug with the social media registration tools. Will update when I have discovered more about the problem.

  9. #49
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default Registration page restored

    The registration page was failing due to faulty human verification. Recapcha was enabled, which is an external service that asks you to select all images with roads in them and various similar checks to verify you are a human and not a bot. As this was failing, I have disabled it and replaced the feature with a simpler text-image check i.e. type what you see in this image - to verify you are human.

    Registration is expected to be fully functional again.

  10. #50
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Outage this morning due to power and internet outages at VIC datacenter.

  11. #51
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Short outage around 7:30PM for 15minutes due a server software fault. Was a significant fault that locked the hypervisor up. I am intending in the near future to apply an update to the hypervisor in the hope that these rare lockup events will be avoided altogether. We've had a small number of them this year.

  12. #52
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Significant outage this morning from early hours through to around 8:30 due to server stall. I am considering moving ozmps to a different server as the primary server has shown less than perfect stability with an uptick in events like this. a couple of years ago it was once every 6 months, which is tolerable given the highly complex device configuration. but currently it's as much as once per month. That may mean removing devices until the culprit is identified; it seems unlikely to be a major component such as mainboard memory cpu or pxi express switch.

  13. #53
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Significant outage from about 10:30PM AEDT last night to 12:00AM today due to flooding. Tripped circuit breakers, still cleaning up.

  14. #54
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Significant outage from sometime last last night through to 2:00PM AEDT today due to disk full and subsequent database failure - event logs had filled disk leaving insufficient storage for the database to operate..
    Will look at creating an auto tidy-up. Left to it's own devices takes best part of a year to fill the disk with logs.

  15. #55
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Significant outage from about 12:01AM this morning through to about 1:44PM due to a VIC state wide ISP outage. Appears to be resolved now.

  16. #56
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Whoops. Access to the website was impaired today because SSL certificate had expired. I had begun renewing it a few days ago but got distracted by problems with the certificate management tools I was using. Fixed that and forgot to actually renew the ozmpsclub certificate. Had to reissue instead of renew. doh

    My bad, sorry for inconvenience

  17. #57
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Outage today for about an hour due to a system lockup. Appears the server hardware needs a firmware update. There will be a restart outage for that when it happens.

  18. #58
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Having some database issues that are causing problems accessing the site. You may see gateway timeout messages. I am currently investigating the problem.

  19. #59
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default

    Quote Originally Posted by Nexus View Post
    Having some database issues that are causing problems accessing the site. You may see gateway timeout messages. I am currently investigating the problem.
    Appears to have choked while running a daily backup. Have cleared disk space and performed database maintenance and all appears well now.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Michelin PS5 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - Corksport turbo manifold - HT 98 octane tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - Bespoke Raspberry Pi Android based GPS/Carplay and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift - 2XS turbo manifold.

  20. #60
    Join Date
    Nov 2008
    Location
    Craigieburn
    Age
    50
    Posts
    1,642
    Blog Entries
    1

    Default Significant Outage

    Hi everyone, apologies significant outage this afternoon/evening for several hours due to failed storage array controller. Server crashed had to swap part out with spare. Good thing I had a spare, although now that it's running again I can think of a way I could have brought it online if I hadn't. No data loss or corruption, although I was concerned there may be for a bit.

    Apologies I've been a bit absent recent weeks, got some really full on shennanigans going down that I have to deal with, but there's light at the end of the tunnel.
    "Blue Meanie" 2007 Aurora Blue MPS 3 - 18x8.5+44 SSR GTX01 - 235/40R18 Michelin PS5 - 3.5" ETS TMIC - CPE stg 2 mount - HKS/CPE BPV - 2XS inlet - 2XS short shift - Corksport turbo manifold - HT 98 octane tune - Leather/Aluminium handbrake - Momo shifty knob - 7" touchscreen - JDM Mazda Retractable dashtop screen assembly - Bespoke Raspberry Pi Android based GPS/Carplay and instrumentation - 36AH reserve battery and C-TEK isolator - TEIN Street Advanced coilovers 1" drop - Superpro bushings - 220Kw/410Nm.

    "Lipstick" 2013 Velocity Red MPS 3 - 18x7.5+48 Enkei RPF1 -225/40R18 Federal RS-RR - CPE TMIC - COBB inlet - CPE stg 2 mount - COBB Stage 1 98 octane tune - COBB shifty knob - 2XS short shift - 2XS turbo manifold.

Page 3 of 4 FirstFirst 1234 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •