Resolved -
The fix we applied 24 hours ago appears to be holding, so we are resolving this incident. We thank everyone for their patience!
Mar 4, 10:54 UTC
Monitoring -
We have made some adjustments based on recommendations from our database vendor and will continue to monitor their impact on system stability.
Mar 3, 08:11 UTC
Update -
Following guidance from our vendor, we've brought #AO3 back online to help troubleshoot. We expect the site will go back down, although we don't know when that will happen. If you choose to post or update your works while AO3 is up, we strongly recommend making sure you have a backup of any text you are attempting to post.
Mar 3, 04:36 UTC
Update -
AO3 is now fully down. After performing a software upgrade a few days ago, we've been unable to keep our server cluster healthy due to what we believe is a bug or issue with the new release of our database server software.
We're working with our vendor's support team to understand the cause and whether reverting or upgrading further will best resolve the issue without continuing the cycle of up and downtimes.
We do not currently have an estimated time for when we will be back up, but we expect it to be multiple hours at minimum. We apologize for the inconvenience and will update you when we know more.
Mar 3, 03:14 UTC
Investigating -
We are investigating reports of errors when trying to load AO3, including errors saying that AO3 is down, which seem to be related to the previous incident. We will update when we have more information!
Mar 2, 22:33 UTC