Feb 2022
Reward redemptions are back to normal.
The issue has been identified and we're working on a fix.
We’re investigating issues related to reward redemptions. Certain reward links appear to be returning “Failed to discover a valid reward”, despite being valid.
This incident has been resolved.
Services are starting to recover. We'll keep monitoring.
Production servers are returning 522. This is related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.
Jan 2022
No notices reported this month
Dec 2021
We are no longer experiencing issues. Our upstream providers haven't marked their issues as resolved, so we're continuing to monitor ourselves.
We are continuing to monitor for any further issues.
Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.
OK, AWS seems to have lost power to one of their data centers in the US-East-1 region. They're working to restore power now, and are starting to see servers come back.
Confirmed that the issue is related to AWS, and so large parts of the internet are down too. We've already contacted our downstream providers, but unfortunately we're going to have to wait for them to figure this out.
Production servers are returning 503 - Application Errors. On first glance, this appears to be related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We're investigating an issue related to degraded performance of our background processing queues. This is resulting in emails, including order confirmations or reward emails, going out on a delayed basis to users. Our engineering team has identified the root cause and is applying a fix now.
Incident has been resolved, all systems back to normal.
All systems appear to be operational now. We are still monitoring some delays in our background processing systems, but there should be minimal impact on component performance.
We've deployed a fix and are monitoring improvements. Performance issues are starting to dissipate, and should be fully back to normal within a few hours.
We're investigating degraded app performance resulting in longer response times and some timeouts. Engineers have identified the root cause and are working to resolve.
Dec 2021 to Feb 2022
