Tremendous - Notice history

All systems operational

API - Operational

99% - uptime
Oct 2021
Nov 2021
Dec 2021

Reward redemptions - Operational

99% - uptime
Oct 2021
Nov 2021
Dec 2021

Marketing site - Operational

100% - uptime
Oct 2021
Nov 2021
Dec 2021

Dashboard - Operational

99% - uptime
Oct 2021
Nov 2021
Dec 2021

Notice history

Dec 2021

AWS outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Update

    We are no longer experiencing issues. Our upstream providers haven't marked their issues as resolved, so we're continuing to monitor ourselves.

  • Monitoring
    Update

    We are continuing to monitor for any further issues.

  • Monitoring
    Update

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Monitoring
    Monitoring

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Identified
    Update

    OK, AWS seems to have lost power to one of their data centers in the US-East-1 region. They're working to restore power now, and are starting to see servers come back.

  • Identified
    Identified

    Confirmed that the issue is related to AWS, and so large parts of the internet are down too. We've already contacted our downstream providers, but unfortunately we're going to have to wait for them to figure this out.

  • Investigating
    Investigating

    Production servers are returning 503 - Application Errors. On first glance, this appears to be related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.

Degraded background processing performance
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    The issue has been identified and a fix is being implemented.

  • Investigating
    Investigating

    We're investigating an issue related to degraded performance of our background processing queues. This is resulting in emails, including order confirmations or reward emails, going out on a delayed basis to users. Our engineering team has identified the root cause and is applying a fix now.

Database degraded performance
  • Resolved
    Resolved

    Incident has been resolved, all systems back to normal.

  • Monitoring
    Update

    All systems appear to be operational now. We are still monitoring some delays in our background processing systems, but there should be minimal impact on component performance.

  • Monitoring
    Monitoring

    We've deployed a fix and are monitoring improvements. Performance issues are starting to dissipate, and should be fully back to normal within a few hours.

  • Identified
    Identified

    We're investigating degraded app performance resulting in longer response times and some timeouts. Engineers have identified the root cause and are working to resolve.

Nov 2021

No notices reported this month

Oct 2021

Database degraded performance
  • Resolved
    Resolved

    Incident has been resolved.

  • Monitoring
    Monitoring

    Engineers have deployed fixes and are seeing improvements; now just monitoring for things to get back to status quo.

  • Identified
    Identified

    We're investigating degraded app performance resulting in longer response times and some timeouts. Engineers have identified the root cause and are working to resolve.

Oct 2021 to Dec 2021

Next