Tremendous - Notice history

All systems operational

API - Operational

99% - uptime
Dec 2021
Jan 2022
Feb 2022

Reward redemptions - Operational

99% - uptime
Dec 2021
Jan 2022
Feb 2022

Marketing site - Operational

100% - uptime
Dec 2021
Jan 2022
Feb 2022

Dashboard - Operational

99% - uptime
Dec 2021
Jan 2022
Feb 2022

Notice history

Feb 2022

Reward redemptions outage
  • Resolved
    Resolved

    Reward redemptions are back to normal.

  • Identified
    Identified

    The issue has been identified and we're working on a fix.

  • Investigating
    Investigating

    We’re investigating issues related to reward redemptions. Certain reward links appear to be returning “Failed to discover a valid reward”, despite being valid.

Service outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Services are starting to recover. We'll keep monitoring.

  • Investigating
    Investigating

    Production servers are returning 522. This is related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.

Jan 2022

No notices reported this month

Dec 2021

AWS outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    We are no longer experiencing issues. Our upstream providers haven't marked their issues as resolved, so we're continuing to monitor ourselves.

  • Update
    Update

    We are continuing to monitor for any further issues.

  • Update
    Update

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Monitoring
    Monitoring

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Update
    Update

    OK, AWS seems to have lost power to one of their data centers in the US-East-1 region. They're working to restore power now, and are starting to see servers come back.

  • Identified
    Identified

    Confirmed that the issue is related to AWS, and so large parts of the internet are down too. We've already contacted our downstream providers, but unfortunately we're going to have to wait for them to figure this out.

  • Investigating
    Investigating

    Production servers are returning 503 - Application Errors. On first glance, this appears to be related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.

Degraded background processing performance
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    The issue has been identified and a fix is being implemented.

  • Investigating
    Investigating

    We're investigating an issue related to degraded performance of our background processing queues. This is resulting in emails, including order confirmations or reward emails, going out on a delayed basis to users. Our engineering team has identified the root cause and is applying a fix now.

Database degraded performance
  • Resolved
    Resolved

    Incident has been resolved, all systems back to normal.

  • Update
    Update

    All systems appear to be operational now. We are still monitoring some delays in our background processing systems, but there should be minimal impact on component performance.

  • Monitoring
    Monitoring

    We've deployed a fix and are monitoring improvements. Performance issues are starting to dissipate, and should be fully back to normal within a few hours.

  • Identified
    Identified

    We're investigating degraded app performance resulting in longer response times and some timeouts. Engineers have identified the root cause and are working to resolve.

Dec 2021 to Feb 2022

Next