Tremendous - AWS outage – Incident details

Reward redemptions experiencing degraded performance

AWS outage

Resolved
Major outage
Started over 4 years agoLasted about 24 hours

Affected

API

Major outage from 12:27 PM to 12:18 PM

Reward redemptions

Major outage from 12:27 PM to 12:18 PM

Dashboard

Major outage from 12:27 PM to 12:18 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Update
    UTC
    Update

    We are no longer experiencing issues. Our upstream providers haven't marked their issues as resolved, so we're continuing to monitor ourselves.

  • Update
    UTC
    Update

    We are continuing to monitor for any further issues.

  • Update
    UTC
    Update

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Monitoring
    UTC
    Monitoring

    Our production servers are back and we'll keep monitoring them to make sure it's stable again as our providers are still working on the issue.

  • Update
    UTC
    Update

    OK, AWS seems to have lost power to one of their data centers in the US-East-1 region. They're working to restore power now, and are starting to see servers come back.

  • Identified
    UTC
    Identified

    Confirmed that the issue is related to AWS, and so large parts of the internet are down too. We've already contacted our downstream providers, but unfortunately we're going to have to wait for them to figure this out.

  • Investigating
    UTC
    Investigating

    Production servers are returning 503 - Application Errors. On first glance, this appears to be related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.