Tremendous - Notice history

All systems operational

API - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Reward redemptions - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Marketing site - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Dashboard - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Sandbox - Operational

Notice history

View current status

Jan 2023

502s during deploy of application upgrades
ResolvedDegraded performance
  • Resolved
    UTC
    Resolved

    We experienced a partial outage during a routine application upgrade this morning, lasting from 9:18am - 9:25am. During this time, a percentage of requests received a 502 Bad Gateway error. Engineers identified the issue (related to a load balancer) and quickly deployed a fix. All services are fully functional again.

Dec 2022

API order creation interrupted
ResolvedDegraded performance
  • Resolved
    UTC
    Resolved

    Some percentage of API orders attempted from 6:54am to 7:17am EST this morning were unable to be created, and our server responded with a 500 response. Our engineers have identified the root cause, a fix has been deployed and the functionality has been fully restored.

Tremendous Dashboard not loading
ResolvedPartial outage54 minutes
  • Resolved
    UTC
    Resolved

    The incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've rolled back some changes and access should be restored. We're monitoring changes now.

  • Identified
    UTC
    Identified

    Unfortunately, the fix we deployed seems to have restored some functionality, but not all. We are working on another fix.

  • Monitoring
    UTC
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    UTC
    Identified

    We're investigating an issue where the Tremendous dashboard isn't loading, and a white page renders instead. We've identified the cause (a change in how our JS is bundled), and are working on a fix. API orders and reward redemptions are not affected.

Short outage during internal system upgrade
ResolvedPartial outage
  • Resolved
    UTC
    Resolved

    During an internal upgrade in our software stack, we experienced a brief (30-60 seconds) outage today. Operations have been fully restored.

Nov 2022

Marketing site experiencing 406 errors
ResolvedDegraded performance15 minutes
  • Update
    UTC
    Update

    We hit the frontpage of Hacker News \(a social media site\) in the wee hours of the morning EST, and received an unexpected influx of traffic to our blog. This overwhelmed our marketing site infrastructure–most likely, Webflow, which we use to host the site. The site started returning 406’s \(which is still a mystery\). We’ve already been working on migrating our marketing site away from Webflow to a Vercel-hosted GatsbyJS site. This should scale much better, and allow for basic engineering features like logging and alerts.

  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've deployed a fix, and the marketing site is back up.

  • Identified
    UTC
    Identified

    It seems like the issue is related to a misbehaving Cloudflare worker. We're double checking something, but should be back up soon....

  • Investigating
    UTC
    Investigating

    We are currently investigating error requests to www.tremendous.com. These seem to be isolated to our marketing site. You can still log into Tremendous by visiting app.tremendous.com. Reward redemptions and our API are also unaffected.

Previous

Nov 2022 to Jan 2023

Next