Tremendous - Notice history

All systems operational

API - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Reward redemptions - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Marketing site - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Dashboard - Operational

100% - uptime
Nov 2022
Dec 2022
Jan 2023

Notice history

Jan 2023

502s during deploy of application upgrades
  • Resolved
    Resolved

    We experienced a partial outage during a routine application upgrade this morning, lasting from 9:18am - 9:25am. During this time, a percentage of requests received a 502 Bad Gateway error. Engineers identified the issue (related to a load balancer) and quickly deployed a fix. All services are fully functional again.

Dec 2022

API order creation interrupted
  • Resolved
    Resolved

    Some percentage of API orders attempted from 6:54am to 7:17am EST this morning were unable to be created, and our server responded with a 500 response. Our engineers have identified the root cause, a fix has been deployed and the functionality has been fully restored.

Tremendous Dashboard not loading
  • Resolved
    Resolved

    The incident has been resolved.

  • Monitoring
    Monitoring

    We've rolled back some changes and access should be restored. We're monitoring changes now.

  • Identified
    Identified

    Unfortunately, the fix we deployed seems to have restored some functionality, but not all. We are working on another fix.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    We're investigating an issue where the Tremendous dashboard isn't loading, and a white page renders instead. We've identified the cause (a change in how our JS is bundled), and are working on a fix. API orders and reward redemptions are not affected.

Short outage during internal system upgrade
  • Resolved
    Resolved

    During an internal upgrade in our software stack, we experienced a brief (30-60 seconds) outage today. Operations have been fully restored.

Nov 2022

Marketing site experiencing 406 errors
  • Update
    Update

    We hit the frontpage of Hacker News \(a social media site\) in the wee hours of the morning EST, and received an unexpected influx of traffic to our blog. This overwhelmed our marketing site infrastructure–most likely, Webflow, which we use to host the site. The site started returning 406’s \(which is still a mystery\). We’ve already been working on migrating our marketing site away from Webflow to a Vercel-hosted GatsbyJS site. This should scale much better, and allow for basic engineering features like logging and alerts.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've deployed a fix, and the marketing site is back up.

  • Identified
    Identified

    It seems like the issue is related to a misbehaving Cloudflare worker. We're double checking something, but should be back up soon....

  • Investigating
    Investigating

    We are currently investigating error requests to www.tremendous.com. These seem to be isolated to our marketing site. You can still log into Tremendous by visiting app.tremendous.com. Reward redemptions and our API are also unaffected.

Nov 2022 to Jan 2023

Next