Tremendous - Notice history

All systems operational

API - Operational

100% - uptime
Oct 2022
Nov 2022
Dec 2022

Reward redemptions - Operational

100% - uptime
Oct 2022
Nov 2022
Dec 2022

Marketing site - Operational

100% - uptime
Oct 2022
Nov 2022
Dec 2022

Dashboard - Operational

100% - uptime
Oct 2022
Nov 2022
Dec 2022

Notice history

Dec 2022

API order creation interrupted
  • Resolved
    Resolved

    Some percentage of API orders attempted from 6:54am to 7:17am EST this morning were unable to be created, and our server responded with a 500 response. Our engineers have identified the root cause, a fix has been deployed and the functionality has been fully restored.

Tremendous Dashboard not loading
  • Resolved
    Resolved

    The incident has been resolved.

  • Monitoring
    Monitoring

    We've rolled back some changes and access should be restored. We're monitoring changes now.

  • Identified
    Identified

    Unfortunately, the fix we deployed seems to have restored some functionality, but not all. We are working on another fix.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    We're investigating an issue where the Tremendous dashboard isn't loading, and a white page renders instead. We've identified the cause (a change in how our JS is bundled), and are working on a fix. API orders and reward redemptions are not affected.

Short outage during internal system upgrade
  • Resolved
    Resolved

    During an internal upgrade in our software stack, we experienced a brief (30-60 seconds) outage today. Operations have been fully restored.

Nov 2022

Marketing site experiencing 406 errors
  • Resolved
    Update

    We hit the frontpage of Hacker News \(a social media site\) in the wee hours of the morning EST, and received an unexpected influx of traffic to our blog. This overwhelmed our marketing site infrastructure–most likely, Webflow, which we use to host the site. The site started returning 406’s \(which is still a mystery\). We’ve already been working on migrating our marketing site away from Webflow to a Vercel-hosted GatsbyJS site. This should scale much better, and allow for basic engineering features like logging and alerts.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've deployed a fix, and the marketing site is back up.

  • Identified
    Identified

    It seems like the issue is related to a misbehaving Cloudflare worker. We're double checking something, but should be back up soon....

  • Investigating
    Investigating

    We are currently investigating error requests to www.tremendous.com. These seem to be isolated to our marketing site. You can still log into Tremendous by visiting app.tremendous.com. Reward redemptions and our API are also unaffected.

Oct 2022

Reward redemptions experiencing issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    We've identified an issue where recipients are unable to successfully complete the reward redemption flow. We've identified the issue (a routine code change that caused a validation during the redemption flow to fail, thereby blocking reward redemptions). A fix is going out, and the issue should be resolved shortly.

Site availability issues
  • Resolved
    Resolved

    An error during a software update of our application servers lead to unexpected downtime from 8:16am EST to 8:41am EST. Our engineering team was able to resolve this issue and subsequently re-apply the update successfully.

Oct 2022 to Dec 2022

Next