tremendous-status - Notice history

Reward redemptions experiencing degraded performance

API - Operational

100.0% uptime
Nov 2022
Dec 2022
Jan 2023

Reward redemptions - Degraded performance

100.0% uptime
Nov 2022
Dec 2022
Jan 2023

Marketing site - Operational

100.0% uptime
Nov 2022
Dec 2022
Jan 2023

Dashboard - Operational

99.99% uptime
Nov 2022
Dec 2022
Jan 2023

Sandbox - Operational

Website - Operational

100.0% uptime
Nov 2022
Dec 2022
Jan 2023

Notice history

View current status

Jan 2023

502s during deploy of application upgrades
ResolvedDegraded performance

  • We experienced a partial outage during a routine application upgrade this morning, lasting from 9:18am - 9:25am. During this time, a percentage of requests received a 502 Bad Gateway error. Engineers identified the issue (related to a load balancer) and quickly deployed a fix. All services are fully functional again.

    UTC

Dec 2022

API order creation interrupted
ResolvedDegraded performance

  • Some percentage of API orders attempted from 6:54am to 7:17am EST this morning were unable to be created, and our server responded with a 500 response. Our engineers have identified the root cause, a fix has been deployed and the functionality has been fully restored.

    UTC
Tremendous Dashboard not loading
ResolvedPartial outage54 minutes
  • ResolvedUTC

    The incident has been resolved.

  • MonitoringUTC

    We've rolled back some changes and access should be restored. We're monitoring changes now.

  • IdentifiedUTC

    Unfortunately, the fix we deployed seems to have restored some functionality, but not all. We are working on another fix.

  • MonitoringUTC

    A fix has been implemented and we are monitoring the results.

  • IdentifiedUTC

    We're investigating an issue where the Tremendous dashboard isn't loading, and a white page renders instead. We've identified the cause (a change in how our JS is bundled), and are working on a fix. API orders and reward redemptions are not affected.

Short outage during internal system upgrade
ResolvedPartial outage

  • During an internal upgrade in our software stack, we experienced a brief (30-60 seconds) outage today. Operations have been fully restored.

    UTC

Nov 2022

Marketing site experiencing 406 errors
ResolvedDegraded performance15 minutes
  • UpdateUTC

    We hit the frontpage of Hacker News \(a social media site\) in the wee hours of the morning EST, and received an unexpected influx of traffic to our blog. This overwhelmed our marketing site infrastructure–most likely, Webflow, which we use to host the site. The site started returning 406’s \(which is still a mystery\). We’ve already been working on migrating our marketing site away from Webflow to a Vercel-hosted GatsbyJS site. This should scale much better, and allow for basic engineering features like logging and alerts.

  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    We've deployed a fix, and the marketing site is back up.

  • IdentifiedUTC

    It seems like the issue is related to a misbehaving Cloudflare worker. We're double checking something, but should be back up soon....

  • InvestigatingUTC

    We are currently investigating error requests to www.tremendous.com. These seem to be isolated to our marketing site. You can still log into Tremendous by visiting app.tremendous.com. Reward redemptions and our API are also unaffected.

Previous

Nov 2022 to Jan 2023

Next