Tremendous - Notice history

All systems operational

API - Operational

100% - uptime
Sep 2022
Oct 2022
Nov 2022

Reward redemptions - Operational

100% - uptime
Sep 2022
Oct 2022
Nov 2022

Marketing site - Operational

100% - uptime
Sep 2022
Oct 2022
Nov 2022

Dashboard - Operational

100% - uptime
Sep 2022
Oct 2022
Nov 2022

Notice history

Nov 2022

Marketing site experiencing 406 errors
  • Update
    Update

    We hit the frontpage of Hacker News \(a social media site\) in the wee hours of the morning EST, and received an unexpected influx of traffic to our blog. This overwhelmed our marketing site infrastructure–most likely, Webflow, which we use to host the site. The site started returning 406’s \(which is still a mystery\). We’ve already been working on migrating our marketing site away from Webflow to a Vercel-hosted GatsbyJS site. This should scale much better, and allow for basic engineering features like logging and alerts.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've deployed a fix, and the marketing site is back up.

  • Identified
    Identified

    It seems like the issue is related to a misbehaving Cloudflare worker. We're double checking something, but should be back up soon....

  • Investigating
    Investigating

    We are currently investigating error requests to www.tremendous.com. These seem to be isolated to our marketing site. You can still log into Tremendous by visiting app.tremendous.com. Reward redemptions and our API are also unaffected.

Oct 2022

Reward redemptions experiencing issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A fix has been implemented and we are monitoring the results.

  • Identified
    Identified

    We've identified an issue where recipients are unable to successfully complete the reward redemption flow. We've identified the issue (a routine code change that caused a validation during the redemption flow to fail, thereby blocking reward redemptions). A fix is going out, and the issue should be resolved shortly.

Site availability issues
  • Resolved
    Resolved

    An error during a software update of our application servers lead to unexpected downtime from 8:16am EST to 8:41am EST. Our engineering team was able to resolve this issue and subsequently re-apply the update successfully.

Sep 2022

Database upgrades
  • Completed
    September 27, 2022 at 9:26 AM
    Completed
    September 27, 2022 at 9:26 AM

    The scheduled maintenance has been completed.

  • In progress
    September 27, 2022 at 8:00 AM
    In progress
    September 27, 2022 at 8:00 AM

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • Update
    September 27, 2022 at 7:00 AM
    Planned
    September 27, 2022 at 7:00 AM

    Maintenance will begin as scheduled in 60 minutes.

  • Update
    September 08, 2022 at 1:57 PM
    Planned
    September 08, 2022 at 1:57 PM

    We will be undergoing scheduled maintenance during this time.

  • Planned
    September 08, 2022 at 1:57 PM
    Planned
    September 08, 2022 at 1:57 PM

    We'll be undergoing a scheduled database upgrade. During this time, all Tremendous services will be down. API requests will return a 503, and UIs will show a "Down for scheduled maintenance" page.

Testflight database upgrade
  • Completed
    September 15, 2022 at 7:42 PM
    Completed
    September 15, 2022 at 7:42 PM

    The scheduled maintenance has been completed.

  • In progress
    September 15, 2022 at 7:31 PM
    In progress
    September 15, 2022 at 7:31 PM

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • Update
    September 15, 2022 at 6:30 PM
    Planned
    September 15, 2022 at 6:30 PM

    Maintenance will begin as scheduled in 60 minutes.

  • Planned
    September 14, 2022 at 8:02 PM
    Planned
    September 14, 2022 at 8:02 PM

    The sandbox environment (testflight.tremendous.com) will be undergoing scheduled maintenance during this time. There will be no impacts on the production environment.

Reward redemptions down
  • Resolved
    Resolved

    Reward redemptions were down for approximately 30 minutes this afternoon. The cause was a hosting configuration change that engineers had deployed just prior. Engineers promptly rolled back the offending configuration change, and things went back to normal. Apologies, and thank you for bearing with us.

Sep 2022 to Nov 2022

Next