tremendous-status - Notice history

Reward redemptions experiencing degraded performance

API - Operational

99.25% uptime
Aug 2022
Sep 2022
Oct 2022

Reward redemptions - Degraded performance

99.93% uptime
Aug 2022
Sep 2022
Oct 2022

Marketing site - Operational

100.0% uptime
Aug 2022
Sep 2022
Oct 2022

Dashboard - Operational

99.95% uptime
Aug 2022
Sep 2022
Oct 2022

Sandbox - Operational

Website - Operational

100.0% uptime
Aug 2022
Sep 2022
Oct 2022

Notice history

View current status

Oct 2022

Reward redemptions experiencing issues
ResolvedMajor outage16 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    A fix has been implemented and we are monitoring the results.

  • IdentifiedUTC

    We've identified an issue where recipients are unable to successfully complete the reward redemption flow. We've identified the issue (a routine code change that caused a validation during the redemption flow to fail, thereby blocking reward redemptions). A fix is going out, and the issue should be resolved shortly.

Site availability issues
ResolvedPartial outage2 hours 45 minutes

  • An error during a software update of our application servers lead to unexpected downtime from 8:16am EST to 8:41am EST. Our engineering team was able to resolve this issue and subsequently re-apply the update successfully.

    UTC

Sep 2022

SEP27
Database upgrades
CompletedMaintenance1 hour 26 minutes
  • CompletedSeptember 27, 2022 at 9:26 AMUTC

    The scheduled maintenance has been completed.

  • In progressSeptember 27, 2022 at 8:00 AMUTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • UpdateSeptember 27, 2022 at 7:00 AMUTC

    Maintenance will begin as scheduled in 60 minutes.

  • UpdateSeptember 08, 2022 at 1:57 PMUTC

    We will be undergoing scheduled maintenance during this time.

  • PlannedSeptember 08, 2022 at 1:57 PMUTC

    We'll be undergoing a scheduled database upgrade. During this time, all Tremendous services will be down. API requests will return a 503, and UIs will show a "Down for scheduled maintenance" page.

SEP15
Testflight database upgrade
CompletedMaintenance12 minutes
  • CompletedSeptember 15, 2022 at 7:42 PMUTC

    The scheduled maintenance has been completed.

  • In progressSeptember 15, 2022 at 7:31 PMUTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • UpdateSeptember 15, 2022 at 6:30 PMUTC

    Maintenance will begin as scheduled in 60 minutes.

  • PlannedSeptember 14, 2022 at 8:02 PMUTC

    The sandbox environment (testflight.tremendous.com) will be undergoing scheduled maintenance during this time. There will be no impacts on the production environment.

Reward redemptions down
ResolvedPartial outage

  • Reward redemptions were down for approximately 30 minutes this afternoon. The cause was a hosting configuration change that engineers had deployed just prior. Engineers promptly rolled back the offending configuration change, and things went back to normal. Apologies, and thank you for bearing with us.

    UTC

Aug 2022

AUG26
Database maintenance
CompletedMaintenance30 minutes
  • CompletedAugust 26, 2022 at 11:52 PMUTC

    The scheduled maintenance has been completed.

  • UpdateAugust 26, 2022 at 11:50 PMUTC

    Verification is currently underway for the maintenance items.

  • In progressAugust 26, 2022 at 11:21 PMUTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  • PlannedAugust 26, 2022 at 11:20 PMUTC

    We are realizing a database maintenance.

Downtime with upstream DNS provider
ResolvedMajor outage1 hour 10 minutes
  • ResolvedUTC

    This incident has been resolved.

  • UpdateUTC

    All seems OK now, we're monitoring for any other issues.

  • MonitoringUTC

    DNS seems to be coming back now.

  • UpdateUTC

    We're standing by, waiting for our hosting and DNS providers to fix their issues. Following along at status.heroku.com

  • IdentifiedUTC

    The issue has been identified and a fix is being implemented.

  • InvestigatingUTC

    Our hosting provider, Heroku, is experiencing DNS issues. These appear to be affecting Tremendous. https://status.heroku.com/

API Key Rotation
ResolvedPartial outage50 hours 29 minutes
  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC

    Vast majority of active API clients have updated their tokens. Thank you all.

  • InvestigatingUTC

    We're currently in the process of rotating customer API keys. If you are receiving 401 Unauthorized responses from the API, you can fix this by obtaining a new API key from the Tremendous Dashboard.

Previous

Aug 2022 to Oct 2022

Next