Duration: 1 hour and 27 minutes

Affected Components: Reward redemptions

Mar 10, 21:24:00 GMT+0
Identified - Recipients were unable to redeem their rewards during this timeframe..

Mar 10, 22:51:00 GMT+0
Resolved - We identified a change in our bot detection as the root cause for this incident, and have rolled back the change. We apologize for this incident. Recipients can now retry their redemption..

Duration: 1 hour and 3 minutes

Affected Components: Reward redemptions, Dashboard, API

Mar 2, 16:25:00 GMT+0
Investigating - The website and the API are unresponsive; we're investigating the cause..

Mar 2, 17:19:54 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 2, 17:28:29 GMT+0
Resolved - This incident has been resolved..

Duration: 5 hours and 18 minutes

Affected Components: Reward redemptions

Jan 30, 09:30:00 GMT+0
Identified - Prepaid card images aren't loading for a small subset of recipients..

Jan 30, 14:48:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 13 hours and 28 minutes

Affected Components: Reward redemptions

Dec 19, 00:38:48 GMT+0
Monitoring - Venmo has fixed the issue and we have released all delayed payouts. We are still working on refunding the incorrectly charged fees..

Dec 19, 17:38:03 GMT+0
Resolved - We've confirmed new payouts work correctly. We're marking this incident as resolved, but we're still working on refunding affected recipients. We will communicate that resolution directly to the affected recipients..

Dec 18, 04:10:00 GMT+0
Identified - We're aware that Venmo is incorrectly charging recipients fees. We are delaying Venmo redemptions while we work to fix this issue. We are also working on a resolution for recipients who received less than the correct amount. We'll update this as soon as we know more..

Duration: 30 minutes

Affected Components: Reward redemptions, Dashboard, API

Nov 22, 10:00:00 GMT+0
Identified - We will upgrade a critical part of our infrastructure, and that will require up to 30 minutes of downtime..

Nov 22, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 22, 10:30:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 3 hours and 3 minutes

Affected Components: Reward redemptions, Marketing site, Dashboard, API, Sandbox

Nov 18, 12:00:00 GMT+0
Investigating - Due to a global Cloudflare outage, we are currently experiencing disruption in our services. We will share updates as soon as more information becomes available. Cloudflare is aware of the issue and is currently investigating on their end: <https://www.cloudflarestatus.com/>.

Nov 18, 15:02:56 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 5 minutes

Affected Components: API, Reward redemptions

Oct 20, 14:32:28 GMT+0
Identified - A small portion of physical and digital prepaid cards are experiencing delays, preventing order placement and some cards from being displayed. The problem has been identified and we are working with our vendor to resolve this issue..

Oct 20, 21:37:03 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours

Affected Components: Sandbox

Sep 15, 07:10:00 GMT+0
Identified - An issue occurred during an automatic upgrade of our infrastructure in Google Cloud that required our manual intervention to fix. The Sandbox environment was not accessible during this period..

Sep 15, 14:10:00 GMT+0
Resolved - We've fixed the issue, and the environment is back up..

Duration: 17 hours and 25 minutes

Affected Components: Reward redemptions

Jun 20, 15:13:42 GMT+0
Resolved - Our provider \*\*Runa\*\* has traced the problem to Cloudflare mistakenly flagging some redemption links. Most customers can redeem normally, but a subset may still encounter errors. For real-time visibility into which gift-card brands are currently redeemable and future updates, please check <https://catalog-status.tremendous.com/>..

Jun 19, 21:48:32 GMT+0
Identified - **Impact:** Some gift-card redemptions processed through our upstream provider **Runa** are currently failing. **Status:** We’re working closely with Runa and monitoring their recovery efforts. We’ll post another update as soon as redemptions are fully restored..

Affected Components: Reward redemptions, Dashboard, API

Apr 24, 00:13:00 GMT+0
Resolved - SendGrid says email delivery is back to 100%. Email that was delayed has now been delivered. [You can also check SendGrid's status page directly for updates.](https://status.sendgrid.com/).

Duration: 37 minutes

Affected Components: API, Reward redemptions, Dashboard

Apr 16, 20:03:02 GMT+0
Resolved - This incident has been resolved..

Apr 16, 19:26:00 GMT+0
Monitoring - We've identified an issue causing high database load which is leading to slow response times across the board. We've identified a fix and are expecting performance to improve shortly..

Duration: 5 hours and 14 minutes

Affected Components: Reward redemptions

Apr 14, 00:07:32 GMT+0
Monitoring - The upstream vendor outage has ended, and we are monitoring things as they resolve to normal..

Apr 13, 21:20:50 GMT+0
Identified - Our primary provider for US-based gift cards is experiencing a production outage. We're monitoring the situation and hope to have reward redemptions reinstated shortly..

Apr 14, 02:34:25 GMT+0
Resolved - This incident has been resolved..

Duration: 32 minutes

Affected Components: Reward redemptions

Apr 8, 21:19:48 GMT+0
Monitoring - We are monitoring the situation..

Apr 8, 21:52:16 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 48 minutes

Affected Components: Reward redemptions

Mar 24, 19:28:51 GMT+0
Investigating - We are currently investigating this incident..

Mar 24, 21:17:18 GMT+0
Resolved - This incident has been resolved..

Duration: 20 minutes

Affected Components: API, Reward redemptions, Dashboard

Jan 13, 21:59:00 GMT+0
Resolved - This incident has been resolved..

Jan 13, 21:39:00 GMT+0
Investigating - Partially degraded performance on the dashboard and API..

Duration: 19 days, 12 hours and 44 minutes

Affected Components: API, Dashboard

Oct 25, 20:03:43 GMT+0
Resolved - We are seeing emails delivered immediately at this point..

Oct 25, 12:15:00 GMT+0
Monitoring - The service provider we use for email delivery, SendGrid, is experiencing an incident that is causing delays in email delivery. This can affect all emails we send, including reward deliveries and transactional emails such as login verification emails. SendGrid is assuring us that no email is lost, but this is currently causing significant delays in emails being delivered. We are monitoring the SendGrid incident and will verify that emails are delivered correctly..

Duration: 17 minutes

Affected Components: Dashboard

Sep 12, 15:32:03 GMT+0
Investigating - We're investigating order placement issues in the dashboard, where the final review screen shows "0" amounts for orders..

Sep 12, 15:34:43 GMT+0
Identified - We identified the problem and are deploying a fix. We expect things should be working again once the new code is in production, perhaps 15-20 minutes..

Sep 12, 15:49:16 GMT+0
Resolved - This incident has been resolved..

Sep 12, 15:50:12 GMT+0
Resolved - The incident has been resolved - if you are still experiencing issues, you may need to do a browser refresh (CMD + R or CTRL + R). Thank you for your patience..

Duration: 14 minutes

Affected Components: API, Marketing site

Aug 7, 19:39:22 GMT+0
Identified - Due to issues with [Vercel](https://www.vercel-status.com/) (our upstream hosting provider), the marketing site on [tremendous.com](http://tremendous.com) is down. The application - dashboard, reward redemptions, and all other product functionality - continues to be functional..

Aug 7, 19:41:33 GMT+0
Identified - Apparently our status page also has a dependency on Vercel, so [status.tremendous.com](http://status.tremendous.com) is also non-functional. In the meanwhile, we're redirecting all requests to [www.tremendous.com](http://www.tremendous.com) to [app.tremendous.com](http://app.tremendous.com) so users are able to log in. .

Aug 7, 19:51:17 GMT+0
Monitoring - We had a brief API outage related to the same issue (from 3:39pm EST to 3:49pm EST), but it's been resolved.. .

Aug 7, 19:53:16 GMT+0
Resolved - This incident has been resolved- Vercel has deployed a fix and is monitoring..

Duration: 5 minutes

Affected Components: API

Jul 24, 19:18:00 GMT+0
Resolved - This incident has been resolved..

Jul 24, 19:12:00 GMT+0
Identified - Today at around 3:13pm ET there was a problem with a database change we deployed that caused a number of actions to start failing across the site. The following actions failed during this time: * Placing new orders * Approving orders * Redemptions This problem lasted for about five minutes from 3:12:30 pm to 3:18:22 pm ET. Failed redemptions retry automatically and recipients should have received their payout by now. **However, you will have to retry newly placed orders as well as order approvals via the API.**.

Duration: 7 days, 5 hours and 56 minutes

Affected Components: Reward redemptions

Jul 19, 12:18:26 GMT+0
Investigating - Due to internal issues with an upstream fulfillment provider, redemptions for 100+ gift cards in the US are experiencing errors, including brands like Amazon, Walmart, Target, Starbucks. We are monitoring the product status with the fulfillment provider and should have redemption become available again soon..

Jul 19, 12:48:13 GMT+0
Identified - Our best guess is that this is related to the [Crowdstrike global outage](https://www.theverge.com/2024/7/19/24201717/windows-bsod-crowdstrike-outage-issue), given it's affecting multiple gift card fulfillment vendors that we rely on. .

Jul 19, 13:12:46 GMT+0
Identified - We're seeing progress from our upstream vendors. Though gift card fulfillment isn't working yet, their APIs seem to be coming online again. .

Jul 19, 13:22:56 GMT+0
Identified - Vendor APIs seem to be coming back online now, and fulfillment is slowly resuming back to normal..

Jul 19, 15:12:21 GMT+0
Resolved - Prepaid card issues seem to have subsided, but we are continuing to monitor for new issues..

Duration: 7 days and 6 hours

Affected Components: API

Jul 16, 13:36:00 GMT+0
Resolved - This incident has been resolved..

Jul 16, 10:42:00 GMT+0
Identified - Over the past year we have been gradually and methodically [migrating customers away from the usage of multi-reward orders](https://developers.tremendous.com/changelog/deprecating-multi-reward-orders). Today, we added a new validation to ensure that no new customers could start using multi-reward orders. Unfortunately there was a bug in our validation and we started rejecting create order payloads that contained only one reward, but did so in an array under the `rewards` key rather than the singular `reward` key. This was not intentional, and orders using that format continue to be valid today and into the future. Any order affected by this issue would have received a `422` response with the error message "Send data for a single reward in the reward field" This affected orders placed between 10:28 and 13:36 UTC (that's 6:42am to 9:36am ET). We apologize profusely for this issue. We are implementing additional tests and monitors to ensure this doesn't happen again. None of the affected orders were placed in our system; please retry them now and they should succeed..

Duration: 30 minutes

Affected Components: API, Marketing site, Reward redemptions, Dashboard

Apr 20, 10:30:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 20, 10:00:00 GMT+0
Identified - We will upgrade a critical part of our infrastructure, and that will require up to 30 minutes of downtime..

Apr 20, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 10 hours and 24 minutes

Affected Components: Reward redemptions

Feb 21, 17:38:33 GMT+0
Resolved - This incident has been resolved..

Feb 21, 07:15:00 GMT+0
Monitoring - Redemptions to PayPal and Venmo are currently delayed. We expect to pay out delayed redemptions starting at 11am ET today..

Feb 21, 16:36:15 GMT+0
Monitoring - New payouts are no longer being delayed and we have started to clear the queue of delayed payouts..

Affected Components: Reward redemptions

Feb 20, 08:00:00 GMT+0
Resolved - Redemptions to PayPal and Venmo were again delayed between 3am ET and noon ET. All delayed payouts were completed by noon ET and new payouts are no longer delayed..

Duration: 12 hours and 27 minutes

Affected Components: Reward redemptions

Feb 19, 12:18:01 GMT+0
Identified - Users will be notified as soon as their reward is available..

Feb 20, 00:44:32 GMT+0
Resolved - We were able to clear the delayed PayPal and Venmo payouts early. All delayed payouts have been completed at this point, and new redemptions are no longer delayed..

Duration: 43 minutes

Affected Components: API, Dashboard, Reward redemptions

Jan 27, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 27, 10:43:24 GMT+0
Completed - Maintenance has completed successfully..

Jan 27, 10:00:00 GMT+0
Identified - We will upgrade a critical part of our infrastructure, and that will require 1 hour of downtime..

Affected Components: Dashboard

Oct 20, 16:52:00 GMT+0
Resolved - There was a temporary problem where customers couldn't put orders through, either by manually adding recipients, or uploading CSVs. This is now resolved..

Duration: 10 hours

Affected Components: Dashboard

Oct 18, 19:47:00 GMT+0
Resolved - This incident has been resolved..

Oct 18, 19:36:00 GMT+0
Monitoring - We're experiencing issues with the Tremendous dashboard (logging in, placing orders, general navigation). The issue started ~10 minutes ago. We've already identified the problem and things should be resolved in the next 5 minutes..

Duration: 2 hours

Affected Components: API, Dashboard, Reward redemptions

Sep 8, 13:57:08 GMT+0
Identified - We'll be undergoing a scheduled database upgrade. During this time, all Tremendous services will be down. API requests will return a 503, and UIs will show a "Down for scheduled maintenance" page..

Sep 8, 13:57:52 GMT+0
Identified - We will be undergoing scheduled maintenance during this time..

Sep 27, 07:00:24 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Sep 27, 08:00:26 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Sep 27, 09:26:12 GMT+0
Completed - The scheduled maintenance has been completed..

Duration: 10 minutes

Affected Components: API, Dashboard

Sep 15, 19:31:41 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Sep 15, 18:30:23 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Sep 14, 20:02:15 GMT+0
Identified - The sandbox environment (testflight.tremendous.com) will be undergoing scheduled maintenance during this time. There will be no impacts on the production environment..

Sep 15, 19:42:34 GMT+0
Completed - The scheduled maintenance has been completed..

Duration: 20 minutes

Affected Components: API, Dashboard, Reward redemptions

Aug 26, 23:20:40 GMT+0
Identified - We are realizing a database maintenance..

Aug 26, 23:21:35 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Aug 26, 23:52:20 GMT+0
Completed - The scheduled maintenance has been completed..

Aug 26, 23:50:42 GMT+0
Identified - Verification is currently underway for the maintenance items..

Duration: 10 minutes

Affected Components: API, Dashboard, Reward redemptions

Mar 27, 11:00:26 GMT+0
Identified - Maintenance will begin as scheduled in 60 minutes..

Mar 25, 11:13:50 GMT+0
Identified - We will be undergoing scheduled maintenance during this time..

Mar 27, 12:10:23 GMT+0
Completed - The scheduled maintenance has been completed..

Mar 27, 12:00:39 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Duration: 1 hour

Affected Components: API, Dashboard, Reward redemptions

Sep 26, 11:38:17 GMT+0
Completed - Everything back to normal..

Sep 26, 11:27:15 GMT+0
Identified - Database upgrades are complete, just monitoring for any anomalies now..

Sep 26, 11:00:15 GMT+0
Identified - Scheduled maintenance is currently in progress. We will provide updates as necessary..

Sep 22, 13:05:56 GMT+0
Identified - We'll be undergoing scheduled maintenance for a database upgrade. During this time, API clients will receive "503 - Service Unavailable" responses. This should be complete within an hour..