Tremendous - Notice history

All systems operational

API - Operational

100% - uptime
Jul 2024
Aug 2024
Sep 2024

Reward redemptions - Operational

100% - uptime
Jul 2024
Aug 2024
Sep 2024

Marketing site - Operational

100% - uptime
Jul 2024
Aug 2024
Sep 2024

Dashboard - Operational

100% - uptime
Jul 2024
Aug 2024
Sep 2024

Notice history

Sep 2024

UI order placement failure
  • Update
    Update

    The incident has been resolved - if you are still experiencing issues, you may need to do a browser refresh (CMD + R or CTRL + R).

    Thank you for your patience.

  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We identified the problem and are deploying a fix. We expect things should be working again once the new code is in production, perhaps 15-20 minutes.

  • Investigating
    Investigating
    We're investigating order placement issues in the dashboard, where the final review screen shows "0" amounts for orders.

Aug 2024

Marketing pages are down
  • Resolved
    Resolved

    This incident has been resolved- Vercel has deployed a fix and is monitoring.

  • Monitoring
    Monitoring

    We had a brief API outage related to the same issue (from 3:39pm EST to 3:49pm EST), but it's been resolved..

  • Update
    Update

    Apparently our status page also has a dependency on Vercel, so status.tremendous.com is also non-functional.

    In the meanwhile, we're redirecting all requests to www.tremendous.com to app.tremendous.com so users are able to log in.

  • Identified
    Identified

    Due to issues with Vercel (our upstream hosting provider), the marketing site on tremendous.com is down. The application - dashboard, reward redemptions, and all other product functionality - continues to be functional.

Jul 2024

Brief outage placing and approving orders via API
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Today at around 3:13pm ET there was a problem with a database change we deployed that caused a number of actions to start failing across the site.

    The following actions failed during this time:

    • Placing new orders

    • Approving orders

    • Redemptions

    This problem lasted for about five minutes from 3:12:30 pm to 3:18:22 pm ET.

    Failed redemptions retry automatically and recipients should have received their payout by now. However, you will have to retry newly placed orders as well as order approvals via the API.

Fulfillment errors for Amazon.com, Visa Cards and other 100+ products
  • Resolved
    Resolved

    Prepaid card issues seem to have subsided, but we are continuing to monitor for new issues.

  • Update
    Update

    Vendor APIs seem to be coming back online now, and fulfillment is slowly resuming back to normal.

  • Update
    Update

    We're seeing progress from our upstream vendors. Though gift card fulfillment isn't working yet, their APIs seem to be coming online again.

  • Identified
    Identified

    Our best guess is that this is related to the Crowdstrike global outage, given it's affecting multiple gift card fulfillment vendors that we rely on.

  • Investigating
    Investigating

    Due to internal issues with an upstream fulfillment provider, redemptions for 100+ gift cards in the US are experiencing errors, including brands like Amazon, Walmart, Target, Starbucks.
    We are monitoring the product status with the fulfillment provider and should have redemption become available again soon.

API incorrectly rejecting some new orders
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    Over the past year we have been gradually and methodically migrating customers away from the usage of multi-reward orders. Today, we added a new validation to ensure that no new customers could start using multi-reward orders.

    Unfortunately there was a bug in our validation and we started rejecting create order payloads that contained only one reward, but did so in an array under the rewards key rather than the singular reward key. This was not intentional, and orders using that format continue to be valid today and into the future.

    Any order affected by this issue would have received a 422 response with the error message "Send data for a single reward in the reward field"

    This affected orders placed between 10:28 and 13:36 UTC (that's 6:42am to 9:36am ET). We apologize profusely for this issue. We are implementing additional tests and monitors to ensure this doesn't happen again.

    None of the affected orders were placed in our system; please retry them now and they should succeed.

Jul 2024 to Sep 2024

Next