At about 12:50 UTC there was a spike in the number of database connections used, to the point of exhausting the maximum number of available database connections. Upon investigation we realized that this was caused by a rapid succession of new app releases, which caused our infrastructure provider Heroku to launch lots of new app dynos. However, the old dynos are not shut down immediately. So with each new release the number of used connections went up. At this time we had already stopped issuing new releases so the issue resolved itself as the old app dynos started being shut down. As remediation, we are updating our process to prevent the issuing of this many new releases in such a short timeframe.

Our background processing framework is processing tasks too slowly. This is resulting in lag for things like sending reward emails, webhooks going out, and new orders / rewards showing up in search bars. It appears to have started yesterday. Engineers have identified the issue and are working on a fix.

We're currently investigating downtime in all services. We've already identified the cause, and are working to deploy a fix.

Over 20 minutes this morning, we were experiencing timeout errors for very few requests to our API due to monthly maintenance work. We are working with the few affected customers directly over the coming hours to ensure all orders get placed as intended.

We’re investigating issues related to reward redemptions. Certain reward links appear to be returning “Failed to discover a valid reward”, despite being valid.

Production servers are returning 522. This is related to downstream issues with our hosting provider, unrelated to the Tremendous application. We are investigating.