Tremendous - Notice history

At about 12:50 UTC there was a spike in the number of database connections used, to the point of exhausting the maximum number of available database connections. Upon investigation we realized that this was caused by a rapid succession of new app releases, which caused our infrastructure provider Heroku to launch lots of new app dynos. However, the old dynos are not shut down immediately. So with each new release the number of used connections went up. At this time we had already stopped issuing new releases so the issue resolved itself as the old app dynos started being shut down. As remediation, we are updating our process to prevent the issuing of this many new releases in such a short timeframe.

Our background processing framework is processing tasks too slowly. This is resulting in lag for things like sending reward emails, webhooks going out, and new orders / rewards showing up in search bars. It appears to have started yesterday. Engineers have identified the issue and are working on a fix.

We're currently investigating downtime in all services. We've already identified the cause, and are working to deploy a fix.

Over 20 minutes this morning, we were experiencing timeout errors for very few requests to our API due to monthly maintenance work. We are working with the few affected customers directly over the coming hours to ensure all orders get placed as intended.