tremendous-status - Degraded performance with background processing framework – Incident details

Reward redemptions experiencing degraded performance

Show current status

Degraded performance with background processing framework

ResolvedDegraded performance43 minutes

Affected

API

Degraded performance from 9:53 PM to 10:36 PM

Reward redemptions

Degraded performance from 9:53 PM to 10:36 PM

Dashboard

Degraded performance from 9:53 PM to 10:36 PM

  • ResolvedUTC

    This issue is now resolved and all background tasks are being picked up quickly again.

  • MonitoringUTC

    Fix has been implemented. Things are steadily going back to normal- we're monitoring to make sure all stays well.

  • IdentifiedUTC

    Our background processing framework is processing tasks too slowly. This is resulting in lag for things like sending reward emails, webhooks going out, and new orders / rewards showing up in search bars. It appears to have started yesterday. Engineers have identified the issue and are working on a fix.