Tremendous - Degraded performance with background processing framework – Incident details

Reward redemptions experiencing degraded performance

Degraded performance with background processing framework

Resolved
Degraded performance
Started over 4 years agoLasted 43 minutes

Affected

API

Degraded performance from 9:53 PM to 10:36 PM

Reward redemptions

Degraded performance from 9:53 PM to 10:36 PM

Dashboard

Degraded performance from 9:53 PM to 10:36 PM

Updates
  • Resolved
    UTC
    Resolved

    This issue is now resolved and all background tasks are being picked up quickly again.

  • Monitoring
    UTC
    Monitoring

    Fix has been implemented. Things are steadily going back to normal- we're monitoring to make sure all stays well.

  • Identified
    UTC
    Identified

    Our background processing framework is processing tasks too slowly. This is resulting in lag for things like sending reward emails, webhooks going out, and new orders / rewards showing up in search bars. It appears to have started yesterday. Engineers have identified the issue and are working on a fix.