Upfirst - Elevated 401 errors for API endpoints – Incident details

All systems operational

Elevated 401 errors for API endpoints

Resolved
Degraded performance
Started 4 months agoLasted 1 day

Affected

OpenAI → Chat Completions API

Operational from 8:40 PM to 11:26 PM, Degraded performance from 11:26 PM to 8:41 AM

Updates
  • Resolved
    UTC
    Resolved

    All impacted services have now fully recovered.

  • Monitoring
    UTC
    Monitoring

    We have applied the mitigation and are monitoring the recovery.

  • Update
    UTC
    Update

    We've identified that intermittent 401 errors on some API endpoints are being caused by timeouts. We have applied mitigations that reduced impact in parts of the system and are continuing to deploy additional fixes and monitor recovery until error rates return to expected levels.

  • Update
    UTC
    Update

    We've identified that intermittent 401 errors on some API endpoints are being caused by timeouts. We have applied mitigations that reduced impact in parts of the system and are continuing to deploy additional fixes and monitor recovery until error rates return to expected levels.

  • Update
    UTC
    Update

    We've identified that intermittent 401 errors on some API endpoints are being caused by timeouts. We have applied mitigations that reduced impact in parts of the system and are continuing to deploy additional fixes and monitor recovery until error rates return to expected levels.

  • Identified
    UTC
    Identified

    We have identified that users are experiencing elevated errors for the impacted services.

    We are working on implementing a mitigation.

  • Monitoring
    UTC
    Monitoring

    We have applied the mitigation and are monitoring the recovery.

  • Investigating
    UTC
    Investigating

    We are investigating the issue for the listed services.

  • Monitoring
    UTC
    Monitoring

    We have applied the mitigation and are monitoring the recovery.

  • Investigating
    UTC
    Investigating

    We are investigating the issue for the listed services.