audit

identity

console

registry

grafana

logs

remote

workloads

wormholes

One of the upstream services that we leverage for container registry is experiencing an outage. Errors are intermittent.

Degraded Image push and pull

logs are being upgraded to provide faster queries and higher availability.

Logs upgrade

Log queries were unable to complete. Log data for workloads was still collected and is once again available to be queried.

Workload logs are not searchable

The Control Plane API was experiencing a high volume of errors. Initial data suggests that the issue was due to a cloud provider networking issue, however, a full root cause investigation is underway.

High error rate

A subset of Workloads in the aws-us-east-2 location were unable to resolve external DNS queries from 5:42am to 5:49am Pacific Time.

An outage occurred on the DNS resolution service in aws-us-east-2 region lasting 6 minutes (05:42:55 - 05:48:55 AM Pacific Time). The incident was triggered by a planned network configuration change intended to modify outbound routing rules. Due to an implementation issue in the change logic, instead of modifying the existing route, the system deleted and recreated the route for one cluster in aws-us-east-2. This deletion and recreation process resulted in a temporary loss of DNS resolution capabilities to external resources.

DNS Resolution Failures

DNS Lookups are failing for some workload replicas in GCP US-EAST1.

Teams are investigating the full root cause of this issue. Preliminary data indicates that a Cloud Provider change caused the network to fail in a non recoverable way on a small number of nodes in this location.

Control Plane Service Health

History of incidents reported by component