Troubleshooting
Step-by-step guides for recovering from common production issues such as non-ready pods, locked Terraform state, full PVCs, and Loki alerts.
These guides cover recovering from problems in a running environment: pods that will not become ready, a Terraform state left locked by a failed release, a PersistentVolumeClaim running out of space, unhealthy Loki ingesters, and Promtail or Loki components being throttled.
Most of them work against the environment's Kubernetes cluster, so they start by downloading its credentials. Rolling back a service to a previously registered image is here too, and rolling back a release itself is covered under Rolling back.
Deployment Has Non-Ready Pods
Identify why a deployment or statefulset has non-ready pods by locating the pod name and using kubectl describe to inspect the events.
Artifact History & Rollback
View the artifact history for a service in the Facets Control Plane and roll back to a previously registered image from the Resource Center.
Expanding a PVC
Check disk utilization and resize a Kubernetes PersistentVolumeClaim, including Azure Managed Disk options with and without downtime.
Rolling Restart of Pods
Trigger a rolling restart of an application or service pods from the Facets UI, which runs kubectl rollout restart on the deployment for you.
Unlocking Terraform State
Recover from a state lock error after a failed release by copying the lock id from the release details and unlocking the Terraform state in Facets.
Setting Up Loki for VMs
Set up Loki to collect logs from VMs by adding a k8s_resource load balancer to your Blueprint and installing and configuring Promtail on the VM.
Handle Loki Alerts
Handle Loki and Promtail alerts for CPU throttling, memory throttling, and OOM killed pods by adjusting resource limits through spec overrides.
Removing Unhealthy Ingester Instances
Resolve too many unhealthy instances in the Loki ring by port-forwarding the ingester service and forgetting unhealthy instances from the ring UI.
External Tools & Usage
Out-of-the-box tools in Facets for monitoring and resource management: Kubernetes Dashboard, Prometheus, Alertmanager, Grafana, and Wetty.
Deployment Has Non-Ready Pods
Identify why a deployment or statefulset has non-ready pods by locating the pod name and using kubectl describe to inspect the events.