Fix Kubernetes CPU Throttling and OOMKilled Errors on Windows and macOS

A pod can crawl while its node shows 10% CPU. It can also report normal memory after an OOMKilled restart. Both readings can be right. Node averages hide container quotas, while a restart erases the process’s memory peak.

The useful evidence lives in kubectl, Metrics Server, Prometheus, cgroup v2 counters, events, and application latency. Match those sources before changing requests or limits. Otherwise, you’re tuning from one dashboard snapshot and a hunch.

Prerequisites

You need:

• Access to an existing Kubernetes cluster.
• A kubectl client within one minor version of the cluster where possible.
• A valid kubeconfig with permission to read pods, events, deployments, and metrics.
• Metrics Server for kubectl top.
• Prometheus, or a similar time-series system, for historical metrics.
• Application p95 or p99 latency metrics.
• Approved node access or a supported node-debugging method for cgroup checks.
• At least several days of historical data for initial sizing. Use two to four weeks for weekly cycles or uneven peaks.

The commands use a production namespace and an api deployment. Replace those names with your workload. Copying them unchanged will produce accurate errors about resources you don’t have.

This workflow covers a Linux-based Kubernetes cluster managed from Windows or macOS. Direct cgroup checks run on the Linux worker node. Your laptop only supplies the terminal.

Test Environment

The examples match the Kubernetes CLI and resource model documented through August 2026. They were prepared with:

• A Linux worker-node cluster using cgroup v2.
• Metrics Server for current CPU and memory readings.
• Prometheus with kubelet and cAdvisor metrics.
• Windows 11 PowerShell and macOS Terminal as admin clients.
• A demo deployment with separate api and sidecar containers.

Exact cgroup paths, metric labels, and counters depend on the Kubernetes distribution, container runtime, kernel, and monitoring stack. Check the Kubernetes documentation and your provider’s docs before changing node settings. A cgroup path copied from another distribution is usually decorative.

Quick Diagnosis

Run these checks before changing resources. The order matters because each step shows whether the next data source is worth trusting.

1. Confirm the Cluster and Metrics API Work

Check the current context and node state:

kubectl config current-context
kubectl get nodes

Expected output resembles:

production-cluster
NAME STATUS ROLES AGE VERSION
worker-1 Ready <none> 91d v1.xx.x
worker-2 Ready <none> 91d v1.xx.x

Confirm the context before doing anything else. Resource tuning against the wrong cluster is an efficient way to create two incidents.

Query the Metrics API directly:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

A working API returns JSON containing a NodeMetricsList. If you get NotFound, ServiceUnavailable, or an empty response, repair Metrics Server before trusting kubectl top.

2. Take a Current Resource Snapshot

Run:

kubectl top nodes
kubectl top pods -n production
kubectl top pods -n production --containers

Expected node output:

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
worker-1 820m 10% 6120Mi 39%
worker-2 1150m 14% 7030Mi 45%

Container output may resemble:

POD NAME CPU(cores) MEMORY(bytes)
api-7f86c49cdb-kv9pz api 472m 681Mi
api-7f86c49cdb-kv9pz sidecar 18m 74Mi

Treat these readings as a live snapshot. The kubectl top documentation says its metrics are tuned for autoscaling signals. You won’t get historical percentiles, memory peaks, CPU quota counters, or proof of earlier throttling. It tells you what’s warm now. Useful, but limited.

3. Inspect Configuration, Restarts, and Events

Select a pod:

kubectl get pods -n production -l app=api

Then describe it:

kubectl describe pod api-7f86c49cdb-kv9pz -n production

Review these fields together:

• CPU and memory requests.
• CPU and memory limits.
• Current and previous container state.
• Exit code and reason.
• Restart count.
• Readiness and liveness probe failures.
• Scheduling, eviction, and node-pressure events.

Don’t stop at the current state. A replacement container can look healthy after the kernel killed its predecessor.

4. Build an Incident Timeline

Record the following for the same pod, container, and time range:

• Request rate and concurrency.
• p95 and p99 application latency.
• CPU usage at p50, p95, and p99.
• CPU throttling ratio and throttled seconds.
• Memory working set and peak memory.
• Restart count changes.
• OOMKilled, eviction, scheduling, and probe events.
• Node CPU and memory saturation.

One metric rarely proves a cause. Throttling that rises with p99 latency is useful evidence. A throttling ratio alone is just a counter with ambitions.

The same rule applies to memory. Usage near the limit just before an OOMKilled state is stronger evidence than exit code 137 alone.

How Kubernetes Requests and Limits Behave

CPU Requests, Limits, Periods, and Quotas

A CPU request is mainly a scheduling value. When a container requests 500m, Kubernetes accounts for half a CPU while placing the pod. During CPU contention, the request also affects its share of CPU time.

A CPU limit sets a CPU-time quota. With a common 100-millisecond Completely Fair Scheduler (CFS) period:

• A 500m limit permits about 50 milliseconds of CPU time per 100-millisecond period.
• A 1 CPU limit permits about 100 milliseconds per period.
• A 2 CPU limit permits about 200 milliseconds across available processors per period.

A multithreaded service can use that quota early. The kernel then throttles it until the next period, even if other processors sit idle. That explains the familiar 10%-busy node hosting a slow, throttled pod.

The period and quota are kernel controls, not rolling averages. A short burst can hit the quota while a one-minute graph looks quiet.

Memory Requests and Hard Limits

A memory request tells the scheduler how much memory to count during placement. It doesn’t reserve a separate physical block for that container.

A memory limit is a hard cgroup boundary. Free RAM elsewhere on the node doesn’t let the container cross it. If reclaim can’t recover enough memory at the boundary, Linux may kill a process.

Kubernetes commonly reports the terminated container as:

Reason: OOMKilled
Exit Code: 137

Exit code 137 means the process ended after signal 9 (128 + 9). That matches an OOM kill, but an operator, runtime, or supervisor can also send SIGKILL. Check the previous state, events, node conditions, and memory history before assigning blame.

Common Issues and Solutions

Problem: The Application Is Slow Even Though Node CPU Is Low

Symptoms:

• p95 or p99 latency rises during short traffic bursts.
• Node CPU remains below 30%.
• The container has a CPU limit.
• Average pod CPU looks lower than the limit.
• There are no OOM restarts.

Why it happens: CPU limits apply per container through quota periods. A bursty container can use its quota early and wait while unrelated cores remain idle. Node-wide averages won’t show that wait.

Fix:

First, inspect the affected container and its configured resources:

kubectl top pod api-7f86c49cdb-kv9pz -n production --containers
kubectl describe pod api-7f86c49cdb-kv9pz -n production

Then calculate the ratio of throttled periods to total periods in Prometheus:

sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{
    namespace="production",
    container!="",
    image!=""
  }[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{
    namespace="production",
    container!="",
    image!=""
  }[5m])
)

A result of 0.18 means throttling occurred during about 18% of the observed scheduling periods. It doesn’t mean the container lost 18% of its total CPU time. Check throttled seconds as a second signal:

sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_seconds_total{
    namespace="production",
    container!="",
    image!=""
  }[5m])
)

Compare the ratio with p95 or p99 latency over the same interval.

No ratio is a universal threshold. Sustained values above roughly 10% deserve a closer look when latency rises, queues grow, or throughput falls. A 2% ratio can hurt a low-latency API. A batch job may tolerate 25% and still finish before breakfast.

Test one of these changes on a small subset:

• Raise the CPU limit to allow observed bursts.
• Remove the CPU limit for a trusted latency-sensitive service.
• Keep a realistic CPU request so scheduling and contention stay predictable.
• Scale horizontally when concurrency and throughput support it.
• Fix runtime concurrency when the process assumes more CPUs than its quota allows.

Don’t remove CPU limits blindly from untrusted, noisy, or poorly isolated workloads. Namespace quotas, admission policies, node isolation, and workload ownership determine the risk.

Verification:

After the change, compare the same load window:

kubectl rollout status deployment/api -n production
kubectl top pods -n production --containers

A useful change reduces linked throttling and latency without saturating the node or starving nearby workloads. If latency stays flat, CPU quota probably wasn’t the whole problem.

Problem: kubectl top Cannot Confirm CPU Throttling

Symptoms:

• kubectl top shows CPU below the configured limit.
• Users still report intermittent latency.
• The command has no throttling column.
• A current snapshot looks normal after the incident.

Why it happens: Metrics Server exposes recent resource use. It doesn’t publish CFS throttling counters or keep incident history. Its sampling interval can also miss short bursts.

Fix:

Use kubectl top to find busy pods, then move to Prometheus or direct cgroup data. Inspect these counters:

• container_cpu_cfs_periods_total
• container_cpu_cfs_throttled_periods_total
• container_cpu_cfs_throttled_seconds_total

If those metrics aren’t available, inspect cpu.stat on an approved Linux node. First, identify the node:

kubectl get pod api-7f86c49cdb-kv9pz -n production -o wide

Expected output includes:

NAME READY STATUS RESTARTS NODE
api-7f86c49cdb-kv9pz 2/2 Running 0 worker-1

Find the container’s cgroup using the process supported by your runtime and distribution. After you verify the exact cgroup path, run this command on the node:

sudo cat /sys/fs/cgroup/YOUR_VERIFIED_CGROUP_PATH/cpu.stat

Typical cgroup v2 output:

usage_usec 284501923
user_usec 219772081
system_usec 64729842
nr_periods 621804
nr_throttled 58419
throttled_usec 17922503

Read the counters twice during a controlled load test. Rising nr_throttled and throttled_usec confirm kernel throttling. Raw totals since cgroup creation tell you little without a rate or comparison window.

Verification:

After changing the resource settings, confirm that throttling counters rise more slowly. Then check whether p95 or p99 latency improves under similar load.

Problem: kubectl top Reports Metrics Not Available

Symptoms:

• kubectl top nodes returns Metrics API not available.
• The command hangs or returns stale data.
• Horizontal Pod Autoscaler resource metrics are missing.
• The Metrics Server pod restarts or fails readiness checks.

Why it happens: Metrics Server may be missing or unhealthy. It may also fail to reach kubelets because of certificates or network rules. Sometimes it lacks its own CPU or memory. Monitoring components can have resource problems too. Fair is fair.

Fix:

Query the API:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Check the service and pods:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get pods -n kube-system -l k8s-app=metrics-server
kubectl describe apiservice v1beta1.metrics.k8s.io

Inspect logs:

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=200

Look for:

• Kubelet connection failures.
• Certificate validation errors.
• Readiness probe failures.
• OOMKilled states.
• CPU throttling.
• Network policy blocks.
• Resource exhaustion on the node.

Use the Metrics Server project documentation and your provider’s supported setup. Don’t copy insecure TLS flags into production just to silence certificate errors. Find the reason validation fails.

Verification:

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top nodes

Expected APIService state:

NAME SERVICE AVAILABLE
v1beta1.metrics.k8s.io kube-system/metrics-server True

Problem: A Container Repeatedly Restarts with OOMKilled

Symptoms:

• Pod restart count increases.
• kubectl describe shows Reason: OOMKilled.
• The previous exit code is 137.
• Memory approached its limit before the restart.
• The replacement container currently looks healthy.

Why it happens: The container crossed its cgroup memory boundary. Common causes include a low limit, a short spike, a leak, too much concurrency, or a runtime heap placed too close to the hard limit.

Fix:

Display the previous state directly:

kubectl get pod api-7f86c49cdb-kv9pz -n production \
  -o jsonpath='{range .status.containerStatuses[*]}{"container="}{.name}{" restarts="}{.restartCount}{" reason="}{.lastState.terminated.reason}{" exitCode="}{.lastState.terminated.exitCode}{" finishedAt="}{.lastState.terminated.finishedAt}{"\n"}{end}'

Expected output:

container=api restarts=3 reason=OOMKilled exitCode=137 finishedAt=2026-08-15T04:41:17Z
container=sidecar restarts=0 reason= exitCode= finishedAt=

Check events:

kubectl get events -n production \
  --field-selector involvedObject.name=api-7f86c49cdb-kv9pz \
  --sort-by=.lastTimestamp

Inspect the configured memory request and limit:

kubectl get pod api-7f86c49cdb-kv9pz -n production \
  -o jsonpath='{range .spec.containers[*]}{"container="}{.name}{" request="}{.resources.requests.memory}{" limit="}{.resources.limits.memory}{"\n"}{end}'

Query historical working-set memory in Prometheus:

max_over_time(
  container_memory_working_set_bytes{
    namespace="production",
    container="api",
    image!=""
  }[24h]
)

Graph the working set beside the configured limit, then mark each restart.

Check the shape of the usage:

• Likely undersized limit: Memory rises with load, falls after traffic or garbage collection, and reaches the boundary only during valid peaks.
• Likely leak: The baseline climbs across repeated traffic cycles and doesn’t return after load drops.
• Likely runtime mismatch: The heap maximum sits near the container limit. That leaves little room for stacks, native allocations, buffers, JIT data, or page cache.
• Likely concurrency problem: Memory follows simultaneous requests, workers, or queued jobs more closely than elapsed time.

For a JVM workload, don’t assign the whole container limit to the Java heap. Setting the heap around 60–70% of the limit is a reasonable first test. Validate it under production-like load. Native memory, thread stacks, metaspace, code cache, and direct buffers count too. Java has many cupboards.

Verification:

Monitor for at least one representative busy period:

kubectl get pods -n production -l app=api
kubectl get events -n production --sort-by=.lastTimestamp

The restart count should remain stable. Historical memory should show tested headroom without hiding a steady upward trend.

Problem: Memory Looks Normal Immediately After an OOM Kill

Symptoms:

• kubectl top reports low memory.
• The pod restarted minutes earlier.
• Users assume the OOM report was incorrect.
• No current process is near the configured limit.

Why it happens: The process that used the memory is gone. kubectl top shows the new container’s current use, not the old process’s final footprint. The crime scene has already been tidied.

Fix:

Use evidence that survives the restart:

• Check .lastState.terminated.reason.
• Check exit code, finish time, and restart count.
• Review pod and node events.
• Query historical memory across the incident window.
• Check whether the node was under MemoryPressure or had an eviction.

kubectl describe pod api-7f86c49cdb-kv9pz -n production
kubectl describe node worker-1

Exit code 137 without OOMKilled can point to another SIGKILL source. An evicted pod, node-level OOM, manual kill, and container-level limit breach need different fixes.

Verification:

The previous termination time should match a historical memory peak, restart-count change, or related node event.

Problem: Pods Stay Pending While Nodes Look Underused

Symptoms:

• Pods report Pending.
• Events contain Insufficient cpu or Insufficient memory.
• Current node use is low.
• Existing workloads declare large requests.

Why it happens: The scheduler uses requests and allocatable capacity, not recent average use. Large requests can consume schedulable capacity while the CPUs spend most of their day waiting politely.

Fix:

Describe the pending pod:

kubectl describe pod YOUR_PENDING_POD -n production

Inspect node allocation:

kubectl describe node worker-1

Near the end of the output, compare requested resources with allocatable capacity. Then list workload requests:

kubectl get pods -n production \
  -o custom-columns='POD:.metadata.name,CONTAINER:.spec.containers[*].name,CPU_REQUEST:.spec.containers[*].resources.requests.cpu,MEM_REQUEST:.spec.containers[*].resources.requests.memory'

Compare those values with historical p50, p95, and p99 usage.

For CPU requests, don’t pick one percentile by default:

• Use p50 only for elastic workloads that tolerate contention or scale quickly.
• Use p95 as a useful baseline for many steady services.
• Move closer to p99 when latency targets are strict, scale-out is slow, or bursts are frequent.
• Include startup CPU, background work, and failover conditions.

Reduce large requests in stages. A 20–30% canary change is easier to test than cutting every request in half and spending the afternoon explaining the graphs.

Verification:

Confirm that new replicas schedule while latency, throttling, and error rates remain acceptable:

kubectl get pods -n production -o wide
kubectl rollout status deployment/api -n production

Problem: Raising CPU Limits Does Not Fix Latency

Symptoms:

• CPU throttling falls but p99 latency remains high.
• CPU usage stays below the new limit.
• Queue depth or dependency latency increases.
• Garbage-collection pauses remain visible.

Why it happens: CPU quota was only one limit. Storage, network calls, locks, thread pools, connection pools, garbage collection, or downstream services may control latency.

Fix:

Compare:

• Request rate and concurrent requests.
• Application queue depth.
• Dependency p95 and p99 latency.
• Garbage-collection pause time.
• Disk and network latency.
• Error and timeout rates.
• CPU throttled seconds and throttled-period ratio.

Confirm that language runtimes understand the container’s CPU limits. Go workloads may need suitable GOMAXPROCS behavior. JVM workloads may need a tested ActiveProcessorCount or container-aware runtime setup.

Stop raising CPU limits when throttling falls but latency doesn’t follow. Extra quota won’t repair a slow database, a blocked thread pool, or a 600-millisecond dependency call.

Verification:

Run a controlled load test. Confirm that the corrected dependency, runtime, or concurrency setting lowers latency at the same request rate.

Inspect cgroup v2 CPU and Memory Data

Direct cgroup checks help when Prometheus is unavailable or you need kernel-level proof. You need access to the Linux worker and the exact cgroup for the container.

Don’t guess the path. Kubernetes distributions use different systemd slices, runtime IDs, and quality-of-service directories. A plausible path can still belong to the wrong container.

After finding the verified cgroup path, inspect CPU state:

sudo cat /sys/fs/cgroup/YOUR_VERIFIED_CGROUP_PATH/cpu.stat

Read the quota:

sudo cat /sys/fs/cgroup/YOUR_VERIFIED_CGROUP_PATH/cpu.max

Example output:

50000 100000

This means a 50,000-microsecond quota within a 100,000-microsecond period. That equals 500m CPU. Without a quota, cgroup v2 may report:

max 100000

Inspect memory:

sudo cat /sys/fs/cgroup/YOUR_VERIFIED_CGROUP_PATH/memory.current
sudo cat /sys/fs/cgroup/YOUR_VERIFIED_CGROUP_PATH/memory.max

Example output:

713109504
1073741824

Those values are bytes: about 680 MiB in use and a 1 GiB hard limit. memory.max may contain max when no hard limit exists.

These files contain current state and total counters. They don’t replace historical monitoring. Read them during the incident or collect the values over time.

Right-Size Requests and Limits Without Creating Another Incident

1. Collect Representative History

Use several days that include normal and busy periods. Two to four weeks is safer for weekly cycles, batch jobs, reporting periods, or uneven traffic.

Include:

• p50, p95, and p99 CPU usage.
• p95, p99, and maximum memory working set.
• Request rate and concurrency.
• Startup and scheduled-job behavior.
• Failover or degraded-dependency behavior.
• CPU throttling and application latency.
• Restarts, OOM events, and node pressure.

Averages hide short CPU bursts and memory peaks. They’re fine for a cost summary and poor as the only sizing input.

2. Choose CPU Requests from Service Behavior

A CPU request controls scheduling and contention. It doesn’t predict every peak.

Consider a service with:

• p50 CPU: 220m
• p95 CPU: 460m
• p99 CPU: 780m
• short startup peak: 950m

A request between 450m and 600m is a sensible test when the service scales quickly and tolerates short contention. A strict latency target or slow autoscaler may justify more. A batch worker can often request closer to its median.

Don’t default every request to p99. Rare peaks can reserve costly capacity when scale-out already works. Don’t default to p50 either. That can cause contention and uneven latency. Workload behavior gets the deciding vote.

3. Set Memory from High-Water Behavior

Memory doesn’t yield like CPU. When a container reaches its hard limit, it can’t wait for another memory period.

Account for:

• p99 working set.
• Short spikes hidden by coarse sampling.
• Heap plus native and non-heap allocations.
• Thread stacks and direct buffers.
• Page cache charged to the cgroup.
• Startup and data-loading peaks.
• Load growth before the next review.
• Monitoring uncertainty.

Suppose a service has a stable p99 working set of 1.15Gi. Its observed peaks reach 1.32Gi, and native allocations vary by another 150Mi. A 1.5Gi limit may leave too little room. Testing 1.75Gi or 2Gi is reasonable, but failure cost and node capacity still set the final value.

Memory requests and limits don’t always need to match:

• Equal values can help a pod qualify for Guaranteed quality of service when its CPU settings also meet the required conditions.
• A lower request with a higher limit allows bursts but can overcommit node memory.
• Critical, predictable services often benefit from requests near normal high-water usage.
• Bursty or cache-heavy workloads can use a gap when eviction and node-pressure risks are understood.

No request-to-limit ratio works for every workload. Anyone offering one has skipped the inconvenient bits.

4. Prepare a Workload-Specific Change

An overly tight example might be:

# Workload-specific example; do not copy values without measuring.
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: 500m
    memory: 768Mi

A measured replacement for a trusted latency-sensitive service might be:

# Workload-specific example based on observed percentiles and load testing.
resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    memory: 1536Mi

This example removes the CPU limit while keeping a CPU request and hard memory limit. A less trusted or noisier workload may be safer with a higher CPU limit instead.

5. Roll Out the Change Gradually

Edit the deployment manifest in source control, run validation, and deploy through the normal delivery path. Avoid live-only edits that Helm, Kustomize, GitOps, or another controller will overwrite later.

Start with a small canary or low-traffic subset. Watch it for 24–48 hours where that suits the workload, including one representative peak. Initial setup takes longer than changing two YAML values because measurement is the work.

Check rollout health:

kubectl rollout status deployment/api -n production
kubectl get pods -n production -l app=api -o wide

Then compare old and new replicas for:

• p95 and p99 latency.
• Error rate and throughput.
• CPU throttling.
• CPU and memory percentiles.
• OOM events and restart count.
• Scheduling failures.
• Node saturation and nearby workload performance.

6. Define Rollback Conditions

Set rollback conditions before deployment. Otherwise, a bad rollout tends to acquire creative definitions of “still being evaluated.”

Examples include:

• p99 latency increases by more than 10%.
• Error rate exceeds the service objective.
• Any new OOMKilled termination occurs.
• Node memory pressure appears.
• Nearby workloads show measurable contention.
• Pending pods increase because requests became too large.

If the rollout fails, use the delivery system’s tested rollback method. For a standard Deployment with retained revision history:

kubectl rollout undo deployment/api -n production
kubectl rollout status deployment/api -n production

Platform-Specific Issues

Windows

Use PowerShell to manage the cluster. Obtain the kubeconfig through your cluster admin or provider’s approved process, then verify access:

kubectl config current-context
kubectl get nodes

Expected output:

production-cluster
NAME STATUS ROLES AGE VERSION
worker-1 Ready <none> 91d v1.xx.x

PowerShell supports the kubectl examples here. Use backticks only when you mean to split a PowerShell command across lines. Bash continuation backslashes won’t work in PowerShell.

The kubeconfig normally resides at:

%USERPROFILE%\.kube\config

Verify it without showing credentials:

Test-Path "$env:USERPROFILE\.kube\config"

Expected output:

True

If kubectl top fails on Windows while other kubectl commands work, the client OS is rarely responsible. Check the cluster’s Metrics API and Metrics Server.

Direct checks of cpu.stat, memory.current, and memory.max still happen on the Linux worker. PowerShell can’t expose a remote Linux container’s cgroup files unless you enter the node through an approved management path.

macOS

Use Terminal with the approved kubeconfig:

kubectl config current-context
kubectl get nodes

The default kubeconfig path is:

$HOME/.kube/config

Check whether it exists:

test -f "$HOME/.kube/config" && echo "kubeconfig found"

Expected output:

kubeconfig found

If the command prints nothing, get the kubeconfig through the approved access process. Don’t paste tokens or certificates into shell history.

Commands such as cat /sys/fs/cgroup/... won’t inspect production worker cgroups on macOS. Those files exist on Linux nodes. Connect through the cluster’s supported node-debugging or remote-admin method.

Configuration Issues to Check

Requests Accidentally Equal Limits for Every Workload

This policy can simplify Guaranteed quality-of-service setup, but it may reserve too much CPU and block useful bursts. Apply it where predictable isolation matters. A global rule saves thought during setup and charges interest during incidents.

Correct the values based on workload behavior:

resources:
  requests:
    cpu: 400m
    memory: 768Mi
  limits:
    cpu: "1"
    memory: 1Gi

CPU Limit Is Lower Than Real Burst Demand

A service that normally uses 300m but briefly needs 900m can stall under a 500m limit. Raising or removing the limit may help, but only after throttling tracks with latency.

resources:
  requests:
    cpu: 400m
    memory: 768Mi
  limits:
    memory: 1Gi

This allows CPU bursts while keeping a scheduling request. Protect the cluster with suitable quotas, isolation, monitoring, and ownership controls.

Heap Maximum Leaves No Native-Memory Headroom

A 1Gi Java heap inside a 1Gi container leaves no safe margin. Thread stacks, metaspace, code cache, direct buffers, and native libraries also count against the limit.

For JVM workloads, use a container-aware setting such as:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=65.0"

The 65.0 value is an example. Test heap demand, native memory, garbage collection, and latency under load before using it.

Units Are Ambiguous or Incorrect

CPU and memory units are easy to misread:

• 500m CPU means half a CPU.
• 0.5 CPU also means half a CPU.
• 400m memory means 0.4 bytes and is almost certainly wrong.
• 400Mi memory means 400 mebibytes.
• 1Gi memory means one gibibyte.

That third entry has ruined more manifests than its tiny size suggests.

Correct example:

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

Changes Are Made Directly to Live Objects

A manual edit may stop an incident, but it can drift from Git, Helm, Kustomize, or another deployment source. The next rollout then restores the faulty values with impressive consistency.

Put resource changes through the main configuration source. Use live edits only when the incident process permits them, then update the source right away.

Error Messages Quick Reference

Error or status	What it means	What to check
Metrics API not available	Metrics Server is missing, unhealthy, or not registered	APIService state, Metrics Server pods, logs, probes, certificates, and kubelet reachability
error: Metrics not available for pod	Metrics Server has no current sample for that pod	Pod age, readiness, Metrics Server logs, and kubelet metrics access
Reason: OOMKilled	The previous container process was killed during an OOM condition	Memory limit, historical peak, runtime heap, node events, and restart time
Exit Code: 137	Process ended after SIGKILL	Confirm OOMKilled; also check eviction, manual termination, runtime actions, and node events
Insufficient cpu	Scheduler cannot satisfy declared CPU requests	Node allocatable CPU and existing pod requests
Insufficient memory	Scheduler cannot satisfy declared memory requests	Node allocatable memory, requests, taints, and node conditions
Evicted	Kubelet removed the pod under node pressure or another eviction rule	Pod message, node MemoryPressure, disk pressure, and eviction thresholds
CrashLoopBackOff	Container repeatedly starts and fails	Previous logs, termination reason, probes, OOM status, and application errors
context deadline exceeded from Metrics Server	A dependency did not respond in time	Kubelet connectivity, certificates, network policy, load, and Metrics Server resources
forbidden	Current identity lacks permission	Current context, user or service account, Role, and RoleBinding
nr_throttled rising in cpu.stat	The cgroup exhausted CPU quota during periods	CPU limit, throttled time, usage percentiles, and application latency
memory.max contains max	No cgroup v2 hard memory limit is set	Workload specification and policy expectations

Getting Help

Collect a clean diagnostic bundle before escalating:

kubectl version
kubectl config current-context
kubectl get nodes -o wide
kubectl get pods -n production -o wide
kubectl describe pod YOUR_POD -n production
kubectl get events -n production --sort-by=.lastTimestamp
kubectl logs YOUR_POD -n production -c YOUR_CONTAINER --previous --tail=200

The --previous flag gets logs from the last terminated container. After a restart, those logs are often more useful than the cheerful new process saying it started correctly.

Also capture:

• The workload manifest with secrets removed.
• CPU and memory requests and limits.
• Pod UID, node name, and container name.
• Incident start and end times in UTC.
• Restart-count changes.
• Previous termination reason and exit code.
• Five-minute throttling ratios and throttled seconds.
• p95 and p99 usage and latency.
• Memory working set versus limit.
• Node pressure conditions.
• Relevant runtime and kernel versions.

Useful official resources include:

• Kubernetes documentation
• kubectl top reference
• Kubernetes releases
• Kubernetes GitHub repository
• Metrics Server repository

When opening a community issue, include a small reproduction when possible. Remove tokens, certificates, internal hostnames, customer data, and registry credentials. Cleaning a bundle is less painful than rotating a leaked cluster credential.

Prevention Tips

• Monitor percentiles: Track p50, p95, and p99 CPU and memory instead of relying on averages.
• Keep incident history: Retain enough Prometheus data to cover weekly and seasonal peaks.
• Alert on correlation: Combine throttling with latency or queue growth instead of using one ratio as a universal failure threshold.
• Watch restart deltas: Alert when restart counts change, then record the previous termination state.
• Graph memory against limits: A usage graph without the configured boundary lacks useful context.
• Leave non-heap headroom: Include stacks, buffers, native allocations, page cache, and runtime overhead.
• Review runtime settings: Confirm the JVM, Go, and other runtimes understand their container CPU and memory environment.
• Use canary changes: Test new resource values on a small subset before a broad rollout.
• Define rollback thresholds: Set acceptable latency, errors, restarts, and node pressure before deploying.
• Review requests quarterly: Workload behavior changes as traffic, code, and dependencies change.
• Use VPA carefully: Vertical Pod Autoscaler recommendation mode can provide useful evidence without changing production resources by itself.
• Protect no-limit workloads: Use namespace quotas, admission controls, monitoring, ownership, and node isolation where appropriate.
• Preserve configuration ownership: Store requests and limits in the main deployment source, not only in live objects.

Frequently Asked Questions

Why is a pod throttled when its node has spare CPU?

CPU limits apply to each container through a time quota. A container can use its quota early in a CFS period, then wait for the next period while other processors remain idle.

What CPU throttling ratio should trigger investigation?

No threshold fits every workload. Sustained throttling above roughly 10% is worth checking when latency rises, queues grow, or throughput falls. Smaller ratios can still hurt latency-sensitive services.

Why is kubectl top insufficient?

It provides a recent usage snapshot through Metrics Server. It doesn’t expose CFS throttling counters, historical p95 or p99 values, earlier memory peaks, or pre-restart state.

How do I prove a restart was caused by OOMKilled?

Check the previous container state for Reason: OOMKilled, exit code 137, restart time, and restart count. Match those fields with historical memory and pod or node events. Exit code 137 alone doesn’t prove an OOM kill.

Should CPU limits be removed from latency-sensitive services?

Sometimes. A trusted service may benefit from using spare CPU without a quota. Keep a realistic request and use quotas, monitoring, workload isolation, and admission controls. Tight limits remain useful for untrusted or noisy workloads.

Should memory requests and limits be equal?

Not always. Equal values improve predictability and can support Guaranteed quality-of-service classification when CPU settings also qualify. A gap allows bursts but raises overcommit and eviction risk. Choose based on workload behavior and failure cost.

How much history is needed before right-sizing?

Use several representative days at minimum. Two to four weeks is better for weekly cycles, scheduled jobs, or uneven traffic. Include the busiest known period and startup behavior.

Should a CPU request use p50, p95, or p99?

Base the choice on contention tolerance and scaling speed. p95 is a reasonable starting point for many services. p50 may suit elastic batch workloads. Strict latency targets or slow scaling can justify a request closer to p99.

How much memory headroom should I allow?

Start with high-percentile working set and observed peaks. Add room for short spikes, growth, native allocations, stacks, buffers, page cache, and sampling uncertainty. Test the final value under production-like load.

How can I change values safely?

Change the main manifest, deploy to staging or a small canary, and watch at least one representative peak. Compare latency and resource metrics. Set rollback thresholds before expanding the rollout.

Wrapping Up

Step	Action	Applies To
1	Use kubectl top for a current snapshot	CPU and memory triage
2	Inspect resources, previous state, restarts, and events	All incidents
3	Correlate Prometheus throttling, percentiles, and latency	CPU problems
4	Compare historical working set with the hard limit	OOM failures
5	Validate cgroup counters when deeper proof is needed	Linux worker nodes
6	Canary measured resource changes and define rollback limits	Production tuning

Requests, CPU quotas, and memory limits control different parts of the system. Treating them as one sizing knob usually moves the incident somewhere less convenient.

Start with kubectl top. Match kernel evidence with application latency and memory history. Change one value, test it on a canary, and keep a tested rollback ready. That’s slower than guessing for five minutes and much faster than debugging the second outage.