How to Debug Kubernetes Networking Issues: A Layer-by-Layer Troubleshooting Guide (2026)

“Connection timed out.” “Name or service not known.” “503 Service Unavailable.” Every Kubernetes networking incident starts with one of these three messages. Each one can hide half a dozen causes: a dead CNI pod, a missing Service endpoint, broken CoreDNS, a bad NetworkPolicy, or a mesh sidecar rewriting iptables rules under your app. None of these announce themselves.

Kubernetes networking is layered, though, and you can test each layer on its own. This guide walks through a symptom-first, layer-by-layer process using kubectl, ephemeral debug containers, and CNI tools like calicoctl and cilium status. Tested on Kubernetes 1.35.x with Calico 3.28, Cilium 1.16, and Istio 1.30.

Quick Diagnosis

Before chasing a specific component, run these five commands. They take under two minutes and usually tell you which layer to focus on.

# 1. Confirm the pod has an IP and is actually Running
kubectl get pods -o wide -n <namespace>

# 2. Check recent events for scheduling or CNI errors
kubectl describe pod <pod-name> -n <namespace>

# 3. Confirm the CNI plugin pods are healthy on every node
kubectl get pods -n kube-system -o wide | grep -E 'calico|cilium|flannel'

# 4. Confirm the Service has healthy backends
kubectl get endpoints <service-name> -n <namespace>

# 5. Confirm CoreDNS is up
kubectl get pods -n kube-system -l k8s-app=kube-dns

Map the symptom to the layer

This is the step most people skip. It’s also the step that saves the most time during an incident. Match your symptom against this table before you start digging.

Symptom	Most likely layer	First command to run
Pod stuck in ContainerCreating or Pending, no IP assigned	CNI plugin	kubectl describe pod | check Events
curl: (7) Failed to connect using a pod IP	CNI routing / kube-proxy	Ephemeral debug pod ping/curl by IP
curl: (6) Could not resolve host using a Service DNS name, but the IP works	CoreDNS	dig from a debug pod
Connection hangs then times out on a specific port only	NetworkPolicy	kubectl get networkpolicy -A
Intermittent 503/upstream connect error after a mesh install	Service mesh sidecar	kubectl logs <pod> -c istio-proxy
Works from some pods, not others, same namespace	NetworkPolicy or node-specific CNI issue	Compare pod labels + node placement
Everything worked yesterday, nothing changed today	Check kubectl get events -A --sort-by=.lastTimestamp first, something changed

The layer-by-layer incident checklist

Run this sequence in order during a live incident. Don’t skip steps, even when you’re sure you know the cause. The point is to rule things out fast instead of guessing. Guessing is how you burn 40 minutes chasing a NetworkPolicy when the real problem is a crashed CNI pod on one node.

• Pod status: Is the pod Running and Ready? Does it have an IP? (kubectl get pods -o wide)
• CNI health: Are all CNI DaemonSet pods Running on every node, including the node this pod is scheduled on?
• Node-local routing: Can you reach the pod IP from another pod on the same node? From a different node?
• Service and Endpoints: Does the Service have matching, ready endpoints? (kubectl get endpoints)
• DNS: Does dig/nslookup resolve the Service name from inside the cluster?
• NetworkPolicy: Are there policies in the namespace, and do they allow this specific traffic (port, protocol, direction)?
• Service mesh: Is a sidecar injected? Are the Istio CNI or sidecar init containers healthy?

If you get through all seven steps with no clear answer, stop guessing and grab packet captures instead (tcpdump inside a netshoot pod). See the “Getting Help” section below.

Common Issues

Pod Stuck Without an IP Address

Symptoms:

• Pod stays in ContainerCreating or Init:0/1 far longer than a few seconds
• kubectl get pods -o wide shows no IP in the IP column
• kubectl describe pod shows repeated FailedCreatePodSandBox events

Cause: The CNI plugin isn’t working right on the node where the pod landed. Usually the CNI DaemonSet pod, Calico’s calico-node, Cilium’s cilium, or Flannel’s kube-flannel, isn’t Running on that node. Sometimes the CNI binary or config is just missing from /opt/cni/bin/ or /etc/cni/net.d/.

Fix:

kubectl describe pod <pod-name> -n <namespace>

Look for a line like this in the Events section:

Warning FailedCreatePodSandBox 12s kubelet
Failed to create pod sandbox: rpc error: code = Unknown desc =
failed to setup network for sandbox: plugin type=”calico” failed
(add): stat /opt/cni/bin/calico: no such file or directory

That error means the Calico binary is missing on this specific node. Confirm the CNI DaemonSet status:

kubectl get pods -n kube-system -o wide | grep calico-node

calico-node-4x9pq 1/1 Running 0 14d 10.0.4.11 node-3 <none> <none>
calico-node-8k2wv 0/1 CrashLoopBackOff 6 22m 10.0.4.12 node-4 <none> <none>

There it is. Node 4’s Calico pod is stuck in a crash loop. That explains the missing IP. Check its logs:

kubectl logs -n kube-system calico-node-8k2wv --previous

For Cilium clusters, use the cilium CLI instead. It gives a more structured health report than raw logs:

cilium status --wait

/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled

DaemonSet cilium Desired: 4, Ready: 4/4, Available: 4/4
Cluster Pods: 18/18 managed by Cilium

Verification: Delete the stuck pod and let it reschedule, or wait for the CNI pod to recover. Confirm the new pod gets an IP:

kubectl get pod <pod-name> -o wide

NAME READY STATUS RESTARTS AGE IP NODE
web-7d9f6 1/1 Running 0 18s 10.244.4.22 node-4

CNI Plugin Pods Unhealthy (Calico / Cilium Node-Level Check)

Symptoms:

• Some nodes work fine, others show pods stuck without IPs
• kubectl get nodes shows all nodes Ready, but networking is inconsistent between them
• Cross-node pod-to-pod traffic fails while same-node traffic works (or vice versa)

Cause: The CNI agent is running but not fully healthy. Often it’s a BGP peering failure (Calico) or an eBPF/datapath issue (Cilium) on a subset of nodes. This one is sneaky: the pod shows Running, but the routing table on that node is wrong. Nothing in the pod status tells you that.

Fix:

For Calico, check BGP peer status with calicoctl (install it as a kubectl plugin or standalone binary):

calicoctl node status

Calico process is running.

IPv4 BGP status
+————–+——————-+——-+———-+————-+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+————–+——————-+——-+———-+————-+
| 10.0.4.11 | node-to-node mesh | up | 14d02h34 | Established |
| 10.0.4.13 | node-to-node mesh | down | 00h02m11 | Active |
+————–+——————-+——-+———-+————-+

A down peer state means that node can’t exchange routes with the rest of the cluster. Traffic to pods on that node black-holes. Check journalctl -u calico-node or kubectl logs -n kube-system <calico-node-pod> -c calico-node on that node. Usual causes: a firewall blocking TCP/179, or an MTU mismatch.

For Cilium, run the built-in connectivity test. It deploys test pods and reports pass/fail per network path:

cilium connectivity test

✅ pod-to-pod
✅ pod-to-service
❌ pod-to-external-1111 (curl-external-node/curl-to-outside)
Connection timed out after 5000ms

Connectivity test failed: 1 tests failed

Verification: Re-run calicoctl node status or cilium connectivity test after fixing the underlying issue (firewall rule, MTU, or node reboot) and confirm all peers show Established / all tests show ✅.

Tip: If you’re not sure whether your cluster runs Calico, Cilium, or Flannel, run kubectl get pods -n kube-system and look at the DaemonSet names. That tells you which commands in this section apply.

Testing Pod-to-Pod Connectivity Without Touching Production Workloads

Symptoms:

• You suspect a routing problem but don’t want to kubectl exec into a production container. It may not even have curl or dig installed, since distroless images rarely do.
• You need to test from the exact network namespace of a running pod, specifically, not “somewhere in the cluster”

Cause: Production images are often minimal, distroless or scratch-based, and skip debugging tools by design. Great for image size. Terrible at 2 AM when you need curl and it’s just not there.

Fix:

For a quick, standalone test pod using netshoot (bundles curl, dig, nslookup, tcpdump, mtr, and more):

kubectl run netshoot --rm -it --restart=Never \
  --image=nicolaka/netshoot -- /bin/bash

# Inside the netshoot pod
curl -sv http://backend-service.default.svc.cluster.local:8080
dig backend-service.default.svc.cluster.local
ping -c 3 10.244.4.22

For a lighter footprint, busybox works for basic ICMP/DNS checks (no curl, but has wget and nslookup):

kubectl run busybox-test --rm -it --restart=Never \
  --image=busybox:1.36 -- sh

When you specifically need to test from inside a struggling production pod’s network namespace, without adding tools to the image or restarting it, use an ephemeral debug container (stable since Kubernetes 1.25):

kubectl debug -it <pod-name> \
  --image=nicolaka/netshoot \
  --target=<container-name> \
  -- /bin/bash

--target attaches the debug container to the same process namespace as the target container. Tools like tcpdump and ss then see exactly what the app container sees. This is the approach the Kubernetes debugging docs recommend. It doesn’t touch the pod spec or restart the workload.

Verification: A successful curl against the target IP or Service name with an HTTP status code (even a 4xx/5xx) confirms the network path is open. You’ve moved from “unknown” to “known application-layer response.” A hang or Connection timed out confirms it’s still a network-layer problem.

curl Works With a Pod IP But Fails With the Service DNS Name

Symptoms:

• curl http://10.244.4.22:8080 succeeds
• curl http://backend-service.default.svc.cluster.local:8080 hangs or returns Could not resolve host
• nslookup inside a debug pod times out or returns SERVFAIL

Cause: This isolates the problem to DNS, not routing. That’s one of the more satisfying moments in networking debugging; now you know where to look. Either CoreDNS pods aren’t healthy, the pod’s /etc/resolv.conf is misconfigured, or CoreDNS’s upstream forwarding is broken.

Fix:

kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME READY STATUS RESTARTS AGE
coredns-6f6b8cc4f6-2x9jp 1/1 Running 0 12d
coredns-6f6b8cc4f6-9k4wc 1/1 Running 0 12d

If both replicas are Running, check the logs for resolution errors:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

[ERROR] plugin/errors: 2 backend-service.default.svc.cluster.local.
A: read udp 10.244.0.5:52134->169.254.169.254:53: i/o timeout

That pattern, a timeout to an upstream resolver, points to CoreDNS’s forward plugin failing to reach its upstream. It’s not a problem with cluster-internal names. Check the Corefile:

kubectl get configmap coredns -n kube-system -o yaml

From a debug pod, confirm cluster-internal resolution specifically (bypassing external forwarding):

dig +short backend-service.default.svc.cluster.local @10.96.0.10
nslookup backend-service.default.svc.cluster.local 10.96.0.10

10.96.0.10 is the default kube-dns Service ClusterIP; confirm yours with kubectl get svc -n kube-system kube-dns.

Also verify the pod’s own resolver config is pointing at the cluster DNS Service, not something injected incorrectly:

kubectl exec <pod-name> -- cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Verification: dig against the cluster DNS Service IP should return an ANSWER SECTION with the correct pod/Service IP within a few hundred milliseconds. If that works but the app’s curl still fails, the DNS layer is fine; go back and re-check routing/NetworkPolicy.

A Service Has No Endpoints Despite Healthy Pods

Symptoms:

• Pods backing a Service are Running and 1/1 Ready
• curl to the Service ClusterIP or DNS name hangs and eventually times out
• kubectl get endpoints <service-name> shows <none>

Cause: The Service’s selector doesn’t match the pod labels, a common typo after a Deployment refactor. Or the pods are failing their readiness probe and never get added as endpoints.

Fix:

kubectl get endpoints backend-service -n default

NAME ENDPOINTS AGE
backend-service <none> 4m12s

Compare the Service selector against actual pod labels:

kubectl get svc backend-service -o jsonpath='{.spec.selector}'

{“app”:”backend”,”tier”:”api”}

kubectl get pods --show-labels -l app=backend

NAME READY STATUS LABELS
backend-7f8d9c 1/1 Running app=backend,version=v2

Notice tier=api is missing from the pod labels. That mismatch means zero pods match the selector. Somebody’s YAML refactor dropped a label, and now the Service is talking to nobody. Fix the Service selector or the pod template labels, then confirm:

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: backend        # matches pod label exactly
  ports:
    - port: 8080
      targetPort: 8080

If labels match but endpoints are still empty, check readiness probes: a failing probe removes a pod from Endpoints even while it stays Running:

kubectl describe pod backend-7f8d9c | grep -A5 Readiness

Verification:

kubectl get endpoints backend-service

NAME ENDPOINTS AGE
backend-service 10.244.2.14:8080,10.244.3.9:8080 9s

NetworkPolicy Blocking Traffic (vs. a CNI Bug)

Symptoms:

• Pods can ping each other by IP (ICMP succeeds) but TCP/HTTP traffic on a specific port hangs
• Traffic works fine from some namespaces but not others
• Nothing in CNI logs indicates an error; routing looks correct

Cause: ICMP is often excluded from a NetworkPolicy’s port rules, which is why ping succeeding while curl fails is the strongest signal you’re looking at a policy problem, not a CNI bug. A CNI bug usually breaks everything on a node uniformly, including ICMP. A NetworkPolicy blocks selectively, by port, protocol, or namespace.

Fix:

List policies in the affected namespace:

kubectl get networkpolicies -n default

NAME POD-SELECTOR AGE
default-deny-all <none> 3d
allow-frontend app=backend 3d

Describe the policy to see exactly what it permits:

kubectl describe networkpolicy allow-frontend -n default

PodSelector: app=backend
Allowing ingress traffic:
To Port: 8080/TCP
From:
PodSelector: app=frontend
Not affecting egress traffic
Policy Types: Ingress

If the pod making the request doesn’t carry the label app=frontend, this policy silently drops it: no error, just a hang until the client-side timeout fires. This is the part that makes NetworkPolicy debugging so miserable; there’s no rejection message, just silence. Confirm the calling pod’s labels:

kubectl get pod <caller-pod> --show-labels

To confirm it’s the policy and not the CNI, remove the policy temporarily in a non-production namespace. Or run kubectl get networkpolicy -n default -o yaml to check for a namespace-wide default-deny with no matching allow rule:

# default-deny-all.yaml — blocks everything unless another policy allows it
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Verification: After adding or correcting an allow rule, re-run the connectivity test from the ephemeral debug pod. A curl that previously hung should now return promptly. If it still hangs after the policy change, the problem is elsewhere; re-check the CNI layer.

Tip: Calico and Cilium both support policy enforcement logging. Cilium’s cilium monitor --type drop shows dropped packets in real time, including the policy verdict that caused the drop, much faster than trial-and-error policy edits.

Service Mesh Sidecar Causing Timeouts or 503s (Istio / Linkerd)

Symptoms:

• Everything worked before Istio or Linkerd was installed; now requests intermittently time out or return 503 upstream connect error or disconnect/reset before headers
• kubectl get pods shows 2/2 containers per pod (app + sidecar) instead of 1/1
• Traffic between pods in the mesh fails, but traffic to pods outside the mesh (or with sidecar injection disabled) works fine

Cause: A service mesh injects a sidecar proxy, istio-proxy for Istio, linkerd-proxy for Linkerd, and rewrites the pod’s iptables rules to route all traffic through it. If the sidecar starts after the app container, traffic gets redirected into a proxy that is still starting up. Same thing happens if the Istio CNI plugin is chained incorrectly with your primary CNI, producing exactly this timeout/503 pattern. This is the mesh’s most common failure mode by a wide margin, so check it first if you just installed one.

Fix:

First, confirm this is mesh-related by checking whether the pod has a sidecar and whether it’s healthy:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'

backend istio-proxy

Check the sidecar’s own logs: a 503 often shows up there before it ever reaches your app:

kubectl logs <pod-name> -c istio-proxy --tail=50

[2026-08-15T14:02:11.442Z] “GET /api/health HTTP/1.1” 503 UF,URX
upstream_reset_before_response_started{connection_termination}

UF (Upstream Failure) and URX (Upstream Retry Limit Exceeded) are Envoy flags. This means the mesh’s proxy is failing to reach the local app container, not a cross-pod networking issue.

If you’re running Istio in CNI mode (recommended over the legacy init-container approach), confirm the Istio CNI DaemonSet is fully ready across every node:

kubectl get daemonset -n istio-system istio-cni-node

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE
istio-cni-node 4 4 4 4 4

Inspect the sidecar’s injected iptables rules directly to confirm redirection is set up correctly (ISTIO_INBOUND chain should redirect to the proxy’s port, typically 15006):

kubectl exec -it <pod-name> -c istio-proxy -- iptables-save -t nat

-A ISTIO_INBOUND -p tcp -m tcp –dport 8080 -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT –to-ports 15006

For Linkerd, the equivalent check is linkerd check --proxy and kubectl logs <pod-name> -c linkerd-proxy, which reports proxy readiness and TLS handshake failures separately from CNI-level errors.

Per the Istio CNI diagnostic docs, also check the Istio CNI pod logs for chaining errors:

kubectl logs -n istio-system -l k8s-app=istio-cni-node --tail=50

Verification: Once the sidecar and CNI DaemonSet both report Ready, re-test with curl from a mesh-injected debug pod (mesh sidecar behavior differs from a plain netshoot pod outside the mesh). Confirm the 503/UF errors stop appearing in the istio-proxy logs.

CNI Plugin Conflicts After Installing a Service Mesh CNI

Symptoms:

• Networking broke specifically after installing Istio’s CNI plugin or another mesh CNI alongside Calico/Cilium
• Some pods route correctly, others don’t, with no obvious pattern
• /etc/cni/net.d/ on a node has more than one config file

Cause: Kubernetes uses the alphabetically first config file in /etc/cni/net.d/ unless plugins are explicitly chained. If the mesh CNI’s config file sorts before your primary CNI’s config, or isn’t marked as chained, it can silently take over pod network setup, or conflict with it. Yes, alphabetical sorting decides your cluster’s networking behavior. It’s exactly as fragile as it sounds.

Fix:

SSH to an affected node (or use a privileged debug pod with hostPath access) and list the CNI config directory:

ls -la /etc/cni/net.d/

10-calico.conflist
YYY-istio-cni.conf

cat /etc/cni/net.d/YYY-istio-cni.conf

Confirm the mesh CNI config includes a plugins chain instead of replacing the primary CNI entirely. Istio’s CNI is designed to be chained; it appends itself to the existing config instead of sitting as a standalone file:

{
  "cniVersion": "0.3.1",
  "name": "istio-cni",
  "type": "istio-cni",
  "chained": true
}

Two competing top-level config files instead of one chained entry is the bug. Reinstall the mesh CNI with chaining enabled, per its docs, or verify your install order matches the vendor’s guide.

Verification: After correcting the config, restart a test pod and confirm both CNI functions apply: the pod gets a routable IP from Calico/Cilium and the mesh’s traffic redirection rules are present (check with the iptables-save command from the previous section).

Error Messages Table

Error message	Layer	Meaning
FailedCreatePodSandBox	CNI	CNI binary/config missing or plugin crashed on that node
NetworkPlugin cni failed to set up pod	CNI	Plugin returned an error during pod network setup
curl: (7) Failed to connect to <IP> port <port>: Connection refused	Routing / app	Port not open, or nothing listening; check ss -tlnp in the container
curl: (6) Could not resolve host	DNS	CoreDNS unreachable or /etc/resolv.conf misconfigured
SERVFAIL (from dig/nslookup)	DNS	CoreDNS upstream forwarding failing or CoreDNS pod unhealthy
Connection hangs, then times out (no error text)	NetworkPolicy or CNI drop	Packet silently dropped; check policies before assuming a CNI bug
503 upstream connect error or disconnect/reset before headers	Service mesh sidecar	Envoy/Linkerd proxy couldn’t reach the local app or upstream
UF,URX (Envoy access log flags)	Service mesh sidecar	Upstream Failure / Retry limit exceeded; proxy-to-app connection issue
Endpoints: <none>	Service/Endpoints	Selector mismatch or failing readiness probe
BGP peer STATE: down (Calico)	CNI routing	Node can’t exchange routes; check firewall on TCP/179

Platform-Specific Issues

On Windows

Kubernetes clusters almost always run on Linux nodes, but your debugging client often runs on Windows. Most “networking” complaints from Windows machines are actually kubectl or kubeconfig issues, not cluster issues. Rule that out before you blame Calico.

kubectl can’t reach the cluster from PowerShell:

kubectl config get-contexts
kubectl cluster-info

If cluster-info hangs, check whether a corporate VPN or proxy is intercepting traffic to the API server port, usually 6443. That’s a Windows network-stack issue, not a Kubernetes one.

calicoctl or cilium CLI not found: confirm the binary was added to your PATH after download. Windows doesn’t do this automatically, the way Homebrew does on macOS:

Get-Command calicoctl
Get-Command cilium

On macOS

Docker Desktop’s local Kubernetes cluster uses a different network model than production clusters. If you’re testing NetworkPolicy behavior locally, know that Docker Desktop doesn’t enforce NetworkPolicies by default. You need a real CNI, Calico or Cilium, installed via kind or minikube to reproduce policy bugs locally. Skip this and you’ll swear a policy works locally, then watch it fail in prod.

kubectl config get-contexts

CURRENT NAME CLUSTER
* docker-desktop docker-desktop
prod-cluster prod-cluster.example.com

Double-check CURRENT before running any debug commands. Running kubectl exec against the wrong context during a production incident is a common, costly mistake.

Homebrew installs for CLI tools are straightforward:

brew install kubectl
brew install cilium-cli

On Web (Browser-Based Cloud Shells)

Many teams run kubectl from a browser-based cloud shell rather than a local terminal. Two issues come up repeatedly:

Session resets losing your kubeconfig context. Browser-based shells are often ephemeral. Closing the tab or hitting an idle timeout can wipe your home directory, including ~/.kube/config. Re-authenticate and re-fetch your kubeconfig at the start of every session instead of assuming it’s still there from yesterday.

kubectl exec/kubectl debug streaming failures behind a corporate proxy. Interactive commands (exec, debug -it, port-forward) use a long-lived streaming connection. If your browser-based terminal sits behind a strict corporate proxy, these commands can hang with no clear error. Simple commands like get and describe still work fine. Test from a direct network path, VPN or bastion host, to confirm it’s a proxy issue, not a cluster problem.

Configuration Issues

Default-deny NetworkPolicy with no matching allow rule. Teams add a namespace-wide default-deny-all policy for security, then forget explicit allows for legitimate traffic, including from ingress controllers or monitoring scrapers. Always pair a default-deny with the allow rules it needs, and test in a staging namespace first.

Service selector doesn’t match pod template labels. This is almost always a copy-paste error after cloning a Deployment. Run kubectl get svc <name> -o jsonpath='{.spec.selector}' next to kubectl get pods --show-labels any time endpoints come back empty.

Multiple CNI config files with ambiguous ordering. As covered above, /etc/cni/net.d/ uses alphabetical ordering to pick the active config unless plugins are explicitly chained. Naming matters: a file like 05-custom.conf can silently override your primary CNI’s 10-calico.conflist.

CoreDNS Corefile misconfigured after a manual edit. A common mistake: editing the forward upstream in the CoreDNS ConfigMap without validating it. A typo’d upstream IP breaks every external DNS lookup, but not cluster-internal ones. Validate with kubectl get configmap coredns -n kube-system -o yaml before and after any change. Restart CoreDNS pods to pick up the new config: kubectl rollout restart deployment coredns -n kube-system.

MTU mismatches on overlay networks. VXLAN and other overlay CNI modes add encapsulation overhead, commonly 50 bytes. If node NICs run a standard 1500 MTU but the CNI’s overlay MTU isn’t adjusted down, large packets get silently fragmented or dropped. That produces intermittent failures that look like flaky NetworkPolicy behavior but are really MTU-related. Check with ip link show on a node, then compare it against your CNI’s documented MTU recommendation.

Getting Help

When the checklist above doesn’t resolve the issue, gather logs and diagnostic bundles before escalating:

• CNI DaemonSet logs: kubectl logs -n kube-system <calico-node-or-cilium-pod>
• Cilium diagnostic bundle: cilium sysdump: packages logs, config, and state from every node into one archive
• Calico diagnostics: calicoctl node diags: collects BGP state, iptables rules, and Felix logs
• Istio bug report: istioctl bug-report: pulls proxy configs, logs, and cluster state for mesh issues
• kubelet logs on a node: journalctl -u kubelet -n 200 (requires node SSH access)
• CoreDNS logs: kubectl logs -n kube-system -l k8s-app=kube-dns

Community resources:

• Kubernetes Slack: #kubernetes-novice and #sig-network channels
• Calico and Cilium both maintain active Slack communities linked from their GitHub repos
• Istio GitHub Discussions for mesh-specific issues
• Official docs: Kubernetes networking concepts, debugging running pods, Istio CNI diagnostics

Prevention Tips

• Pair every default-deny NetworkPolicy with documented allow rules: keep them in the same PR/commit so reviewers see both halves together.
• Run cilium connectivity test or calicoctl node diags as a scheduled health check, including outside of incidents. Catching a down BGP peer at 2 AM in a dashboard beats catching it during a customer-facing outage.
• Alert on CNI and mesh DaemonSet readiness, in addition to pod counts. A DaemonSet showing 3/4 Ready for more than a few minutes should page someone.
• Test mesh upgrades in a canary namespace first. Sidecar injection changes (proxy version bumps, iptables rule changes) are a leading cause of “it broke after the upgrade” incidents.
• Keep a compatibility note for your CNI + mesh combination. Istio CNI chaining requirements differ between Calico and Cilium. Document which mode you’re running so the next person debugging doesn’t have to rediscover it.
• Run istioctl analyze before applying mesh config changes: it catches common misconfigurations (missing chained CNI, conflicting VirtualServices) before they hit production.

Wrapping Up

Kubernetes networking failures feel chaotic until you treat them as a stack, not a mystery: pod IP, then CNI, then Service/Endpoints, then DNS, then NetworkPolicy, then mesh. Work the layers in order with the commands above. Most “the network is broken” incidents resolve in minutes instead of hours.

Step	Action	Applies To
1	Check pod status and IP	All incidents
2	Verify CNI DaemonSet health (Calico/Cilium)	Missing IPs, routing failures
3	Test with netshoot/ephemeral debug container	Pod-to-pod connectivity
4	Check CoreDNS and dig resolution	Name resolution failures
5	Review NetworkPolicies for the namespace	Selective port/protocol failures
6	Check sidecar logs and Istio CNI DaemonSet	Post-mesh-install 503s/timeouts
7	Confirm Endpoints match Service selector	“Healthy pods, unreachable Service”

If you’re just standing up a cluster, run cilium connectivity test (or Calico’s equivalent BGP health check) once as a baseline. You’ll know immediately if something’s off, instead of finding out during your first real incident, a much worse time to learn your BGP mesh has a dead peer.

Last updated: 2026-08-15 | Applies to Kubernetes 1.35.x with Calico 3.28, Cilium 1.16, and Istio 1.30 on Linux nodes