Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container

Ask Question
We are trying to create POD but the Pod's status struck at ContainerCreating for long time.
This is the output we got after running the command: kubectl describe pod
Name:           demo-6c59fb8f77-9x6sr
Namespace:      default
Priority:       0
Node:           k8-slave2/10.0.0.5
Start Time:     Wed, 23 Dec 2020 10:16:23 +0000
Labels:         app=demo
                pod-template-hash=6c59fb8f77
Annotations:    <none>
Status:         Pending
IPs:            <none>
Controlled By:  ReplicaSet/demo-6c59fb8f77
Containers:
  private-docker-registry:
    Container ID:
    Image:          private-docker-registry:5000/mahin/mof-docker-demo:v1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p94zw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p94zw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p94zw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
      Type     Reason                  Age                  From               Message
      ----     ------                  ----                 ----               -------
      Normal   Scheduled               10m                  default-scheduler  Successfully assigned default/demo-6c59fb8f77-9x6sr to k8-slave2
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "95e72bfc6f6c13de7f5c96eb76b012c2e6639ca03f4c2f270b23ed1a09b90413" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "566370012e4a1d32af2ef9035ff64d743cd81f36f25d2724e7b033e393b8247e" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d499e40f572cfc29ecfb44f8376493df56a44213b1c1e9333b65499a0c288cd" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "53241e64de1e4470712b4061e2c82f44916d654bc532f8f1d12e5d5d4e136914" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fd168faab4546f988dc38fc56df2f71cf80c922e86d3f869be15a43f08328f99" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e578afe329abb0cba64802dfa480e00f2bbbb8c80be537791c24a31c853eb62f" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a3cb32dba55907ca907fc4f38f7ca05ef6db10a6af2dd1fa3c4db166e4ab9ffe" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7e4368ba8ec460b3c94de24ab0a04b6c799eb28df885cbbacfc3bb3ffa8c1e67" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Warning  FailedCreatePodSandBox  10m (x4 over 10m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c4aaa8f8cd2dc1eff788baf04774c4ecc845568d00ed1b386df311ec224eb6f3" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
      Normal   SandboxChanged          56s (x551 over 10m)  kubelet            Pod sandbox changed, it will be killed and re-created.
azureuser@k8-master:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
default                demo-6c59fb8f77-2jq6k                        0/1     ContainerCreating   0          5m23s
kube-system            coredns-f9fd979d6-q8s9b                      1/1     Running             2          27h
kube-system            coredns-f9fd979d6-qnm4j                      1/1     Running             2          27h
kube-system            etcd-k8-master                               1/1     Running             2          27h
kube-system            kube-apiserver-k8-master                     1/1     Running             3          27h
kube-system            kube-controller-manager-k8-master            1/1     Running             3          27h
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
kube-system            kube-proxy-4mb47                             1/1     Running             2          27h
kube-system            kube-proxy-54m9b                             1/1     Running             2          27h
kube-system            kube-proxy-wdxfz                             1/1     Running             1          27h
kube-system            kube-scheduler-k8-master                     1/1     Running             3          27h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-zmlvs   0/1     ContainerCreating   0          27h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-cnsvn         0/1     ContainerCreating   0          6h3m
To fix the flannel crashloopbackoff we did Kubeadm reset and after some time this problem showed up again.
Current we are working with one master and two worker node.
My cluster details as follows:
azureuser@k8-master:~$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://52.150.11.168:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
Docker version:
azureuser@k8-master:~$ sudo docker version
[sudo] password for azureuser: 
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        369ce74a3c
 Built:             Wed Oct 14 19:00:27 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       369ce74a3c
  Built:            Wed Oct 14 16:52:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu1~18.04.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:
kubeadm version :
azureuser@k8-master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:15:05Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
The flannel is crashing whenever I tried to schedule pod creation.
                Please paste error output as text, not as picture. Can you share more details like kubernetes version, your environment? Did you try other CNI than flanner, for example calico? Are you able to provide some steps to replicate this issue? Did you change number of nodes in your cluster? Could you also provide describe some pods of other affected pods? Do you have enough resources (RAM, CPU)? Any taints, firewall changes?
– PjoterS
                Dec 23, 2020 at 12:43
                No we haven't used any other CNI other than Flannel. And have made changes to post that you have mentioned. And yes we have enough resources.
– Giridharan M
                Dec 23, 2020 at 13:19
                Did you check if you have file in mentioned directory? /run/flannel/subnet.env ? If no you can create it manually as mention in this article or redeploy flannel. Could you also share affected flannel pod logs kubectl logs <flannelPod> -n kube-system ? Did you deploy flannel before you use kubeadm init ?
– PjoterS
                Dec 23, 2020 at 15:16
Background
I think your issue is cased by your 2 Flannel CNI pods CrashLoopBackOff status.
Your error
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
is pointing that pod cannot be created due to lack of /run/flannel/subnet.env file.
In Flannel Github document you can find:
Flannel runs a small, single binary agent called flanneld on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space.
Meaning, to proper work, Flannel pod should be running on each node as it contains subnets information. From your outputs I can see that only 1 is working properly out of 3 Flannel pods.
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE
kube-system            kube-flannel-ds-kqz4t                        0/1     CrashLoopBackOff    92         27h
kube-system            kube-flannel-ds-szqzn                        1/1     Running             3          27h
kube-system            kube-flannel-ds-v9q47                        0/1     CrashLoopBackOff    142        27h
If mentioned pod was scheduled on node where flannel pod is not working it won't be created due to CNI network issues. Besides your demo pod, also kubernetes-dashboard pods have the same issue with ContainerCreating status.
Conclusion
Your demo pod cannot be scheduled as Kubernetes encounter some network issues related with flannel configuration file (...network: open /run/flannel/subnet.env: no such file or directory).
Your flannel pods restarts counts is very high as for 27 hours. You have to determine why and fix it. It might be lack of resources, network issues with your infrastructure or many other reasons. Once all flannel pods will be working correctly, your shouldn't encounter this error.
Solution
You have to make flannel pods works correctly on each node.
Additional Troubleshooting Details
For detailed investigation please provide
$ kubectl describe kube-flannel-ds-kqz4t -n kube-system
$ kubectl describe kube-flannel-ds-v9q47 -n kube-system
Logs details would be also helpful
$ kubectl logs kube-flannel-ds-kqz4t -n kube-system
$ kubectl logs kube-flannel-ds-v9q47 -n kube-system
Please replace kubectl get pods --all-namespaces with kubectl get pods -o wide -A and output of kubectl get nodes -o wide.
If you will provide those information, it should be possible to determine root cause of flannel pods issues and I will edit this answer with exact solution.
                hi @PjoterS, I have same issue, mind if you provide some assistance? my output for k get po -o wide -A shows coredns 'Completed' and flannel-ds 'CrashLoopBackOff'; the second command shows all nodes 'Ready'
– Spencer Trinh
                Feb 28, 2021 at 21:06
                @Spencer Trinh it would be better if you would create new question on stack with some details (like describe o flanne pod, env details and steps to replicate issue).
– PjoterS
                Mar 1, 2021 at 7:06
                I found the solution at github.com/kubernetes/kubernetes/issues/70202, however every reboot I need to re-create this file. Is that normal behaviour?
– Spencer Trinh
                Mar 6, 2021 at 21:26
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.