You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
By clicking “Sign up for GitHub”, you agree to our
terms of service
and
privacy statement
. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
infinite error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
#3768
infinite error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
daiguocheng
opened this issue
Feb 16, 2023
· 16 comments
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
snap version
snap 2.57.6-2.el7
snapd 2.57.6-2.el7
series 16
centos 7
microk8s version
MicroK8s v1.26.0 revision 4390
cat /proc/version
Linux version 5.4.224-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC))
#1
SMP Tue Nov 8 17:24:56 EST 2022
cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
====================================================================================
journalctl -xefu snap.microk8s.daemon-containerd
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.437992358+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-qxfnb,Uid:2ce7a7a9-0bbb-4b69-b92b-783fef50bc89,Namespace:kube-system,Attempt:0,}"
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.491393765+08:00" level=warning msg=""io.containerd.runc.v1" is deprecated since containerd v1.4, consider using "io.containerd.runc.v2""
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.516462748+08:00" level=info msg="loading plugin "io.containerd.event.v1.publisher"..." runtime=io.containerd.runc.v1 type=io.containerd.event.v1
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.516507608+08:00" level=info msg="loading plugin "io.containerd.ttrpc.v1.task"..." runtime=io.containerd.runc.v1 type=io.containerd.ttrpc.v1
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.516518427+08:00" level=info msg="loading plugin "io.containerd.internal.v1.shutdown"..." runtime=io.containerd.runc.v1 type=io.containerd.internal.v1
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.516592674+08:00" level=info msg="starting signal loop" namespace=k8s.io path=/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1 pid=15025 runtime=io.containerd.runc.v1
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.582474045+08:00" level=info msg="shim disconnected" id=5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.582514866+08:00" level=warning msg="cleaning up after shim disconnected" id=5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1 namespace=k8s.io
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.582521878+08:00" level=info msg="cleaning up dead shim"
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.587916743+08:00" level=error msg="failed to delete" cmd="/snap/microk8s/4390/bin/containerd-shim-runc-v1 -namespace k8s.io -address /var/snap/microk8s/common/run/containerd.sock -publish-binary /var/lib/snapd/snap/microk8s/4390/bin/containerd -id 5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1 -bundle /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1 delete" error="exit status 1"
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.589264339+08:00" level=warning msg="failed to clean up after shim disconnected" error="io.containerd.runc.v1: remove /run/containerd/s/2d664a2b0a48ca9805195e85ceb5dbcda1d78a06b52d9f8d7213f9092cfeb9b8: no such file or directory: exit status 1" id=5f07447107967e7f8c4e73ac2cf4df66273f1574562a6f05d380ae5e26a0bcd1 namespace=k8s.io
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.589308100+08:00" level=error msg="copy shim log" error="read /proc/self/fd/20: file already closed"
2月 16 15:29:27 k8s-node-3 microk8s.daemon-containerd[1070]: time="2023-02-16T15:29:27.619287858+08:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-qxfnb,Uid:2ce7a7a9-0bbb-4b69-b92b-783fef50bc89,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
Hello,
I am getting the exact same error message in Centos 7.9. I think Ubuntu apt-get will resolve this but Centos 7.9 still has problem. How to fix in Centos.
$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
MicroK8s v1.26.1 revision 4595
$ snap version
snap 2.57.6-2.el7
snapd 2.57.6-2.el7
series 16
centos 7
kernel 3.10.0-1160.81.1.el7.x86_64
$ journalctl -xefu snap.microk8s.daemon-containerd
Feb 21 21:17:38 xxxxxxx microk8s.daemon-containerd[22714]: time="2023-02-21T21:17:38.123998675-05:00" level=warning msg="failed to clean up after shim disconnected" error="io.containerd.runc.v1: remove /run/containerd/s/404689d605a31b459d01cfa56ebace2e654acc46d6cd01b6aa0eabe2626c5cab: no such file or directory: exit status 1" id=73cff542ac1b5a9d93fd6471361c76f723896b87d5ac795235bf2c3c4f26c39f namespace=k8s.io
Feb 21 21:17:38 xxxxxxxx microk8s.daemon-containerd[22714]: time="2023-02-21T21:17:38.124048482-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed"
Feb 21 21:17:38 xxxxxxxx microk8s.daemon-containerd[22714]: time="2023-02-21T21:17:38.131711254-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-tp97h,Uid:993b0cf9-000a-480e-9cc1-8236d3b7a47e,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
$ snap logs microk8s
2023-02-21T21:27:40-05:00 microk8s.daemon-containerd[23595]: time="2023-02-21T21:27:40.568314680-05:00" level=error msg="copy shim log" error="read /proc/self/fd/19: file already closed"
2023-02-21T21:27:40-05:00 microk8s.daemon-containerd[23595]: time="2023-02-21T21:27:40.576555356-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-tp97h,Uid:993b0cf9-000a-480e-9cc1-8236d3b7a47e,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
2023-02-21T21:27:40-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:40.577432 23735 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"
2023-02-21T21:27:40-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:40.577494 23735 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown" pod="kube-system/calico-node-tp97h"
2023-02-21T21:27:40-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:40.577536 23735 kuberuntime_manager.go:782] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown" pod="kube-system/calico-node-tp97h"
2023-02-21T21:27:40-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:40.577626 23735 pod_workers.go:965] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "calico-node-tp97h_kube-system(993b0cf9-000a-480e-9cc1-8236d3b7a47e)" with CreatePodSandboxError: "Failed to create sandbox for pod \"calico-node-tp97h_kube-system(993b0cf9-000a-480e-9cc1-8236d3b7a47e)\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown"" pod="kube-system/calico-node-tp97h" podUID=993b0cf9-000a-480e-9cc1-8236d3b7a47e
2023-02-21T21:27:43-05:00 microk8s.daemon-kubelite[23735]: W0221 21:27:43.666933 23735 manager.go:1174] Failed to process watch event {EventType:0 Name:/kubepods/burstable/pod993b0cf9-000a-480e-9cc1-8236d3b7a47e/2e51105b164e12dd99695912faf634ed71b8551fc56f6de0768c792769450a76 WatchSource:0}: container "2e51105b164e12dd99695912faf634ed71b8551fc56f6de0768c792769450a76" in namespace "k8s.io": not found
2023-02-21T21:27:44-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:44.764783 23735 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
2023-02-21T21:27:46-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:46.722401 23735 summary_sys_containers.go:48] "Failed to get system container stats" err="failed to get cgroup stats for "/system.slice/snap.microk8s.daemon-kubelite.service": failed to get container info for "/system.slice/snap.microk8s.daemon-kubelite.service": unknown container "/system.slice/snap.microk8s.daemon-kubelite.service"" containerName="/system.slice/snap.microk8s.daemon-kubelite.service"
2023-02-21T21:27:49-05:00 microk8s.daemon-kubelite[23735]: E0221 21:27:49.765654 23735 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Is the version of cglib not new enough? (2.25 required, but only 2.17 available on centos-7.9)
microk8s.reset --destroy-storage
Disabling all addons
Disabling addon : core/cert-manager
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/kube-ovn
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/minio
Disabling addon : core/observability
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Restarting cluster
Stopped.
sudo: 在加载插件“sudoers_policy”时在 /etc/sudo.conf 第 19 行出错
sudo: 无法加载 /usr/libexec/sudo/sudoers.so：/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /snap/microk8s/4390/usr/lib/x86_64-linux-gnu/libgnutls.so.30)
sudo: 致命错误，无法加载插件
Setting up the CNI
ldd --version
ldd (GNU libc) 2.17
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
由 Roland McGrath 和 Ulrich Drepper 编写。
Yes I did yum update and I got same version on centos 7.9.2009 :
$ ldd --version
ldd (GNU libc) 2.17
But same issue supposed to be fixed here but not sure why:
#2181
It was something in the upgrade process that did it for me. It might have
been the container.d run time or something else
On Tue, Feb 21, 2023, 10:39 PM GC ***@***.***> wrote:
Yes I did yum update and I got same version:
$ ldd --version
ldd (GNU libc) 2.17
Reply to this email directly, view it on GitHub
<
#3768 (comment)
>,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ABDASECZMSTHD4HBLMR55BLWYWC5TANCNFSM6AAAAAAU5Z24U4
>
You are receiving this because you commented.Message ID:
***@***.***>
Just an update that I removed microk8s 1.26/stable and tried to install 1.25/stable but same error. Next I installed 1.24/stable and microk8s started without errors.
$ snap remove microk8s
$ snap install microk8s --classic --channel=1.24/stable
microk8s (1.24/stable) v1.24.10 from Canonical✓ installed
$ microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
So looks like there is GLIBC_2.25' not found issue in 1.25/stable and later versions on Centos 7.9 .
Hi
@neoaggelos
yes I was able to get another identical CentOS Linux release 7.9.2009 VM which doesn't have snap/microk8s yet and we can test specific build. I didn't want to disturb the existing running Microk8s installed: v1.24.10 (4561) 224MB classic which is also running fine Jupyterhub.
Hi
@ebebpl
I believe the GLIBC_2.25 error is the reason why you are seeing this issue. Can you instead attempt to install MicroK8s with:
sudo snap install microk8s --classic --channel latest/edge/test-containerd
And give it some time to work things out. If you see any errors related to locale settings, see
# in case you see locale issues
echo '
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
' | sudo tee -a /etc/environment
sudo reboot
Let me know if this solves your issue.
Hi @neoaggelos , I installed but ran into different issue now. Will try and see how to fix this new issue and let you know:
[root@]$ sudo snap install microk8s --classic --channel latest/edge/test-containerd
2023-03-06T11:39:57-05:00 INFO Waiting for automatic snapd restart...
microk8s (edge/test-containerd) v1.26.1 from Canonical✓ installed
[root@]$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
[root@]$ microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat ‘/var/snap/microk8s/4840/var/kubernetes/backend/cluster.yaml’: No such file or directory
cp: cannot stat ‘/var/snap/microk8s/4840/var/kubernetes/backend/localnode.yaml’: No such file or directory
cp: cannot stat ‘/var/snap/microk8s/4840/var/kubernetes/backend/info.yaml’: No such file or directory
I wonder if there is stale data that might be causing this. Can you try
sudo snap remove microk8s --purge
sudo snap install microk8s --channel latest/edge/test-containerd
This time I tried to install the old version 1.24/stable which was working in another server. But this new server seems to have problem with 1.24 also. Let me dig further it could be a network issue maybe:
[root@]$ sudo snap remove microk8s --purge
microk8s removed
[root@]$ snap install microk8s --classic --channel=1.24/stable
microk8s (1.24/stable) v1.24.10 from Canonical✓ installed
[root@]$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
[root@]$ snap logs microk8s
2023-03-06T12:37:52-05:00 microk8s.daemon-k8s-dqlite[27474]: + '[' -f /var/snap/microk8s/4561/var/lock/no-k8s-dqlite ']'
2023-03-06T12:37:52-05:00 microk8s.daemon-k8s-dqlite[27474]: + '[' -e /var/snap/microk8s/4561/var/lock/low-memory-guard.lock ']'
2023-03-06T12:37:52-05:00 microk8s.daemon-k8s-dqlite[27474]: + app=k8s-dqlite
2023-03-06T12:37:52-05:00 microk8s.daemon-k8s-dqlite[27474]: + '[' -e /var/snap/microk8s/4561/args/k8s-dqlite ']'
2023-03-06T12:37:52-05:00 microk8s.daemon-k8s-dqlite[27474]: + sleep 6
2023-03-06T12:37:53-05:00 microk8s.daemon-kubelite[27388]: W0306 12:37:53.065299 27388 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {unix:///var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379 kine.sock 0 }. Err: connection error: desc = "transport: Error while dialing dial unix /var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379: connect: no such file or directory". Reconnecting...
2023-03-06T12:37:53-05:00 microk8s.daemon-kubelite[27388]: W0306 12:37:53.533739 27388 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {unix:///var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379 kine.sock 0 }. Err: connection error: desc = "transport: Error while dialing dial unix /var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379: connect: no such file or directory". Reconnecting...
2023-03-06T12:37:54-05:00 microk8s.daemon-kubelite[27388]: W0306 12:37:54.493426 27388 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {unix:///var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379 kine.sock 0 }. Err: connection error: desc = "transport: Error while dialing dial unix /var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379: connect: no such file or directory". Reconnecting...
2023-03-06T12:37:55-05:00 microk8s.daemon-apiserver-kicker[23278]: Setting up the CNI
2023-03-06T12:37:55-05:00 microk8s.daemon-kubelite[27388]: W0306 12:37:55.975382 27388 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {unix:///var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379 kine.sock 0 }. Err: connection error: desc = "transport: Error while dialing dial unix /var/snap/microk8s/4561/var/kubernetes/backend/kine.sock:12379: connect: no such file or directory". Reconnecting...
This issue should be fixed (for some time now, sorry for not updating the issue) with #3839 in MicroK8s 1.25 and newer.
If you encounter again, please open a new issue, thanks!
