Hi I am installed Rancher with RKE2 on SLE Micro using the R rancher-users #general

Hi,I am installed Rancher with RKE2 on SLE Micro using the RPM package. I did this two times before, and it always worked great, installing the package,

systemctl start rke2-server

, and waiting for it to come up. This time, the service will not come up - in the kubelet.log file I find several entries of this:

Copy code

E1205 13:53:15.683661    2726 remote_runtime.go:209] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"
E1205 13:53:15.683713    2726 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683746    2726 kuberuntime_manager.go:833] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown" pod="kube-system/etcd-rancher-har-nue-01"
E1205 13:53:15.683802    2726 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-rancher-har-nue-01_kube-system(e18aa5e5b83a5a3c56d78e4054612394)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown\"" pod="kube-system/etcd-rancher-har-nue-01" podUID=e18aa5e5b83a5a3c56d78e4054612394
E1205 13:53:15.723238    2726 kubelet.go:2466] "Error getting node" err="node \"rancher-har-nue-01\" not found"

Am I right in thinking that this is my issue? If yes, any ideas what is happening here and where that "invalid argument: unknown" could come from?

creamy-pencil-82913

12/05/2022, 5:53 PM

Can you read through the comments at https://github.com/rancher/rke2/issues/851 and see if anything here matches what you’ve done or are experiencing? Sounds very similar.

thankful-balloon-877

12/05/2022, 6:10 PM

Thanks Brandon, I actually found that issue, but figured it does not really match my use case, because I do not have SELinux enforced and neither do I use any non-standard network plugins

creamy-pencil-82913

12/05/2022, 6:26 PM

Are you using your own containerd?

thankful-balloon-877

12/05/2022, 6:26 PM

Nothing of the sorts, it's a stock installation of SLE Micro

This is what I wrote myself together from the previous attempts .. hence I am a bit confused what's different this time 🙂 https://w3.nue.suse.com/~gpfuetzenreuter/init-rke-node.sh.txt

creamy-pencil-82913

12/05/2022, 7:00 PM

you might look at the containerd log file and see if there’s anything else in there that might suggest whats gone wrong

thankful-balloon-877

12/05/2022, 7:16 PM

is it possible I don't even have containerd?

Copy code

rancher-har-nue-01:~ # ls /var/log/containers/ 
rancher-har-nue-01:~ # rpm -qa|grep container 
container-selinux-2.188.0-150400.1.8.noarch

creamy-pencil-82913

12/05/2022, 7:25 PM

The containerd log, not the container logs

/var/lib/rancher/rke2/agent/containerd/containerd.log

so you do have selinux stuff on here but it’s not in enforcing mode?

what mode is it in?

thankful-balloon-877

12/05/2022, 7:28 PM

oh, sorry, I forgot containerd was shipped together with rke2. containerd.log shows some of this:

Copy code

time="2022-12-05T19:26:40.803544362Z" level=warning msg="cleanup warnings time=\"2022-12-05T19:26:40Z\" level=info msg=\"starting signal loop\" namespace=k8s.
io pid=17802\ntime=\"2022-12-05T19:26:40Z\" level=warning msg=\"failed to read init pid file\" error=\"open /run/k3s/containerd/io.containerd.runtime.v2.task/
<http://k8s.io/3e36f5b6a0971ee3b62b5597f9c9931d8e58edc45b9a55ecf509272f6bb5a1a2/init.pid|k8s.io/3e36f5b6a0971ee3b62b5597f9c9931d8e58edc45b9a55ecf509272f6bb5a1a2/init.pid>: no such file or directory\"\n" 
time="2022-12-05T19:26:40.803953105Z" level=error msg="copy shim log" error="read /proc/self/fd/21: file already closed" 
time="2022-12-05T19:26:40.818880484Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-rancher-har-nue-01,Uid:e18aa5e5b83a5a3c56d78e4054612394
,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim: OCI runtime create failed: runc create faile
d: unable to start container process: error during container init: write /proc/self/attr/keycreate: invalid argument: unknown"

if you want I can upload the full file, but it seems similar to kubelet log?

SELinux shows as permissive, I have not configured anything in regards to it

Copy code

rancher-har-nue-01:~ # getenforce 
Permissive

creamy-pencil-82913

12/05/2022, 7:29 PM

what versions of SLE Micro and RKE2 are you using?

thankful-balloon-877

12/05/2022, 7:29 PM

Copy code

rancher-har-nue-01:~ # rpm -qa|grep rke 
rke2-selinux-0.11-1.sle.noarch 
rke2-common-1.23.14~rke2r1-0.x86_64 
rke2-server-1.23.14~rke2r1-0.x86_64 
rancher-har-nue-01:~ # grep PRETTY /etc/os-release 
PRETTY_NAME="SUSE Linux Enterprise Micro 5.3"

creamy-pencil-82913

12/05/2022, 7:32 PM

I think it is related to weird selinux stuff. Either make sure you have all the selinux-related packages installed (latest rke2-selinux) and rke2 started with

selinux: true

or remove the other selinux bits.

https://github.com/containerd/containerd/issues/5864#issuecomment-898625687 suggests that it is related to using selinux contexts that don’t exist, which would make sense if you were missing some selinux bits. Permissive mode doesn’t block anything, but you still need to set up the contexts properly.

thankful-balloon-877

12/05/2022, 7:54 PM

interesting. I now tried it with

selinux: true

added to config.yaml, but that seems to get stuck with the same loop. what I notice is that on one of my existing installations (same setup just slightly older versions)

Copy code

ls -RZ /var/lib/rancher/|grep container_var_lib
system_u:object_r:container_var_lib_t:s0 rke2 
    system_u:object_r:container_var_lib_t:s0 agent 
    system_u:object_r:container_var_lib_t:s0 bin 
    system_u:object_r:container_var_lib_t:s0 server
where as on the new one
 Copy code
ls -RZ /var/lib/rancher/|grep container_var_lib
<empty>
rancher-har-nue-01:~ # restorecon -Rvn /var/lib/rancher/|grep container_var_lib
<empty>


                              the old/working one has these versions
                              
                               
                                
                                 
                                 Copy code
                                
                               
                              
                              rancher-prv-01:~ # rpm -qa|egrep 'selinux|rke' 
selinux-policy-targeted-20210716-150400.2.3.noarch 
patterns-microos-selinux-5.3.3-150400.1.1.x86_64 
libselinux1-3.1-150400.1.69.x86_64 
selinux-policy-20210716-150400.2.3.noarch 
container-selinux-2.188.0-150400.1.2.noarch 
selinux-tools-3.1-150400.1.69.x86_64 
rke2-selinux-0.9-1.sle.noarch 
rke2-common-1.23.9~rke2r1-0.x86_64 
rke2-server-1.23.9~rke2r1-0.x86_64

creamy-pencil-82913


                            12/05/2022, 7:57 PM


                              did you install the rke2-selinux package that I mentioned above?

thankful-balloon-877


                            12/05/2022, 7:57 PM


                              yes, it is installed

creamy-pencil-82913


                            12/05/2022, 7:57 PM


                              ah but no container-selinux


                               that should be a dependency for the rke2-selinux package


                                did you get any errors when installing it?


                                 you appear to be missing a bunch of selinux related stuff, compared to your working node at least

thankful-balloon-877


                               12/05/2022, 7:58 PM


                                 it is.
                                 
                                  
                                   
                                    
                                    Copy code
                                   
                                  
                                 
                                 rancher-har-nue-01:~ # rpm -q container-selinux 
container-selinux-2.188.0-150400.1.8.noarch 
rancher-har-nue-01:~ # rpm -q rke2-selinux 
rke2-selinux-0.11-1.sle.noarch
rancher-har-nue-01:~ # rpm -q --requires rke2-selinux |grep container 
container-selinux >= 2.164.2-1.1

creamy-pencil-82913


                               12/05/2022, 7:58 PM


                                 ah ok. I didn’t see it in the package list you posted above

thankful-balloon-877


                               12/05/2022, 7:58 PM


                                 I think I confused you by using different grep patterns, sorry
                                 
                                 😄

creamy-pencil-82913


                               12/05/2022, 7:58 PM


                                 yeah


                                  I would probably just compare packages between the two, see if there’s anything else you’re missing. It is very odd that you are not getting any container contexts set on the binaries


                                   what contexts DO the rke2 binaries have?

thankful-balloon-877


                                 12/05/2022, 8:01 PM


                                   the package sets seem to be the same.. just the versions are newer, but I guess others are using the same packages too..
on the binary there's another obscurity indeed:
                                   
                                    
                                     
                                      
                                      Copy code
                                     
                                    
                                   
                                   old:
rancher-prv-01:~ # ls -Z /usr/bin/rke2 
system_u:object_r:container_runtime_exec_t:s0 /usr/bin/rke2