Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

after updating my server, I can't restart kubernetes.

Feb  6 10:34:26 chgvas99 kubelet: F0206 10:34:26.662744   27634 server.go:189] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Feb  6 10:34:26 chgvas99 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Feb  6 10:34:26 chgvas99 systemd: Unit kubelet.service entered failed state.
Feb  6 10:34:26 chgvas99 systemd: kubelet.service failed.
i checked on the directory and indeed there is no config.yaml i've the same error on my nodes i cant restart them 
server : 3.10.0-957.5.1.el7.x86_64
kubernetes : Major:"1", Minor:"13", GitVersion:"v1.13.3"
              GoVersion:"go1.11.5"
                You have not specified if "my server" means the master, or a worker node, or whether this machine ever was working and mysteriously stopped or what. But, having said that, you might be able to just rm -rf /etc/kubernetes /var/lib/kubelet and re-run kubeadm join ${the_master} --token ${the_token} and reprovision the node
– mdaniel
                Feb 7, 2019 at 6:38
                thanks for your reply @matthew l Daniel  . the problems happened on all of my k8s ! master and nodes after i've update the os and restart it . i lost the file config.yaml
– morla
                Feb 7, 2019 at 8:19
You're using kubeadm - So The fact that /var/lib/kubelet/config.yaml is empty is probably related to the worker node not being joined to the cluster.
This might be related to networking issues - but lets try step by step:
1 ) Create a valid token for the worker node to join the cluster:
 Run:  sudo kubeadm token create --print-join-command --v=5 and make sure you receive an output command like:
kubeadm join <master-node-ip>:6443 --token aa334.. --discovery-token-ca-cert-hash sha256:..
2 ) Run the provided command in the worker node.
3 ) If Everything is OK - the /var/lib/kubelet/config.yaml should be populated and the status of sudo systemctl status kubelet should look good.
4 ) If you can an error - try running the ame join command with --v=5 - you'll probably see some networking issues.
4.A ) If you got an error like dial tcp <master-ip>:6443: connect: no route to host - make sure that you have a communication between your nodes - run curl <master-node-ip>:6443 from worker node - you'll probably get the same no route error.

Go to the master node and open the 6443 port (I'll assume you're working on secured private network) and try the connectivity again.
4.B ) If opening port in master succeed and you're able to curl from worker to master you should receive a response from the API server like: Client sent an HTTP request to an HTTPS server. 

5 ) If curl succeed but you're still facing connectivity problems try:
5.A ) Comparing the .kube/config files of master and worker nodes - make sure the IP of the API server is correct.
5.B ) Make sure you enabled bridge networking mode on all nodes: sudo sysctl net.bridge.bridge-nf-call-iptables=1.
5.C ) Make sure you have an SDN solution like calico, flannel or weave and that you see that the relevant kube-system pods are running:
$kubectl -n kube-system get pods
NAME                                      READY   STATUS    RESTARTS   AGE
coredns-f9fd979d6-lpdlc                   1/1     Running   2          7d12h
coredns-f9fd979d6-vcs7g                   1/1     Running   2          7d12h
etcd-master-node-k8s                      1/1     Running   2          7d12h
kube-apiserver-master-node-k8s            1/1     Running   2          7d12h
kube-controller-manager-master-node-k8s   1/1     Running   2          7d12h
kube-proxy-kh2lc                          1/1     Running   2          7d12h
kube-proxy-lfmc4                          1/1     Running   0          4m36s
kube-scheduler-master-node-k8s            1/1     Running   2          7d12h
weave-net-59r5b                           2/2     Running   6          7d11h <-- Here 
weave-net-c44d6                           2/2     Running   1          4m36s <-- Here
6 ) If nothing works - try running kubeadm reset on the worker node.
My Env: 3 master and forefront loadbalancer while initializing cluster
Had to adjust my load balancer and take out other master nodes and "kubeadm init"  and issue was gone , Network error possibly .
Then readd all the other master nodes.
Had a similar issue where got that error on control join command. It ended up being the load balancer ip assigned to the joining control as secondary  from a previous installation (which has nothing running behind it). Run ip a and make sure that the joining node does not have secondary load balancer ip.
echo y | kubeadm reset || true
rm -rf /etc/cni/net.d || true
rm -rf /var/lib/etcd || true
rm -rf ~/.kube || true
ip address delete {{ k8s_apiserver_vip_cidr }} dev {{ k8s_interface }}
kubeadm join ...
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.