相关文章推荐
没人理的绿豆  ·  SQL Server 和Oracle ...·  1 年前    · 
逼格高的蜡烛  ·  How do I send and ...·  1 年前    · 
高大的荔枝  ·  QGraphicsScene管理QGraph ...·  1 年前    · 
Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

I want to create a secret for my kubernetes cluster. So I composed following dummy-secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

When I run kubectl create -f dummy-secret.yaml I receive back following message:

Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

Not sure why it happens.

As I understood, I need to encode all values under the data key in the yaml file. So I did base64 encoding, but kubernetes still doesn't handle the yaml secret file as I expect.

UPDATE:

I used this command to encode data values on my mac:

echo -n 'mega_secret_key' | openssl base64

I got the decoded values "mega_secret_key" and "really_secret_value1" from from your encoded data. Seems they are not encoded in right way. So, encode your data in right way:

$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==
$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK

Then check whether they are encoded properly:

$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1

So they are ok. Now use them in your dummy-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK

And run $ kubectl create -f dummy-secret.yaml.

UPDATE on 11-02-2022:

The newer versions of Kubernetes support the optional stringData property where one can provide the value against any key without decoding.

All key-value pairs in the stringData field are internally merged into the data field. If a key appears in both the data and the stringData field, the value specified in the stringData field takes precedence.

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
stringData:
  API_KEY: mega_secret_key
  API_SECRET: really_secret_value1

UPDATE:

If you use -n flag while running $ echo "some_text", it will trim the trailing \n (newline) from the string you are printing.

$ echo "some_text"
some_text
$ echo -n "some_text"
some_text⏎

Just try it,

# first encode
$ echo -n "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5
$ echo -n "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTE=
# then decode and check whether newline is stripped
$ echo "bWVnYV9zZWNyZXRfa2V5" | base64 -d
mega_secret_key⏎
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTE=" | base64 -d
really_secret_value1⏎

You can use these newly (without newline) decoded data in your secret instead. That also should fine.

$ cat - <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
secret/dummy-secret created

At the time of update, my kubernetes version is,

Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} Server
Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} ```
                Still receive back: Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...
– Alex Fruzenshtein
                Nov 20, 2018 at 15:30
                apiVersion: v1 kind: Secret metadata:   name: dummy-secret type: Opaque data:   API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==   API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK
– Alex Fruzenshtein
                Nov 20, 2018 at 15:43
                Your commands here will include a newline in the values, which is usually not what you want.
– matte
                Jun 11, 2019 at 16:18

This was already answered but for future reference, there is no need to encode the strings by using stringData instead of data field as shown below:

#secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
stringData:
  API_KEY: "STRING_IN_CLEAR_TEXT"
  API_SECRET: "STRING_IN_CLEAR_TEXT"
                When using python kubernetes API client, how to change data to stringData cause it defaults to k v1.ObjectMeta.TypeMeta: Kind: Data not Kind: stringData
– Arsen
                May 15, 2020 at 16:50

After a while I want to return back to this question and leave an answer with a reference to official kubernetes docs:

echo -n 'admin' | base64
YWRtaW4=
echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm

Pay extra attention to -n, because it guaranties that after decoding your secret key will not contain 'new line symbol'.

$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

If I use your original it works fine:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

Then:

$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created

I'm using the following version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

This may also happen when trying to remove the new line characters in a wrong way (the correct way is to remove the suffix "Cg==").

I used base64 from cli although there are workarounds to avoid the NL, like

https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334

they don't work in MacOS, I found it simpler to use python like this

import base64
data = "abc123!?$*&()'-=@~"
# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")

Or still using a pure bash-based solution:

echo 'secret_to_encode' | tr -d \\n | base64
    sudo mkdir backup 
    sudo cp -R /etc/kubernetes backup/
    sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/

look to dir /etc/kubernetes/

ls -l
total 80
-rw------- 1 root root 5440 Mar  3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17  2020 audit-policy
-rw-r--r-- 1 root root  368 Mar  4  2020 calico-config.yml
-rw-r--r-- 1 root root  270 Mar  4  2020 calico-crb.yml
-rw-r--r-- 1 root root  341 Mar  4  2020 calico-cr.yml
-rw-r--r-- 1 root root  147 Mar  4  2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar  4  2020 calico-node.yml
-rw------- 1 root root 5472 Mar  3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14  2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar  3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar  4  2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14  2020 manifests
lrwxrwxrwx 1 root root   28 Mar  4  2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar  3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar  3 10:20 ssl

try to find k8s cluster config, in my case this is kubeadm-config.v1alpha3.yaml if same file absent you can generate

kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml

also in my case absent folder /etc/kubernetes/pki and exist ~/ssl I create symlink /etc/kubernetes/pki from /etc/kubernetes/ssl

ln -s /etc/kubernetes/ssl /etc/kubernetes/pki

let regenerate certs

kubeadm alpha phase certs apiserver  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing apiserver certificate and key.
kubeadm alpha phase certs apiserver-kubelet-client
I0303 13:12:24.543254   40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.
kubeadm alpha phase certs front-proxy-client
I0303 13:12:35.660672   40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.
kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]
kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing etcd/server certificate and key.
kubeadm alpha phase certs  etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/healthcheck-client certificate and key.
kubeadm alpha phase certs  etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]

check actuality certs

find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:29 2022 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:52 2022 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:06:48 2022 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:39:56 2022 GMT
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 10:29:43 2030 GMT
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 19:40:13 2022 GMT
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:36:38 2022 GMT

next step to generate new config files: admin.conf, controller-manager.conf, kubelet.conf, scheduler.conf, in first step move old files to /tmp

cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/
kubeadm alpha phase kubeconfig all  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml 
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"

after restart master kubelet and check status after restart

sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet
systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 53001 (kubelet)
   Memory: 51.2M
   CGroup: /system.slice/kubelet.service

check access master node

kubectl get nodes
kubectl get ns
NAME                  STATUS   AGE
default               Active   464d
product1-mvp          Active   318d
infra-logging         Active   315d
infra-nginx-ingress   Active   386d
kube-public           Active   464d
kube-system           Active   464d
pg                    Active   318d

check cert

notAfter=Mar  3 07:40:43 2022 GMT

repeat procedure for all master nodes.

Now we ready to renew certs for work nodes. In first step we must delete or move kubelet.conf

cd /etc/kubernetes/
mv kubelet.conf kubelet.conf_old

after change bootstrap-kubelet.conf

**apiVersion: v1 clusters:

  • cluster: certificate-authority-data: | LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ETX server: https://192.168.4.201:6443 name: product1 contexts:
  • context: cluster: product1 user: tls-bootstrap-token-user name: tls-bootstrap-token-user@product1 current-context: tls-bootstrap-token-user@product1 kind: Config preferences: {} users:
  • name: tls-bootstrap-token-user user: token: fgz9qz.lujw0bwsdfhdsfjhgds**
  • where: certificate-authority-data – root certificate PKI CA master, may take from /etc/kubernetes/kubelet.conf on master nodes

  • server: https://192.168.4.201:6443 - ip api server master node virtual balance ip
  • token: fgz9qz.lujw0bwsdfhdsfjhgds - token, generate on master node

    kubeadm token create
    

    after restart and check kubelet work node and work node come back to cluster:

    systemctl restart kubelet
    systemctl status kubelet -l
    ● kubelet.service - Kubernetes Kubelet Server
       Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
       Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
         Docs: https://github.com/GoogleCloudPlatform/kubernetes
      Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
     Main PID: 54621 (kubelet)
       Memory: 52.1M
       CGroup: /system.slice/kubelet.service
    

    check update certs may be ls in dir

    ls -las /var/lib/kubelet/pki/
    total 24
    4 -rw-------. 1 root root 1135 Mar  3 14:06 kubelet-client-2021-03-03-14-06-34.pem
    0 lrwxrwxrwx. 1 root root   59 Mar  3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
    4 -rw-r--r--. 1 root root 2267 Mar  2 10:40 kubelet.crt
    4 -rw-------. 1 root root 1679 Mar  2 10:40 kubelet.key
    

    after repeat this procedure by all work nodes.

    I encountered the same issue.
    In my case the encoding was fine - I just copied the output without the last character of "=".

    Additional scenarios that might lead to the same error:

  • Misplaced whitespaces or broken lines in the base64 copied output.

  • Not specifying the type field in the secret.

    This gave me the right hint. I used Notepad++ to encode a string as Base64 and the resulting string had no padding characters. After adding two padding characters ("=") the API request got accepted. – turbolocust Jun 20, 2022 at 11:09

    Thanks for contributing an answer to Stack Overflow!

    • Please be sure to answer the question. Provide details and share your research!

    But avoid

    • Asking for help, clarification, or responding to other answers.
    • Making statements based on opinion; back them up with references or personal experience.

    To learn more, see our tips on writing great answers.

  •