手动部署k8s(十二)-部署worker节点kubelet组件 > 原教程来自 [github/opsnull](https://github.com/opsnull/follow-me-install-kubernetes-cluster), 现在此基础上记录自己搭建遇到的问题 ### 下载和分发 kubelet 二进制文件 参考 [手动部署k8s(八)-部署 master 节点](https://kkwen.cn/index.php/archives/30/) ### 安装依赖包 参考 [(六)手动部署k8s之-部署flannel网络](https://kkwen.cn/index.php/archives/18/) ### 创建 kubelet bootstrap kubeconfig 文件 ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_name in slave-34 slave-35 slave-36 do echo ">>> ${node_name}" # 创建 token export BOOTSTRAP_TOKEN=$(kubeadm token create \ --description kubelet-bootstrap-token \ --groups system:bootstrappers:${node_name} \ --kubeconfig ~/.kube/config) # 设置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/cert/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置上下文参数 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置默认上下文 kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig done ``` - 向 kubeconfig 写入的是 token,bootstrap 结束后 kube-controller-manager 为 kubelet 创建 client 和 server 证书 查看 kubeadm 为各节点创建的 token ``` [root@ _38_ /opt/k8s/work]# kubeadm token list --kubeconfig ~/.kube/config TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 41in74.vbutfdhlhq17flhy 23h 2019-08-06T13:41:51+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:slave-34 4h412a.ezo1f6iafyko8yul 23h 2019-08-06T13:41:53+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:slave-36 y82yeg.8ucvez5eabnzizfi 23h 2019-08-06T13:41:53+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:slave-35 [root@ _39_ /opt/k8s/work]# ``` - token 有效期为 1 天,超期后将不能再被用来 boostrap kubelet,且会被 kube-controller-manager 的 tokencleaner 清理; - kube-apiserver 接收 kubelet 的 bootstrap token 后,将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers,后续将为这个 group 设置 ClusterRoleBinding 查看各 token 关联的 Secret ``` [root@ _39_ /opt/k8s/work]# kubectl get secrets -n kube-system|grep bootstrap-token bootstrap-token-41in74 bootstrap.kubernetes.io/token 7 116s bootstrap-token-4h412a bootstrap.kubernetes.io/token 7 114s bootstrap-token-y82yeg bootstrap.kubernetes.io/token 7 114s ``` ### 分发 bootstrap kubeconfig 文件到所有 worker 节点 ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_name in slave-34 slave-35 slave-36 do echo ">>> ${node_name}" scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig done ``` ### 创建和分发 kubelet 参数配置文件 从 v1.10 开始,部分 kubelet 参数需在配置文件中配置,kubelet --help 会提示 ``` DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag ``` 创建 kubelet 参数配置文件模板(可配置项参考代码中注释 ) ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh cat > kubelet-config.yaml.template <>> ${node_ip}" sed -e "s/##NODE_IP##/${node_ip}/" kubelet-config.yaml.template > kubelet-config-${node_ip}.yaml.template scp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/etc/kubernetes/kubelet-config.yaml done ``` ### 创建和分发 kubelet systemd unit 文件 创建 kubelet systemd unit 文件模板 ``` cd /opt/k8s/work source /opt/k8s/bin/environment.sh cat > kubelet.service.template <>> ${node_name}" sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.service scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service done ``` ### Bootstrap Token Auth 和授予权限 - kubelet 启动时查找 --kubeletconfig 参数对应的文件是否存在,如果不存在则使用 --bootstrap-kubeconfig 指定的 kubeconfig 文件向 kube-apiserver 发送证书签名请求 (CSR)。 - kube-apiserver 收到 CSR 请求后,对其中的 Token 进行认证,认证通过后将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers,这一过程称为 Bootstrap Token Auth。 默认情况下,这个 user 和 group 没有创建 CSR 的权限,kubelet 启动失败,错误日志如下 ``` $ sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests' May 26 12:13:41 zhangjun-k8s01 kubelet[128468]: I0526 12:13:41.798230 128468 certificate_manager.go:366] Rotating certificates May 26 12:13:41 zhangjun-k8s01 kubelet[128468]: E0526 12:13:41.801997 128468 certificate_manager.go:385] Failed while requesting a signed certificate from the master: cannot cre ate certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:82jfrm" cannot create resource "certificatesigningrequests" i n API group "certificates.k8s.io" at the cluster scope May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.044828 128468 kubelet.go:2244] node "zhangjun-k8s01" not found May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.078658 128468 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Unauthor ized May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.079873 128468 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Unauthorize d May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.082683 128468 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Unau thorized May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.084473 128468 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unau thorized May 26 12:13:42 zhangjun-k8s01 kubelet[128468]: E0526 12:13:42.088466 128468 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: U nauthorized ``` 解决办法是:创建一个 clusterrolebinding,将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定 ``` kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers ``` ### 启动 kubelet 服务 ``` source /opt/k8s/bin/environment.sh for node_ip in ${NODE_IPS[@]} do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/" ssh root@${node_ip} "/usr/sbin/swapoff -a" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet" done ``` - 启动服务前必须先创建工作目录; - 关闭 swap 分区,否则 kubelet 会启动失败 ### 修改部分 - 将work节点的/etc/kubernetes/kubelet-bootstrap.kubeconfig中127.0.0.1:8443改为master节点地址 192.168.1.31或192.168.1.32或192.168.1.33 - 将master节点上的kube-nginx监听的地址127.0.0.1:8443 改为 listen 0.0.0.0:8443 - 重启kube-nginx ### 自动 approve CSR 请求 创建三个 ClusterRoleBinding,分别用于自动 approve client、renew client、renew server 证书 ``` cd /opt/k8s/work cat > csr-crb.yaml < 58s v1.14.2 slave-35 Ready 77s v1.14.2 slave-36 Ready 74s v1.14.2 ``` kube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥 ``` [root@ _77_ /opt/k8s/work]# ls -l /etc/kubernetes/kubelet.kubeconfig -rw------- 1 root root 2313 8月 5 15:26 /etc/kubernetes/kubelet.kubeconfig [root@ _78_ /opt/k8s/work]# ls -l /etc/kubernetes/cert/|grep kubelet -rw------- 1 root root 1273 8月 5 15:30 kubelet-client-2019-08-05-15-30-11.pem lrwxrwxrwx 1 root root 59 8月 5 15:30 kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2019-08-05-15-30-11.pem ``` - 没有自动生成 kubelet server 证书 ### 手动 approve server cert csr 基于安全性考虑,CSR approving controllers 不会自动 approve kubelet server 证书签名请求,需要手动 approve ``` [root@ _84_ /opt/k8s/work]# kubectl get csr NAME AGE REQUESTOR CONDITION csr-27shb 3m17s system:node:slave-36 Pending csr-2fl9z 6m38s system:bootstrap:41in74 Approved,Issued csr-6xdh5 6m56s system:bootstrap:y82yeg Approved,Issued csr-8hf5g 3m19s system:node:slave-35 Pending csr-jk257 3m1s system:node:slave-34 Pending csr-n4c99 9m17s system:bootstrap:41in74 Approved,Issued csr-t442g 9m22s system:bootstrap:y82yeg Approved,Issued csr-v4tph 6m53s system:bootstrap:4h412a Approved,Issued csr-vbjsl 9m18s system:bootstrap:4h412a Approved,Issued [root@ _85_ /opt/k8s/work]# kubectl certificate approve csr-27shb certificatesigningrequest.certificates.k8s.io/csr-27shb approved [root@ _86_ /opt/k8s/work]# kubectl certificate approve csr-8hf5g certificatesigningrequest.certificates.k8s.io/csr-8hf5g approved [root@ _87_ /opt/k8s/work]# kubectl certificate approve csr-jk257 certificatesigningrequest.certificates.k8s.io/csr-jk257 approved [root@ _79_ /opt/k8s/work]# ls -l /etc/kubernetes/cert/kubelet-* -rw------- 1 root root 1273 8月 5 15:30 /etc/kubernetes/cert/kubelet-client-2019-08-05-15-30-11.pem lrwxrwxrwx 1 root root 59 8月 5 15:30 /etc/kubernetes/cert/kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2019-08-05-15-30-11.pem -rw------- 1 root root 1313 8月 5 15:34 /etc/kubernetes/cert/kubelet-server-2019-08-05-15-34-17.pem lrwxrwxrwx 1 root root 59 8月 5 15:34 /etc/kubernetes/cert/kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-2019-08-05-15-34-17.pem ``` ### kubelet 提供的 API 接口 kubelet 启动后监听多个端口,用于接收 kube-apiserver 或其它客户端发送的请求 ``` [root@ _80_ /opt/k8s/work]# netstat -lnpt|grep kubelet tcp 0 0 192.168.1.34:10248 0.0.0.0:* LISTEN 19220/kubelet tcp 0 0 192.168.1.34:10250 0.0.0.0:* LISTEN 19220/kubelet tcp 0 0 127.0.0.1:35982 0.0.0.0:* LISTEN 19220/kubelet ``` - 10248: healthz http 服务; - 10250: https 服务,访问该端口时需要认证和授权(即使访问 /healthz 也需要); 未开启只读端口 10255; - 从 K8S v1.10 开始,去除了 --cadvisor-port 参数(默认 4194 端口),不支持访问 cAdvisor UI & API 例如执行 kubectl exec -it nginx-ds-5rmws -- sh 命令时,kube-apiserver 会向 kubelet 发送如下请求: ``` POST /exec/default/nginx-ds-5rmws/my-nginx?command=sh&input=1&output=1&tty=1 ``` kubelet 接收 10250 端口的 https 请求,可以访问如下资源: - /pods、/runningpods - /metrics、/metrics/cadvisor、/metrics/probes - /spec - /stats、/stats/container - /logs - /run/、/exec/, /attach/, /portForward/, /containerLogs/ 由于关闭了匿名认证,同时开启了 webhook 授权,所有访问 10250 端口 https API 的请求都需要被认证和授权。 预定义的 ClusterRole system:kubelet-api-admin 授予访问 kubelet 所有 API 的权限(kube-apiserver 使用的 kubernetes 证书 User 授予了该权限): ``` [root@ _89_ /opt/k8s/work]# kubectl describe clusterrole system:kubelet-api-admin Name: system:kubelet-api-admin Labels: kubernetes.io/bootstrapping=rbac-defaults Annotations: rbac.authorization.kubernetes.io/autoupdate: true PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- nodes/log [] [] [*] nodes/metrics [] [] [*] nodes/proxy [] [] [*] nodes/spec [] [] [*] nodes/stats [] [] [*] nodes [] [] [get list watch proxy] ``` ### kubelet api 认证和授权 kubelet 配置了如下认证参数: - authentication.anonymous.enabled:设置为 false,不允许匿名�访问 10250 端口; - authentication.x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTPs 证书认证; - authentication.webhook.enabled=true:开启 HTTPs bearer token 认证; 同时配置了如下授权参数 - authroization.mode=Webhook:开启 RBAC 授权 kubelet 收到请求后,使用 clientCAFile 对证书签名进行认证,或者查询 bearer token 是否有效。如果两者都没通过,则拒绝请求,提示 Unauthorized ``` [root@ _81_ /opt/k8s/work]# curl -s --cacert /etc/kubernetes/cert/ca.pem https://192.168.1.34:10250/metrics Unauthorized ``` 通过认证后,kubelet 使用 SubjectAccessReview API 向 kube-apiserver 发送请求,查询证书或 token 对应的 user、group 是否有操作资源的权限(RBAC); ### 证书认证和授权 ``` # 权限不足的证书 [root@ _93_ /opt/k8s/work]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://192.168.1.34:10250/metrics Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics) # 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书 [root@ _94_ /opt/k8s/work]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.1.34:10250/metrics|head # HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 ``` - --cacert、--cert、--key 的参数值必须是文件路径,如上面的 ./admin.pem 不能省略 ./,否则返回 401 Unauthorized ### bear token 认证和授权 创建一个 ServiceAccount,将它和 ClusterRole system:kubelet-api-admin 绑定,从而具有调用 kubelet API 的权限 ``` [root@ _95_ /opt/k8s/work]# kubectl create sa kubelet-api-test serviceaccount/kubelet-api-test created [root@ _96_ /opt/k8s/work]# kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test clusterrolebinding.rbac.authorization.k8s.io/kubelet-api-test created [root@ _97_ /opt/k8s/work]# SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}') [root@ _98_ /opt/k8s/work]# TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}') [root@ _99_ /opt/k8s/work]# echo ${TOKEN} eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Imt1YmVsZXQtYXBpLXRlc3QtdG9rZW4tbndiaDgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3ViZWxldC1hcGktdGVzdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjhkZjQyZjg3LWI3NTUtMTFlOS04ODllLTAwMGMyOTNkMWRlNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Omt1YmVsZXQtYXBpLXRlc3QifQ.Hh4xl6e8e2GRIQOjZDVPwSM3sQLoDFsFqVX3Entg84dTrERpESe1ST0yCcE22QdB5mnm5b8LHY0j4Uu6W14a0Pzkip3OORshAwDlNqa3hpiBuMo84gdzR0VfkC4SrKMnnNzaC2GLD9eFiMVsX6ZbAsLYziH7Az_81DIz8OH7IH6wTyWJmtlJ70q3AbEGeKjlpbsgEeV7uj1XsrifnIGt_n_Lx4osbjDE6kvlQDEsZZBjQ0aAwusyfu7YjhGOKObvpoc58RF64uWb84x7S0C51eSZS2cxWUtxfcd9itffXvJSWf91nOUNYqkWirHqsOVmJEawBC3ELZaQj9qAemos_g ``` ``` [root@ _100_ /opt/k8s/work]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://192.168.1.34:10250/metrics|head # HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 # HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 ``` ### cadvisor 和 metrics cadvisor 是内嵌在 kubelet 二进制中的,统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况的服务。 浏览器访问 https://192.168.1.34:10250/metrics 和 https://192.168.1.34:10250/metrics/cadvisor 分别返回 kubelet 和 cadvisor 的 metrics。 kubelet.config.json 设置 authentication.anonymous.enabled 为 false,不允许匿名证书访问 10250 的 https 服务; 需要浏览器中手动导入证书,具体方法查看链接 [参考A.浏览器访问kube-apiserver安全端口.md](https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/A.%E6%B5%8F%E8%A7%88%E5%99%A8%E8%AE%BF%E9%97%AEkube-apiserver%E5%AE%89%E5%85%A8%E7%AB%AF%E5%8F%A3.md),创建和导入相关证书,然后访问上面的 10250 端口 ### 获取 kubelet 的配置 从 kube-apiserver 获取各节点 kubelet 的配置 ``` # 使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书; [root@ _104_ /opt/k8s/work]# curl -sSL --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem ${KUBE_APISERVER}/api/v1/nodes/slave-34/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' { "syncFrequency": "1m0s", "fileCheckFrequency": "20s", "httpCheckFrequency": "20s", "address": "192.168.1.34", "port": 10250, "rotateCertificates": true, "serverTLSBootstrap": true, "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/cert/ca.pem" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "registryPullQPS": 0, "registryBurst": 20, "eventRecordQPS": 0, "eventBurst": 20, "enableDebuggingHandlers": true, "enableContentionProfiling": true, "healthzPort": 10248, "healthzBindAddress": "192.168.1.34", "oomScoreAdj": -999, "clusterDomain": "cluster.local", "clusterDNS": [ "10.254.0.2" ], "streamingConnectionIdleTimeout": "4h0m0s", "nodeStatusUpdateFrequency": "10s", "nodeStatusReportFrequency": "1m0s", "nodeLeaseDurationSeconds": 40, "imageMinimumGCAge": "2m0s", "imageGCHighThresholdPercent": 85, "imageGCLowThresholdPercent": 80, "volumeStatsAggPeriod": "1m0s", "cgroupsPerQOS": true, "cgroupDriver": "cgroupfs", "cpuManagerPolicy": "none", "cpuManagerReconcilePeriod": "10s", "runtimeRequestTimeout": "10m0s", "hairpinMode": "promiscuous-bridge", "maxPods": 220, "podCIDR": "172.30.0.0/16", "podPidsLimit": -1, "resolvConf": "/etc/resolv.conf", "cpuCFSQuota": true, "cpuCFSQuotaPeriod": "100ms", "maxOpenFiles": 1000000, "contentType": "application/vnd.kubernetes.protobuf", "kubeAPIQPS": 1000, "kubeAPIBurst": 2000, "serializeImagePulls": false, "evictionHard": { "memory.available": "100Mi" }, "evictionPressureTransitionPeriod": "5m0s", "enableControllerAttachDetach": true, "makeIPTablesUtilChains": true, "iptablesMasqueradeBit": 14, "iptablesDropBit": 15, "failSwapOn": true, "containerLogMaxSize": "20Mi", "containerLogMaxFiles": 10, "configMapAndSecretChangeDetectionStrategy": "Watch", "enforceNodeAllocatable": [ "pods" ], "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1" } ``` Last modification:August 5th, 2019 at 05:08 pm © 允许规范转载 Support 如果觉得我的文章对你有用 ×Close Appreciate the author Sweeping payments