What system:bootstrappers:kubeadm:default-node-token？

system:bootstrappers:kubeadm:default-node-token是个Group(常量值），谁在用？通过实验发现这个Group只有通过kubeadm init 安装后的集群才有，通过二进制安装的集群是没有。

通过kubeadm token list 和 kubectl get secrets 可以查看到bootstrap-token的auth-extra-groups为system:bootstrappers:kubeadm:default-node-token

kubeadm token list

TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION                                                EXTRA GROUPS
f723b1.e6cf92ad0c6cac77   <forever>   <never>   authentication,signing   kubelet-bootstrap-token                                    system:bootstrappers:kubeadm:default-node-token



kubectl get secrets -n kube-system bootstrap-token-f723b1 -o jsonpath='{.data.auth-extra-groups}' |base64 -d
system:bootstrappers:kubeadm:default-node-token

1）从rolebindings查看看

kubectl get rolebindings -A -o json |grep "system:bootstrappers:kubeadm:default-node-token" | wc -l
4

kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[0].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4
kube-proxy
kubeadm:kubeadm-certs
kubeadm:nodes-kubeadm-config

只有三条？继续再改下

kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[1].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4
kubeadm:kubelet-config

2）从clusterrolebindings查看看

kubectl get clusterrolebindings -o json |grep "system:bootstrappers:kubeadm:default-node-token" | wc -l
3

kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.subjects[0].kind=="Group") | select(.subjects[0].name=="system:bootstrappers:kubeadm:default-node-token")' | grep -A2 metadata | grep name |cut -d \" - -f4

kubeadm:get-nodes
kubeadm:kubelet-bootstrap
kubeadm:node-autoapprove-bootstrap

export rolebinding clusterrolebinding role and clusterrole

PAGER="cat"
command -v "kubectl-neat" &>/dev/null && PAGER="kubectl neat"

file=/dev/stdout

(
echo "---"
kubectl get rolebinding kube-proxy -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:kubeadm-certs  -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:nodes-kubeadm-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get rolebinding kubeadm:kubelet-config -n kube-system  -o yaml | $PAGER

echo "---"
kubectl get clusterrolebinding kubeadm:get-nodes -o yaml | $PAGER
echo "---"
kubectl get clusterrolebinding kubeadm:kubelet-bootstrap -o yaml | $PAGER
echo "---"
kubectl get clusterrolebinding kubeadm:node-autoapprove-bootstrap -o yaml | $PAGER

echo "---"
kubectl get role kube-proxy -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:kubeadm-certs  -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:nodes-kubeadm-config -n kube-system -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:kubelet-config -n kube-system  -o yaml | $PAGER

echo "---"
kubectl get clusterrole kubeadm:get-nodes -o yaml | $PAGER
echo "---"
kubectl get clusterrole system:node-bootstrapper -o yaml | $PAGER
echo "---"
kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient -o yaml | $PAGER
) > $file

导出结果

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-proxy
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resourceNames:
  - kube-proxy
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubeadm:kubeadm-certs
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resourceNames:
  - kubeadm-certs
  resources:
  - secrets
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubeadm:nodes-kubeadm-config
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resourceNames:
  - kubeadm-config
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubeadm:kubelet-config
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resourceNames:
  - kubelet-config
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeadm:get-nodes
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node-bootstrapper
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - create
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/nodeclient
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-proxy
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-proxy
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubeadm:kubeadm-certs
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:kubeadm-certs
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubeadm:nodes-kubeadm-config
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:nodes-kubeadm-config
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubeadm:kubelet-config
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:kubelet-config
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:get-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeadm:get-nodes
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

# Kubernetes <1.24
JWT_TOKEN_DEFAULT_DEFAULT=$(kubectl get secrets \
    $(kubectl get serviceaccounts/default -o jsonpath='{.secrets[0].name}') \
    -o jsonpath='{.data.token}' | base64 --decode)

# Kubernetes 1.24+
JWT_TOKEN_DEFAULT_DEFAULT=$(kubectl create token default)

export KUBE_API="https://192.168.122.200:8443"
curl $KUBE_API/apis/apps/v1/namespaces/default/deployments \
--cacert /etc/kubernetes/pki/ca.crt --header "Authorization: Bearer $JWT_TOKEN_DEFAULT_DEFAULT"

shell we need system:anonymous binding cluster-admin?

查询下谁(rolebindings或clusterrolebindings)绑定了system:anonymous这个User?

kubectl get rolebindings -A -o json | jq -r '.items[] | select(.subjects[0].kind=="User")' |grep -C20 system:anonymous

可以查询到在kube-public空间有个叫kubeadm:bootstrap-signer-clusterinfo的rolebinding

kubectl get role,rolebindings -n kube-public
NAME                                                                  CREATED AT
role.rbac.authorization.k8s.io/kubeadm:bootstrap-signer-clusterinfo   2022-12-30T01:46:36Z
role.rbac.authorization.k8s.io/system:controller:bootstrap-signer     2022-12-30T01:46:33Z

NAME                                                                         ROLE                                        AGE
rolebinding.rbac.authorization.k8s.io/kubeadm:bootstrap-signer-clusterinfo   Role/kubeadm:bootstrap-signer-clusterinfo   21d
rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer     Role/system:controller:bootstrap-signer     21d

利用脚本将这些roles和rolebindings导出

PAGER="cat"
command -v "kubectl-neat" &>/dev/null && PAGER="kubectl neat"

file=/dev/stdout

(
echo "---"
kubectl get rolebinding kubeadm:bootstrap-signer-clusterinfo -n kube-public -o yaml | $PAGER
echo "---"
kubectl get rolebinding system:controller:bootstrap-signer  -n kube-public -o yaml | $PAGER
echo "---"
kubectl get role kubeadm:bootstrap-signer-clusterinfo -n kube-public -o yaml | $PAGER
echo "---"
kubectl get role system:controller:bootstrap-signer -n kube-public  -o yaml | $PAGER
)>$file

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubeadm:bootstrap-signer-clusterinfo
  namespace: kube-public
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:bootstrap-signer-clusterinfo
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:controller:bootstrap-signer
  namespace: kube-public
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: system:controller:bootstrap-signer
subjects:
- kind: ServiceAccount
  name: bootstrap-signer
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubeadm:bootstrap-signer-clusterinfo
  namespace: kube-public
rules:
- apiGroups:
  - ""
  resourceNames:
  - cluster-info
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:controller:bootstrap-signer
  namespace: kube-public
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resourceNames:
  - cluster-info
  resources:
  - configmaps
  verbs:
  - update
- apiGroups:
  - ""
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
  - update

经过分析，kubeadm将system:anonymous用户赋予了kubeadm:bootstrap-signer-clusterinfo角色，
kubeadm安装的kubelet启动的token的auth-extra-groups必须为system:bootstrappers:kubeadm:default-node-token，如果kubelet bootstrap-token-secret中的auth-extra-groups不是system:bootstrappers:kubeadm:default-node-token则会报错

kubelet[11247]: E0118 15:40:29.064946 11247 kubelet_node_status.go:92] "Unable to register node with API server" err="nodes is forbidden: User \"system:anonymous\" cannot create resource \"nodes\" in API group \"\" at the cluster scope"

有一个解决办法是将system:anonymous用户授予最高权限则可通过，但此方法不是安全的。

以下脚本来自网络

kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous
# kubectl delete clusterrolebinding system:anonymous
kubectl auth can-i --list --as=system:anonymous -n default

Resources   Non-Resource URLs   Resource Names   Verbs
*.*         []                  []               [*]
            [*]                 []               [*]
            [/healthz]          []               [get]
            [/livez]            []               [get]
            [/readyz]           []               [get]
            [/version/]         []               [get]
            [/version]          []               [get]

ref

• How to view members of subject with Group kind
• create-a-bootstrap-token
• How To Call Kubernetes API using Simple HTTP Client