_

前一篇《搭建Kubernetes集群》里说到kubekey+podman。那是因为新版本docker 抽风影响默认路由。后来重装了老版本docker desktop后解决了,我装的是v4.23.0。 于是像试试docker 自带的kubernetes

安装k8s

2024-01-25T142537 进入Settings > Kubernetes ,钩上然后点Apply & restart
2024-01-25T142713 看到这个变绿就装好了,对,就是那么简单

安装dashboard

1
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

2024-01-25T142949 装好后长这样

生token

admin-user.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
# apply 后执行: kubectl -n kubernetes-dashboard create token admin-user
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
---
# kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={".data.token"} | base64 -d
apiVersion: v1
kind: Secret
metadata:
name: admin-user
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin-user"
type: kubernetes.io/service-account-token

apply上面这个manifests 用这个命令获取登录token kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={".data.token"} | base64 -d

访问dashboard

kubectl patch svc kubernetes-dashboard -n kubernetes-dashboard -p '{"spec": {"type": "NodePort"}}'

这里要提到docker k8s的好处,它不用ingress 或者LoadBalancer 只要绑个节点端口,本地就可以访问了。

2024-01-25T144518

https://localhost:30324 这就是dashboard的访问地址了。

2024-01-25T144612 输入刚才获得的token 就可以登录了。

新版Docker desktop会起一个def 路由,导致宿主机不能上网 》 上面这篇提到了新版本的问题,于是用了podman,实际上podman真不怎么好用。于是研究了一下。

新版本 2.6.X 的问题是docker起来后会新添加一个默认路由,把流量路由到bridge100这个设备上导致上网问题。 在网上找到一个issue Strange network routes after upgrading to 4.24.x 里面提到了和我一样的问题,但是人家是linux。他的解决方案也比较简单

1
Downgrading again to version 4.23.0 completely solves that issue - routes are gone.

于是我试了一下,装了docker desktop 4.23 确实有效。

实验环境

我在公司内网的PVE上安装了3个VM,k[1:3] 虚拟机的os 是ubuntu 22.04.3 装了docker-ce,ubuntu 上的containerd 是docker 提供的。

1
2
3
4
5
# 记得把swap关了
swapoff -a

# 把自己(ubuntu)加到docker 组里,这样调用docker的时候就不用sudo了。newgrp docker是切组,也可以退出当前会话再登录。
sudo usermod -aG docker $USER && newgrp docker

kubeadm

这个东西真的蛮复杂的,按照官方的教程我确实跑起了一个,但是因为我起的时候没有安装网络组件(cni)搞了2天各种问题,所以放弃了。下面是步骤:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#containerd
# 预先要添加docker的apt repo https://docs.docker.com/engine/install/ubuntu/
sudo apt install -y containerd.io

# 这里装了containerd.io后还需要把 /etc/containerd/config.toml 配置里的下面这行注释掉:
# disabled_plugins = ["cri"]
# 然后重启服务

# 装kube三兄弟
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt install kubeadm kubelet kubectl
# 锁定包版本,解锁的时候用unhold
sudo apt-mark hold kubelet kubeadm kubectl


# 起集群
sudo kubeadm init --pod-network-cidr=10.244.0.0/18 --service-cidr=10.244.64.0/18

# 得到下面的输出就是ok了

# Your Kubernetes control-plane has initialized successfully!
# To start using your cluster, you need to run the following as a regular user:
# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# Alternatively, if you are the root user, you can run:
# export KUBECONFIG=/etc/kubernetes/admin.conf
# You should now deploy a pod network to the cluster.
# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
# https://kubernetes.io/docs/concepts/cluster-administration/addons/
# Then you can join any number of worker nodes by running the following on each as root:
# kubeadm join 172.17.0.220:6443 --token 22tkpm.0eetw95la7mtv6qm \
# --discovery-token-ca-cert-hash sha256:****************************************************************

# 添加节点就按上面输出的最后一行 kubeadm join来

祝君好运~

另寻出路

发现自己手挫不成后就找找别的工具。看了一下kops和rancher都是基于云的,我的目标是本地裸机不符合 于是我看了最早的kubesphere的一个工具kubekey。

kubekey

这货也是坑,不过比kubeadm好多了。 kubekey可以只用来部署k8s 不安装kubesphere,参照这个文章 How to Install Kubernetes the Easy Way Using KubeKey 确实是可以跑起一个集群的。

小坑

k8s版本问题,我贴一下我的config,这里的版本是我从1.29.0一点一点试下去的!因为用的registry是kubesphere团队自己维护的并不是registry.k8s.io所以有坑。倒是可以试试把registy切换到google,但是我没有精力折腾了,有缘人如果有心可以给我留言。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: node1, address: 172.17.0.220, internalAddress: 172.17.0.220, user: ubuntu, privateKeyPath: "~/.ssh/id_rsa"}
- {name: node2, address: 172.17.0.221, internalAddress: 172.17.0.221, user: ubuntu, privateKeyPath: "~/.ssh/id_rsa"}
- {name: node3, address: 172.17.0.222, internalAddress: 172.17.0.222, user: ubuntu, privateKeyPath: "~/.ssh/id_rsa"}
roleGroups:
etcd:
- node1
control-plane:
- node1
worker:
- node1
- node2
- node3
controlPlaneEndpoint:
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.22.17
clusterName: cluster.local
autoRenewCerts: true
etcd:
type: kubekey
network:
plugin: calico
kubePodsCIDR: 10.244.64.0/18
kubeServiceCIDR: 10.244.0.0/18
multusCNI:
enabled: false
registry:
privateRegistry: ""
namespaceOverride: ""
registryMirrors: []
insecureRegistries: []
addons: []

额外的

我是在我本机执行的kk create cluster 。其实完全可以在节点里找一台做主控来开集群。但是kk 本来就是个远程的playbook,他核心是ansible 只要有ssh权限就ok。在集群起来后需要把集群的管理配置拉到本地和本机的配置合并一下然。

1
2
3
4
# k1 是control-plane,看我上面的配置
scp k1:~/.kube/config ./my-cluster
KUBECONFIG=~/.kube/config:./my-cluster kubectl config view --flatten > new-kubeconfig
mv new-kubeconfig ~/.kube/config

列出所有可控集群

1
2
3
4
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
kind-kind-cluster kind-kind-cluster kind-kind-cluster
* [email protected] cluster.local kubernetes-admin

切换上下文

1
$ kubectl config use-context [email protected]

因为要测试一个grpc的服务,我需要压力测试。协作单位有专门的压测部门我就直接用他们的技术栈了。

准备

  • 用pip装locust
  • 用pip 装 grpc-interceptor

少量编程

新建个python项目

grpc_user.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import time
from typing import Any, Callable
import grpc
import grpc.experimental.gevent as grpc_gevent
from grpc_interceptor import ClientInterceptor
from locust import User
from locust.exception import LocustError

# patch grpc so that it uses gevent instead of asyncio
grpc_gevent.init_gevent()


class LocustInterceptor(ClientInterceptor):
def __init__(self, environment, *args, **kwargs):
super().__init__(*args, **kwargs)

self.env = environment

def intercept(
self,
method: Callable,
request_or_iterator: Any,
call_details: grpc.ClientCallDetails,
):
response = None
exception = None
start_perf_counter = time.perf_counter()
response_length = 0
try:
response = method(request_or_iterator, call_details)
response_length = response.result().ByteSize()
except grpc.RpcError as e:
exception = e

self.env.events.request.fire(
request_type="grpc",
name=call_details.method,
response_time=(time.perf_counter() - start_perf_counter) * 1000,
response_length=response_length,
response=response,
context=None,
exception=exception,
)
return response


class GrpcUser(User):
abstract = True
stub_class = None

def __init__(self, environment):
super().__init__(environment)
for attr_value, attr_name in ((self.host, "host"), (self.stub_class, "stub_class")):
if attr_value is None:
raise LocustError(f"You must specify the {attr_name}.")

self._channel = grpc.insecure_channel(self.host)
interceptor = LocustInterceptor(environment=environment)
self._channel = grpc.intercept_channel(self._channel, interceptor)

self.stub = self.stub_class(self._channel)

locustfile.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from locust import task, constant

import AccountService_pb2
import AccountService_pb2_grpc
import grpc_user


class GrpcUser(grpc_user.GrpcUser):
wait_time = constant(0)
# wait_time = constant_throughput(2)
host = '172.18.14.130:50051'
stub_class = AccountService_pb2_grpc.AccountStub

@task
def updateCharacter(self):
input_data = AccountService_pb2.characterRequest(user_id='123', server='test.tooqing.com:6100',
character_id='123', nickname='seven', avatar='1',
level='99')
response = self.stub.updateCharacter(input_data)
return response

运行

python -m locust

在webui上设定用户数和创建频率,就可以上路了。 2023-11-04T105021

从剪切板粘贴文本到vscode 的terminal里会贴2次。

原因是我在 .zprofile里用了set -a 解决方案也很简单,在文件最后加 set +a 就好了。

本地环境是Mac

用了tungstenite-rs这个crate来做websocket支持,过程中发现发包时候长度和我写入的预期长度不符合。

文档里看到这么一段 原文

pub const fn with_fixed_int_encoding(self) -> Configuration<E, Fixint, A, L>

Fixed-size integer encoding.

  • Fixed size integers are encoded directly
  • Enum discriminants are encoded as u32
  • Lengths and usize are encoded as u64

于是在代码中,构造config 的地方加上后,问题得到解决

1
2
let config = config::standard()
.with_fixed_int_encoding();

今天老婆开着沪C的小车绕了大半个上海回娘家了。路上有点坎坷,特别是最近还一直下雨,辛苦了。

很久没有搭理这个blog了,现在工作上不是特别忙我打算年后把blog继续写起来。今天整理了一下blog相关的一些东西github pages ~ hexo ~ cloudflare 解析等等,很多。看到这篇文章的时候应该是已经都搞定了,yep这是个测试用的post =)

年前在研究Rust-lang,我会在这里记录一些技术零星。

2022新年快乐!

0%