k8s之纵向扩缩容Vpa

k8s之纵向扩缩容Vpa

调优 kubernetes
· 约 5 分钟

vpa

纵向容器自动缩放器（VPA）使用户无需设置最新的资源限制和对容器中容器的要求。配置后，它将根据使用情况自动设置请求，从而允许在节点上进行适当的调度，以便为每个Pod提供适当的资源量。它还将保持限制和初始容器配置中指定的请求之间的比率。

它既可以根据资源的使用情况来缩减对资源过度使用的Pod，也可以对资源需求不足的向上扩展Pod。

Kubernetes VPA 包含以下组件：

  • Recommender：用于根据监控指标结合内置机制给出资源建议值
  • Updater：用于实时更新 pod resource requests
  • History Storage：用于采集和存储监控数据
  • Admission Controller: 用于在 pod 创建时修改 resource requests

架构图

VPA体系结构图

主要流程是：Recommender在启动时从History Storage获取历史数据，根据内置机制修改VPA API object资源建议值。Updater监听VPA API object，依据建议值动态修改 pod resource requests。VPA Admission Controller则是用于 pod 创建时修改 pod resource requests。History Storage则是通过Kubernetes Metrics API采集和存储监控数据。

CPU 和内存的建议值均是依据历史数据+固定机制计算而成。

在 Kubernetes VPA 中缺少资源回收的机制，但Recommender却可以配合Updater动态修改 pod resource requests 的值。也就是说 pod resource requests - 推荐值 = 资源回收值。这间接实现了资源回收的功能。

VPA工作流程

空值

  1. VPA连续检查您在设置过程中配置的指标值，默认间隔为10秒
  2. 如果达到阈值，VPA会尝试更改分配的内存和/或CPU
  3. VPA主要更新部署或复制控制器规范中的资源
  4. 重新启动Pod后，新资源将全部应用于创建的实例。

使用VPA时需要考虑以下几点：

  • 如果不重新启动Pod，将无法更改资源。到目前为止，主要的合理性在于，这样的变化可能会导致很多不稳定。因此，考虑重新启动Pod并根据新分配的资源对其进行调度。
  • VPA和HPA尚不兼容，不能在同一pod上使用。如果要在同一群集中同时使用它们，请确保在设置中分开它们的作用域。
  • VPA仅根据观察到的过去和当前资源使用情况来调整容器的资源请求。它没有设置资源限制。对于行为不当的应用程序可能会出现问题，这些应用程序开始使用越来越多的资源，导致pod被Kubernetes杀死。

安装

当前默认版本是Vertical Pod Autoscaler 0.8.0

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler
./hack/vpa-up.sh

注意：该脚本当前读取环境变量：$REGISTRY和$TAG。除非要使用VPA的非默认版本，否则请确保未设置它们。

如果出现如下报错，我们可以升级openssl到支持-addext选项的版本，或在下载via-release-0.8这个分支的代码并使用./hack/vpa-up.sh

ERROR: Failed to create CA certificate for self-signing. If the error is "unknown option -addext", update your openssl version or deploy VPA from the vpa-release-0.8 branch.
deployment.apps/vpa-admission-controller created

安装好之后会创建三个pod，这三个pod使用的镜像需要翻墙下载

[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system  get pod 
NAME                                                READY   STATUS    RESTARTS   AGE
calico-kube-controllers-688c5dc8c7-rcn2t            1/1     Running   0          59d
calico-node-n7sj8                                   1/1     Running   7          89d
calico-node-vk9c9                                   1/1     Running   1          89d
coredns-6955765f44-mgcc4                            1/1     Running   0          59d
coredns-6955765f44-vwzxt                            1/1     Running   0          59d
csi-disk-provisioner-0                              2/2     Running   0          17d
csi-diskplugin-jmw74                                2/2     Running   0          17d
csi-diskplugin-mv99m                                2/2     Running   0          17d
dex-1591193803-6844b4846c-4wwzg                     1/1     Running   1          19d
dex-k8s-authenticator-1591193977-54f66bd94f-tv2rl   1/1     Running   1          59d
etcd-master01                                       1/1     Running   8          89d
kube-apiserver-master01                             1/1     Running   0          19d
kube-controller-manager-master01                    1/1     Running   0          13d
kube-proxy-s98fh                                    1/1     Running   0          59d
kube-proxy-zn2l6                                    1/1     Running   0          59d
kube-scheduler-master01                             1/1     Running   0          13d
kube-sealyun-lvscare-node01                         1/1     Running   0          89d
kuboard-85d499bd9b-xmvtp                            1/1     Running   0          15d
metrics-server-6b4d999dcb-xzx4p                     1/1     Running   0          13d
snapshot-controller-0                               1/1     Running   0          14d
traefik-0                                           1/1     Running   0          20d
vpa-admission-controller-6b88678dcb-d5vb6           1/1     Running   0          7s
vpa-recommender-5fc9b4c4cb-d5j8l                    1/1     Running   0          32s
vpa-updater-865849df48-rcl94                        1/1     Running   0          40s
[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system  get deployments.apps  -o wide
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                IMAGES     
....
vpa-admission-controller           1/1     1            1           62m   admission-controller      us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-admission-controller:0.8.0   app=vpa-admission-controller
vpa-recommender                    1/1     1            1           62m   recommender               us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-recommender:0.8.0            app=vpa-recommender
vpa-updater                        1/1     1            1           62m   updater                   us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-updater:0.8.0                app=vpa-updater

这里可以使用 misterli/vpa-recommender:0.8.0,misterli/vpa-updater:0.8.0,misterli/vpa-admission-controller:0.8.0这三个镜像进行替换

使用

  • VPA 有以下四种更新策略：

    •Initial：仅在 Pod 创建时修改资源请求，以后都不再修改。

    •Auto：默认策略，在 Pod 创建时修改资源请求，并且在 Pod 更新时也会修改。

    •Recreate：类似 Auto，在 Pod 的创建和更新时都会修改资源请求，不同的是，只要Pod 中的请求值与新的推荐值不同，VPA 都会驱逐该 Pod，然后使用新的推荐值重新启一个。因此，一般不使用该策略，而是使用 Auto，除非你真的需要保证请求值是最新的推荐值。

    •Off：不改变 Pod 的资源请求，不过仍然会在 VPA 中设置资源的推荐值。

我们部署一个使用vpa的示例：

---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
  name: hamster-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: hamster
  updatePolicy:
    updateMode: "Auto"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hamster
spec:
  selector:
    matchLabels:
      app: hamster
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:
      containers:
        - name: hamster
          image: ubuntu:16.04
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

我们发现稍等片刻vpa会对pod的资源进行动态更新

[root@master01 vertical-pod-autoscaler]# kubectl apply -f test.yaml 
verticalpodautoscaler.autoscaling.k8s.io/hamster-vpa created
deployment.apps/hamster created
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w 
NAME                              READY   STATUS    RESTARTS   AGE
event-exporter-56b4dbc8fb-gsmx4   1/1     Running   0          40d
hamster-bb5bb8754-krljt           1/1     Running   0          21s
hamster-bb5bb8754-nvhf2           1/1     Running   0          21s
hamster-bb5bb8754-krljt           1/1     Terminating   0          92s
hamster-bb5bb8754-qs8dl           0/1     Pending       0          0s
hamster-bb5bb8754-qs8dl           0/1     Pending       0          0s
hamster-bb5bb8754-qs8dl           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-qs8dl           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-qs8dl           1/1     Running             0          1s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m3s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m4s
hamster-bb5bb8754-krljt           0/1     Terminating         0          2m4s
hamster-bb5bb8754-nvhf2           1/1     Terminating         0          2m32s
hamster-bb5bb8754-d2rnh           0/1     Pending             0          0s
hamster-bb5bb8754-d2rnh           0/1     Pending             0          0s
hamster-bb5bb8754-d2rnh           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-d2rnh           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-d2rnh           1/1     Running             0          2s

查看一下vpa的详情可以看到vpa建议的值

[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name:         hamster-vpa
Namespace:    default
.....
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     579m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     4554m
        Memory:  262144k
Events:          <none>

查看当前pod的资源限制，pod的requests已经被动态调整了

[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-kp9jk  -o yaml 
apiVersion: v1
kind: Pod
....
    name: hamster
    resources:
      requests:
        cpu: 587m
        memory: 262144k
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-jqvcc
      readOnly: true
.....

还可以限制动态调整的范围

---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
  name: hamster-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: hamster
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 1
          memory: 500Mi
        controlledResources: ["cpu", "memory"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hamster
spec:
  selector:
    matchLabels:
      app: hamster
  replicas: 2
  template:
    metadata:
      labels:
        app: hamster
    spec:
      containers:
        - name: hamster
          image: ubuntu:16.04
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
          command: ["/bin/sh"]
          args:
            - "-c"
            - "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

上面的yaml创建了一个包含2个Pod的deployment，每个Pod运行一个请求100m cpu 的容器，并尝试使用略高于500m cpu的容器。该命令还会创建一个指向deployment的VPA配置。VPA将观察Pod的行为，大约5分钟后，它们应使用更高的CPU请求进行更新（请注意，VPA不会在deployment中修改模板，但Pod的实际请求会被更新）。要查看VPA配置和当前推荐的资源请求:

[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name:         hamster-vpa
Namespace:    default
...
Status:
  Conditions:
    Last Transition Time:  2020-09-01T03:55:47Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     552m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     1
        Memory:  370459537
Events:          <none>

查看pod 详情，发现此时 resources字段已经被更改

[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-gcrbc  -o yaml 
apiVersion: v1
kind: Pod
   ....
    image: ubuntu:16.04
    imagePullPolicy: IfNotPresent
    name: hamster
    resources:
      requests:
        cpu: 587m
        memory: 262144k
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    ...
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w 
NAME                              READY   STATUS    RESTARTS   AGE
event-exporter-56b4dbc8fb-gsmx4   1/1     Running   0          40d
hamster-bb5bb8754-ghggr           1/1     Running   0          5s
hamster-bb5bb8754-xnndg           1/1     Running   0          5s
hamster-bb5bb8754-xnndg           1/1     Terminating   0          62s
hamster-bb5bb8754-gcrbc           0/1     Pending       0          0s
hamster-bb5bb8754-gcrbc           0/1     Pending       0          0s
hamster-bb5bb8754-gcrbc           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-gcrbc           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-gcrbc           1/1     Running             0          1s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          93s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          94s
hamster-bb5bb8754-xnndg           0/1     Terminating         0          94s
hamster-bb5bb8754-ghggr           1/1     Terminating         0          2m2s
hamster-bb5bb8754-gwxgw           0/1     Pending             0          0s
hamster-bb5bb8754-gwxgw           0/1     Pending             0          0s
hamster-bb5bb8754-gwxgw           0/1     ContainerCreating   0          0s
hamster-bb5bb8754-gwxgw           0/1     ContainerCreating   0          1s
hamster-bb5bb8754-gwxgw           1/1     Running             0          2s

event事件

0s          Normal    EvictedByVPA        pod/hamster-bb5bb8754-xnndg     Pod was evicted by VPA Updater to apply resource recommendation.
0s          Normal    SuccessfulCreate    replicaset/hamster-bb5bb8754    Created pod: hamster-bb5bb8754-gcrbc
0s          Normal    Scheduled           pod/hamster-bb5bb8754-gcrbc     Successfully assigned default/hamster-bb5bb8754-gcrbc to node01
0s          Normal    Killing             pod/hamster-bb5bb8754-xnndg     Stopping container hamster
0s          Normal    Pulled              pod/hamster-bb5bb8754-gcrbc     Container image "ubuntu:16.04" already present on machine
0s          Normal    Created             pod/hamster-bb5bb8754-gcrbc     Created container hamster
0s          Normal    Started             pod/hamster-bb5bb8754-gcrbc     Started container hamster
0s          Normal    EvictedByVPA        pod/hamster-bb5bb8754-ghggr     Pod was evicted by VPA Updater to apply resource recommendation.
0s          Normal    Killing             pod/hamster-bb5bb8754-ghggr     Stopping container hamster
0s          Normal    SuccessfulCreate    replicaset/hamster-bb5bb8754    Created pod: hamster-bb5bb8754-gwxgw
0s          Normal    Scheduled           pod/hamster-bb5bb8754-gwxgw     Successfully assigned default/hamster-bb5bb8754-gwxgw to node01
0s          Normal    Pulled              pod/hamster-bb5bb8754-gwxgw     Container image "ubuntu:16.04" already present on machine
0s          Normal    Created             pod/hamster-bb5bb8754-gwxgw     Created container hamster
0s          Normal    Started             pod/hamster-bb5bb8754-gwxgw     Started container hamster