Skip to content

Scaling Guidelines#

The method for scaling KOF depends on the type of expansion:

Increase volumes size#

Don't use this method if your storage class doesn't have allowVolumeExpansion.

With allowVolumeExpansion you can increase the storage size for:

vmcluster 
kof-regional: # or kof-mothership if you have it installed on mothership cluster
  values:
    victoriametrics:
      vmcluster:
        spec:
          vmstorage:
            storage:
              volumeClaimTemplate:
                spec:
                  resources:
                    requests:
                      storage: <new size>
victoria-logs-cluster 
victoria-logs-cluster:
  vlstorage:
    persistentVolume:
      size: <new size>

victoria-traces-cluster:
  vtstorage:
    persistentVolume:
      size: <new size>

For vmcluster you can just run helm upgrade with new values, VictoriaMetrics operator will handle volume expansion automatically.

helm upgrade -i --reset-values --wait \
  --create-namespace -n kof kof \
  -f kof-values.yaml \
  oci://ghcr.io/k0rdent/kof/charts/kof \
  --version 1.8.0

For victoria-logs-cluster and victoria-traces-cluster you need to delete the stateful set and expand PVCs manually before running helm upgrade:

victoria-logs-cluster volumes expansion

  1. If you have installed the kof-storage chart on the mothership cluster via kof-umbrella helm chart, suspend the flux helmrelease firstly 

    kubectl patch helmrelease kof-storage \
-n kof \
--type=merge \
-p '{"spec":{"suspend": true}}'

  2. Delete the current stateful set: 

    # --cascade=orphan keeps the Pods and PVCs alive while removing the StatefulSet object
kubectl delete statefulset kof-storage-victoria-logs-cluster-vlstorage \
  -n kof --cascade=orphan

  3. Expand the size for each PVC from 0 to the number of replicas 

    kubectl patch pvc vlstorage-volume-kof-storage-victoria-logs-cluster-vlstorage-0 \
-n kof \
--type=merge \
-p '{"spec":{"resources":{"requests":{"storage":"<new size>"}}}}'

  4. Verify that PVCs are expanded 

    kubectl get pvc -n kof -w

  5. Run helm upgrade with values set to the new size

Regional Expansion#

  1. Deploy a regional cluster in the new region.
  2. Configure child clusters in this region to point to this regional cluster.

Adding a New Child Cluster#

  1. Apply templates, as in the child cluster section.
  2. Verify the data flow.
  3. Configure any custom dashboards.

You Must Construct Additional Pylons#

  1. Change the replicaCount of components like victoria-logs-cluster as documented in the regional cluster section.
  2. Change the replicas number of components like opencost as documented in the child cluster section.