Installing k0rdent Observability and FinOps#

Options#

KOF may be installed using different options:

flowchart TD
    S((Start))
    S --> AWS[AWS]
    S --> AZ[Azure]
    S --> OS[OpenStack<br>and others]
    AWS -.-> AG[**Aig Gap**]
    AZ -.-> AG
    OS --> AG
    AWS --> NAG[No Air Gap]
    AZ --> NAG
    OS --> NAG
    AG --> I[**Istio**]
    AG --> MD[Manual DNS]
    AG -.-> AD[Auto DNS]
    NAG --> I
    NAG --> MD
    NAG --> AD
    I --> R[KOF Regional:<br><br>own ClusterDeployment<br>or shared with KCM Region<br>or with Management<br>or just ConfigMap]
    MD --> R
    AD --> R
    R --> M2[Store KOF data<br>from Management...]
    M2 --> M2M[**...to Management**]
    M2 --> M2R[...to Regional<br>with Istio<br>or without]
    M2 --> M2TP[...to Third-party]

Opinionated default installation we plan to simplify in the next release is shown in bold style.

Prerequisites#

Before beginning KOF installation, you should have the following components in place:

A k0rdent management cluster - You can get instructions to create one in the quickstart guide.
You will also need your infrastructure provider credentials, such as those shown in the guide for AWS.
- Note that you should skip the "Create your ClusterDeployment" and later sections.
Finally, select one of the options:
- DNS auto-config to automate the process for all regional clusters.
- Manual DNS config is applied later for each regional cluster separately and manually.
- Istio service mesh for secure connectivity between clusters. This is the only option which does not need access to create external DNS records for service endpoints such as kof.example.com.

For for information on long-term storage planning, review the KOF Retention page, which covers retention and replication strategies.

DNS auto-config#

To avoid manual configuration of DNS records for service endpoints later, you can automate the process now using external-dns. Examples by cloud provider:

AWS

For AWS in production, use the Node IAM Role or IRSA methods in production.

Just for the sake of this demo based on the aws-standalone template, however, you can use the most straightforward (though less secure) static credentials method:

Create an external-dns IAM user with this policy.

Create an access key and external-dns-aws-credentials file, as in:

[default]
aws_access_key_id = <EXAMPLE_ACCESS_KEY_ID>
aws_secret_access_key = <EXAMPLE_SECRET_ACCESS_KEY>

Create the external-dns-aws-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-aws-credentials \
  --from-file external-dns-aws-credentials

Azure

To enable DNS auto-config on Azure, use DNS Zone Contributor.

Create an Azure service principal with the DNS Zone Contributor permissions. You can find an example here.

Create the azure.json text file containing the service principal configuration data:

{
  "tenantId": "<EXAMPLE_TENANT_ID>",
  "subscriptionId": "<EXAMPLE_SUBSCRIPTION_ID>",
  "resourceGroup": "<EXAMPLE_RESOURCE_GROUP>",
  "aadClientId": "<EXAMPLE_SP_APP_ID>",
  "aadClientSecret": "<EXAMPLE_SP_PASSWORD>"
}

Create the external-dns-azure-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-azure-credentials \
  --from-file azure.json

See external-dns Azure documentation for more details.

OpenStack and others

Please check external-dns tutorials and new providers to find instructions on how to create a secret.

For example the external-dns-openstack-credentials could be created by applying the external-dns-openstack-webhook docs.

Istio#

If you've selected to skip both DNS auto-config now and Manual DNS config later, you can apply these steps to enable the Istio service mesh:

Istio

Check the overview in the k0rdent/istio repository and this video:

Create and label the kof namespace to allow Istio to inject its sidecars:

kubectl create namespace kof
kubectl label namespace kof istio-injection=enabled

Install the k0rdent/istio charts to the management cluster:

helm upgrade -i --reset-values --wait --create-namespace -n istio-system k0rdent-istio \
  oci://ghcr.io/k0rdent/istio/charts/k0rdent-istio \
  --version 0.4.0 \
  --set cert-manager-service-template.enabled=false \
  --set "istiod.meshConfig.extensionProviders[0].name=otel-tracing" \
  --set "istiod.meshConfig.extensionProviders[0].opentelemetry.port=4317" \
  --set "istiod.meshConfig.extensionProviders[0].opentelemetry.service=kof-collectors-daemon-collector.kof.svc.cluster.local" \
  --set-json 'gateway.resource.spec.servers[0]={"port":{"number":15443,"name":"tls","protocol":"TLS"},"tls":{"mode":"AUTO_PASSTHROUGH"},"hosts":["{clusterName}-vmauth.kof.svc.cluster.local"]}'

Please use helm v3 until we fix k0rdent/istio issue #40 leading to UPGRADE FAILED: conflict occurred... when using helm v4.

Grafana#

Grafana installation and automatic configuration are now disabled in KOF by default.
Review the Using KOF and Grafana in KOF guides to make a decision.

Management Cluster#

To install KOF on the management cluster, look through the default values of the kof chart, and apply this example, or use it as a reference:

Create an empty kof-values.yaml file.
Check requirements to the storage class.

If you want to use a default storage class, but kubectl get sc shows no (default), create it. Otherwise you can use a non-default storage class in the kof-values.yaml file:
Non-default storage class
```
kof-mothership:
  values:
    global:
      storageClass: <EXAMPLE_STORAGE_CLASS>
```
If you plan to enable kof-storage component (for local/single-cluster deployments), also set:
```
kof-storage:
  values:
    global:
      storageClass: <EXAMPLE_STORAGE_CLASS>
```
If kubectl get sc shows nothing or just kubernetes.io/no-provisioner in the PROVISIONER column, apply OpenEBS or similar.

If you've applied the DNS auto-config section, add the information about external-dns credentials to the kof-values.yaml file. Examples by cloud provider:

AWS

kof-mothership:
  values:
    kcm:
      kof:
        mcs:
          kof-aws-dns-secrets:
            matchLabels:
              k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
            secrets:
              - external-dns-aws-credentials

Azure

kof-mothership:
  values:
    kcm:
      kof:
        mcs:
          kof-azure-dns-secrets:
            matchLabels:
              k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
            secrets:
              - external-dns-azure-credentials

OpenStack

kof-mothership:
  values:
    kcm:
      kof:
        mcs:
          kof-openstack-dns-secrets:
            matchLabels:
              k0rdent.mirantis.com/kof-openstack-dns-secrets: "true"
            secrets:
              - external-dns-openstack-credentials

This enables auto-distribution of the DNS secret to regional clusters.

Examples of ClusterDeployments in Regional Cluster and Child Cluster sections are both using namespace: kcm-system for a child cluster to connect to a regional cluster easily.

If you plan to have multiple tenants/namespaces in the management cluster, please note they are isolated by default: a child cluster in namespace: tenantA will be able to connect to a regional cluster in namespace: tenantA only.

If you want to allow a child cluster in one namespace to connect to a regional cluster in another namespace, enable the crossNamespace value in the kof-values.yaml file:
crossNamespace
```
kof-mothership:
  values:
    kcm:
      kof:
        operator:
          crossNamespace: true
```
Check the high-level default values of the kof umbrella chart, and more detailed default values of other charts, to customize your kof-values.yaml file where needed, for example:
If your regional clusters already have cert-manager and ingress-nginx, disable them in KOF:
```
kof-regional:
  values:
    cert-manager:
      enabled: false
    ingress-nginx:
      enabled: false
```
To customize storage for all regional clusters at once now:
```
kof-regional:
  values:
    storage:
      # ...
```
For example, if all your regional clusters will use OpenStack, you may set the OpenStack-specific values described in step 10 of the Regional Cluster section under the storage: key here.
To customize collectors for all child clusters at once now:
```
kof-child:
  values:
    collectors:
      collectors:
        # ...
```
Note: the first collectors key is to reference the subchart, and the second collectors key is part of its values.

Check the examples of OpenTelemetry collectors custom configuration.
If you're upgrading KOF from an earlier version, check the Upgrading KOF guide.
Install the KOF umbrella chart, which orchestrates the installation of operators, mothership, and shared config for all regional and child clusters:
```
helm upgrade -i --reset-values --wait \
  --create-namespace -n kof kof \
  -f kof-values.yaml \
  oci://ghcr.io/k0rdent/kof/charts/kof \
  --version 1.8.0
```
The chart uses FluxCD to manage sequential deployment of all KOF components.

If helm v4 failed to call webhook, apply the workaround from k0rdent/kof issue #715.

Wait for all HelmReleases to be ready:

kubectl wait --for=condition=Ready helmreleases --all -n kof --timeout=10m
kubectl get helmreleases -n kof

Wait for all pods to show that they're Running:
```
kubectl get pod -n kof
```
Check options to store metrics of the management cluster in the Storing KOF data guide.

Regional Cluster#

Note

To use the new KCM Regional Clusters with KOF, please apply the KCM Region With KOF guide on top of the steps below.

To install KOF on the regional cluster, apply this example for AWS, or use it as a reference:

Set your KOF variables using your own values:

REGION=us-east-2
REGIONAL_CLUSTER_NAME=aws-$REGION
ADMIN_EMAIL=$(git config user.email)
echo "$REGION, $REGIONAL_CLUSTER_NAME, $ADMIN_EMAIL"

If you have not applied the Istio section, set the REGIONAL_DOMAIN too, using your own domain:

REGIONAL_DOMAIN=$REGIONAL_CLUSTER_NAME.kof.example.com
echo "$REGIONAL_DOMAIN"

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-1-0-22

Compose the regional ClusterDeployment. Examples by cloud provider:

AWS

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.xlarge
EOF

Azure

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

OpenStack

FLAVOR=m1.large
IMAGE=ubuntu-24.04-x86_64

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: openstack-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    identityRef:
      cloudName: openstack
      region: $REGION
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    controlPlaneNumber: 1
    controlPlane:
      flavor: $FLAVOR
      image:
        filter:
          name: $IMAGE
    workersNumber: 1
    worker:
      flavor: $FLAVOR
      image:
        filter:
          name: $IMAGE
EOF

If you've applied the DNS auto-config section, add it to the .metadata.labels in the regional-cluster.yaml file. Examples by cloud provider:
AWS
```
k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
```
Azure
```
k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
```
OpenStack
```
k0rdent.mirantis.com/kof-openstack-dns-secrets: "true"
```
See also "OpenStack extra requirement for DNS auto-config" on step 10.

If you've applied the Istio section, update the regional-cluster.yaml file:

Istio

Add these lines:

k0rdent.mirantis.com/istio-role: member
k0rdent.mirantis.com/istio-gateway: "true"

Delete these lines:

k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL

This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. If you add them, please copy .metadata.labels as well.
The aws-standalone-cp template provides the default storage class ebs-csi-default-sc for AWS. The k0rdent quickstart guide provides the default storage class managed-csi for Azure. If you want to use a non-default storage class, add it to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:
```
k0rdent.mirantis.com/kof-storage-class: <EXAMPLE_STORAGE_CLASS>
```

The kof-operator creates and configures PromxyServerGroup and GrafanaDatasource (if enabled) automatically. It uses the endpoints listed below by default. If you want to disable the built-in metrics, logs, and traces to use your own existing instances instead, add custom endpoints to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:

Endpoints

k0rdent.mirantis.com/kof-write-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/insert/0/prometheus/api/v1/write
k0rdent.mirantis.com/kof-read-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/select/0/prometheus
k0rdent.mirantis.com/kof-write-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vli/insert/opentelemetry/v1/logs
k0rdent.mirantis.com/kof-read-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vls
k0rdent.mirantis.com/kof-write-traces-endpoint: https://vmauth.$REGIONAL_DOMAIN/vti/insert/opentelemetry/v1/traces
k0rdent.mirantis.com/kof-read-traces-endpoint: https://vmauth.$REGIONAL_DOMAIN/vts/select/jaeger

If you've applied the Istio section, default endpoints are:

k0rdent.mirantis.com/kof-write-metrics-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vm/insert/0/prometheus/api/v1/write
k0rdent.mirantis.com/kof-read-metrics-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vm/select/0/prometheus
k0rdent.mirantis.com/kof-write-logs-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vli/insert/opentelemetry/v1/logs
k0rdent.mirantis.com/kof-read-logs-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vls
k0rdent.mirantis.com/kof-write-traces-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vti/insert/opentelemetry/v1/traces
k0rdent.mirantis.com/kof-read-traces-endpoint: http://$REGIONAL_CLUSTER_NAME-vmauth:8427/vts/select/jaeger

If you want to skip creation of the regional cluster completely, but still let child clusters discover your custom endpoints, you may create a regional ConfigMap instead of a regional ClusterDeployment:

Regional ConfigMap

cat >regional-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: regional
data:
  regional_cluster_name: $REGIONAL_CLUSTER_NAME
  regional_cluster_namespace: kcm-system
  regional_cluster_cloud: aws
  aws_region: $REGION
  azure_location: ""
  openstack_region: ""
  vsphere_datacenter: ""
  istio_role: ""
  kof_http_config: ""
  write_metrics_endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/insert/0/prometheus/api/v1/write
  read_metrics_endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/select/0/prometheus
  write_logs_endpoint: https://vmauth.$REGIONAL_DOMAIN/vli/insert/opentelemetry/v1/logs
  read_logs_endpoint: https://vmauth.$REGIONAL_DOMAIN/vls
  write_traces_endpoint: https://vmauth.$REGIONAL_DOMAIN/vti/insert/opentelemetry/v1/traces
EOF

cat regional-configmap.yaml
kubectl apply -f regional-configmap.yaml

Note that a similar ConfigMap is generated automatically from a regional ClusterDeployment too.

If you need a custom http client configuration for PromxyServerGroup and GrafanaDatasource (if enabled), add it to the regional-cluster.yaml file in the .metadata.annotations:
Custom http client configuration
```
k0rdent.mirantis.com/kof-http-config: '{"dial_timeout": "10s", "tls_config": {"insecure_skip_verify": true}}'
```

MultiClusterService named kof-regional-cluster configure and install cert-manager, ingress-nginx, kof-operators, kof-storage, and kof-collectors charts automatically.

To pass any custom values to the kof-storage chart or its subcharts, such as victoria-logs-cluster, add them to the regional-cluster.yaml file in the .spec.config.clusterAnnotations. Examples:

VictoriaLogs replicaCount

k0rdent.mirantis.com/kof-storage-values: |
  victoria-logs-cluster:
    vlinsert:
      replicaCount: 2

OpenStack extra requirement for DNS auto-config

k0rdent.mirantis.com/kof-storage-values: |
  external-dns:
    provider:
      name: webhook
      webhook:
        image:
          repository: ghcr.io/inovex/external-dns-openstack-webhook
          tag: 1.1.0
        extraVolumeMounts:
          - name: oscloudsyaml
            mountPath: /etc/openstack/
    extraVolumeMounts: null
    extraVolumes:
      - name: oscloudsyaml
        secret:
          secretName: external-dns-openstack-credentials

Verify and apply the Regional ClusterDeployment:

cat regional-cluster.yaml

kubectl apply -f regional-cluster.yaml

Wait for the ClusterDeployment to become Ready:

kubectl wait --for=condition=Ready --timeout=20m \
  cld -n kcm-system $REGIONAL_CLUSTER_NAME

Child Cluster#

Note

To use the new KCM Regional Clusters with KOF, please apply the KCM Region With KOF guide on top of the steps below.

To install KOF on the actual cluster to be monitored, apply this example for AWS, or use it as a reference:

Ensure you still have the variables set in the Regional Cluster section, and set your own value for the CHILD_CLUSTER_NAME variable:
```
CHILD_CLUSTER_NAME=$REGIONAL_CLUSTER_NAME-child1
echo "$CHILD_CLUSTER_NAME"
```

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-1-0-22

Compose the child ClusterDeployment. Examples by cloud provider:

AWS

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.medium
EOF

Azure

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

OpenStack

FLAVOR=m1.large
IMAGE=ubuntu-24.04-x86_64

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: openstack-cluster-identity-cred
  cleanupOnDeletion: true
  config:
    identityRef:
      cloudName: openstack
      region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      flavor: $FLAVOR
      image:
        filter:
          name: $IMAGE
    workersNumber: 1
    worker:
      flavor: $FLAVOR
      image:
        filter:
          name: $IMAGE
EOF

If you've applied the Istio section, update the child-cluster.yaml file:
Istio

Add this line:
```
k0rdent.mirantis.com/istio-role: member
```
There is no need to create a gateway in the child cluster using the k0rdent.mirantis.com/istio-gateway: "true" label. The child cluster uses the gateway in the regional cluster to store KOF data.
This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. If you add them, please copy .metadata.labels as well.
The kof-operator discovers the regional cluster by the location of the child cluster. If you have more than one regional cluster in the same AWS region / Azure location / etc., and you want to connect the child cluster to specific regional cluster, add this regional cluster name to the child-cluster.yaml file in the .metadata.labels:
```
k0rdent.mirantis.com/kof-regional-cluster-name: $REGIONAL_CLUSTER_NAME
```
If you've enabled the crossNamespace value, specify the namespace too, for example:
```
k0rdent.mirantis.com/kof-regional-cluster-namespace: kcm-system
```
MultiClusterService named kof-child-cluster configures and installs cert-manager, kof-operators, and kof-collectors charts automatically.

To pass any custom values to the kof-collectors chart or its subcharts, such as opencost, add them to the child-cluster.yaml file in the .spec.config. Example:
OpenCost replicas
```
clusterAnnotations:
  k0rdent.mirantis.com/kof-collectors-values: |
    opencost:
      opencost:
        exporter:
          replicas: 2
```
Note: the first opencost key is to reference the subchart, and the second opencost key is part of its values.

Verify and apply the ClusterDeployment:

cat child-cluster.yaml

kubectl apply -f child-cluster.yaml

Wait for the ClusterDeployment to become Ready:

kubectl wait --for=condition=Ready --timeout=20m \
  cld -n kcm-system $CHILD_CLUSTER_NAME

Now that you have installed KOF, please verify it.