Installing k0rdent Observability and FinOps#

Prerequisites#

Before beginning KOF installation, you should have the following components in place:

A k0rdent management cluster - You can get instructions to create one in the quickstart guide.
- To test on macOS you can install using: brew install kind && kind create cluster -n k0rdent
You will also need your infrastructure provider credentials, such as those shown in the guide for AWS.
- Note that you should skip the "Create your ClusterDeployment" and later sections.
Finally, select one of the options:
- DNS auto-config to automate the process for all regional clusters.
- Manual DNS config is applied later for each regional cluster separately and manually.
- Istio service mesh for secure connectivity between clusters. This is the only option which does not need the access to create external DNS records for service endpoints such as kof.example.com.

DNS auto-config#

To avoid manual configuration of DNS records for service endpoints later, you can automate the process now using external-dns.

AWS#

For AWS, use the Node IAM Role or IRSA methods in production.

For now, however, just for the sake of this demo based on the aws-standalone template, you can use the most straightforward (though less secure) static credentials method:

Create an external-dns IAM user with this policy.

Create an access key and external-dns-aws-credentials file, as in:

[default]
aws_access_key_id = <EXAMPLE_ACCESS_KEY_ID>
aws_secret_access_key = <EXAMPLE_SECRET_ACCESS_KEY>

Create the external-dns-aws-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-aws-credentials \
  --from-file external-dns-aws-credentials

Azure#

To enable DNS auto-config on Azure, use DNS Zone Contributor.

Create an Azure service principal with the DNS Zone Contributor permissions. You can find an example here.

Create the azure.json text file containing the service principal configuration data:

{
  "tenantId": "<EXAMPLE_TENANT_ID>",
  "subscriptionId": "<EXAMPLE_SUBSCRIPTION_ID>",
  "resourceGroup": "<EXAMPLE_RESOURCE_GROUP>",
  "aadClientId": "<EXAMPLE_SP_APP_ID>",
  "aadClientSecret": "<EXAMPLE_SP_PASSWORD>"
}

Create the external-dns-azure-credentials secret in the kof namespace:

kubectl create namespace kof
kubectl create secret generic \
  -n kof external-dns-azure-credentials \
  --from-file azure.json

See external-dns Azure documentation for more details.

Istio#

If you've selected to skip both DNS auto-config now and Manual DNS config later, apply the steps below to enable Istio service mesh:

Check the overview in the kof/docs/istio.md and this video:

Create and label the kof namespace to allow Istio to inject its sidecars:

kubectl create namespace kof
kubectl label namespace kof istio-injection=enabled

Install the kof-istio chart to the management cluster:
```
helm upgrade -i --reset-values --wait \
  --create-namespace -n istio-system kof-istio \
  oci://ghcr.io/k0rdent/kof/charts/kof-istio --version 0.3.0
```
You may want to customize collectors by passing values to kof-istio-child profile for all child clusters at once now, or for each child cluster later, or just use the default values.

Management Cluster#

To install KOF on the management cluster, look through the default values of the kof-mothership and kof-operators charts, and apply this example, or use it as a reference:

Install kof-operators required by kof-mothership:

helm upgrade -i --reset-values --wait \
  --create-namespace -n kof kof-operators \
  oci://ghcr.io/k0rdent/kof/charts/kof-operators --version 0.3.0

Create the mothership-values.yaml file:
```
kcm:
  installTemplates: true
```
This enables installation of ServiceTemplates such as cert-manager and kof-storage, to make it possible to reference them from the regional and child MultiClusterServices.
If you want to use a default storage class, but kubectl get sc shows no (default), create it. Otherwise you can use a non-default storage class in the mothership-values.yaml file:
```
global:
  storageClass: <EXAMPLE_STORAGE_CLASS>
```

If you've applied the DNS auto-config section, add to the kcm: object in the mothership-values.yaml file.

For AWS, add:

  kof:
    clusterProfiles:
      kof-aws-dns-secrets:
        matchLabels:
          k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
        secrets:
          - external-dns-aws-credentials

For Azure, add:

  kof:
    clusterProfiles:
      kof-azure-dns-secrets:
        matchLabels:
          k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
        secrets:
          - external-dns-azure-credentials

This enables Sveltos to auto-distribute DNS secret to regional clusters.

Two secrets are auto-created by default:
- storage-vmuser-credentials is a secret used by VictoriaMetrics. You don't need to use it directly. It is auto-distributed to other clusters by clusterProfiles here.
- grafana-admin-credentials is a secret that we will use in the Grafana section. It is auto-created here.

Install kof-mothership:

helm upgrade -i --reset-values --wait -n kof kof-mothership \
  -f mothership-values.yaml \
  oci://ghcr.io/k0rdent/kof/charts/kof-mothership --version 0.3.0

Wait until the value of VALID changes to true for all ServiceTemplates:
```
kubectl get svctmpl -A
```
If you have not applied the Istio section:
- Look through the MultiClusterServices in the kof-regional and kof-child charts.
- If your regional clusters already have ingress-nginx and cert-manager, you can ask kof-regional chart to not install them by setting Helm values ingress-nginx.enabled and cert-manager.enabled to false below.
- You may want to customize collectors for all child clusters at once now, or for each child cluster later, or just use the default values.
- Install these charts into the management cluster with default or custom values:
```
helm upgrade -i --reset-values --wait -n kof kof-regional \
  oci://ghcr.io/k0rdent/kof/charts/kof-regional --version 0.3.0

helm upgrade -i --reset-values --wait -n kof kof-child \
  oci://ghcr.io/k0rdent/kof/charts/kof-child --version 0.3.0
```
Wait for all pods to show that they're Running:
```
kubectl get pod -n kof
```

Regional Cluster#

To install KOF on the regional cluster, look through the default values of the kof-storage chart, and apply this example for AWS, or use it as a reference:

Set your KOF variables using your own values:

REGION=us-east-2
REGIONAL_CLUSTER_NAME=aws-$REGION
REGIONAL_DOMAIN=$REGIONAL_CLUSTER_NAME.kof.example.com
ADMIN_EMAIL=$(git config user.email)
echo "$REGIONAL_CLUSTER_NAME, $REGIONAL_DOMAIN, $ADMIN_EMAIL"

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-0-3-0

Compose the regional ClusterDeployment:

For AWS:

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1alpha1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.xlarge
EOF

For Azure:

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >regional-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1alpha1
kind: ClusterDeployment
metadata:
  name: $REGIONAL_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: regional
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    clusterAnnotations:
      k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
      k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

If you've applied the DNS auto-config section, add to the .metadata.labels in the regional-cluster.yaml file.

For AWS, add:
```
k0rdent.mirantis.com/kof-aws-dns-secrets: "true"
```
For Azure, add:
```
k0rdent.mirantis.com/kof-azure-dns-secrets: "true"
```
If you've applied the Istio section, update the regional-cluster.yaml file:
- Replace this line:
```
k0rdent.mirantis.com/kof-storage-secrets: "true"
```
  with this line:
```
k0rdent.mirantis.com/istio-role: child
```
  Note that child is not a typo and kof- prefix is not missing in istio-role. This makes your config compatible with possible migration of Istio support from KOF to the upstream k0rdent, where child is a generic term described in the Cluster Deployments concept.
- Delete these lines:
```
k0rdent.mirantis.com/kof-regional-domain: $REGIONAL_DOMAIN
k0rdent.mirantis.com/kof-cert-email: $ADMIN_EMAIL
```
This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. Only if you add them, please copy .metadata.labels there too.
The aws-standalone-cp template provides the default storage class ebs-csi-default-sc for AWS. The k0rdent quickstart guide provides the default storage class managed-csi for Azure. If you want to use a non-default storage class, add it to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:
```
k0rdent.mirantis.com/kof-storage-class: <EXAMPLE_STORAGE_CLASS>
```

The kof-operator creates and configures PromxyServerGroup and GrafanaDatasource automatically. It uses the endpoints listed below by default. Only if you want to disable the built-in metrics, logs, and traces to use your own existing instances instead, add custom endpoints to the regional-cluster.yaml file in the .spec.config.clusterAnnotations:

k0rdent.mirantis.com/kof-write-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/insert/0/prometheus/api/v1/write
k0rdent.mirantis.com/kof-read-metrics-endpoint: https://vmauth.$REGIONAL_DOMAIN/vm/select/0/prometheus
k0rdent.mirantis.com/kof-write-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vls/insert/opentelemetry/v1/logs
k0rdent.mirantis.com/kof-read-logs-endpoint: https://vmauth.$REGIONAL_DOMAIN/vls
k0rdent.mirantis.com/kof-write-traces-endpoint: https://jaeger.$REGIONAL_DOMAIN/collector

The MultiClusterService named kof-regional-cluster configures and installs cert-manager, ingress-nginx, kof-storage, kof-operators, and kof-collectors charts automatically. To pass any custom values to the kof-storage chart or its subcharts like the victoria-logs-single, add them to the regional-cluster.yaml file in the .spec.config.clusterAnnotations, for example:
```
k0rdent.mirantis.com/kof-storage-values: |
  victoria-logs-single:
    server:
      replicaCount: 2
```

Verify and apply the Regional ClusterDeployment:

cat regional-cluster.yaml

kubectl apply -f regional-cluster.yaml

Watch how the cluster is deployed to AWS until all values of READY are True:

clusterctl describe cluster -n kcm-system $REGIONAL_CLUSTER_NAME \
  --show-conditions all

Child Cluster#

To install KOF on the actual cluster to be monitored, look through the default values of the kof-operators and kof-collectors charts, and apply this example for AWS, or use it as a reference:

Set your own value below, verifing the variables:

CHILD_CLUSTER_NAME=$REGIONAL_CLUSTER_NAME-child1
echo "$CHILD_CLUSTER_NAME, $REGIONAL_DOMAIN"

Use the up-to-date ClusterTemplate, as in:

kubectl get clustertemplate -n kcm-system | grep aws
TEMPLATE=aws-standalone-cp-0-3-0

Compose the child ClusterDeployment:

For AWS:

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1alpha1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: aws-cluster-identity-cred
  config:
    clusterIdentity:
      name: aws-cluster-identity
      namespace: kcm-system
    region: $REGION
    controlPlaneNumber: 1
    controlPlane:
      instanceType: t3.large
    workersNumber: 3
    worker:
      instanceType: t3.medium
EOF

For Azure:

AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
echo "AZURE_SUBSCRIPTION_ID=$AZURE_SUBSCRIPTION_ID"

cat >child-cluster.yaml <<EOF
apiVersion: k0rdent.mirantis.com/v1alpha1
kind: ClusterDeployment
metadata:
  name: $CHILD_CLUSTER_NAME
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/kof-storage-secrets: "true"
    k0rdent.mirantis.com/kof-cluster-role: child
spec:
  template: $TEMPLATE
  credential: azure-cluster-identity-cred
  config:
    clusterIdentity:
      name: azure-cluster-identity
      namespace: kcm-system
    subscriptionID: $AZURE_SUBSCRIPTION_ID
    location: $REGION
    controlPlaneNumber: 1
    controlPlane:
      vmSize: Standard_A4_v2
    workersNumber: 3
    worker:
      vmSize: Standard_A4_v2
EOF

If you've applied the Istio section, update the child-cluster.yaml file:

replace this line:
```
k0rdent.mirantis.com/kof-storage-secrets: "true"
```
with this line:
```
k0rdent.mirantis.com/istio-role: child
```
This ClusterDeployment uses propagation of its .metadata.labels to the resulting Cluster because there are no .spec.config.clusterLabels here. Only if you add them, please copy .metadata.labels there too.
The kof-operator discovers the regional cluster by the location of the child cluster. Only if you have more than one regional cluster in the same AWS region / Azure location / etc, and you want to connect the child cluster to specific regional cluster, add this regional cluster name to the child-cluster.yaml file in the .metadata.labels:
```
k0rdent.mirantis.com/kof-regional-cluster-name: $REGIONAL_CLUSTER_NAME
```
The MultiClusterService named kof-child-cluster configures and installs cert-manager, kof-operators, and kof-collectors charts automatically. To pass any custom values to the kof-collectors chart or its subcharts like the opencost, add them to the child-cluster.yaml file in the .spec.config, for example:
```
clusterAnnotations:
  k0rdent.mirantis.com/kof-collectors-values: |
    opencost:
      opencost:
        exporter:
          replicas: 2
```
Note: the first opencost key is to reference the subchart, and the second opencost key is part of its values.

Verify and apply the ClusterDeployment:

cat child-cluster.yaml

kubectl apply -f child-cluster.yaml

Watch while the cluster is deployed to AWS until all values of READY are True:

clusterctl describe cluster -n kcm-system $CHILD_CLUSTER_NAME \
  --show-conditions all