Architecture

High-level

From a high-level perspective, KOF consists of three layers:

• the Collection layer, where the statistics and events are gathered,
• the Regional layer, which includes storage to keep track of those statistics and events,
• and the Management layer, where you interact through the UI.

flowchart TD
    A[Management UI, promxy] 
    A --> C[Storage Region 1]
    A --> D[Storage Region 2]
    C --> E[Collect Child 1]
    C --> F[Collect Child 2]
    D ==> G[...]

Mid-level

Getting a little bit more detailed, it's important to undrestand that data flows upwards, from observed objects to centralized Grafana on the Management layer:

Management Cluster

kof-mothership chart

grafana-operator

victoria-metrics-operator

cluster-api-visualizer

sveltos-dashboard

kof-operator

k0rdent service templates

promxy

kof-operators chart

opentelemetry-operator

prometheus-operator-crds

kof-collectors chart

opencost

kube-state-metrics

prometheus-node-exporter

Either kof-istio

Certificates

ClusterProfiles

Or kof-regional and kof-child

MultiClusterServices

Cloud 1..N

Region 1..M

Regional Cluster

kof-storage chart

grafana-operator

victoria-metrics-operator

victoria-logs-single

jaeger-operator

external-dns

kof-operators chart

opentelemetry-operator

prometheus-operator-crds

kof-collectors chart

opencost

kube-state-metrics

prometheus-node-exporter

cert-manager

ingress-nginx or kof-istio

Child Cluster 1

cert-manager

Optional kof-istio

kof-operators chart

opentelemetry-operator

prometheus-operator-crds

kof-collectors chart

opencost

kube-state-metrics

prometheus-node-exporter

observed objects

Low-level

At a low level, you can see how metrics, logs, and traces work their way around the system.

Helm Charts

KOF is deployed as a series of Helm charts at various levels.

kof-mothership

• Centralized Grafana dashboard, managed by grafana-operator
• Local VictoriaMetrics storage for alerting rules only, managed by victoria-metrics-operator
• cluster-api-visualizer for insight into multicluster configuration
• Sveltos dashboard, automatic secret distribution
• kof-operator (don't confuse it with the kof-operators chart) for auto-configuration
• k0rdent service templates used by kof-regional and kof-child charts
• Promxy for aggregating Prometheus metrics from regional clusters

kof-regional

• MultiClusterService which configures and installs kof-storage and other charts to regional clusters

kof-child

• MultiClusterService which configures and installs kof-collectors and other charts to child clusters

kof-storage

• Regional Grafana dashboard, managed by grafana-operator
• Regional VictoriaMetrics storage with main data, managed by victoria-metrics-operator
• vmauth entrypoint proxy for VictoriaMetrics components
• vmcluster for high-available fault-tolerant version of VictoriaMetrics database
• victoria-logs-single for high-performance, cost-effective, scalable logs storage
• Regional Jaeger tracing platform, managed by jaeger-operator
• external-dns to communicate with other clusters

kof-istio

• Optional Istio support for secure connectivity between clusters without external DNS

kof-operators

• prometheus-operator-crds required to create OpenTelemetry collectors, also required to monitor kof-mothership itself
• OpenTelemetry collectors below, managed by opentelemetry-operator

kof-collectors

• prometheus-node-exporter for hardware and OS metrics
• kube-state-metrics for metrics about the state of Kubernetes objects
• OpenCost "shines a light into the black box of Kubernetes spend"