Enable Audit Logging on a Management Cluster

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

Audit policy defines rules about which events are recorded and what data they include. For child clusters, k0rdent provides a consistent mechanism for managing cluster audit policy through the ClusterAuditPolicy resource. For details, see Enabling Audit Logging.

For management clusters, audit policy is configured by modifying the cluster's API server configuration. This configuration must be applied before starting the control plane on each control plane machine.

For example, when using k0s as the Kubernetes distribution, you can modify the API server configuration by changing the spec.api.extraArgs field in the ClusterConfig resource.

Prerequisites:

1. Audit policy file exists on control plane machines.
2. (Optional) If audit logs will be stored on control plane machines, the log directory must exist with appropriate permissions.
3. k0s configuration file exists on control plane machines and is configured to use the audit policy file and optionally the log directory.

Example Configuration

1. Create an audit policy file on control plane machines. For example, create /etc/k0s/audit-policy.yaml with the following minimal content:

   # Log all requests at the Metadata level.
   apiVersion: audit.k8s.io/v1
   kind: Policy
   rules:
   - level: Metadata
   

2. Create a k0s configuration file.

   mkdir -p /etc/k0s
   k0s config create > /etc/k0s/k0s.yaml
   

3. Edit the k0s configuration file and add the audit-policy-* parameters to spec.api.extraArgs:

   apiVersion: k0s.k0sproject.io/v1beta1
   kind: ClusterConfig
   metadata:
     name: k0s
     namespace: kube-system
   spec:
     api:
       extraArgs:
         audit-policy-file: "/etc/k0s/audit-policy.yaml"
         audit-log-path: "/var/log/k0s/audit.log" # if set to "-", logs will be written to standard output
         audit-log-maxage: "7"
         audit-log-maxbackup: "5"
         audit-log-maxsize: "100"
   

For details about the k0s configuration, see the k0s documentation.

1. Ensure the audit logs directory exists with appropriate permissions:

   mkdir -p /var/log/k0s
   sudo chown kube-apiserver:kube-apiserver /var/log/k0s
   

   Warning

   If your k0s configuration has non-default users, the audit logs directory must be owned by the kubeAPIserverUser (default: kube-apiserver).

2. Proceed with installing and starting the k0s control plane.

Audit Policy for Management Clusters

The audit policy below is a baseline policy you can use as-is for k0rdent management clusters. It audits core Kubernetes API requests, including requests from k0rdent and its dependent components. It also excludes select high-volume, low-risk requests to reduce noise and performance overhead. You can customize this policy by adding or modifying rules as needed.

For details about the policy fields, see the Kubernetes Policy configuration reference.

Baseline Audit policy for the Management cluster

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services", "services/status"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes", "nodes/status"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:cloud-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces", "namespaces/status", "namespaces/finalize"]
  # Don't log leader election lease updates (high-volume from all controllers).
  - level: None
    verbs: ["get", "update"]
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]
  # Don't log HPA fetching metrics.
  - level: None
    users:
      - system:kube-controller-manager
      - system:cloud-controller-manager
    verbs: ["get", "list"]
    resources:
      - group: "metrics.k8s.io"
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Don't log events requests because of performance impact.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"]
    verbs: ["update","patch"]
    resources:
      - group: "" # core
        resources: ["nodes/status", "pods/status"]
    omitStages:
      - "RequestReceived"
  - level: Request
    userGroups: ["system:nodes"]
    verbs: ["update","patch"]
    resources:
      - group: "" # core
        resources: ["nodes/status", "pods/status"]
    omitStages:
      - "RequestReceived"
  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    users: ["system:serviceaccount:kube-system:namespace-controller"]
    verbs: ["deletecollection"]
    omitStages:
      - "RequestReceived"
  # Secrets, ConfigMaps, TokenRequest and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
    omitStages:
      - "RequestReceived"
  # Get responses can be large; skip them.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "coordination.k8s.io"
      - group: "discovery.k8s.io"
      - group: "extensions"
      - group: "flowcontrol.apiserver.k8s.io"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "node.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "scheduling.k8s.io"
      - group: "storage.k8s.io"
      # k0s
      - group: "k0s.k0sproject.io"
      - group: "helm.k0sproject.io"
      - group: "autopilot.k0sproject.io"
      # CAPI
      - group: "cluster.x-k8s.io"
      - group: "infrastructure.cluster.x-k8s.io"
      - group: "controlplane.cluster.x-k8s.io"
      - group: "bootstrap.cluster.x-k8s.io"
      - group: "addons.cluster.x-k8s.io"
      - group: "ipam.cluster.x-k8s.io"
      # Flux
      - group: "source.toolkit.fluxcd.io"
      - group: "helm.toolkit.fluxcd.io"
      - group: "kustomize.toolkit.fluxcd.io"
      # Sveltos
      - group: "lib.projectsveltos.io"
      - group: "config.projectsveltos.io"
      # k0rdent
      - group: "k0rdent.mirantis.com"
    omitStages:
      - "RequestReceived"
  # Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "coordination.k8s.io"
      - group: "discovery.k8s.io"
      - group: "extensions"
      - group: "flowcontrol.apiserver.k8s.io"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "node.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "scheduling.k8s.io"
      - group: "storage.k8s.io"
      # k0s
      - group: "k0s.k0sproject.io"
      - group: "helm.k0sproject.io"
      - group: "autopilot.k0sproject.io"
      # CAPI
      - group: "cluster.x-k8s.io"
      - group: "infrastructure.cluster.x-k8s.io"
      - group: "controlplane.cluster.x-k8s.io"
      - group: "bootstrap.cluster.x-k8s.io"
      - group: "addons.cluster.x-k8s.io"
      - group: "ipam.cluster.x-k8s.io"
      # Flux
      - group: "source.toolkit.fluxcd.io"
      - group: "helm.toolkit.fluxcd.io"
      - group: "kustomize.toolkit.fluxcd.io"
      # Sveltos
      - group: "lib.projectsveltos.io"
      - group: "config.projectsveltos.io"
      # k0rdent
      - group: "k0rdent.mirantis.com"
    omitStages:
      - "RequestReceived"
  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - "RequestReceived"

For details about Kubernetes auditing, see Kubernetes Auditing.