Skip to content

Cloud provider credentials propagation#

Some components in the cluster deployment require cloud provider credentials to be passed for proper functioning. For example, Cloud Controller Manager (CCM) requires provider credentials to create load balancers and provide other functionality.

This poses a challenge for credentials delivery. Currently cloud-init is used to pass all necessary credentials, but this approach has several problems:

  • Credentials are stored unencrypted in the instance metadata.
  • Rotation of the credentials is impossible without complete instance redeployment.
  • Possible leaks, since credentials are copied to several Secret objects related to bootstrap data.

To solve these problems in k0rdent we're using the Sveltos controller, which can render the CCM template with all necessary data from the CAPI provider resources (like ClusterIdentity) and can create secrets directly on the cluster deployment.

Note

CCM template examples can be found in *-credentials.yaml here. Look for the ConfigMap object that has the projectsveltos.io/template: "true" annotation and *-resource-template as the object name.

This eliminates the need to pass anything credentials-related to cloud-init and makes it possible to rotate credentials automatically without the need for instance redeployment.

Also, this automation makes it possible to separate roles and responsibilities so that only the lead engineer has access to credentials, and other engineers can use them without seeing values and even any access to underlying infrastructure platform.

The process is fully automated and credentials will be propagated automatically within the ClusterDeployment reconciliation process; user only needs to provide the correct Credential object.

Provider-specific notes#

Since this feature depends on the provider, it's important to review any provider-specific notes and clarifications.

Note

More detailed research notes can be found here.

AWS#

Since AWS uses roles, which are assigned to instances, no additional credentials will be created.

The AWS provider supports 3 types of ClusterIdentity and, which one to use depends on your specific use case. More information regarding CAPA ClusterIdentity resources can be found in the CRD Reference.

Azure#

Currently the Cluster API Azure (CAPZ) provider creates azure.json Secret objects in the same namespace as the Cluster object. By design they should be referenced in the cloud-init YAML later during bootstrap process.

In k0rdent these Secret objects aren't used and will not be added to the cloud-init, but engineers can access them without restrictions, which is a security issue.

OpenStack#

For OpenStack, CAPO relies on a clouds.yaml file. In k0rdent, you provide this file in a Kubernetes Secret that references OpenStack credentials (ideally application credentials for enhanced security). During reconciliation, KCM automatically generates the cloud-config required by OpenStack’s cloud-controller-manager.

For more details, refer to the KCM OpenStack Credential Propagation doc.

Adopted clusters#

Credentials for adopted clusters consist of a secret containing a kubeconfig file to access the existing kubernetes cluster. The kubeconfig file for the cluster should be contained in the value key of the secret object. The following is an example of a secret that contains the kubeconfig for an adopted cluster. To create this secret, first create or obtain a kubeconfig file for the cluster that is being adopted and then run the following command to base64 encode it:

cat kubeconfig | base64 -w 0

Once you have obtained a base64 encoded kubeconfig file create a secret:

apiVersion: v1
data:
  value: <base64 encoded kubeconfig file>
kind: Secret
metadata:
  name: adopted-cluster-kubeconf
  namespace: <namespace>
type: Opaque

The Credential Distribution System#

k0rdent provides a mechanism to distribute Credential objects across namespaces using the AccessManagement object. This object defines a set of accessRules that determine how credentials are distributed.

Each access rule specifies:

  1. The target namespaces where credentials should be delivered.
  2. A list of Credential names to distribute to those namespaces.

The KCM controller copies the specified Credential objects from the system (defaults to kcm-system) namespace to the target namespaces based on the accessRules in the AccessManagement spec.

Note

Starting from v1.5.0 KCM controller also copies all the cluster identity resources and all the referenced identity objects across namespaces. Follow Cluster Identity Distribution System for details.

Info

Access rules can also include Cluster and Service Template Chains (ClusterTemplateChain objects and ServiceTemplateChain objects) to distribute templates to target namespaces. For more details, read: Template Life Cycle Management.

How to Configure Credential Distribution#

To configure the distribution of Credential objects:

  1. Edit the AccessManagement object.
  2. Populate the .spec.accessRules field with the list of Credential names and the target namespaces.

Here’s an example configuration:

spec:
  accessRules:
  - targetNamespaces:
      list:
        - dev
        - test
    credentials:
      - aws-demo
      - azure-demo

In this example, the aws-demo and azure-demo Credential objects will be distributed to the dev and test namespaces.

Regional Credential Distribution#

To deploy clusters correctly, the ClusterIdentity object (for example, AWSClusterStaticIdentity) and all of its referenced resources (such as Secrets) must exist in the same cluster where the CAPI objects are created. When a cluster is deployed in a specific region, all required ClusterIdentity objects must be present in the corresponding regional cluster.

Starting from v1.5.0, when creating regional Credentials (with spec.region), all ClusterIdentity objects and their referenced resources are automatically synchronized with the regional cluster.

For example, when you create the following Credential object:

apiVersion: k0rdent.mirantis.com/v1beta1
kind: Credential
metadata:
  name: aws-cluster-credential
  namespace: test
spec:
  region: region1
  description: "Credential Example"
  identityRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSClusterStaticIdentity
    name: aws-cluster-identity

and AWSClusterStaticIdentity references the aws-cluster-identity-secret secret:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSClusterStaticIdentity
metadata:
  name: aws-cluster-identity
spec:
  secretRef: aws-cluster-identity-secret
  allowedNamespaces: {}

The KCM controller will do the following:

  1. Copy the aws-cluster-identity-secret from the system namespace of the management cluster to the system namespace of the regional cluster.
  2. Copy the aws-cluster-identity AWSClusterStaticIdentity object from the management cluster to the regional cluster.

Warning

For Cluster Identity distribution to function correctly, the corresponding ProviderInterface must be properly configured. Providers included with k0rdent already have a preconfigured ProviderInterface as part of the ProviderTemplate. If you are using a custom or Bring-Your-Own provider, you must properly configure the ProviderInterface to enable Cluster Identity distribution. For detailed instructions, refer to the Cluster Identity Distribution System documentation.