Skip to main content

Embedded Cluster Config (Beta)

This topic is a reference for the Replicated Embedded Cluster Config custom resource. For more information about Embedded Cluster, see Use Embedded Cluster.

Limitations​

  • Most of the fields in the Config spec are plain YAML and do not support Go template functions, including Replicated template functions. The values field for each entry in extensions.helmCharts does support the use of template functions.

  • There are additional, property-specific limitations. For more information, see the following sections.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
metadata:
  name: my-embedded-cluster-config
spec:
  version: "3.0.0-beta.1+k8s-1.34"
  domains:
    proxyRegistryDomain: proxy.example.com
    replicatedAppDomain: updates.example.com
  unsupportedOverrides:
    k0s: |
      config:
        metadata:
          name: foo
        spec:
          telemetry:
            enabled: false
          workerProfiles:
            - name: ip-forward
              values:
                allowedUnsafeSysctls:
                  - net.ipv4.ip_forward
  extensions:
    helmCharts:
      - chart:
          name: ingress-nginx
          chartVersion: "4.11.3"
        releaseName: ingress-nginx
        namespace: ingress-nginx
        values: |
          controller:
            service:
              type: NodePort
              nodePorts:
                http: "80"
                https: "443"
            image:
              repository: 'repl{{ ReplicatedImageName "registry.k8s.io/ingress-nginx/controller" }}'
              tag: 'repl{{ HelmValue ".Values.controller.image.tag" }}'
              digest: ""
              digestChroot: ""
            admissionWebhooks:
              patch:
                image:
                  repository: 'repl{{ ReplicatedImageName "registry.k8s.io/ingress-nginx/kube-webhook-certgen" }}'
                  tag: 'repl{{ HelmValue ".Values.controller.admissionWebhooks.patch.image.tag" }}'
                  digest: ""
      - chart:
          name: goldpinger
          chartVersion: "6.1.2"
        releaseName: goldpinger
        namespace: goldpinger
        weight: 11
        values: |
          image:
            repository: 'repl{{ ReplicatedImageName "docker.io/bloomberg/goldpinger" }}'

version (Required)​

Specify the versions of Embedded Cluster and Kubernetes to install.

Embedded Cluster appends the version of Kubernetes to the version label in the format +k8s-1.34. For example, Embedded Cluster 3.0.0-alpha-26+k8s-1.34 uses Embedded Cluster 3.0.0-alpha-26 and Kubernetes 1.34.

Each version of Embedded Cluster includes specific versions of components like OpenEBS. For more information, see the Embedded Cluster Release Notes.

Replicated recommends that you update the version frequently to ensure that you are using the latest version of Embedded Cluster.

important

Don't downgrade the Embedded Cluster version or the Kubernetes version. Also, only increase the Kubernetes version by one minor version at a time. For more information, see Limitations in Embedded Cluster Overview.

roles (Beta)​

You can customize node roles in the Embedded Cluster Config using the roles key.

A common use case for customizing node roles is to assign workloads to specific nodes. For example, if your application has graphics processing unit (GPU) workloads, you could declare a custom role that applies a gpu=true label to any node assigned that role. You can then schedule GPU workloads on nodes labeled gpu=true.

When the roles key is configured, users select one or more roles to assign to a node when it is installed or joined to the cluster. For more information about assigning roles when joining a node, see Join nodes in Access and manage embedded clusters.

If the roles key is not configured, all nodes joined to the cluster are assigned the default controller role. The controller role designates nodes that run the Kubernetes control plane. Controller nodes can also run other workloads, such as application workloads.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  roles:
    controller:
      # Optionally change the display name for the default controller role
      name: app
      labels:
        app: "true" # Label applied to nodes assigned the controller role
    # Custom roles
    custom:
      - name: db
        labels:
          db: "true" # Label applied to "db" nodes
      - name: gpu
        labels:
          gpu: "true" # Label applied to "gpu" nodes

Limitations​

  • Defining node roles with the roles key is Beta.

  • The first node added to the cluster must include the controller role. It may additionally be assigned any custom roles.

  • A node's role assignment cannot be changed after the node is joined to the cluster. To change a node's roles, reset the node and add it again with the new role selection.

roles.controller​

In the roles.controller key, you can customize the default controller role:

  • name: Sets the display name for the controller role. By default, the controller role is named controller.

    note

    If you plan to declare any custom roles, Replicated recommends that you change the default name for the controller role to a term that is easy to understand (such as app). When custom roles are present, both the controller role name and any custom role names are displayed to the user when they install or join a node.

  • labels: Kubernetes labels that Embedded Cluster applies to every node assigned the controller role.

roles.custom​

In the roles.custom key, you can declare additional roles. Each custom role includes the following fields:

  • name: (Required) The name of the custom role.

  • labels: Kubernetes labels that Embedded Cluster applies to every node assigned the role.

Multiple roles may be assigned to the same node — the joined node carries the union of each role's labels. If two roles define the same label key, the role that appears later in the user's selection wins.

domains​

Configure the domains key so that Embedded Cluster uses your custom domains for the Replicated proxy registry and Replicated app service.

Embedded Cluster uses the custom domains specified in domains.proxyRegistryDomain and domains.replicatedAppDomain when making requests to the given service.

You must add the custom domains that you specify in the domains.proxyRegistryDomain and domains.replicatedAppDomain fields to the Vendor Portal before Embedded Cluster can use them. For more information, see Add a Custom Domain in the Vendor Portal in Using Custom Domains.

If domains.proxyRegistryDomain and domains.replicatedAppDomain are not set, Embedded Cluster uses the default Replicated domains. For more information about aliasing Replicated endpoints with custom domains, see About Custom Domains.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  domains:
    proxyRegistryDomain: proxy.mycompany.com
    replicatedAppDomain: updates.mycompany.com

extensions​

Embedded Cluster installs Helm charts before your application as part of Embedded Cluster itself. Specify these charts in spec.extensions.helmCharts. Each item describes one chart release.

helmCharts​

Each object in helmCharts can include the following fields:

  • chart: (Required) Identifies the chart.
    • name: Chart name.
    • chartVersion: Chart version string to install.
  • releaseName: (Required) Helm release name.
  • namespace: (Required) Namespace for the release.
  • weight: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights.
  • values: Multi-line string of Helm values. This field supports Replicated template functions. Embedded Cluster updates Helm extensions when users deploy new versions of your application. For example, you can change values from one release to another, and those changes apply when the user deploys the new version.

Requirements​

  • You must provide chart.chartVersion. Omitting it can cause upgrade issues.

  • For air gap bundles, images used by Helm extensions must not refer to a multi-architecture image by digest. Air gap bundles include only x64 images, and the digest for the x64 image will be different from the digest for the multi-architecture image, preventing Kubernetes from locating the image in the bundle. For an example of how to set digests to empty strings and pull by tag only, see the ingress-nginx Example.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  extensions:
    helmCharts:
      - chart:
          name: ingress-nginx
          chartVersion: "4.11.3"
        releaseName: ingress-nginx
        namespace: ingress-nginx
        values: |
          controller:
            service:
              type: NodePort
              nodePorts:
                http: "80"
                https: "443"
            image:
              digest: ""
              digestChroot: ""

unsupportedOverrides​

important

Advanced users who understand the risks and ramifications of changing the default configuration should use this feature with caution.

Use unsupported overrides to change Embedded Cluster's default configuration, including the k0s config and Helm values for built-in extensions. Use overrides carefully. Replicated has not tested these combinations and they can disrupt clusters. Replicated does not support issues caused by unsupported overrides.

While you should use them with caution, unsupported overrides are useful if you need to make changes that Embedded Cluster does not otherwise expose.

Override the k0s Config​

By default, Embedded Cluster uses a k0s config that Replicated has tested and confirmed works for Embedded Clusters. In some circumstances, you might want to change the k0s config.

For more information on the k0s config, see Configuration options in the k0s documentation.

The value of unsupportedOverrides.k0s is a string containing YAML that becomes (or patches) the k0s configuration Embedded Cluster uses. How deeply Embedded Cluster merges that YAML with its defaults depends on the field. Treat overrides as high risk and validate in a non-production cluster.

For example, you can enable WireGuard-based encryption. You might need to add other configuration. See spec.network.calico in the k0s documentation:

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          network:
            calico:
              wireguard: true

Limitations​

  • You cannot change the spec.api and spec.storage keys in the k0s config after installation. Whereas most keys in the k0s config apply to the whole cluster, k0s sets these two keys for each node individually. Embedded Cluster cannot update these keys on each individual node during updates, so you cannot change them after installation.

  • Overrides can replace whole sections of the k0s configuration rather than deep-merge with every default. When overriding a list (for example, workerProfiles), you may need to include entries that Embedded Cluster relies on by default, or behavior can change in ways that are hard to predict. Confirm the effective k0s config for your release if you rely on defaults plus overrides.

Override the Helm values for built-in extensions​

Embedded Cluster deploys built-in extensions with Helm. You can patch their values with unsupportedOverrides.builtInExtensions. Each item sets name (the chart to patch) and values (a multi-line YAML string). Embedded Cluster merges your values into the defaults it uses.

Configure the Kubelet​

You can configure the Kubelet to customize worker nodes with Embedded Cluster. One common use case is raising the per-node pod limit (maxPods). Another is tuning parallel image pulls (maxParallelImagePulls).

Add a worker profile in unsupportedOverrides.k0s at config.spec.workerProfiles[]. When you define a worker profile, Embedded Cluster uses it for every node during initial installation and when joining nodes.

Each profile requires:

  • name: Profile name. Do not use default because k0s reserves that name.
  • values: Kubelet settings. See KubeletConfiguration in the Kubernetes documentation.

For more information, see spec.workerProfiles in the k0s documentation.

You can combine worker profiles with other k0s settings in the same k0s string (for example, metadata, telemetry, or network).

Limitations​

  • Embedded Cluster sets the worker profile during initial installation. You cannot change or add worker profiles on upgrade.
  • Embedded Cluster supports only one worker profile and applies it to all nodes. If you add more than one profile in workerProfiles[], Embedded Cluster applies only the first profile.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          workerProfiles:
          - name: custom-maxpods
            values:
              maxPods: 150
On this page