Concepts

• 1: Architecture
• 2: Known Metadata
• 3: Templating

1 - Architecture

Key Decisions

Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As Cluster Administrator, ask yourself:

• 🔑 How much ownership can be delegated to Tenant Owners (Platform Users)?

The answer to this question may be influenced by the following aspects:

• Are the Cluster Administrators willing to grant permissions to Tenant Owners?

  • You might have a problem with know-how and probably your organisation is not yet pushing Kubernetes itself enough as a key strategic platform. The key here is enabling Platform Users through good UX and know-how transfers

• Who is responsible for the deployed workloads within the Tenants?

  • If Platform Administrators are still handling this, a true “shift left” has not yet been achieved.

• Who gets paged during a production outage within a Tenant’s application?

  • You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant.

• Are your customers technically capable of working directly with the Kubernetes API?

  • If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp.

Personas

In Capsule, we introduce a new persona called the Tenant Owner. The goal is to enable Cluster Administrators to delegate tenant management responsibilities to Tenant Owners. Here’s how it works:

Capsule Administrators

Configure Capsule Administrators. The ClusterRoles assigned to Administrators can be configured in the CapsuleConfiguration as well.

They are promoted to Tenant-Owners for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants.

Note: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster’s multi-tenant environment. When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.

Capsule Administrators can:

• Create and manage namespaces via labels in any tenant.
• Create namespaces outside of tenants.
• Delete namespaces in any tenant.

Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants.

Capsule Users

Configure Capsule Users

Any entity which needs to interact with tenants and their namespaces must be defined as a Capsule User. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users).

Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission..

You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with Tenant Owners, the status will always show the effective users:

kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq

[
  {
    "kind": "Group",
    "name": "oidc:kubernetes:admin"
  },
  {
    "kind": "Group",
    "name": "projectcapsule.dev"
  },
  {
    "kind": "User",
    "name": "test-user"
  }
]

Tenant Owners

Configure Tenant Owners

Every Tenant Owner must be a Capsule User. By using the TenantOwner CRD this is automatically handeled.

They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant.

Tenant Owners can:

• Create and manage namespaces within their tenant.
• Delete namespaces within their tenant.

Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management.

Layouts

Let’s discuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case.

Tenant As A Service

With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsibility. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant’s namespaces.

Scheduling

Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it’s essential to preserve maximum flexibility for your platform administrators. This means ensuring that:

• Nodes can be drained or deleted at any time.
• Cluster updates can be performed at any time.
• The number of worker nodes can be scaled up or down as needed.

If your cluster architecture prevents any of these capabilities, or if certain applications block the enforcement of these policies, you should reconsider your approach.

Dedicated

Strong tenant isolation, ensuring that any noisy neighbor effects remain confined within individual tenants (tenant responsibility). This approach may involve higher administrative overhead and costs compared to shared compute. It also provides enhanced security by dedicating nodes to a single customer/application. It is recommended, at a minimum, to separate the cluster’s operator workload from customer workloads.

Shared

With this approach you share the nodes amongst all Tenants, therefore giving you more potential for optimizing resources on a node level. It’s a common pattern to separate the controllers needed to power your Distribution (operators) from the actual workload. This ensures smooth operations for the cluster

Overview:

• ✅ Designed for cost efficiency .
• ✅ Suitable for applications that typically experience low resource fluctuations and run with multiple replicas.
• ❌ Not ideal for applications that are not cloud-native ready, as they may adversely affect the operation of other applications or the maintenance of node pools.
• ❌ Not ideal if strong isolation is required

We provide the concept of ResourcePools or CustomQuotas to manage resources cross namespaces. There’s some further aspects you must think about with shared approaches:

• PriorityClasses
• ResourceQuotas
• LimitRanges
• Descheduling/Rebalancing

2 - Known Metadata

Labels

Labels commonly used in Capsule items are listed below. These labels are applied to the corresponding resources by the Capsule controller and can be used for filtering and selection purposes.

capsule.clastix.io/tenant

projectcapsule.dev/tenant

capsule.clastix.io/managed-by

projectcapsule.dev/managed-by

projectcapsule.dev/created-by

| Description | Target Objects | Audience | |—|—| | Indicator which controller of capsule or Custom Resource is responsible for managing the corresponding Object. Mainly used in Replications to establish that objects were originally created by a Replication. | * Any Object being influenced by Replications | Controller |

projectcapsule.dev/name

projectcapsule.dev/cordoned

projectcapsule.dev/pool

3 - Templating

Fast Templates

For simple template cases we provide a fast templating engine. With this engine, you can use Go templates syntax to reference Tenant and Namespace fields. There are no operators or anything else supported.

Available fields are:

• {{tenant.name}}: The Name of the Tenant
• {{namespace}}: The Name of the namespace within the tenant (current context)

Sprout Templating

Our template library is mainly based on the upstream implementation from Sprout. You can find the all available functions here:

• https://docs.atom.codes/sprout/registries/list-of-all-registries

We have removed certain functions which could exploit runtime information. Therefor the following functions are not available:

• env
• expandEnv

Data

You can provide structured data for each Tenant which can be used in templating:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  data:
    bool: true
    foo: bar
    list:
    - a
    - b
    number: 123
    obj:
      nested: value

Function Library

Custom Functions we provide in our template package.

deterministicUUID

deterministicUUID generates a deterministic, RFC-4122–compliant UUID (version 5 + RFC4122 variant) from a set of input strings. It is designed for use in templates where you need stable, repeatable IDs derived from meaningful inputs (e.g. cluster name, tenant, role name), instead of random UUIDs.

This is especially useful for:

• Crossplane / IaC resources that must not change IDs across reconciles

The function takes any number of strings and turns them into a UUID in a fully deterministic way.

What that means in practice:

• If you call it twice with the same values, you get the same UUID
• If any input changes, the UUID changes too
• There is no randomness involved
• The output is always a valid UUID

So from the outside, it behaves just like a normal UUID, just deterministic.

deterministicUUID(parts ...string) string

Example usage:

{{ deterministicUUID "cluster-a" "app-123" "tenant-x" "some-role" }}

generateAgeKey

generateAgeKey generates a new age X25519 key pair for use with age. It returns both the private identity and the public recipient key.

This is useful in templates where a resource needs to create an age-compatible encryption identity, for example when generating Kubernetes Secrets that are later used for encrypting or decrypting data.

This is especially useful for:

• Bootstrap secrets that need an age identity
• Generating encryption keys during initial provisioning
• Creating age recipient keys for systems that need to encrypt data for a generated identity

The function does not return a plain string. It returns an object with two fields:

• Identity: the private age identity, e.g. AGE-SECRET-KEY-1...
• Recipient: the public age recipient, e.g. age1...

What that means in practice:

• Each call generates a new key pair
• The identity is the private key and must be treated as secret
• The recipient is the public key and can be shared with systems that need to encrypt data
• The output is not deterministic
• Calling this function during every reconcile may rotate the generated key unless the result is persisted

generateAgeKey() any

Example usage:

{{ $key := generateAgeKey }}
apiVersion: v1
kind: Secret
metadata:
  name: age-key
type: Opaque
stringData:
  identity: {{ $key.Identity | quote }}
  recipient: {{ $key.Recipient | quote }}

Output:

apiVersion: v1
kind: Secret
metadata:
  name: age-key
type: Opaque
stringData:
  identity: "AGE-SECRET-KEY-1..."
  recipient: "age1..."

Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates.

generateAgePQKey

generateAgePQKey generates a new age post-quantum hybrid key pair for use with age. It returns both the private identity and the public recipient key.

This is useful in templates where a resource needs to create an age-compatible encryption identity using the newer hybrid recipient format.

This is especially useful for:

• Bootstrap secrets that should use age hybrid keys
• Generating encryption keys during initial provisioning
• Creating public recipient keys for systems that need to encrypt data for a generated hybrid identity
• Future-facing age encryption setups where hybrid keys are preferred

The function does not return a plain string. It returns an object with two fields:

• Identity: the private age hybrid identity, e.g. AGE-SECRET-KEY-PQ-1...
• Recipient: the public age recipient, e.g. age1...

What that means in practice:

• Each call generates a new key pair
• The identity is the private key and must be treated as secret
• The recipient is the public key and can be shared with systems that need to encrypt data
• The output is not deterministic
• Calling this function during every reconcile may rotate the generated key unless the result is persisted

generateAgePQKey() any

Example Usage:

{{ $key := generateAgePQKey }}
apiVersion: v1
kind: Secret
metadata:
  name: age-pq-key
type: Opaque
stringData:
  identity: {{ $key.Identity | quote }}
  recipient: {{ $key.Recipient | quote }}

Output:

apiVersion: v1
kind: Secret
metadata:
  name: age-pq-key
type: Opaque
stringData:
  identity: "AGE-SECRET-KEY-PQ-1..."
  recipient: "age1..."

Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates.