This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Day-2 Operations

Monitoring, backup and restore, and troubleshooting for production Capsule clusters

1 - Monitoring

Monitoring Capsule Items and Tenants

The Capsule dashboard allows you to track the health and performance of Capsule manager and tenants, with particular attention to resources saturation, server responses, and latencies. Prometheus and Grafana are requirements for monitoring Capsule.

OpenTelemetry Tracing

Capsule can export OpenTelemetry traces for admission webhook requests through the OTLP gRPC exporter. Enable tracing with manager.options.tracing.enabled=true and set manager.options.tracing.endpoint to your collector or tracing backend, for example tempo.monitoring-system.svc:4317. The exporter supports insecure transport for local clusters, TLS server name overrides, gzip compression, export timeouts, custom OTLP headers, and basic authentication. For production, prefer manager.options.tracing.basicAuth.existingSecret so credentials are loaded from a Kubernetes Secret instead of being rendered into controller arguments. Use manager.options.tracing.sampleRatio to control trace volume.

Tenants

Instrumentation for Tenants.

Metrics

The following Metrics are exposed and can be used for monitoring:

# HELP capsule_tenant_condition Provides per tenant condition status for each condition
# TYPE capsule_tenant_condition gauge
capsule_tenant_condition{condition="Cordoned",tenant="solar"} 0
capsule_tenant_condition{condition="Ready",tenant="solar"} 1


# HELP capsule_tenant_namespace_condition Provides per namespace within a tenant condition status for each condition
# TYPE capsule_tenant_namespace_condition gauge
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="earth",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="fire",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="foild",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="green",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="solar",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="wind",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Ready",target_namespace="earth",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="fire",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="foild",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="green",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="solar",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="wind",tenant="solar"} 1

# HELP capsule_tenant_namespace_count Total number of namespaces currently owned by the tenant
# TYPE capsule_tenant_namespace_count gauge
capsule_tenant_namespace_count{tenant="solar"} 6

# HELP capsule_tenant_namespace_relationship Mapping metric showing namespace to tenant relationships
# TYPE capsule_tenant_namespace_relationship gauge
capsule_tenant_namespace_relationship{target_namespace="earth",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="fire",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="soil",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="green",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="solar",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="wind",tenant="solar"} 1

# HELP capsule_tenant_resource_limit Current resource limit for a given resource in a tenant
# TYPE capsule_tenant_resource_limit gauge
capsule_tenant_resource_limit{resource="limits.cpu",resourcequotaindex="0",tenant="solar"} 2
capsule_tenant_resource_limit{resource="limits.memory",resourcequotaindex="0",tenant="solar"} 2.147483648e+09
capsule_tenant_resource_limit{resource="pods",resourcequotaindex="1",tenant="solar"} 7
capsule_tenant_resource_limit{resource="requests.cpu",resourcequotaindex="0",tenant="solar"} 2
capsule_tenant_resource_limit{resource="requests.memory",resourcequotaindex="0",tenant="solar"} 2.147483648e+09

# HELP capsule_tenant_resource_usage Current resource usage for a given resource in a tenant
# TYPE capsule_tenant_resource_usage gauge
capsule_tenant_resource_usage{resource="limits.cpu",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="limits.memory",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="namespaces",resourcequotaindex="",tenant="solar"} 2
capsule_tenant_resource_usage{resource="pods",resourcequotaindex="1",tenant="solar"} 0
capsule_tenant_resource_usage{resource="requests.cpu",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="requests.memory",resourcequotaindex="0",tenant="solar"} 0

Custom Metrics

You can gather more information based on the status of the tenants. These can be scrapped via Kube-State-Metrics CustomResourcesState Metrics. With these you have the possibility to create custom metrics based on the status of the tenants.

Here as an example with the kube-prometheus-stack chart, set the following values:

kube-state-metrics:
  rbac:
    extraRules:
      - apiGroups: [ "capsule.clastix.io" ]
        resources: ["tenants"]
        verbs: [ "list", "watch" ]
  customResourceState:
    enabled: true
    config:
      spec:
        resources:
          - groupVersionKind:
              group: capsule.clastix.io
              kind: "Tenant"
              version: "v1beta2"
            labelsFromPath:
              name: [metadata, name]
            metrics:
              - name: "tenant_size"
                help: "Count of namespaces in the tenant"
                each:
                  type: Gauge
                  gauge:
                    path: [status, size]
                commonLabels:
                  custom_metric: "yes"
                labelsFromPath:
                  capsule_tenant: [metadata, name]
                  kind: [ kind ]
              - name: "tenant_state"
                help: "The operational state of the Tenant"
                each:
                  type: StateSet
                  stateSet:
                    labelName: state
                    path: [status, state]
                    list: [Active, Cordoned]
                commonLabels:
                  custom_metric: "yes"
                labelsFromPath:
                  capsule_tenant: [metadata, name]
                  kind: [ kind ]
              - name: "tenant_namespaces_info"
                help: "Namespaces of a Tenant"
                each:
                  type: Info
                  info:
                    path: [status, namespaces]
                    labelsFromPath:
                      tenant_namespace: []
                commonLabels:
                  custom_metric: "yes"
                labelsFromPath:
                  capsule_tenant: [metadata, name]
                  kind: [ kind ]

This example creates three custom metrics:

  • tenant_size is a gauge that counts the number of namespaces in the tenant.
  • tenant_state is a state set that shows the operational state of the tenant.
  • tenant_namespaces_info is an info metric that shows the namespaces of the tenant.

2 - Backup & Restore

Run Backups and Restores of Tenants

Velero is a backup and restore solution that performs data protection, disaster recovery and migrates Kubernetes cluster from on-premises to the Cloud or between different Clouds.

When coming to backup and restore in Kubernetes, we have two main requirements:

  • Configurations backup
  • Data backup

The first requirement aims to backup all the resources stored into etcd database, for example: namespaces, pods, services, deployments, etc. The second is about how to backup stateful application data as volumes.

The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided “as a service” to the users. This means that the cluster admin needs to take care of users’ backup.

Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for:

  • Tenant cluster resources
  • Namespaces belonging to each tenant, including their PersistentVolumeClaims (PVCs)

Create backup of a tenant

Create a backup of the tenant solar. It consists in two different backups:

  • backup of the tenant resource
  • backup of all the resources belonging to the tenant, including the volume data

Prepare Velero and the Tenant

The volume data is backed up with CSI snapshots and the Velero data mover (tested with Velero 1.18). snapshotMoveData only moves the data of a snapshot, it does not set up the storage. This requires:

  • A CSI driver with snapshot support and a VolumeSnapshotClass labeled velero.io/csi-volumesnapshot-class: "true".
  • The Velero CSI feature (--features=EnableCSI, or configuration.features: EnableCSI in the Helm chart) and the Velero node agent. See the Velero documentation on CSI snapshot data movement.

Add the following rule to the Tenant before you back it up, so that it is part of the Tenant backup. During a restore with the data mover, Velero first restores the data into a temporary PersistentVolume (PV) whose claimRef points into the velero namespace. The Velero server then patches the tenant PVC to set spec.volumeName to that PV. The PV does not carry the capsule.clastix.io/tenant label yet, so without the rule Capsule rejects the patch as a cross-tenant mount and the restore hangs with the DataDownload stuck in InProgress:

admission webhook "pvc.validating.projectcapsule.dev" denied the request: Preventing a cross-tenant mount for PersistentVolume pvc-1b6d1f5c-...

Grant the Velero ServiceAccount the volume exception with a storage rule on the Tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  rules:
    - audience:
        - kind: ServiceAccount
          name: system:serviceaccount:velero:velero-server
      enforce:
        action: allow
        conditions:
          - name: restore-rebind
            expression: >-
              request.operation == 'UPDATE' &&
              volume != null &&
              has(volume.spec.claimRef) &&
              volume.spec.claimRef.namespace == 'velero'              
        storage:
          volumes:
            - name: velero-dynamic-pv-restore
              selector: {}

If Velero does not run in the velero namespace as velero-server, change both the namespace and name in the audience and the namespace in the claimRef.namespace condition, otherwise the condition never matches the temporary PV. The condition limits the exception to Velero’s temporary volumes, other Tenants’ labeled PVs stay protected, see Ownership and label protection. After the PVC patch is admitted, Capsule labels the PV with the Tenant label as soon as its claimRef points into a Tenant namespace.

Back up the Tenant and its namespaces

To back up the solar tenant selectively, label the tenant as:

kubectl label tenant solar capsule.clastix.io/tenant=solar

and create the backup of the Tenant, including the rule from above:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: solar-tenant
  namespace: velero
spec:
  includeClusterResources: true
  includedNamespaces:
  - '*'
  includedResources:
  - tenants.capsule.clastix.io
  labelSelector:
    matchLabels:
      capsule.clastix.io/tenant: solar
  storageLocation: default
  ttl: 720h0m0s

Create a backup of all the resources belonging to the solar tenant namespaces. Set snapshotMoveData to back up the content of the PVCs to the object storage:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: solar-namespaces
  namespace: velero
spec:
  includeClusterResources: false
  includedNamespaces:
  - solar-production
  - solar-development
  - solar-marketing
  snapshotMoveData: true
  storageLocation: default
  ttl: 720h0m0s

Restore a tenant from the backup

To recover the tenant after a disaster, or to migrate it to another cluster, restore the Tenant first:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: solar-tenant
  namespace: velero
spec:
  backupName: solar-tenant

If the Tenant backup was taken without the volume rule, add the rule to the restored Tenant before you restore the namespaces.

Create the namespaces

The namespaces have to exist before the restore starts. Velero recreates a missing namespace with its labels but without the OwnerReference that binds it to the Tenant. Capsule denies such a namespace and the restore ends as PartiallyFailed:

admission webhook "namespaces.validating.projectcapsule.dev" denied the request: namespace has tenant label "solar" but no Tenant ownerReference

Create the desired namespaces as a tenant owner. A Capsule administrator can also create the namespace instead of a tenant owner. In that case the tenant label has to be added:

kubectl create namespace solar-production
kubectl label namespace solar-production capsule.clastix.io/tenant=solar

For all options to add a namespace to a tenant, see adding Namespace to a Tenant.

Velero skips resources that already exist, and this includes the namespace itself, even with existingResourcePolicy: update. The labels and annotations of the original namespace, for example the ones used by Capsule rules or replication selectors, are therefore not restored. Add them again when you create the namespace, taken from the backup or from the source of your manifests, while keeping the Capsule tenant label:

kubectl label namespace solar-production team=blue
kubectl annotate namespace solar-production note=hello

Restore the resources

Velero restores the objects into the existing namespace. Capsule recreates the objects it manages itself, for example the RoleBindings of the Tenant owners or the ConfigMaps replicated by a TenantResource. They are protected by Capsule’s webhooks and can only be written by the Capsule controller or a Capsule administrator, so restoring them with Velero is denied and ends the restore as PartiallyFailed. Capsule labels every object it manages with projectcapsule.dev/managed-by. Skip these objects with the operator DoesNotExist on this label:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: solar-namespaces
  namespace: velero
spec:
  backupName: solar-namespaces
  labelSelector:
    matchExpressions:
      - key: projectcapsule.dev/managed-by
        operator: DoesNotExist

After these steps, all resources will be restored in the namespaces.