This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Capsule Proxy

Improve the UX even more with the Capsule Proxy

Capsule Proxy is an add-on for Capsule Operator addressing some RBAC issues when enabling multi-tenancy in Kubernetes since users cannot list the owned cluster-scoped resources. One solution to this problem would be to grant all users LIST permissions for the relevant cluster-scoped resources (eg. Namespaces). However, this would allow users to list all cluster-scoped resources, which is not desirable in a multi-tenant environment and may lead to security issues. Kubernetes RBAC cannot list only the owned cluster-scoped resources since there are no ACL-filtered APIs. For example:

Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope

The reason, as the error message reported, is that the RBAC list action is available only at Cluster-Scope and it is not granted to users without appropriate permissions.

To overcome this problem, many Kubernetes distributions introduced mirrored custom resources supported by a custom set of ACL-filtered APIs. However, this leads to radically change the user’s experience of Kubernetes by introducing hard customizations that make it painful to move from one distribution to another.

With Capsule, we took a different approach. As one of the key goals, we want to keep the same user experience on all the distributions of Kubernetes. We want people to use the standard tools they already know and love and it should just work.

1 - ProxySettings

Configure proxy settings for your tenants

Primitives

Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.

Primitives are strongly considered for tenants, therefor

The proxy setting kind is an enum accepting the supported resources:

EnumDescriptionEffective Operations
TenantUsers are able to LIST this tenant- LIST
StorageClassesPerform operations on the allowed StorageClasses for the tenant- LIST
  • Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed

  • StorageClasses: Perform actions on the allowed StorageClasses for the tenant

  • IngressClasses: Perform actions on the allowed IngressClasses for the tenant

  • PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant PriorityClasses

  • RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant

  • PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant

    GatewayClassesProxy ProxyServiceKind = “GatewayClasses” TenantProxy ProxyServiceKind = “Tenant”

Each Resource kind can be granted with several verbs, such as:

  • List
  • Update
  • Delete

Cluster Resources

This approach is for more generic cluster scoped resources.

TBD

Proxy Settings

Tenants

The Capsule Proxy is a multi-tenant application. Each tenant is a separate instance of the Capsule Proxy. The tenant is identified by the tenantId in the URL. The tenantId is a unique identifier for the tenant. The tenantId is used to identify the tenant in the Capsule Proxy.

2 - Installation

Installation guide for the capsule-proxy

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

Running outside a Kubernetes cluster is also viable, although a valid KUBECONFIG file must be provided, using the environment variable KUBECONFIG or the default file in $HOME/.kube/config.

A Helm Chart is available here.

Exposure

Depending on your environment, you can expose the capsule-proxy by:

  • Ingress
  • NodePort Service
  • LoadBalance Service
  • HostPort
  • HostNetwork

Here how it looks like when exposed through an Ingress Controller:

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

3 - Controller Options

Configure the Capsule Proxy Controller

You can customize the Capsule Proxy with the following configuration

Flags

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

Feature GateDefault ValueDescription
ProxyAllNamespacedfalseProxyAllNamespaced allows to proxy all the Namespaced objects. When enabled, it will discover apis and ensure labels are set for resources in all tenant namespaces resulting in increased memory. However this feature helps with user experience.
SkipImpersonationReviewfalseSkipImpersonationReview allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems.
ProxyClusterScopedfalseProxyClusterScoped allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings