Capsule Proxy
Improve the UX even more with the Capsule Proxy
Capsule Proxy is an add-on for Capsule Operator addressing some RBAC issues when enabling multi-tenancy in Kubernetes since users cannot list the owned cluster-scoped resources. One solution to this problem would be to grant all users LIST
permissions for the relevant cluster-scoped resources (eg. Namespaces
). However, this would allow users to list all cluster-scoped resources, which is not desirable in a multi-tenant environment and may lead to security issues. Kubernetes RBAC cannot list only the owned cluster-scoped resources since there are no ACL-filtered APIs. For example:
Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope
The reason, as the error message reported, is that the RBAC list action is available only at Cluster-Scope and it is not granted to users without appropriate permissions.
To overcome this problem, many Kubernetes distributions introduced mirrored custom resources supported by a custom set of ACL-filtered APIs. However, this leads to radically change the user’s experience of Kubernetes by introducing hard customizations that make it painful to move from one distribution to another.
With Capsule, we took a different approach. As one of the key goals, we want to keep the same user experience on all the distributions of Kubernetes. We want people to use the standard tools they already know and love and it should just work.
1 - ProxySettings
Configure proxy settings for your tenants
Primitives
Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.
Primitives are strongly considered for tenants, therefor
The proxy setting kind is an enum accepting the supported resources:
Enum | Description | Effective Operations |
---|
Tenant | Users are able to LIST this tenant | - LIST |
StorageClasses | Perform operations on the allowed StorageClasses for the tenant | - LIST |
Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
StorageClasses: Perform actions on the allowed StorageClasses for the tenant
IngressClasses: Perform actions on the allowed IngressClasses for the tenant
PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant
PriorityClasses
RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant
GatewayClassesProxy ProxyServiceKind = “GatewayClasses”
TenantProxy ProxyServiceKind = “Tenant”
Each Resource kind can be granted with several verbs, such as:
Cluster Resources
This approach is for more generic cluster scoped resources.
TBD
Proxy Settings
Tenants
The Capsule Proxy is a multi-tenant application. Each tenant is a separate instance of the Capsule Proxy. The tenant is identified by the tenantId
in the URL. The tenantId
is a unique identifier for the tenant. The tenantId
is used to identify the tenant in the Capsule Proxy.
2 - Installation
Installation guide for the capsule-proxy
Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.
The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.
Running outside a Kubernetes cluster is also viable, although a valid KUBECONFIG file must be provided, using the environment variable KUBECONFIG or the default file in $HOME/.kube/config.
A Helm Chart is available here.
Exposure
Depending on your environment, you can expose the capsule-proxy by:
Ingress
NodePort Service
LoadBalance Service
HostPort
HostNetwork
Here how it looks like when exposed through an Ingress Controller:
Distribute CA within the Cluster
The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy
in the capsule-system
namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.
Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines
by using kubectl
and jq
:
kubectl get secret capsule-proxy -n capsule-system -o json \
| jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
| kubectl apply -n tekton-pipelines -f -
This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:
3 - Controller Options
Configure the Capsule Proxy Controller
You can customize the Capsule Proxy with the following configuration
Flags
Feature Gates
Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:
Feature Gate | Default Value | Description |
---|
ProxyAllNamespaced | false | ProxyAllNamespaced allows to proxy all the Namespaced objects. When enabled, it will discover apis and ensure labels are set for resources in all tenant namespaces resulting in increased memory. However this feature helps with user experience. |
SkipImpersonationReview | false | SkipImpersonationReview allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems. |
ProxyClusterScoped | false | ProxyClusterScoped allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings |