This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Capsule Proxy

Improve the UX even more with the Capsule Proxy

1: ProxySettings
2: Installation
3: Controller Options
4: API Reference

Capsule Proxy is an add-on for Capsule Operator addressing some RBAC issues when enabling multi-tenancy in Kubernetes since users cannot list the owned cluster-scoped resources. One solution to this problem would be to grant all users LIST permissions for the relevant cluster-scoped resources (eg. Namespaces). However, this would allow users to list all cluster-scoped resources, which is not desirable in a multi-tenant environment and may lead to security issues. Kubernetes RBAC cannot list only the owned cluster-scoped resources since there are no ACL-filtered APIs. For example:

Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope

The reason, as the error message reported, is that the RBAC list action is available only at Cluster-Scope and it is not granted to users without appropriate permissions.

To overcome this problem, many Kubernetes distributions introduced mirrored custom resources supported by a custom set of ACL-filtered APIs. However, this leads to radically change the user’s experience of Kubernetes by introducing hard customizations that make it painful to move from one distribution to another.

With Capsule, we took a different approach. As one of the key goals, we want to keep the same user experience on all the distributions of Kubernetes. We want people to use the standard tools they already know and love and it should just work.

1 - ProxySettings

Configure proxy settings for your tenants

Primitives

Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.

Primitives are strongly considered for tenants, therefor

The proxy setting kind is an enum accepting the supported resources:

Enum	Description	Effective Operations
`Tenant`	Users are able to `LIST` this tenant	- `LIST`
`StorageClasses`	Perform operations on the allowed StorageClasses for the tenant	- `LIST`

Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
StorageClasses: Perform actions on the allowed StorageClasses for the tenant
IngressClasses: Perform actions on the allowed IngressClasses for the tenant
PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant PriorityClasses
RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant
GatewayClassesProxy ProxyServiceKind = “GatewayClasses” TenantProxy ProxyServiceKind = “Tenant”

Each Resource kind can be granted with several verbs, such as:

List
Update
Delete

Cluster Resources

This approach is for more generic cluster scoped resources.

TBD

Proxy Settings

Tenants

The Capsule Proxy is a multi-tenant application. Each tenant is a separate instance of the Capsule Proxy. The tenant is identified by the tenantId in the URL. The tenantId is a unique identifier for the tenant. The tenantId is used to identify the tenant in the Capsule Proxy.

2 - Installation

Installation guide for the capsule-proxy

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

Running outside a Kubernetes cluster is also viable, although a valid KUBECONFIG file must be provided, using the environment variable KUBECONFIG or the default file in $HOME/.kube/config.

A Helm Chart is available here.

Exposure

Depending on your environment, you can expose the capsule-proxy by:

Ingress
NodePort Service
LoadBalance Service
HostPort
HostNetwork

Here how it looks like when exposed through an Ingress Controller:

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

Kubernetes Reflector

3 - Controller Options

Configure the Capsule Proxy Controller

You can customize the Capsule Proxy with the following configuration

Flags

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

Feature Gate	Default Value	Description
`ProxyAllNamespaced`	`false`	`ProxyAllNamespaced` allows to proxy all the Namespaced objects. When enabled, it will discover apis and ensure labels are set for resources in all tenant namespaces resulting in increased memory. However this feature helps with user experience.
`SkipImpersonationReview`	`false`	`SkipImpersonationReview` allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems.
`ProxyClusterScoped`	`false`	`ProxyClusterScoped` allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings

4 - API Reference

API Reference

Packages:

capsule.clastix.io/v1beta1

capsule.clastix.io/v1beta1

Resource Types:

ProxySetting

ProxySetting

ProxySetting is the Schema for the proxysettings API.

Name	Type	Description	Required
apiVersion	string	capsule.clastix.io/v1beta1	true
kind	string	ProxySetting	true
metadata	object	Refer to the Kubernetes API documentation for the fields of the `metadata` field.	true
spec	object	ProxySettingSpec defines the additional Capsule Proxy settings for additional users of the Tenant. Resource is Namespace-scoped and applies the settings to the belonged Tenant.	false

ProxySetting.spec

ProxySettingSpec defines the additional Capsule Proxy settings for additional users of the Tenant. Resource is Namespace-scoped and applies the settings to the belonged Tenant.

Name	Type	Description	Required
subjects	[]object	Subjects that should receive additional permissions.	true

ProxySetting.spec.subjects[index]

Name	Type	Description	Required
kind	enum	Kind of tenant owner. Possible values are “User”, “Group”, and “ServiceAccount” Enum: User, Group, ServiceAccount	true
name	string	Name of tenant owner.	true
clusterResources	[]object	Cluster Resources for tenant Owner.	false
proxySettings	[]object	Proxy settings for tenant owner.	false

ProxySetting.spec.subjects[index].clusterResources[index]

Name	Type	Description	Required
apiGroups	[]string	APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against any resource listed will be allowed. ‘*’ represents all resources. Empty string represents v1 api resources.	true
operations	[]enum	Operations which can be executed on the selected resources. Default: [List]	true
resources	[]string	Resources is a list of resources this rule applies to. ‘*’ represents all resources.	true
selector	object	Select all cluster scoped resources with the given label selector.	true

ProxySetting.spec.subjects[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector.

Name	Type	Description	Required
matchExpressions	[]object	matchExpressions is a list of label selector requirements. The requirements are ANDed.	false
matchLabels	map[string]string	matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is “key”, the operator is “In”, and the values array contains only “value”. The requirements are ANDed.	false

ProxySetting.spec.subjects[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

Name	Type	Description	Required
key	string	key is the label key that the selector applies to.	true
operator	string	operator represents a key’s relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.	true
values	[]string	values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.	false

ProxySetting.spec.subjects[index].proxySettings[index]

Name	Type	Description	Required
kind	enum	Enum: Nodes, StorageClasses, IngressClasses, PriorityClasses, RuntimeClasses, PersistentVolumes	true
operations	[]enum		true