Proxy

• 1: Installation
• 2: ProxySettings
• 3: Controller Options
• 4: API Reference

Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope

1 - Installation

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

We only support the installation via helm-chart, you can find the chart here:

• https://artifacthub.io/packages/helm/capsule-proxy/capsule-proxy

Considerations

Consdierations when deploying capsule-proxy

Exposure

Depending on your environment, you can expose the capsule-proxy by:

• Ingress
• NodePort Service
• LoadBalance Service
• HostPort
• HostNetwork

Here how it looks like when exposed through an Ingress Controller:

                +-----------+          +-----------+         +-----------+
 kubectl ------>|:443       |--------->|:9001      |-------->|:6443      |
                +-----------+          +-----------+         +-----------+
                ingress-controller     capsule-proxy         kube-apiserver

User Authentication

The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server.

It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.

If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

• Kubernetes Reflector

HTTP Support

NOTE: kubectl will not work against a http server.

Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.

After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.

For example:

TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces

Metrics

Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.

The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.

Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.

capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:

path: the HTTP path of every single request that Capsule Proxy passes to the upstream capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.

path: the HTTP path of every single request that Capsule Proxy passes to the upstream status: the HTTP status code of the request

2 - ProxySettings

The configuration for the Proxy is also declarative via CRDs. This allows both Administrators and Tenant Owners to create flexible rules.

GlobalProxysettings

As an administrator, you might have the requirement to allow users to query cluster-scoped resources which are not directly linked to a tenant or anything like that. In that case you grant cluster-scoped LIST privileges to any subject, no matter what their tenant association is. For example:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - "*"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

With this rule the User alice can list any cluster-scoped resource which match the selector condition. The apiGroups and resources work the same as known from Kubernetes ClusterRoles. All of these are valid expressions:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - ""
      resources:
      - "pods"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev
    - apiGroups:
      - "kyverno.io/v1"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

A powerful tool to enhance the user-experience for all your users.

Proxysettings

ProxySettings are created in a namespace of a tenant, if it’s not in a namespace of a tenant it’s not regarded as valid. With the ProxySettings Tenant Owners can further improve the experience for their fellow tenant users.

apiVersion: capsule.clastix.io/v1beta1
kind: ProxySettings
metadata:
  name: solar-proxy
  namespace: solar-prod
spec:
  subjects:
  - kind: User
    name: alice
    proxySettings:
    - kind: IngressClasses
      operations:
      - List

Primitives

This will be refactored

Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.

The proxy setting kind is an enum accepting the supported resources:

• Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
• StorageClasses: Perform actions on the allowed StorageClasses for the tenant
• IngressClasses: Perform actions on the allowed IngressClasses for the tenant
• PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant PriorityClasses
• RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
• PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant

Each Resource kind can be granted with several verbs, such as:

• List
• Update
• Delete

Special routes for kubectl describe

When issuing a kubectl describe node, some other endpoints are put in place:

• api/v1/pods?fieldSelector=spec.nodeName%3D{name}
• /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/{name}

These are mandatory to retrieve the list of the running Pods on the required node and provide info about its lease status.

3 - Controller Options

You can customize the Capsule Proxy with the following configuration

Flags

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

4 - API Reference

Packages:

• capsule.clastix.io/v1beta1

capsule.clastix.io/v1beta1

Resource Types:

• GlobalProxySettings

• ProxySetting

GlobalProxySettings

GlobalProxySettings is the Schema for the globalproxysettings API.

GlobalProxySettings.spec

GlobalProxySettingsSpec defines the desired state of GlobalProxySettings.

GlobalProxySettings.spec.rules[index]

GlobalProxySettings.spec.rules[index].subjects[index]

GlobalProxySettings.spec.rules[index].clusterResources[index]

GlobalProxySettings.spec.rules[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

GlobalProxySettings.spec.rules[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting

ProxySetting is the Schema for the proxysettings API.

ProxySetting.spec

ProxySettingSpec defines the additional Capsule Proxy settings for additional users of the Tenant. Resource is Namespace-scoped and applies the settings to the belonged Tenant.

ProxySetting.spec.subjects[index]

ProxySetting.spec.subjects[index].clusterResources[index]

ProxySetting.spec.subjects[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

ProxySetting.spec.subjects[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting.spec.subjects[index].proxySettings[index]