Proxy

• 1: Installation
• 2: ProxySettings
• 3: Controller Options
• 4: Integrations
• 4.1: Kubernetes Dashboard
• 4.2: Lens
• 4.3: Tekton
• 4.4: Teleport
• 5: API Reference

Error from server (Forbidden): namespaces is forbidden:
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope

1 - Installation

Capsule Proxy is an optional add-on of the main Capsule Operator, so make sure you have a working instance of Capsule before attempting to install it. Use the capsule-proxy only if you want Tenant Owners to list their Cluster-Scope resources.

The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.

We only support the installation via helm-chart, you can find the chart here:

• https://artifacthub.io/packages/helm/capsule-proxy/capsule-proxy

Considerations

Consdierations when deploying capsule-proxy

Exposure

Depending on your environment, you can expose the capsule-proxy by:

• Ingress
• NodePort Service
• LoadBalance Service
• HostPort
• HostNetwork

Here how it looks like when exposed through an Ingress Controller:

                +-----------+          +-----------+         +-----------+
 kubectl ------>|:443       |--------->|:9001      |-------->|:6443      |
                +-----------+          +-----------+         +-----------+
                ingress-controller     capsule-proxy         kube-apiserver

User Authentication

The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server.

It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.

If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.

Distribute CA within the Cluster

The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.

Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:

 kubectl get secret capsule-proxy -n capsule-system -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
 | kubectl apply -n tekton-pipelines -f -

This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:

• Kubernetes Reflector

HTTP Support

NOTE: kubectl will not work against a http server.

Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.

After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.

For example:

TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces

Metrics

Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.

The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.

Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.

capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:

path: the HTTP path of every single request that Capsule Proxy passes to the upstream capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.

path: the HTTP path of every single request that Capsule Proxy passes to the upstream status: the HTTP status code of the request

2 - ProxySettings

The configuration for the Proxy is also declarative via CRDs. This allows both Administrators and Tenant Owners to create flexible rules.

GlobalProxysettings

As an administrator, you might have the requirement to allow users to query cluster-scoped resources which are not directly linked to a tenant or anything like that. In that case you grant cluster-scoped LIST privileges to any subject, no matter what their tenant association is. For example:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - "*"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

With this rule the User alice can list any cluster-scoped resource which match the selector condition. The apiGroups and resources work the same as known from Kubernetes ClusterRoles. All of these are valid expressions:

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: global-proxy-settings
spec:
  rules:
  - subjects:
    - kind: User
      name: alice
    clusterResources:
    - apiGroups:
      - ""
      resources:
      - "pods"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev
    - apiGroups:
      - "kyverno.io/v1"
      resources:
      - "*"
      operations:
      - List
      selector:
        matchLabels:
          app.kubernetes.io/type: dev

A powerful tool to enhance the user-experience for all your users.

Proxysettings

ProxySettings are created in a namespace of a tenant, if it’s not in a namespace of a tenant it’s not regarded as valid. With the ProxySettings Tenant Owners can further improve the experience for their fellow tenant users.

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: solar-proxy
  namespace: solar-prod
spec:
  subjects:
  - kind: User
    name: alice
    proxySettings:
    - kind: IngressClasses
      operations:
      - List

Primitives

This will be refactored

Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.

The proxy setting kind is an enum accepting the supported resources:

• Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
• StorageClasses: Perform actions on the allowed StorageClasses for the tenant
• IngressClasses: Perform actions on the allowed IngressClasses for the tenant
• PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant PriorityClasses
• RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
• PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant

Each Resource kind can be granted with several verbs, such as:

• List
• Update
• Delete

Special routes for kubectl describe

When issuing a kubectl describe node, some other endpoints are put in place:

• api/v1/pods?fieldSelector=spec.nodeName%3D{name}
• /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/{name}

These are mandatory to retrieve the list of the running Pods on the required node and provide info about its lease status.

3 - Controller Options

You can customize the Capsule Proxy with the following configuration

Flags

Feature Gates

Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:

4 - Integrations

4.1 - Kubernetes Dashboard

This guide describes how to integrate the Kubernetes Dashboard and Capsule Proxy with OIDC authorization.

OIDC Authentication

Your cluster must also be configured to use OIDC Authentication for seemless Kubernetes RBAC integration. In a such scenario, you should have in the kube-apiserver.yaml manifest the following content:

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --oidc-issuer-url=https://${OIDC_ISSUER}
    - --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
    - --oidc-client-id=${OIDC_CLIENT_ID}
    - --oidc-username-claim=preferred_username
    - --oidc-groups-claim=groups
    - --oidc-username-prefix=-

Where ${OIDC_CLIENT_ID} refers to the client ID that all tokens must be issued.

For this client we need: 1. Check Valid Redirect URIs: in the oauth2-proxy configuration we set redirect-url: “https://${DASHBOARD_URL}/oauth2/callback”, it needs to add this path to the Valid Redirect URIs 2. Create a mapper with Mapper Type ‘Group Membership’ and Token Claim Name ‘groups’. 3. Create a mapper with Mapper Type ‘Audience’ and Included Client Audience and Included Custom Audience set to your client name (${OIDC_CLIENT_ID}).

OAuth2 Proxy

To enable the proxy authorization from the Kubernetes dashboard to Keycloak, we need to use an OAuth proxy. In this article, we will use oauth2-proxy and install it as a pod in the Kubernetes Dashboard namespace. Alternatively, we can install oauth2-proxy in a different namespace or use it as a sidecar container in the Kubernetes Dashboard deployment.

Prepare the values for oauth2-proxy:

cat > values-oauth2-proxy.yaml <<EOF
config:
  clientID: "${OIDC_CLIENT_ID}"
  clientSecret: ${OIDC_CLIENT_SECRET}

extraArgs:
  provider: "keycloak-oidc"
  redirect-url: "https://${DASHBOARD_URL}/oauth2/callback"
  oidc-issuer-url: "https://${KEYCLOAK_URL}/auth/realms/${OIDC_CLIENT_ID}"
  pass-access-token: true
  set-authorization-header: true
  pass-user-headers: true

ingress:
  enabled: true
  path: "/oauth2"
  hosts:
    - ${DASHBOARD_URL}
  tls:
    - hosts:
      - ${DASHBOARD_URL}
EOF

More information about the keycloak-oidc provider can be found on the oauth2-proxy documentation. We’re ready to install the oauth2-proxy:

helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy -n ${KUBERNETES_DASHBOARD_NAMESPACE} -f values-oauth2-proxy.yaml

Configuring Keycloak

The Kubernetes cluster must be configured with a valid OIDC provider: for our guide, we’re giving for granted that Keycloak is used, if you need more info please follow the OIDC Authentication section.

In a such scenario, you should have in the kube-apiserver.yaml manifest the following content:

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --oidc-issuer-url=https://${OIDC_ISSUER}
    - --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
    - --oidc-client-id=${OIDC_CLIENT_ID}
    - --oidc-username-claim=preferred_username
    - --oidc-groups-claim=groups
    - --oidc-username-prefix=-

Where ${OIDC_CLIENT_ID} refers to the client ID that all tokens must be issued.

For this client we need:

1. Check Valid Redirect URIs: in the oauth2-proxy configuration we set redirect-url: "https://${DASHBOARD_URL}/oauth2/callback", it needs to add this path to the Valid Redirect URIs
2. Create a mapper with Mapper Type ‘Group Membership’ and Token Claim Name ‘groups’.
3. Create a mapper with Mapper Type ‘Audience’ and Included Client Audience and Included Custom Audience set to your client name(OIDC_CLIENT_ID).

Configuring Kubernetes Dashboard

If your Capsule Proxy uses HTTPS and the CA certificate is not the Kubernetes CA, you need to add a secret with the CA for the Capsule Proxy URL.

cat > ca.crt<< EOF
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
EOF

kubectl create secret generic certificate --from-file=ca.crt=ca.crt -n ${KUBERNETES_DASHBOARD_NAMESPACE}

Prepare the values for the Kubernetes Dashboard:

cat > values-kubernetes-dashboard.yaml <<EOF
extraVolumes:
  - name: token-ca
    projected:
      sources:
        - serviceAccountToken:
            expirationSeconds: 86400
            path: token
        - secret:
            name: certificate
            items:
              - key: ca.crt
                path: ca.crt
extraVolumeMounts:
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: token-ca

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/auth-signin: https://${DASHBOARD_URL}/oauth2/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-url: https://${DASHBOARD_URL}/oauth2/auth
    nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
  hosts:
    - ${DASHBOARD_URL}
  tls:
    - hosts:
      - ${DASHBOARD_URL}

extraEnv:
  - name: KUBERNETES_SERVICE_HOST
    value: '${CAPSULE_PROXY_URL}'
  - name: KUBERNETES_SERVICE_PORT
    value: '${CAPSULE_PROXY_PORT}'
EOF

To add the Certificate Authority for the Capsule Proxy URL, we use the volume token-ca to mount the ca.crt file. Additionally, we set the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to route requests to the Capsule Proxy.

Now you can install the Kubernetes Dashboard:

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard -n ${KUBERNETES_DASHBOARD_NAMESPACE} -f values-kubernetes-dashboard.yaml

4.2 - Lens

With Capsule extension for Lens, a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.

Features

Capsule extension for Lens provides these capabilities:

• List all tenants
• See tenant details and change through the embedded Lens editor
• Check Resources Quota and Budget at both the tenant and namespace level

Please, see the README for details about the installation of the Capsule Lens Extension.

4.3 - Tekton

With Capsule extension for Lens, a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.

Prerequisites

Tekton must be already installed on your cluster, if that’s not the case consult the documentation here:

• Tekton

Cluster Scoped Permissions

Tekton Dashboard

Now for the enduser experience we are going to deploy the tekton dashboard. When using oauth2-proxy we can deploy one single dashboard, which can be used for all tenants. Refer to the following guide to setup the dashboard with the oauth2-proxy:

• Tekton Dashboard

Once that is done, we need to make small adjustments to the tekton-dashboard service account.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
patches:
  # Adjust the service for the capsule-proxy according to your installation
  # The used values are compatbile with the default installation values
  - target:
      version: v1
      kind: Deployment
      name: tekton-dashboard
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: KUBERNETES_SERVICE_HOST
          value: "capsule-proxy.capsule-system.svc"
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: KUBERNETES_SERVICE_PORT
          value: "9001"      

  # Adjust the CA certificate for the capsule-proxy according to your installation
  - target:
      version: v1
      kind: Deployment
      name: tekton-dashboard
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts
        value: []
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts/-
        value:
          mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
          name: token-ca
      - op: add
        path: /spec/template/spec/volumes
        value: []
      - op: add
        path: /spec/template/spec/volumes/-
        value:
          name: token-ca
          projected:
            sources:
              - serviceAccountToken:
                  expirationSeconds: 86400
                  path: token
              - secret:
                  name: capsule-proxy
                  items:
                    - key: ca
                      path: ca.crt      

This patch assumes there’s a secret called capsule-proxy with the CA certificate for the Capsule Proxy URL.

Apply the given kustomization:

extraEnv:

• name: KUBERNETES_SERVICE_HOST value: ‘${CAPSULE_PROXY_URL}’
• name: KUBERNETES_SERVICE_PORT value: ‘${CAPSULE_PROXY_PORT}’

Tekton Operator

When using the Tekton Operator, you need to add the following to the TektonConfig:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  dashboard:
    readonly: false
    options:
      disabled: false
      deployments:
        tekton-dashboard:
          spec:
            template:
              spec:
                volumes:
                  - name: token-ca
                    projected:
                      sources:
                        - serviceAccountToken:
                            expirationSeconds: 86400
                            path: token
                        - secret:
                            name: capsule-proxy
                            items:
                              - key: ca
                                path: ca.crt
                containers:
                  - name: tekton-dashboard
                    volumeMounts:
                      - mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
                        name: token-ca
                    env:
                      - name: KUBERNETES_SERVICE_HOST
                        value: "capsule-proxy.capsule-system.svc"
                      - name: KUBERNETES_SERVICE_PORT
                        value: "9001"

See for reference the options spec

4.4 - Teleport

With Capsule extension for Lens, a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.

Features

Capsule extension for Lens provides these capabilities:

• List all tenants
• See tenant details and change through the embedded Lens editor
• Check Resources Quota and Budget at both the tenant and namespace level

Please, see the README for details about the installation of the Capsule Lens Extension.

5 - API Reference

Packages:

• capsule.clastix.io/v1beta1

capsule.clastix.io/v1beta1

Resource Types:

• GlobalProxySettings

• ProxySetting

GlobalProxySettings

GlobalProxySettings is the Schema for the globalproxysettings API.

GlobalProxySettings.spec

GlobalProxySettingsSpec defines the desired state of GlobalProxySettings.

GlobalProxySettings.spec.rules[index]

GlobalProxySettings.spec.rules[index].subjects[index]

GlobalProxySettings.spec.rules[index].clusterResources[index]

GlobalProxySettings.spec.rules[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

GlobalProxySettings.spec.rules[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting

ProxySetting is the Schema for the proxysettings API.

ProxySetting.spec

ProxySettingSpec defines the additional Capsule Proxy settings for additional users of the Tenant. Resource is Namespace-scoped and applies the settings to the belonged Tenant.

ProxySetting.spec.subjects[index]

ProxySetting.spec.subjects[index].clusterResources[index]

ProxySetting.spec.subjects[index].clusterResources[index].selector

Select all cluster scoped resources with the given label selector. Defining a selector which does not match any resources is considered not selectable (eg. using operation NotExists).

ProxySetting.spec.subjects[index].clusterResources[index].selector.matchExpressions[index]

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

ProxySetting.spec.subjects[index].proxySettings[index]