With Sveltos, a management cluster is used to manage add-ons in tens of clusters.
When managing tens of cluster, multi-tenancy plays an important role.
Common forms of multi-tenancy are:
- share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. These workloads frequently need to communicate with each other, and with other workloads located on the same or different clusters;
- one (or more) cluster(s) fully reserved for an organization.
In both forms, we can define:
- platform admin is the admin with cluster-admin access to all the managed clusters;
- tenant admin is the admin with access to the clusters/namespaces assigned to them by the platform admin. Tenant admin manages applications for a tenant.
Sveltos wants to provide a solution so that:
- platform admin onboards tenant admins and easily defines what each tenant can do in which clusters;
- tenant admins manage tenant applications from a single place, the management cluster.
RoleRequest CRD
RoleRequest is the CRD introduced by Sveltos to allow platform admin to grant permissions to various tenant admins.
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: full-access
spec:
clusterSelector: dep=eng
admin: eng
roleRefs:
- name: full-access
namespace: default
kind: ConfigMap
where:
- admin: identifies the tenant admin to whom permissions are granted;
- clusterSelector: is a Kubernetes label selector. Sveltos uses it to detect all the clusters where permissions need to be granted;
- roleRefs: references ConfigMaps/Secrets each containing one or more Kubernetes ClusterRoles/Roles defining permissions being granted to the tenant admin.
An example of a ConfigMap containing ClusterRole granting full edit permissions:
apiVersion: v1
kind: ConfigMap
metadata:
name: full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: eng-full-access
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
ClusterProfile
After a tenant is onboarded by platform admin, tenant admin can create ClusterProfiles and Sveltos will take care of deploying them to all matching clusters.
Sveltos expects following label to be set on each ClusterProfile created by a tenant admin:
where admin must match RoleRequest.Spec.Admin field.
If:
- each tenant admin is a ServiceAccount in the management cluster;
- Kyverno is deployed in the management cluster;
Sveltos suggests using following Kyverno ClusterPolicy, which will take care of adding proper label to each ClusterProfile at creation time.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-labels
annotations:
policies.kyverno.io/title: Add Labels
policies.kyverno.io/description: >-
Adds projectsveltos.io/admin-name label on each ClusterProfile
created by tenant admin. It assumes each tenant admin is
represented in the management cluster by a ServiceAccount.
spec:
validationFailureAction: enforce
background: false
rules:
- name: add-labels
match:
resources:
kinds:
- ClusterProfile
mutate:
patchStrategicMerge:
metadata:
labels:
projectsveltos.io/admin-name: "{{serviceAccountName}}"
Fully reserving clusters to a tenant
In the following example, all clusters matching Kubernetes label selector org=foo.io will be assigned to tenant foo:
apiVersion: v1
kind: ConfigMap
metadata:
name: full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: foo-full-access
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: full-access
spec:
clusterSelector: org=foo.io
admin: foo
roleRefs:
- name: full-access
namespace: default
kind: ConfigMap
As soon as tenant foo posts following ClusterProfile, Sveltos will deploy Kyverno in any cluster matching org=foo.io label selector.
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno
labels:
projectsveltos.io/admin-name: foo
spec:
clusterSelector: org=foo.io
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v2.6.0
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
If the same tenant foo tries to deploy Kyverno in a cluster not assigned to it, Sveltos will fail the deployment.
For instance if ClusterProfile.Spec.ClusterSelector is set to org=bar.io deployment will fail.
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno
labels:
projectsveltos.io/admin-name: foo
spec:
clusterSelector: org=bar.io
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v2.6.0
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
Sharing cluster between multiple tenants
In following examples, all clusters matching label selector env=internal
are shared between two tenants:
- tenant foo is granted full access to namespaces foo-eng and foo-hr
- tenant bar is granted full access to namespace bar-resource
# ConfigMap contains a Role which gives
# full access to namespace ci-cd and build
apiVersion: v1
kind: ConfigMap
metadata:
name: foo-shared-access
namespace: default
data:
ci_cd_role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: foo-eng
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
build_role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: foo-hr
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
# RoleRequest gives admin 'eng' access to namespaces
# 'ci-cd' and 'cuild' in all clusters matching the label
# selector env=internal
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: foo-access
spec:
clusterSelector: env=internal
admin: foo
roleRefs:
- name: foo-shared-access
namespace: default
kind: ConfigMap
---
# ConfigMap contains a Role which gives
# full access to namespace human-resource
apiVersion: v1
kind: ConfigMap
metadata:
name: bar-shared-access
namespace: default
data:
ci_cd_role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: bar-resource
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
# RoleRequest gives admin 'hr' access to namespace
# 'human-resource' in all clusters matching the label
# selector env=internal
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: bar-access
spec:
clusterSelector: env=internal
admin: bar
roleRefs:
- name: bar-shared-access
namespace: default
kind: ConfigMap
Display tenant admin permissions
One of main Sveltos focuses is visibility. So it should not be a surprise that Sveltos offers a CLI for displaying tenant admin RBACs.
Sveltosctl can be used to display permissions granted to each tenant admin in each managed cluster.
If we have two clusters, a ClusterAPI powered one and a SveltosCluster, both matching label selector
env=internal and we post RoleRequests, we get:
./bin/sveltosctl show admin-rbac
+---------------------------------------------+-------+----------------+------------+-----------+----------------+-------+
| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |
+---------------------------------------------+-------+----------------+------------+-----------+----------------+-------+
| Cluster:default/sveltos-management-workload | eng | build | * | * | * | * |
| Cluster:default/sveltos-management-workload | eng | ci-cd | * | * | * | * |
| Cluster:default/sveltos-management-workload | hr | human-resource | * | * | * | * |
| SveltosCluster:gke/prod-cluster | eng | build | * | * | * | * |
| SveltosCluster:gke/prod-cluster | eng | ci-cd | * | * | * | * |
| SveltosCluster:gke/prod-cluster | hr | human-resource | * | * | * | * |
+---------------------------------------------+-------+----------------+------------+-----------+----------------+-------+
