TL;DR

• What goes into self service tenant creation?
• Automate namespace creation using a self service portal
• Let tenants control access on their own
• Remove administrators from user management
• Separate authentication and authorization

Automating Namespaces

In our last blog post we integrated Active Directory into EKS using the Orchestra login portal. We're able to authenticate using our enterprise Active Directory to EKS and use groups from Active Directory to access EKS. Next, we need to determine what goes into automating tenant creation. The most basic aspects of tenant creation are:

• Who is responsible? - Who owns the tenant? Is there a cost center for charges generated by the tenant? Who is responsible for determining access to the namespace?
• What policies are applicable? - Does the namespace need to be isolated? Does access need to be re-certified on a periodic basis?
• What else besides a namespace needs to be created? - Network policies? RBAC bindings? Pipelines? There's more to a tenant then a namespace.
• How will you audit tenant access? - Once your tenants are created, how do you know what they're creating and managing their access?

Making a few assumptions, a basic tenant will want to:

1. Create a namespace
2. Create RBAC bindings for namespace administrators
3. Create RBAC bindings for namespace viewers

This is as basic as it gets. You'll likely also want to manage resource quotas, add network policies and provision external resources such as pipelines, registries, etc. We'll cover these parts in our next blog post.

Authorization vs Authentication From an Enterprise Perspective

Our cluster currently uses Active Directory for authorizations. This is an elegant solution since we don't need to update RBAC bindings every time a user is added. In a multi-tenant environment it can have one major downside. What happens if you don't have write access to your forest? When you create a namespace, can you create groups that correspond to access to them then add/remove users as needed? It's not unusual for the answer to be "no". In all likelihood, the people who run your Active Directory are not the same people responsible for your cluster. They have different responsibilities and so are an additional stakeholder that needs to provide a process that you are dependent on. Maybe it's as simple as getting your own OU or container in a forest, but that's usually not the case.

Authorizations are generally tied pretty closely to business process. Who has access and why? The "why" being dependent on your tenant. It can be advantageous to separate your authorization data from your authentication data. Keeping it in Kubernetes is clunky, but you still want to store it outside of Active Directory when you don't have control.

Deploy The Orchestra Automation Portal for Kubernetes

You could write a script to create the needed yaml from a template then apply it to your cluster, but that means you're doing something. It also means you're running some kind of manual action. You're spending a considerable amount of time and effort to get an automation platform running, why would you want a manual process for tenant on-boarding? For day two operations, you'll also need a way to provision access and a way to audit your tenants. The Orchestra Automation Portal for Kubernetes gives you a starting point to build a self-service portal:

• Self service portal for requesting a new namespace be created
• An approval process for creating the namespace and tracking approvals
• Creates groups in a local database to manage access to the new namespace
• Provision the namespace and RBAC bindings
• Provide a self service portal for users to gain access to namespaces without administrator intervention

To get started, we'll need two additional pieces of infrastructure:

• MySQL / MariaDB - Stores our audit data and workflow state. Also used to store our authorization groups.
• SMTP - Notify namespace owners and cluster administrators they have open requests to act on

The diagram is slightly different from the last post. In this diagram we're going to integrate a database and an SMTP server. The database stores:

• Authorization Data - Each namespace will get groups to manage who is an administrator, viewer, and access approver. Removing the authorization data from Active Directory cuts down on a dependency and makes it easier to create new namespaces.
• Audit Data - Every transaction, whether its an attribute on a user being updated or an object being created in the cluster, is tracked in the audit database.

By default the database needs to be MariaDB or MySQL. AuroraDB works great too. Other databases are supported, including Postgres and SQL Server.

The SMTP server is used to send notifications. When someone requests that a namespace be created or access to a namespace be granted OpenUnison will send out a notification via email. If you don't have an SMTP server but want to do some testing, try out the blackhole SMTP container we publish.

First, create a database in mariadb and give it a user. If you already deployed the login portal, it would be a good idea to remove it:

$ helm delete orchestra -n openunison

Next, edit the orchestra-secrets-source secret in the openunison namespace, it will look like:

apiVersion: v1
data:
  AD_BIND_PASSWORD: c3RhcnQxMjM=
  K8S_DB_SECRET: aW0gYSBzZWNyZXQ=
  unisonKeystorePassword: aW0gYSBzZWNyZXQ=
kind: Secret
metadata:
  creationTimestamp: "2020-05-04T18:25:37Z"
  name: orchestra-secrets-source
  namespace: openunison
  resourceVersion: "397465"
  selfLink: /api/v1/namespaces/openunison/secrets/orchestra-secrets-source
  uid: 2d9f7faf-4ae6-4684-8a1b-00ca3d5f6615
type: Opaque

Add to values to the data section, one for the password for the database and one for the password for your SMTP server. If your SMTP server doesn't require a password, use empty quotes.

apiVersion: v1
data:
  AD_BIND_PASSWORD: c3RhcnQxMjM=
  K8S_DB_SECRET: aW0gYSBzZWNyZXQ=
  OU_JDBC_PASSWORD: c3RhcnR0MTIz
  SMTP_PASSWORD: ""
  unisonKeystorePassword: aW0gYSBzZWNyZXQ=
kind: Secret
metadata:
  creationTimestamp: "2020-05-04T18:25:37Z"
  name: orchestra-secrets-source
  namespace: openunison
  resourceVersion: "3615758"
  selfLink: /api/v1/namespaces/openunison/secrets/orchestra-secrets-source
  uid: 2d9f7faf-4ae6-4684-8a1b-00ca3d5f6615
type: Opaque

Next, update values.yaml to point to the image to docker.io/tremolosecurity/openunison-k8s-activedirectory:latest and include the database and smtp sections:

network:
  openunison_host: "k8sou.apps.blog-demo.tremolo.dev"
  dashboard_host: "k8sdb.apps.blog-demo.tremolo.dev"
  api_server_host: "k8sapi.apps.blog-demo.tremolo.dev"
  session_inactivity_timeout_seconds: 900
  k8s_url: ""

cert_template:
  ou: "Kubernetes"
  o: "MyOrg"
  l: "My Cluster"
  st: "State of Cluster"
  c: "MyCountry"

image: "docker.io/tremolosecurity/openunison-k8s-activedirectory:latest"
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: kubernetes
enable_impersonation: true

dashboard:
  namespace: "kubernetes-dashboard"
  cert_name: "kubernetes-dashboard-certs"
  label: "k8s-app=kubernetes-dashboard"
  service_name: kubernetes-dashboard
certs:
  use_k8s_cm: false

trusted_certs:
  - name: ldaps
    pem_b64: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXekNDQWtPZ0F3SUJBZ0lFVHZDL0tEQU5CZ2txaGtpRzl3MEJBUXNGQURCZU1RMHdDd1lEVlFRR0V3UjAKWlhOME1RMHdDd1lEVlFRSUV3UjBaWE4wTVEwd0N3WURWUVFIRXdSMFpYTjBNUTB3Q3dZRFZRUUtFd1IwWlhOMApNUTB3Q3dZRFZRUUxFd1IwWlhOME1SRXdEd1lEVlFRREV3aGhjR0ZqYUdWa2N6QWVGdzB5TURBMU1ETXhNRFE1Ck1URmFGdzB6TURBMU1ERXhNRFE1TVRGYU1GNHhEVEFMQmdOVkJBWVRCSFJsYzNReERUQUxCZ05WQkFnVEJIUmwKYzNReERUQUxCZ05WQkFjVEJIUmxjM1F4RFRBTEJnTlZCQW9UQkhSbGMzUXhEVEFMQmdOVkJBc1RCSFJsYzNReApFVEFQQmdOVkJBTVRDR0Z3WVdOb1pXUnpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDCkFRRUFsZkZFcmtXRUdvakhUZ2w3WWgwVnFET0FMTUVIdWpVRWNMaFJVRGxvb0FiUm1meDJQNTg1blYxcEY3ZUoKbUdLSGNHbisyeFJDNjgySGUvMEsvYm9RcWFwRGNONThEbGd4RG5MZm93dVo4Q2JiUEhHTWlmRVJXT3RSYmFPZwpIYnBIUEdwWTJzS1Y0Z0xFSkRTSWxFeXVwMER6YlNZdGZHa2JsbjMvOStLWXFqMXZlOGRyQkpIVzNwdVJZQTV0CnEyaG5HUGhBL3dmOTNDU2x4NzNvdmNxTkRVK0ZKNGJPaUx3dU1JNFZhSkpmM0k5S0Y0V0QzSlVBYktjc3lBV0QKV3pTRG9Cb254NWpBYTFSUnNkMzVsV2NYWnpEYmNOSjdzTUMyMHNyLzZyQllEaE1KL1Z3SUFpd1ZmRDVxTkE4WQpVM1ZkaG9rQUh4N05pRkVVT1J0cnNzUmh6UUlEQVFBQm95RXdIekFkQmdOVkhRNEVGZ1FVY2srSE9kUG90TDZWCjhpdzNDUk9HYlphZlBWSXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSWloQkgrZlE4NEJvOTZJVVhJYWhydDgKSUFYdGo3NDhGWmZUOXM1Z0pHMm84NnBjWE9QVlM0MWlFRHhzOEo0THpWRy9HK2ZrYW9keWliTEFLTUpBYmRCRgpiM0UxNUZUTU9WZUE2N3BodzcvVzFwRWF0elIwODB4N0dkTUpQQXVOODQ3V3dRdlZ3bnVGKzRwK1g3RkxDU3ZXCnlWWGhOb3F1Z0xLbzJPSExCUEUrV3FUY1k2dTNOOTRTK1JwUGpLZG1SeXM0NldicHBuaU5FREFHMERBTWpOSk0KNThKcWJzOWtlSm5qTXNoWnI4dGYvQytScXhoZ1JYaGMwTFNtSGIrZy84V2R4ci9Kd1lneFo5RDRwcnpZRGw1ZgpnMC94bXFsOTB1UmlmN2tpclVQRUUyUmVNcG9jNjZ2YlhSVDRQUmRFa0RpbWJpMU9xdjEzYmJ4VlBKbmo0djA9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KCg==

monitoring:
  prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s

active_directory:
  base: cn=users,dc=domain,dc=com
  host: "apacheds.apacheds.svc.cluster.local"
  port: "636"
  bind_dn: "cn=ldapsa,cn=users,dc=domain,dc=com"
  con_type: ldaps
  srv_dns: "false"
       
database:
  hibernate_dialect: org.hibernate.dialect.MySQL5InnoDBDialect
  quartz_dialect: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
  driver: com.mysql.jdbc.Driver
  url: jdbc:mysql://mariadb.mariadb.svc.cluster.local:3306/unison
  user: unison
  validation: SELECT 1

smtp:
  host: blackhole.blackhole.svc.cluster.local
  port: 25
  user: donotreply@domain.com
  from: donotreply@domain.com
  tls: false
 

Finally, deploy the helm chart. Note that we're not deploying the login portal chart anymore:

helm install orchestra tremolo/openunison-k8s-activedirectory --namespace openunison -f /path/to/values.yaml

In a few minutes there will be two new pods running:

• amq-orchestra - A hardened version of Apache ActiveMQ for reliable messaging and workflow execution.
• openunison-orchestra - The OpenUnison pod.

All of the tables needed in the unison database have been populated by the operator and by OpenUnison. At this point you should be able to login by going to the host.

Bootstrap Initial User

OpenUnison doesn't come with a default administrative user. Default accounts can often be leveraged to compromise an application. In order to initialize your user as an administrative user, login to your database and run:

insert into userGroups (userId,groupId) values (2,1);

Logout of OpenUnison and log back in. There will be two new "badges" on the front page. The ActiveMQ management console is for managing the built in ActieMQ that comes with OpenUnison. The Operator's Console provides you a way to search for users and update their permissions. There are no badges for accessing the dashboard or tokens because your user isn't yet authorized for access. The next task is to make yourself a cluster administrator. Click on the Operator's Console and when it is loaded, check off "Login Id" and put in your username, then click "Search":

After clicking "Search", hit the check box next to your user. Choose "Kubernetes Administration" and click the "Run" next to "Cluster Administrator". Give a reason next to "Reason for Request" and click "Attempt Preapproval?". Make sure "Approved" is select and provide a reason. Finally click "Submit Workflow".

Once done, click on any on any of your user's attributes at the top of the screen (ie first name, last name, etc:

The group "k8s-cluster-administrators" is now on your profile. For the role to take effect, logout and log back in. Once logged back in, badges for the dashboard and tokens are available.

Creating New Namespaces

When a new namespace needs to be onboarded, as user can login to OpenUnison and request that it be created. The new namespace is approved by the cluster administrators. Once approved OpenUnison:

1. Creates the namespace
2. Creates groups in the database to control who can view or administrate the namespace
3. Creates RoleBinding objects for the cluster-admin and view ClusterRoles for the new groups
4. Create a group in the database that identifies who can approve access to the namespace
5. Adds the requester to the approval group and the namespace administrator group
6. Creates an audit trail of all the objects created

Once the namespace is created, administrators are no longer required to manage access to the namespace. Namespace owners are responsible for managing access. To demonstrate, logout of OpenUnison and login with a new user. The new user will only have the "New Kubernetes Namespace" badge available. Click on that badge:

The new screen will ask for the name of the namespace and the reason for its creation. Provide the information and click "Submit Registration". Once submitted, the request needs to be approved. Before logging out, Click on "Reports" in the title bar then choose "My Open Requests":

A report will come up letting you know who needs to approve the request for a new namespace:

Next, logout and login as your administrative user. In the upper right there will be a small red circle with a "1" in it next to the word "Open Approvals". Click on "Open Approvals".

Click on "Review" on the left next to the open approval:

After clicking on Review, go to the bottom of the page, provide a reason for the approval and click "Approve".

Next click on the "Confirm" button to complete the approval. In a few moments the new namespace is available and the owner will be notified that the namespace has been created.

On-boarding Users

One of the goals of this work is to easily onboard new users into namespaces without administrators having to field tickets or run kubectl commands. Using self service, where the users and namespace owners interact but the system administrators are not needed. Logout of OpenUnison and login as a new user. Once logged in click on "Request Access". Click on "Kubernetes Namespace Administrators" then click "Add to Cart" next to the newly created namespace.

After clicking "Add to Cart", go to "Check Out" and provide a reason and submit. Going to the "Reports" section shows that the creator of your namespace is assigned your access approval. That user would have received an email by now notifying them there's a pending request. Login with that user (the one that request the namespace be created), click on "Open Approvals" and approve the access request. Once the approval is submitted the user who requested access will get a notification telling them their access has been approved. Logout and log back in as this user. The user now has the dashboard and token badges available. Clicking on the dashboard will show several errors. Go to the namespace the user was approved access for and the user has administrative access just to that namespace!

Off-Boarding Users

At some point, there will be a need to off-board users. Namespace owners are able to do this without involving administrators. Login as the namespace owner (not the system administrator). Go to "Request Access", choose the namespace and click "Add to Cart". Go to the cart to check-out.

OpenUnison lets you request access on behalf of others, and if authorized preempt the approval decision. By denying access once the workflow completes the user will be removed the namespace's access group and the user will be notified. Preempting the authorization decision avoids the user needing to go through a second set of steps.

Audit and Reporting

In addition to generating the objects needed to manage access to your cluster, OpenUnison also generates audit artifacts that can be used to track the who, what, and why of each namespace. Login to OpenUnison as an administrator and click on "Reports". There will be a section called "Audit Reports". Click on it and the standard reports will be displayed.

Click on any of these reports will provide information directly out of OpenUnison's audit database. Reports can be exported to Excel. The reports are simple SQL queries. Using any business intelligence or reporting tool will work as well to provide finer controller reporting access.

In addition to SQL reports, OpenUnison sends all events to the text based logs that can be tracked and analyzed by any aggregation service. For instance in the logs:

[2020-06-01 17:52:26,570][ActiveMQ Session Task-207] INFO  ProvisioningEngineImpl - target=jitdb entry=false Delete user=rroberts workflow=ProjectAdministrators approval=4 group='k8s-namespace-administrators-jjackson-sandbox'

Shows when the user rroberts was removed from the k8s-namespace-administrators-jjackson-sandbox group in the jitdb datbaase as part of the ProjectAdministrators workflow.

What's Next?

This post focused on creating a baseline platform for self service on-boarding of namespaces and users. The next post in this series will focus on automating your pipeline creation. Amazon has several services that will make your EKS cluster work better that all require automation to be used effectively. We'll extend our automation outside of the cluster to our AWS account to build a more complete platform.

If you're interested in using OpenUnison to automate your EKS cluster access for identity sources other then Active Directory, take a look at the github repo for spins that support SAML2 and OpenID Connect.