Self Managed Camunda Running on an AKS Cluster on Azure

In this post I will setup an AKS cluster, install NGINX as the ingress controller, install a standalone Keycloak instance and then install the self managed Camunda plaform.

You can learn more about Camunda here

Required Software

Ensure you have an Azure subscription you can use. If you have an MSDN subscription don’t forget to activate your monthly Azure Subscription you get with that MSDN.

You will also need the following software installed

Azure CLI
Helm
Kubectl
Valid TLS/SSL Certificate. One can be obtained from LetsEncrypt - see my previous post here

I develop on a windows machine and use the Windows PowerShell ISE for development. I am sure there are more funkier IDEs, but this does the job for me! For ease of simplicity I will create one large PowerShell script that can be executed line by line so you can see what is going on.

This script can be found here - again step through it line by line to exectute the script.

Variables

Lets start with declaring the variables we will use and that can be customised based on your needs.

## Set Variables

$AZ_RESOURCE_GROUP="demo-camunda" #Your resource group
$AZ_LOCATION="northeurope" #What location you want your resources created in
$AZ_CLUSTER_NAME="demo-camunda-aks" #Name of your AKS Cluster
$AZ_VNET_NAME="demo-camunda-vnet" #Name of your VNET
$AZ_VNET_CIDR="10.222.0.0/16" #CIDR of your VNET
$AZ_AKS_CIDR="10.222.0.0/22" #CIDR of your AKS Cluster Nodes
$AZ_AKS_NAME="subnet-aks" 
$AZ_SVC_LB_CIDR="10.222.4.0/28" #CIDR of your Load Balancer
$AZ_SVC_LB_NAME="subnet-lb"
$AZ_AKS_Service_CIDR="10.223.0.0/16" #CIDR of the AKS Services
$AZ_AKS_DNS_CIDR="10.223.0.10" #IP address of the internal AKS DNS Server
$AZ_USER_ASSIGNED_IDENTITY_NAME="demo-camunda-managed-id" #Managed Identity name. Another Managed identity gets created in node pool resource group when creating the AKS cluster
$AZ_SERVICE_ACCOUNT_NAME="workload-identity-sa" # Workflow Identity service account that is created in the AKS Cluster
$AZ_SERVICE_ACCOUNT_NAMESPACE="default" #Namespace to use for your applications and service account
$AZ_FEDERATED_IDENTITY_CREDENTIAL_NAME="demo-camunda-federated-credential" #Federated identity credential name that will be linked with the workflow service account
$AZ_KeyVault_Name="demo-camunda-keyvault" #Keyvault name
$AZ_PublicIP_Name="demo-camunda-ingress-pip" #Public IP address name
$AZ_DNS_Label="demo-camunda" #DNS Label to attach to the public IP address
$AZ_IP_Address_Allow="" #Your IP address to whitelist in the network settings
$AZ_CertificateName="" #Your Certificate name for TLS termination
$AZ_CertificatePath="" #Path to your TLS certificate. The cert needs to be in pfx format.
$AZ_KeyVault_Certificates_Officer="" #Your azure account so you can add the certificate to the key vault

Account and Subscription

Open up your IDE/Terminal of choice and login in to Azure.

az login
az account show

Then set your default account to the subscription you want to use. Make sure you have the correct permission on the subscription to create resources.

az account set -s {Subscription Id}

Create Resources

az group create --resource-group $AZ_RESOURCE_GROUP --location $AZ_LOCATION

# Create Vnet
az network vnet create -g $AZ_RESOURCE_GROUP -n $AZ_VNET_NAME --address-prefix $AZ_VNET_CIDR

# Create Azure AKS Cluster Subnet
az network vnet subnet create --resource-group $AZ_RESOURCE_GROUP --vnet-name $AZ_VNET_NAME --name $AZ_AKS_NAME --address-prefix $AZ_AKS_CIDR

# Create the subnet for Kubernetes Service Load Balancers
az network vnet subnet create --resource-group $AZ_RESOURCE_GROUP --vnet-name $AZ_VNET_NAME --name $AZ_SVC_LB_NAME --address-prefix $AZ_SVC_LB_CIDR

#Add the service endpoints needed for all services to the AKS Subnet
az network vnet subnet update -g $AZ_RESOURCE_GROUP -n $AZ_AKS_NAME --vnet-name $AZ_VNET_NAME --service-endpoints Microsoft.KeyVault

#Create Keyvault for storing your TLS Certificate
az keyvault create -l $AZ_LOCATION -g $AZ_RESOURCE_GROUP -n $AZ_KeyVault_Name --enable-rbac-authorization --sku Standard --default-action Deny

#Get the scope to the Key Vault.
$AZ_KeyVault_Scope=$(az keyvault show --resource-group $AZ_RESOURCE_GROUP --name $AZ_KeyVault_Name --query id --output tsv)

#Get the SubnetID for use later in allowing access to the key vault
$AZ_SUBNET_ID=$(az network vnet show -g $AZ_RESOURCE_GROUP -n $AZ_VNET_NAME -o tsv --query "subnets[?name=='$AZ_AKS_NAME'].id")

#Grant access to the Keyvault that has the TLS Certificate to the AKS Subnet
az keyvault network-rule add -n $AZ_KeyVault_Name  -g $AZ_RESOURCE_GROUP  --subnet $AZ_SUBNET_ID

#Grant access to your IP Address
az keyvault network-rule add -n $AZ_KeyVault_Name -g $AZ_RESOURCE_GROUP --ip-address $AZ_IP_Address_Allow

#Grant your account access to key vault to import certificates
az role assignment create --assignee $AZ_KeyVault_Admin --role "Key Vault Administrator" --scope $AZ_KeyVault_Scope

You should now have your keyvault created and limited access to the VNET and your IP address. You can in the Azure portal.

Next steps are to create the AKS cluster with the azure-keyvault-secrets-provider enabled. This will allow pods on your cluster access your key vault secrets. More information can be found here

#Create Public IP
az network public-ip create -g $AZ_RESOURCE_GROUP -n $AZ_PublicIP_Name --allocation-method Static --sku Standard --dns-name $AZ_DNS_Label

#Get that Public IP to use later in the Igress setup
$AZ_LB_IP=$(az network public-ip show -g $AZ_RESOURCE_GROUP -n $AZ_PublicIP_Name -o tsv --query ipAddress)

#Create Azure Kubernetes Service cluster. Note the VM size as per the recommendations per Camunda
az aks create --resource-group $AZ_RESOURCE_GROUP --name $AZ_CLUSTER_NAME --generate-ssh-keys --vm-set-type VirtualMachineScaleSets `
  --node-vm-size "Standard_D4as_v4" `
  --load-balancer-sku standard `
  --enable-managed-identity `
  --enable-oidc-issuer `
  --enable-addons azure-keyvault-secrets-provider `
  --network-plugin azure `
  --network-policy azure `
  --vnet-subnet-id $AZ_SUBNET_ID `
  --node-count 4 `
  --zones 1 `
  --service-cidr $AZ_AKS_Service_CIDR `
  --dns-service-ip $AZ_AKS_DNS_CIDR `

Once this command completes you should have a running AKS cluster with managed identity and the azure-keyvault-secrets-provider enabled.

We will now create a managed identity - see here for more information on managed identities.

#Create the Managed Identity
az identity create --name $AZ_USER_ASSIGNED_IDENTITY_NAME --resource-group $AZ_RESOURCE_GROUP --location $AZ_LOCATION --subscription $AZ_SUBSCRIPTION

#Get the Client ID of the managed Identity
$AZ_USER_ASSIGNED_CLIENT_ID=$(az identity show -n $AZ_USER_ASSIGNED_IDENTITY_NAME -g $AZ_RESOURCE_GROUP --query "clientId" -otsv)


#Get the AKS OIDC_Issuer URl
$AZ_AKS_OIDC_ISSUER=$(az aks show -n $AZ_CLUSTER_NAME -g $AZ_RESOURCE_GROUP --query oidcIssuerProfile.issuerUrl -o tsv)

#Create the federated identity credential between the managed identity, service account issuer
az identity federated-credential create --name $AZ_FEDERATED_IDENTITY_CREDENTIAL_NAME --identity-name $AZ_USER_ASSIGNED_IDENTITY_NAME --resource-group $AZ_RESOURCE_GROUP --issuer $AZ_AKS_OIDC_ISSUER --subject system:serviceaccount:${AZ_SERVICE_ACCOUNT_NAMESPACE}:${AZ_SERVICE_ACCOUNT_NAME}

#Grant the managed-identity access to Key Vault  where TLS cert is stored
$AZ_KeyVault_Scope=$(az keyvault show --resource-group $AZ_RESOURCE_GROUP --name $AZ_KeyVault_Name --query id --output tsv)

az role assignment create --role "Key Vault Administrator" --assignee $AZ_USER_ASSIGNED_CLIENT_ID --scope $AZ_KeyVault_Scope 

#Need to give the managed idenity the delegated permissions to the resource group the public IP address resides in
#Give the managed id reader role to the resource group
$AZ_RESOURCE_GROUP_SCOPE=$(az group show -n $AZ_RESOURCE_GROUP --query id -o tsv)
$AZ_Client_Identity=$(az aks show -n $AZ_CLUSTER_NAME -g $AZ_RESOURCE_GROUP --query identity.principalId -o tsv)

az role assignment create --assignee $AZ_Client_Identity --role "Network Contributor" --scope $AZ_RESOURCE_GROUP_SCOPE

We will now get the AKS Cluster credentials and add it to our local kubeconfig


#Set the AKS Cluster
az aks get-credentials -n $AZ_CLUSTER_NAME -g $AZ_RESOURCE_GROUP
kubectl config use-context $AZ_CLUSTER_NAME

#Create ingress namespace - can be whatever you want to call it
kubectl create namespace ingress-basic

#Get the Azure Tenant ID for your subscription
$AZ_AZURE_TENANT_ID=$(az account show -s $AZ_SUBSCRIPTION --query tenantId -otsv)

Now we are helming so install Helm and add it to yuor Path environment variables. When installed and ready to use continue with the next step


#Install the mutating webhook for the Azure Workload Identity
helm repo add azure-workload-identity https://azure.github.io/azure-workload-identity/charts
helm repo update
helm install workload-identity-webhook azure-workload-identity/workload-identity-webhook `
   --namespace azure-workload-identity-system `
   --create-namespace `
   --set azureTenantID="$AZ_AZURE_TENANT_ID"


#Create the workload identity service acccount in AKS
$ServiceAccountString = @"
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: $AZ_USER_ASSIGNED_CLIENT_ID
  name: $AZ_SERVICE_ACCOUNT_NAME
  namespace: $AZ_SERVICE_ACCOUNT_NAMESPACE
"@

$ServiceAccountString | kubectl apply -f -