Azure Deployment

This document describes the deployment of the required micro services into your Azure subscription. It is intended to be followed in linear order, skipping any steps not relevant to the particular deployment.

Architecture

arch-azure

Deployment Prerequisites

In order to get started, your Atolio support team will do the following on your behalf:

Grant access to Client ACR repos (for pulling Docker images) to your Azure subscription and provide image pull secrets.
Add your Deployment Engineer as a collaborator to the Atolio GitHub repository (lumen-infra), which contains:
- Deployment documentation
- Terraform for the Atolio stack infrastructure
- Configuration files for Atolio services
- Maintenance scripts

The following deployment prerequisites will help streamline your deployment process.

Determine Azure subscription

You can either choose to deploy Atolio into an existing Azure subscription or an existing one. Atolio will deploy into a new Azure Resource Group (RG), with another RG created automatically by Azure Kubernetes Service (AKS) for the cluster. When the subscription & RG are available, share the details with your Atolio support team.

We recommend:

Ensuring that Service Quotas within your Azure subscription allow for a minimum of 64 vCPU under the Total Regional vCPUs quota.
Raising any other organizational Azure policies / restrictions (e.g. networking, containers) with your Atolio support team ahead of the deployment call.

Determine Deployment Model

We offer both Atolio managed and customer managed deployment models for you to choose from. Please review the comparison and requirements for each approach on our Deployment Model Overview page and inform your Atolio support team which method you’d like to use for your deployment.

Atolio Managed Deployment Prerequisites

The Microsoft Role policy document we provision with our support role script is as follows:

{
    "Name": "Atolio Support Access",
    "Description": "Custom role for Atolio support engineers with minimal required permissions including AKS credential access",
    "AssignableScopes": [
        "/subscriptions/your-subscription-id"
    ],
    "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/write",
        "Microsoft.ContainerService/managedClusters/delete",
        "Microsoft.ContainerService/managedClusters/agentPools/read",
        "Microsoft.ContainerService/managedClusters/agentPools/write",
        "Microsoft.ContainerService/managedClusters/agentPools/delete",
        "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
        "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action",
        "Microsoft.Network/virtualNetworks/read",
        "Microsoft.Network/virtualNetworks/write",
        "Microsoft.Network/virtualNetworks/delete",
        "Microsoft.Network/virtualNetworks/subnets/read",
        "Microsoft.Network/virtualNetworks/subnets/write",
        "Microsoft.Network/virtualNetworks/subnets/delete",
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/publicIPAddresses/read",
        "Microsoft.Network/publicIPAddresses/write",
        "Microsoft.Network/publicIPAddresses/delete",
        "Microsoft.Network/applicationGateways/read",
        "Microsoft.Network/applicationGateways/write",
        "Microsoft.Network/applicationGateways/delete",
        "Microsoft.Network/dnsZones/read",
        "Microsoft.Network/dnsZones/write",
        "Microsoft.Network/dnsZones/delete",
        "Microsoft.Network/dnsZones/SOA/read",
        "Microsoft.Storage/storageAccounts/read",
        "Microsoft.Storage/storageAccounts/write",
        "Microsoft.Storage/storageAccounts/delete",
        "Microsoft.Storage/storageAccounts/listKeys/action",
        "Microsoft.Storage/storageAccounts/fileServices/read",
        "Microsoft.Storage/storageAccounts/fileServices/shares/read",
        "Microsoft.Storage/storageAccounts/blobServices/read",
        "Microsoft.Storage/storageAccounts/blobServices/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/read",
        "Microsoft.Storage/storageAccounts/blobServices/containers/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/delete",
        "Microsoft.Compute/disks/read",
        "Microsoft.Compute/disks/write",
        "Microsoft.Compute/disks/delete",
        "Microsoft.ManagedIdentity/userAssignedIdentities/read",
        "Microsoft.ManagedIdentity/userAssignedIdentities/write",
        "Microsoft.ManagedIdentity/userAssignedIdentities/delete",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/read",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/write",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/delete",
        "Microsoft.Authorization/roleAssignments/read",
        "Microsoft.Authorization/roleAssignments/write",
        "Microsoft.Authorization/roleAssignments/delete",
        "Microsoft.Resources/subscriptions/resourceGroups/read",
        "Microsoft.Resources/subscriptions/resourcegroups/write",
        "Microsoft.Resources/subscriptions/resourcegroups/delete"
    ],
    "NotActions": [
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Authorization/roleDefinitions/write",
        "Microsoft.Authorization/roleDefinitions/delete"
    ],
    "DataActions": [],
    "NotDataActions": [],
    "condition": "@iPAddress() matches '^52.43.209.253$'",
    "conditionVersion": "2.0"
}

Access will be limited to a client support machine that is only accessible to Atolio support engineers and that has a static IP of 52.43.209.253 assigned to aid in identifying activity from Atolio’s team.

If you opt to allow Atolio’s deployment support team to manage the deployment on your behalf the steps to enable this for your Azure subscription are as follows:

Clone our lumen-infra GitHub repository

git clone git@github.com:atolio/lumen-infra.git

Run our Azure support role script against the Azure subscription you’d like us to deploy into
```
./lumen-infra/deploy/terraform/azure/scripts/atolio-support-role.sh
```

Review the output from the script and provide the details to your Atolio support team

Operation completed successfully.
Please securely share the following details with your Atolio support engineer:

Tenant ID: <tenant ID>
Subscription ID: <subsciption ID>
Application (Client) ID: <client app ID>
Client Secret: <client secret>
Secret Expiry Date: <expiry date>

These credentials grant access to the specified Azure subscription with the
custom role 'Atolio Support Access'. The client secret will expire on <expiry date>.

Determine Atolio DNS name

Before the deployment call, you may want to decide on your desired Atolio web location. An Azure DNS Zone will be created in the Azure subscription used for hosting the Atolio stack (e.g. search.example.com.): this will be the DNS name (without the trailing dot) for the Atolio Web application (e.g. https://search.example.com).

For the remainder of this document, we will use https://search.example.com in the examples, but it is expected for you to replace with your own DNS name.

Determine Cloud Networking Options

By default, Atolio’s Terraform code will create a VNet in your Azure subscription. However, you may choose to use an existing VNet and subnets within your Azure subscription. In this case, set create_vnet to false in the config.hcl file during the deployment.

Then, configure all VPC related variables. See this example from config.hcl:

// Uncomment these lines and update the values in case you want to deploy in a
// pre-existing VNet (by default a new VNet will be created).
//
// create_vnet = false
// vnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}"
// vnet_private_subnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{privateSubnetName}"
// vnet_public_subnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{publicSubnetName}"

Optionally, when create_vnet is not set to false you can customize the default VNet and Subnet CIDR blocks if you have specific requirements. See this example from config.hcl:

// Uncomment these lines and update the values if you want to customize the
// default CIDR blocks for the VNet and Subnets created automatically.
//
// Note: this only applies for when create_vnet hasn't been set to false and
// any values modified here will be ignored when create_vnet=false.
//
// vnet_cidr = "10.42.0.0/20"
// pod_subnet_block = "10.42.0.0/22"
// appgw_subnet_block = "10.42.4.0/24"

If custom networking configuration will be necessary be sure to provide these details to the engineer performing the deployment.

Setup Authentication

Atolio supports single sign-on (SSO) authentication through Okta, Microsoft Entra ID, and Google using the OpenID Connect (OIDC) protocol.

Refer to Configuring Authentication for more details on the steps to complete in your desired SSO provider in order to obtain the necessary OIDC configuration values.

If using Microsoft Entra ID, the oidc_client_id and oidc_client_secret will be the respective values created and saved during Azure AD - Create New App Registration.

Setup local environment

Finally, if an engineer from your team will be performing the deployment, ensure they have the following utilities installed:

Setup Terraform on your local machine as described on the HashiCorp docs site - we require v1.5.0 at a minimum.
Install the Azure Command Line Interface
Install kubectl
Install Helm
Install atolioctl

Note: If you are running on Windows, you may also need to install the Windows Subsystem for Linux.

Provide Deployment Engineer with Configuration

At this point if you’re proceeding with an Atolio managed deployment you’ll need to provide the information from these prerequisite steps to your Atolio deployment support team. Otherwise please ensure the information is provided to the engineer from your organization who will be performing the deployment with Atolio’s support.

To recap, provide these details:

Atolio DNS name
OIDC Configuration
(Optional) Custom Cloud Networking Options

If you’re opting to have Atolio manage the deployment you can disregard the remainder of this documentation as these steps will be performed by your Atolio deployment engineer. Otherwise be sure to share this documentation with the engineer from your organization that will be performing the deployment so they can familarize theirself with the steps required.

Create Cloud Infrastructure

Note: Atolio requires an Azure region with 3 availability zones. You can check which regions include support for multiple availability zones here.

The Terraform configuration requires an Azure storage account to store state. A script is available to automate the whole process (including running Terraform). Before running the script, create a config.hcl file based on the provided config.hcl.template:

cd deploy/terraform/azure
cp ./config.hcl.template config.hcl

Atolio Domain Name and Image Pull Secrets

Update the copied file with appropriate values. At a minimum, it should look something like this:

// Domain name for Atolio stack (same as hosted zone name without trailing ".")
lumen_domain_name = "search.example.com"

// The registry in which to obtain containers for services
container_registry = "atolioimages.azurecr.io"
image_pull_username = "provided-by-atolio"
image_pull_password = "provided-by-atolio"

Your Atolio support team will share the appropriate values for the image_pull_username and image_pull_password values.

Application Helm Value Options

Next copy the Helm template and update the values as directed.

cp ./templates/values-lumen-admin.yaml values-lumen.yaml
cp ./templates/values-vespa-admin.yaml values-vespa.yaml

# Default values for lumen.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates (provided by admin user).

# JWT secret key for API call (signature) verification (at least 256 bits / 32 chars)
# Can be generated by running `openssl rand -base64 32` in your terminal
jwtSecretKey: "add-your-jwt-secret-key-here"

# Secret salts for generating Vespa document IDs
# Can be generated by running `openssl rand -base64 32` in your terminal
secretSalts: "add-your-secret-salts-here"

# See also scripts/config-oidc.sh helper script to obtain some of the values below
oidc:
  provider: "add-your-provider-here"
  endpoint: "add-your-endpoint-here"
  clientId: "add-your-id-here"
  clientSecret: "add-your-secret-here"

# Use the Atolio provided image pull secrets for accessing Atolio container images
imagePullSecret:
  create: true

# If running behind a reverse proxy, this should be set to the URL the end user will
# use to access the product.
reverseProxyUrl: ""

# The ACME Cluster Issuer for LetsEncrypt requires an email address to be provided
# for certificate notifications. This is required for LetsEncrypt certificates to work properly.
# Common examples would be an admin or technical support email address for your organization. ex: "admin@example.com" or "engineering@example.com".
letsencrypt:
  email: "letsencrypt@example.com"
  dns01:
    azureDNS:
      environment: AzurePublicCloud

For the jwtSecretKey and secretSalts values any 256 bit (32 character) string can be used. These values are used to sign JWT tokens used by the web application and atolioctl tool and salt document IDs in Atolio’s database. They should be well guarded secrets that are unique to the deployment.

If your users will be accessing the web interface via a reverse proxy (e.g. such as StrongDM), then be sure to set the reverseProxyUrl field to reflect the URL they will actually enter into their browser to access Atolio, which will be different to the hostname defined in lumen_domain_name. Leave this field empty if not using a reverse proxy.

Deployment with create-infra.sh script

Once you have all variables configured to your environment’s requirements you can create the infrastructure and deploy the k8s cluster. From the deploy/terraform/azure directory:

./scripts/create-infra.sh --name=deployment-name

This will create the infrastructure in the westus3 Azure region. If you want to deploy in another region parameter (e.g. eastus) an additional parameter can be provided:

./scripts/create-infra.sh --name=deployment-name --region=eastus

The deployment-name argument is used to define a deployment name for collecting resources into an Azure Resource Group containing the kubernetes cluster, networking, storage, etc. We recommend making it unique across all deployments, i.e. using a globally unique deployment name. Typically this is set to your company name with an optional suffix if specifying environment (e.g. acmecorp or acmecorp-qa).

The script automates the following steps (parameterized based on the provided deployment name):

Create an Azure Blob Storage to store Terraform state
Create a terraform.tfvars file for Terraform
Run terraform init
Run terraform apply (using input variables in generated terraform.tfvars)

Post-deployment steps

With the infrastructure created, you’ll want to update your local kubeconfig with a context for the Atolio cluster (this is also output via Terraform as update_kubeconfig_command):

az aks get-credentials --name=lumen-{deployment-name} --resource-group lumen-{deployment-name}

Delegate responsibility for Atolio subdomain

The parent domain (e.g. example.com) needs to delegate traffic to the new Atolio subdomain (search.example.com). This is achieved by adding an NS record to the parent domain with the 4 name servers copied from the new subdomain.

These nameservers can be retrieved post-creation with terraform output:

terraform output --json name_servers | jq -r '.[]'

At this point you should be able to interact with the kubernetes cluster, e.g.

kubectl get po -n atolio-svc

Note, Atolio specific services run on the following namespaces:

atolio-svc (Services)
atolio-db (Database)
atolio-ctl (Control Plane)

When you have validated that the infrastructure is available, the next step is to configure sources.