Cloud Deployment

• 1: AWS Deployment
• 2: Azure Deployment
• 3: GCP Deployment
• 4: Hardware Requirements
• 5: Atolio Managed vs. Customer Managed Deployment Models
• 6: Operations Best Practices

1 - AWS Deployment

Architecture

Deployment Prerequisites

In order to get started, your Atolio support team will do the following on your behalf:

1. Grant your AWS account access to the Client ECR repos (for pulling Docker images).
2. Add your Deployment Engineer as a collaborator to the Atolio GitHub repository (lumen-infra), which contains:
   • Deployment documentation
   • Terraform for the Atolio stack infrastructure
   • Configuration files for Atolio services
   • Maintenance scripts

The following deployment prerequisites will help streamline your deployment process.

Determine AWS account

You can either choose to deploy Atolio into an existing AWS account or a new account. Atolio also supports deploying to your own AWS Virtual Private Cloud (VPC). When the account is available, share the AWS account number with your Atolio support team.

We recommend:

• Ensuring that Service Quotas within your AWS account allow for a minimum of 64 vCPU for On-Demand Standard instances. See Hardware Requirements for the full list of recommended quotas.
• Raising any other organizational AWS policies / restrictions (e.g. networking, containers) with your Atolio support team ahead of the deployment call.

Determine Deployment Model

We offer both Atolio managed and customer managed deployment models for you to choose from. Please review the comparison and requirements for each approach on our Deployment Model Overview page and inform your Atolio support team which method you’d like to use for your deployment.

Atolio Managed Deployment Prerequisites

The exact permissions being delegated will be presented to the engineer running the script prior to executing. The IAM policies included are:

arn:aws:iam::aws:policy/PowerUserAccess
arn:aws:iam::aws:policy/IAMFullAccess

Access will be limited to a client support machine that is only accessible to Atolio support engineers and that has a static IP of 52.43.209.253 assigned to aid in identifying activity from Atolio’s team.

If you opt to allow Atolio’s deployment support team to manage the deployment on your behalf the steps to enable this for your AWS account are as follows:

1. Clone our lumen-infra GitHub repository

   git clone git@github.com:atolio/lumen-infra.git
   

2. Run our AWS support role script against the AWS account you’d like us to deploy into

   ./lumen-infra/deploy/terraform/aws/scripts/atolio-support-role.sh
   

3. Review the output from the script and provide the Role ARN to your Atolio support team

   Operation completed successfully.
   Role ARN: arn:aws:iam::123456789012:role/AtolioDeploymentAccess
   

Determine Atolio DNS name

Before the deployment call, you may want to decide on your desired Atolio web location. An AWS Route 53 hosted zone will be created in the AWS account used for hosting the Atolio stack (e.g. search.example.com.): this will be the DNS name (without the trailing dot) for the Atolio web application (e.g. https://search.example.com)

This hosted zone allows the deployment (i.e. the External DNS controller) to add records to link host names (e.g. search.example.com, feed.search.example.comand relay.search.example.com) to the load balancer as created by the AWS ALB controller.

For the remainder of this document, we will use https://search.example.com in the examples, but it is expected for you to replace with your own DNS name.

Determine Cloud Networking Options

By default, Atolio’s Terraform code will create a VPC. However, you may choose to use an existing VPC and subnets within your AWS account. In this case, set create_vpc to false.

Then, configure all VPC related variables. See below sample:

// Uncomment these lines and update the values in case you want to deploy in a
// pre-existing VPC (by default a new VPC will be created).
//
// Note that automatic subnet discovery for the ALB controller will only work
// if the subnets are tagged correctly as documented here:
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/deploy/subnet_discovery/
// create_vpc             = false
// vpc_id                 = "vpc-000"
// vpc_cidr_block         = "10.42.0.0/20"
// vpc_private_subnet_ids = ["subnet-1111", "subnet-2222"]
// vpc_public_subnet_ids  = ["subnet-3333", "subnet-4444"]
// vespa_az                = "us-west-2a"
// vespa_private_subnet_id = "subnet-1111"

Additional notes regarding existing VPC usage:

• As per above sample, subnets must be tagged corectly as documented in subnet discovery.
• When specifying vespa_private_subnet_id, the referenced subnet ID must also be in the vpc_private_subnet_ids array.
• In terms of VPC sizing, the default of 10.42.0.0/20 (4,096 addresses) is consistent across AWS, Azure, and GCP deployments and provides ample room for growth. VPC subnet IP addresses are primarily allocated to the EKS cluster and ALB, with AWS reserving several for internal services. We recommend a subnet of /22 (1,024 IPs) as the minimum to ensure enough available IP addresses for Kubernetes to assign to pods as your deployment scales — a /20 VPC fits four /22 subnets, which covers the standard public/private split across availability zones. Smaller subnet sizes may lead to IPv4 address exhaustion as pod counts increase.
• Ensure specified subnets have available IPv4 Addresses.

If custom networking configuration will be necessary be sure to provide these details to the engineer performing the deployment.

Setup authentication

Atolio supports single sign-on (SSO) authentication through Okta, Microsoft Entra ID, and Google using the OpenID Connect (OIDC) protocol.

Refer to Configuring Authentication for more details on the steps to complete in your desired SSO provider in order to obtain the necessary OIDC configuration values.

Setup local environment

Finally, if an engineer from your team will be performing the deployment, ensure they have the following utilities installed:

• Setup Terraform on your local machine as described on the HashiCorp docs site - we require v1.5.0 at a minimum.
• Install the AWS Command Line Interface
• Install kubectl
• Install Helm
• Install atolioctl

Provide Deployment Engineer with Configuration

At this point if you’re proceeding with an Atolio managed deployment you’ll need to provide the information from these prerequisite steps to your Atolio deployment support team. Otherwise please ensure the information is provided to the engineer from your organization who will be performing the deployment with Atolio’s support.

To recap, provide these details:

• Atolio DNS name
• OIDC Configuration
• (Optional) Custom Cloud Networking Options

If you’re opting to have Atolio manage the deployment you can disregard the remainder of this documentation as these steps will be performed by your Atolio deployment engineer. Otherwise be sure to share this documentation with the engineer from your organization that will be performing the deployment so they can familarize theirself with the steps required.

Create Cloud Infrastructure

The Terraform configuration requires an external (S3) bucket to store state. A script is available to automate the whole process (including running Terraform). Before running the script, create a config.hcl file based on the provided config.hcl.template:

cd deploy/terraform/aws
cp ./config.hcl.template config.hcl

Atolio Domain Name

Update the copied file with appropriate values. At a minimum, it should look something like this:

# Domain name for Atolio stack (same as hosted zone name without trailing ".")
lumen_domain_name = "search.example.com"

Application Helm Value Options

Next copy the Helm template and update the values as directed.

cp ./templates/values-lumen-admin.yaml values-lumen.yaml
cp ./templates/values-vespa-admin.yaml values-vespa.yaml

# Default values for lumen.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates (provided by admin user).

# JWT secret key for API call (signature) verification (at least 256 bits / 32 chars)
# Can be generated by running `openssl rand -base64 32` in your terminal
jwtSecretKey: "add-your-jwt-secret-key-here"

# Secret salts for generating Vespa document IDs
# Can be generated by running `openssl rand -base64 32` in your terminal
secretSalts: "add-your-secret-salts-here"

# See also scripts/config-oidc.sh helper script to obtain some of the values below
oidc:
  provider: "add-your-provider-here"
  endpoint: "add-your-endpoint-here"
  clientId: "add-your-id-here"
  clientSecret: "add-your-secret-here"

# If running behind a reverse proxy, this should be set to the URL the end user will
# use to access the product.
reverseProxyUrl: ""

# The ACME Cluster Issuer for LetsEncrypt requires an email address to be provided
# for certificate notifications. This is required for LetsEncrypt certificates to work properly.
# Common examples would be an admin or technical support email address for your organization. ex: "admin@example.com" or "engineering@example.com".
letsencrypt:
  email: "letsencrypt@example.com"

For the jwtSecretKey and secretSalts values any 256 bit (32 character) string can be used. These values are used to sign JWT tokens used by the web application and atolioctl tool and salt document IDs in Atolio’s database. They should be well guarded secrets that are unique to the deployment.

If your users will be accessing the web interface via a reverse proxy (e.g. such as StrongDM), then be sure to set the reverseProxyUrl field to reflect the URL they will actually enter into their browser to access Atolio, which will be different to the hostname defined in lumen_domain_name. Leave this field empty if not using a reverse proxy.

Deployment with create-infra.sh script

Once you have all variables configured, you can create the infrastructure and deploy the k8s cluster. From the deploy/terraform/aws directory:

./scripts/create-infra.sh --name=deployment-name

This will create the infrastructure in the us-west-2 AWS region. If you want to deploy in another region parameter (e.g. us-east-1) an additional parameter can be provided:

./scripts/create-infra.sh --name=deployment-name --region=us-east-1

The deployment-name argument is used to generate a deployment name for e.g. tagging resources and naming e.g. the kubernetes cluster and S3 buckets. So make sure it is unique across all deployments. (i.e. using a globally unique deployment name). Typically this is set to your company name with an optional suffix if specifying environment (e.g. acmecorp or acmecorp-qa).

The script automates the following steps (parameterized based on the provided deployment name):

1. Create an S3 bucket to store Terraform state
2. Create a terraform.tfvars file for Terraform
3. Run terraform init
4. Run terraform apply (using input variables in generated terraform.tfvars)

Post-deployment steps

With the infrastructure created, you’ll want to update your local kubeconfig with a context for the Atolio cluster (this is also output via Terraform as update_kubeconfig_command):

aws --profile {atolio profile} eks update-kubeconfig --region us-west-2 --name lumen-{deployment-name}

Delegate responsibility for Atolio subdomain

The parent domain (e.g. example.com) needs to delegate traffic to the new Atolio subdomain (search.example.com). This is achieved by adding an NS record to the parent domain with the 4 name servers copied from the new subdomain (similar to what is described here).

These nameservers can be retrieved post-creation with terraform output:

terraform output --json name_servers | jq -r '.[]'

At this point you should be able to interact with the kubernetes cluster, e.g.

kubectl get po -n atolio-svc

Note, Atolio specific services run on the following namespaces:

• atolio-svc (Services)
• atolio-db (Database)
• atolio-ctl (Control Plane)

When you have validated that the infrastructure is available, the next step is to configure sources.

2 - Azure Deployment

Architecture

Deployment Prerequisites

In order to get started, your Atolio support team will do the following on your behalf:

1. Grant access to Client ACR repos (for pulling Docker images) to your Azure subscription and provide image pull secrets.
2. Add your Deployment Engineer as a collaborator to the Atolio GitHub repository (lumen-infra), which contains:
   • Deployment documentation
   • Terraform for the Atolio stack infrastructure
   • Configuration files for Atolio services
   • Maintenance scripts

The following deployment prerequisites will help streamline your deployment process.

Determine Azure subscription

You can either choose to deploy Atolio into an existing Azure subscription or an existing one. Atolio will deploy into a new Azure Resource Group (RG), with another RG created automatically by Azure Kubernetes Service (AKS) for the cluster. When the subscription & RG are available, share the details with your Atolio support team.

We recommend:

• Ensuring that Service Quotas within your Azure subscription allow for a minimum of 64 vCPU under the Total Regional vCPUs quota. See Hardware Requirements for the full list of recommended quotas.
• Raising any other organizational Azure policies / restrictions (e.g. networking, containers) with your Atolio support team ahead of the deployment call.

Determine Deployment Model

We offer both Atolio managed and customer managed deployment models for you to choose from. Please review the comparison and requirements for each approach on our Deployment Model Overview page and inform your Atolio support team which method you’d like to use for your deployment.

Atolio Managed Deployment Prerequisites

The Microsoft Role policy document we provision with our support role script is as follows:

{
    "Name": "Atolio Support Access",
    "Description": "Custom role for Atolio support engineers with minimal required permissions including AKS credential access",
    "AssignableScopes": [
        "/subscriptions/your-subscription-id"
    ],
    "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/write",
        "Microsoft.ContainerService/managedClusters/delete",
        "Microsoft.ContainerService/managedClusters/agentPools/read",
        "Microsoft.ContainerService/managedClusters/agentPools/write",
        "Microsoft.ContainerService/managedClusters/agentPools/delete",
        "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
        "Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action",
        "Microsoft.Network/virtualNetworks/read",
        "Microsoft.Network/virtualNetworks/write",
        "Microsoft.Network/virtualNetworks/delete",
        "Microsoft.Network/virtualNetworks/subnets/read",
        "Microsoft.Network/virtualNetworks/subnets/write",
        "Microsoft.Network/virtualNetworks/subnets/delete",
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/publicIPAddresses/read",
        "Microsoft.Network/publicIPAddresses/write",
        "Microsoft.Network/publicIPAddresses/delete",
        "Microsoft.Network/applicationGateways/read",
        "Microsoft.Network/applicationGateways/write",
        "Microsoft.Network/applicationGateways/delete",
        "Microsoft.Network/dnsZones/read",
        "Microsoft.Network/dnsZones/write",
        "Microsoft.Network/dnsZones/delete",
        "Microsoft.Network/dnsZones/SOA/read",
        "Microsoft.Storage/storageAccounts/read",
        "Microsoft.Storage/storageAccounts/write",
        "Microsoft.Storage/storageAccounts/delete",
        "Microsoft.Storage/storageAccounts/listKeys/action",
        "Microsoft.Storage/storageAccounts/fileServices/read",
        "Microsoft.Storage/storageAccounts/fileServices/shares/read",
        "Microsoft.Storage/storageAccounts/blobServices/read",
        "Microsoft.Storage/storageAccounts/blobServices/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/read",
        "Microsoft.Storage/storageAccounts/blobServices/containers/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/delete",
        "Microsoft.Compute/disks/read",
        "Microsoft.Compute/disks/write",
        "Microsoft.Compute/disks/delete",
        "Microsoft.ManagedIdentity/userAssignedIdentities/read",
        "Microsoft.ManagedIdentity/userAssignedIdentities/write",
        "Microsoft.ManagedIdentity/userAssignedIdentities/delete",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/read",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/write",
        "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/delete",
        "Microsoft.Authorization/roleAssignments/read",
        "Microsoft.Authorization/roleAssignments/write",
        "Microsoft.Authorization/roleAssignments/delete",
        "Microsoft.Resources/subscriptions/resourceGroups/read",
        "Microsoft.Resources/subscriptions/resourcegroups/write",
        "Microsoft.Resources/subscriptions/resourcegroups/delete"
    ],
    "NotActions": [
        "Microsoft.Authorization/elevateAccess/Action",
        "Microsoft.Authorization/roleDefinitions/write",
        "Microsoft.Authorization/roleDefinitions/delete"
    ],
    "DataActions": [],
    "NotDataActions": [],
    "condition": "@iPAddress() matches '^52.43.209.253$'",
    "conditionVersion": "2.0"
}

Access will be limited to a client support machine that is only accessible to Atolio support engineers and that has a static IP of 52.43.209.253 assigned to aid in identifying activity from Atolio’s team.

If you opt to allow Atolio’s deployment support team to manage the deployment on your behalf the steps to enable this for your Azure subscription are as follows:

1. Clone our lumen-infra GitHub repository

   git clone git@github.com:atolio/lumen-infra.git
   

2. Run our Azure support role script against the Azure subscription you’d like us to deploy into

   ./lumen-infra/deploy/terraform/azure/scripts/atolio-support-role.sh
   

3. Review the output from the script and provide the details to your Atolio support team

   Operation completed successfully.
   Please securely share the following details with your Atolio support engineer:
   
   Tenant ID: <tenant ID>
   Subscription ID: <subsciption ID>
   Application (Client) ID: <client app ID>
   Client Secret: <client secret>
   Secret Expiry Date: <expiry date>
   
   These credentials grant access to the specified Azure subscription with the
   custom role 'Atolio Support Access'. The client secret will expire on <expiry date>.
   

Determine Atolio DNS name

Before the deployment call, you may want to decide on your desired Atolio web location. An Azure DNS Zone will be created in the Azure subscription used for hosting the Atolio stack (e.g. search.example.com.): this will be the DNS name (without the trailing dot) for the Atolio Web application (e.g. https://search.example.com).

For the remainder of this document, we will use https://search.example.com in the examples, but it is expected for you to replace with your own DNS name.

Determine Cloud Networking Options

By default, Atolio’s Terraform code will create a VNet in your Azure subscription. However, you may choose to use an existing VNet and subnets within your Azure subscription. In this case, set create_vnet to false in the config.hcl file during the deployment.

Then, configure all VPC related variables. See this example from config.hcl:

// Uncomment these lines and update the values in case you want to deploy in a
// pre-existing VNet (by default a new VNet will be created).
//
// create_vnet = false
// vnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}"
// vnet_private_subnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{privateSubnetName}"
// vnet_public_subnet_id = "/subscriptions/{tenantID}/resourceGroups/{rgID}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{publicSubnetName}"

Optionally, when create_vnet is not set to false you can customize the default VNet and Subnet CIDR blocks if you have specific requirements. See this example from config.hcl:

// Uncomment these lines and update the values if you want to customize the
// default CIDR blocks for the VNet and Subnets created automatically.
//
// Note: this only applies for when create_vnet hasn't been set to false and
// any values modified here will be ignored when create_vnet=false.
//
// vnet_cidr = "10.42.0.0/20"
// pod_subnet_block = "10.42.0.0/22"
// appgw_subnet_block = "10.42.4.0/24"

If custom networking configuration will be necessary be sure to provide these details to the engineer performing the deployment.

Setup Authentication

Atolio supports single sign-on (SSO) authentication through Okta, Microsoft Entra ID, and Google using the OpenID Connect (OIDC) protocol.

Refer to Configuring Authentication for more details on the steps to complete in your desired SSO provider in order to obtain the necessary OIDC configuration values.

If using Microsoft Entra ID, the oidc_client_id and oidc_client_secret will be the respective values created and saved during Azure AD - Create New App Registration.

Setup local environment

Finally, if an engineer from your team will be performing the deployment, ensure they have the following utilities installed:

• Setup Terraform on your local machine as described on the HashiCorp docs site - we require v1.5.0 at a minimum.
• Install the Azure Command Line Interface
• Install kubectl
• Install Helm
• Install atolioctl

Note: If you are running on Windows, you may also need to install the Windows Subsystem for Linux.

Provide Deployment Engineer with Configuration

At this point if you’re proceeding with an Atolio managed deployment you’ll need to provide the information from these prerequisite steps to your Atolio deployment support team. Otherwise please ensure the information is provided to the engineer from your organization who will be performing the deployment with Atolio’s support.

To recap, provide these details:

• Atolio DNS name
• OIDC Configuration
• (Optional) Custom Cloud Networking Options

If you’re opting to have Atolio manage the deployment you can disregard the remainder of this documentation as these steps will be performed by your Atolio deployment engineer. Otherwise be sure to share this documentation with the engineer from your organization that will be performing the deployment so they can familarize theirself with the steps required.

Create Cloud Infrastructure

Note: Atolio requires an Azure region with 3 availability zones. You can check which regions include support for multiple availability zones here.

The Terraform configuration requires an Azure storage account to store state. A script is available to automate the whole process (including running Terraform). Before running the script, create a config.hcl file based on the provided config.hcl.template:

cd deploy/terraform/azure
cp ./config.hcl.template config.hcl

Atolio Domain Name and Image Pull Secrets

Update the copied file with appropriate values. At a minimum, it should look something like this:

// Domain name for Atolio stack (same as hosted zone name without trailing ".")
lumen_domain_name = "search.example.com"

// The registry in which to obtain containers for services
container_registry = "atolioimages.azurecr.io"
image_pull_username = "provided-by-atolio"
image_pull_password = "provided-by-atolio"

Your Atolio support team will share the appropriate values for the image_pull_username and image_pull_password values.

Application Helm Value Options

Next copy the Helm template and update the values as directed.

cp ./templates/values-lumen-admin.yaml values-lumen.yaml
cp ./templates/values-vespa-admin.yaml values-vespa.yaml

# Default values for lumen.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates (provided by admin user).

# JWT secret key for API call (signature) verification (at least 256 bits / 32 chars)
# Can be generated by running `openssl rand -base64 32` in your terminal
jwtSecretKey: "add-your-jwt-secret-key-here"

# Secret salts for generating Vespa document IDs
# Can be generated by running `openssl rand -base64 32` in your terminal
secretSalts: "add-your-secret-salts-here"

# See also scripts/config-oidc.sh helper script to obtain some of the values below
oidc:
  provider: "add-your-provider-here"
  endpoint: "add-your-endpoint-here"
  clientId: "add-your-id-here"
  clientSecret: "add-your-secret-here"

# Use the Atolio provided image pull secrets for accessing Atolio container images
imagePullSecret:
  create: true

# If running behind a reverse proxy, this should be set to the URL the end user will
# use to access the product.
reverseProxyUrl: ""

# The ACME Cluster Issuer for LetsEncrypt requires an email address to be provided
# for certificate notifications. This is required for LetsEncrypt certificates to work properly.
# Common examples would be an admin or technical support email address for your organization. ex: "admin@example.com" or "engineering@example.com".
letsencrypt:
  email: "letsencrypt@example.com"
  dns01:
    azureDNS:
      environment: AzurePublicCloud
    

For the jwtSecretKey and secretSalts values any 256 bit (32 character) string can be used. These values are used to sign JWT tokens used by the web application and atolioctl tool and salt document IDs in Atolio’s database. They should be well guarded secrets that are unique to the deployment.

If your users will be accessing the web interface via a reverse proxy (e.g. such as StrongDM), then be sure to set the reverseProxyUrl field to reflect the URL they will actually enter into their browser to access Atolio, which will be different to the hostname defined in lumen_domain_name. Leave this field empty if not using a reverse proxy.

Deployment with create-infra.sh script

Once you have all variables configured to your environment’s requirements you can create the infrastructure and deploy the k8s cluster. From the deploy/terraform/azure directory:

./scripts/create-infra.sh --name=deployment-name

This will create the infrastructure in the westus3 Azure region. If you want to deploy in another region parameter (e.g. eastus) an additional parameter can be provided:

./scripts/create-infra.sh --name=deployment-name --region=eastus

The deployment-name argument is used to define a deployment name for collecting resources into an Azure Resource Group containing the kubernetes cluster, networking, storage, etc. We recommend making it unique across all deployments, i.e. using a globally unique deployment name. Typically this is set to your company name with an optional suffix if specifying environment (e.g. acmecorp or acmecorp-qa).

The script automates the following steps (parameterized based on the provided deployment name):

1. Create an Azure Blob Storage to store Terraform state
2. Create a terraform.tfvars file for Terraform
3. Run terraform init
4. Run terraform apply (using input variables in generated terraform.tfvars)

Post-deployment steps

With the infrastructure created, you’ll want to update your local kubeconfig with a context for the Atolio cluster (this is also output via Terraform as update_kubeconfig_command):

az aks get-credentials --name=lumen-{deployment-name} --resource-group lumen-{deployment-name}

Delegate responsibility for Atolio subdomain

The parent domain (e.g. example.com) needs to delegate traffic to the new Atolio subdomain (search.example.com). This is achieved by adding an NS record to the parent domain with the 4 name servers copied from the new subdomain.

These nameservers can be retrieved post-creation with terraform output:

terraform output --json name_servers | jq -r '.[]'

At this point you should be able to interact with the kubernetes cluster, e.g.

kubectl get po -n atolio-svc

Note, Atolio specific services run on the following namespaces:

• atolio-svc (Services)
• atolio-db (Database)
• atolio-ctl (Control Plane)

When you have validated that the infrastructure is available, the next step is to configure sources.

3 - GCP Deployment

Architecture

Deployment Prerequisites

Before getting started, Atolio needs to have done the following on your behalf:

1. Granted your GCP project access to the Client Container Registry repos (for pulling Atolio container images)
2. Add your Deployment Engineer as a collaborator to the Atolio GitHub repository (lumen-infra), which contains:
   • Deployment documentation
   • Terraform for the Atolio stack infrastructure
   • Configuration files for Atolio services
   • Maintenance scripts
3. Ensure your GCP project has sufficient quota for the required resources, particularly Compute Engine API and Google Kubernetes Engine API. See Hardware Requirements for the full list of recommended quotas.

The following deployment prerequisites will help streamline your deployment process.

Determine GCP project

We recommend you create a new GCP project to host Atolio, as this will help keep your Atolio deployment and other resources separate from your production environment. When the project is available, share the GCP project ID and number with your Atolio support team.

Determine Deployment Model

We offer both Atolio managed and customer managed deployment models for you to choose from. Please review the comparison and requirements for each approach on our Deployment Model Overview page and inform your Atolio support team which method you’d like to use for your deployment.

Determine Atolio DNS name

Before the deployment call, you may want to decide on your desired Atolio web address. A GCP Cloud DNS zone will be created in the GCP project used for hosting the Atolio stack (e.g. search.example.com.): this will be the DNS name (without the trailing dot) for the Atolio web application (e.g. https://search.example.com).

This Cloud DNS zone allows the deployment control plane (i.e. the External DNS controller) to add records to link host names (e.g. search.example.com, feed.search.example.comand relay.search.example.com) to the Ingress controller used for the deployment.

For the remainder of this document, we will use https://search.example.com in the examples, but it is expected for you to replace with your own DNS name.

Setup authentication

Atolio supports single sign-on (SSO) authentication through Okta, Microsoft Entra ID, and Google using the OpenID Connect (OIDC) protocol.

Refer to Configuring Authentication for more details on the steps to complete in your desired SSO provider in order to obtain the necessary OIDC configuration values.

Setup local environment

Finally, if an engineer from your team will be performing the deployment, ensure they have the following utilities installed:

• Setup Terraform on your local machine as described on the HashiCorp docs site - we require v1.6.0 at a minimum.
• Install the Google Cloud SDK
• Install kubectl
• Install Helm
• Install atolioctl

Provide Deployment Engineer with Configuration

At this point if you’re proceeding with an Atolio managed deployment you’ll need to provide the information from these prerequisite steps to your Atolio deployment support team. Otherwise please ensure the information is provided to the engineer from your organization who will be performing the deployment with Atolio’s support.

To recap, provide these details:

• Atolio DNS name
• OIDC Configuration

If you’re opting to have Atolio manage the deployment you can disregard the remainder of this documentation as these steps will be performed by your Atolio deployment engineer. Otherwise be sure to share this documentation with the engineer from your organization that will be performing the deployment so they can familarize theirself with the steps required.

Create Cloud Infrastructure

The Terraform configuration requires an external (Google Cloud Storage) bucket to store state. A script is available to automate the whole process (including running Terraform). Before running the script, create a config.hcl file based on the provided config.hcl.template:

cd deploy/terraform/gcp
cp ./config.hcl.template config.hcl

Update the copied file with appropriate values. At a minimum, it should specify the zone you’re deploying to:

// Core Configuration
zone = "us-central1-f"

Atolio Domain Name

Update the copied file with subdomain you will use for your Atolio deployment values:

# Domain name for Atolio stack (same as hosted zone name without trailing ".")
lumen_domain_name = "search.example.com"

Application Helm Value Options

Next copy the Helm template and update the values as directed.

cp ./templates/values-lumen-admin.yaml values-lumen.yaml
cp ./templates/values-vespa-admin.yaml values-vespa.yaml

# Default values for lumen.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates (provided by admin user).

# JWT secret key for API call (signature) verification (at least 256 bits / 32 chars)
# Can be generated by running `openssl rand -base64 32` in your terminal
jwtSecretKey: "add-your-jwt-secret-key-here"

# Secret salts for generating Vespa document IDs
# Can be generated by running `openssl rand -base64 32` in your terminal
secretSalts: "add-your-secret-salts-here"

# See also scripts/config-oidc.sh helper script to obtain some of the values below
oidc:
  provider: "add-your-provider-here"
  endpoint: "add-your-endpoint-here"
  clientId: "add-your-id-here"
  clientSecret: "add-your-secret-here"

# If running behind a reverse proxy, this should be set to the URL the end user will
# use to access the product.
reverseProxyUrl: ""

# The ACME Cluster Issuer for LetsEncrypt requires an email address to be provided
# for certificate notifications. This is required for LetsEncrypt certificates to work properly.
# Common examples would be an admin or technical support email address for your organization. ex: "admin@example.com" or "engineering@example.com".
letsencrypt:
  email: "letsencrypt@example.com"

For the jwtSecretKey and secretSalts values any 256 bit (32 character) string can be used. These values are used to sign JWT tokens used by the web application and atolioctl tool and salt document IDs in Atolio’s database. They should be well guarded secrets that are unique to the deployment.

If your users will be accessing the web interface via a reverse proxy (e.g. such as StrongDM), then be sure to set the reverseProxyUrl field to reflect the URL they will actually enter into their browser to access Atolio, which will be different to the hostname defined in lumen_domain_name. Leave this field empty if not using a reverse proxy.

The ACME Cluster Issuer for LetsEncrypt requires an email address to be provided for certificate notifications. If the default @example.com email address is used, then the Cluster Issuer will fail to register for certificate management so be sure to provide a valid email address.

Deployment with create-infra.sh script

Once you have all variables configured, you can create the infrastructure and deploy the GKE cluster. From the deploy/terraform/gcp directory:

./scripts/create-infra.sh --name=acmecorp

This will create the infrastructure in the us-central1 GCP region. If you want to deploy in another region parameter (e.g. us-west2) an additional parameter can be provided:

./scripts/create-infra.sh --name=acmecorp --region=us-west2

The --name argument is used to generate a deployment name for tagging resources and naming the GCP resources. So make sure it is unique across all deployments. (i.e. using a globally unique deployment name). Typically this is set to your company name with an optional suffix if specifying environment (e.g. acmecorp or acmecorp-qa).

The script automates the following steps (parameterized based on the provided deployment name):

1. Create a Google Cloud Storage bucket to store Terraform state
2. Create a terraform.tfvars file for Terraform based on the config.hcl file
3. Run terraform init
4. Run terraform apply (using input variables in generated terraform.tfvars)

Post-deployment steps

With the infrastructure created, you’ll want to update your local kubeconfig with a context for the Atolio cluster (this is also output via Terraform as update_kubeconfig_command):

gcloud container clusters get-credentials lumen-{deployment-name} --region us-central1 --project {your-gcp-project-id}

Delegate responsibility for Atolio subdomain

The parent domain (e.g. example.com) needs to delegate traffic to the new Atolio subdomain (search.example.com). This is achieved by adding an NS record to the parent domain with the 4 name servers copied from the new subdomain.

These nameservers can be retrieved post-creation with terraform output:

terraform output --json name_servers | jq -r '.[]'

Deployment Verification

At this point you should be able to interact with the Kubernetes cluster and see Atolio pods deployed, e.g.

kubectl get po -n atolio-svc

Note, Atolio specific services run on the following namespaces:

• atolio-svc (Application Services)
• atolio-db (Database)
• atolio-ctl (Control Plane)

When you have validated that the infrastructure is available, the next step is to configure sources.

4 - Hardware Requirements

Atolio runs entirely inside your infrastructure. There is no Atolio-hosted SaaS layer. This page covers hardware sizing for on-premise, airgapped, and bring-your-own-Kubernetes (BYOK) deployments, where you supply the cluster and node hardware.

Atolio also supports managed deployments on AWS, Azure, and GCP, where Atolio’s Terraform provisions sized node groups for you. For per-cloud instance types, quotas, and networking detail, see the deployment guides for AWS, Azure, and GCP.

Overview

A deployment requires:

• General compute nodes for application services and the search database.
• At least one GPU node for the embedding service. Production requires GPU-backed embedding; CPU-only is for proof-of-concept or staging.
• Persistent block storage for Vespa content (index) nodes.
• S3-compatible object storage (MinIO, Ceph RGW, or any S3-compatible store) for assets such as user avatars and shared logs.
• An ingress gateway for HTTPS. Inbound connectivity is required for connectors that receive webhook updates.

Query-time LLM inference is deployed independently of the Atolio cluster. It can run entirely on-premise for fully sovereign deployments, on an OpenAI-compatible endpoint you operate, or via an external managed provider. Sizing the inference layer is outside the scope of this page. Only embedding requires an in-cluster GPU.

The major workloads:

Three things drive sizing: active connector count, indexed corpus size, and concurrent users. The figures below are baselines. Atolio’s deployment team will check them against your expected scale before you provision.

Pod-level Resource Requirements

This is the canonical sizing view. Use it to size worker nodes in your Kubernetes or OpenShift cluster.

The 2 Vespa content pods and 3 Vespa admin pods must each run on physically separate machines (admin pods can be co-located with other services). The remaining services can share nodes wherever capacity exists.

Cluster Requirements

Atolio runs on any conformant Kubernetes 1.33+ cluster, including airgapped environments. OpenShift (including ROSA) is fully supported and tested; vanilla Kubernetes, Rancher/RKE2, and EKS-Anywhere style installs work as well.

BYOK fits when data-residency or compliance rules require infrastructure inside your perimeter, when the cluster is airgapped, when you already operate a hardened Kubernetes platform, or when the LLM, embedder, index, and connectors must all run on customer-owned hardware.

What Atolio provides

• Helm charts: atolio-db (Vespa) and atolio-svc (Lumen application services).
• Container images suitable for an internal or airgapped registry.
• Deployment runbooks and direct support during install.

What you provide

• A Kubernetes or OpenShift cluster sized per the Pod-level Resource Requirements, with the role split below.
• A CSI block storage class for Vespa PVCs.
• An S3-compatible object store (MinIO, Ceph RGW, or any S3-compatible store).
• A reachable container registry plus credentials. For airgapped installs, your internal mirror.
• An ingress controller and TLS certificate (or cert-manager).
• An OIDC provider for SSO. Atolio supports Okta, Microsoft Entra ID, Google, and Keycloak.

Node role split

x86_64 is the typical on-prem architecture. ARM64 images are also published and supported.

Airgapped deployments

In airgapped mode, all components run inside your perimeter, with no outbound traffic at runtime. Atolio’s deployment team can supply an offline image bundle and matching runbook on request.

Storage

Encryption-at-rest is provided by your CSI driver / storage class. Mark Vespa content and admin StorageClasses and PVCs as Retain so their data survives chart redeploys.

GPU Requirements

The embedder is the only in-cluster service that requires a GPU. It generates dense vector embeddings for indexed content; running on GPU keeps ingest throughput up during large connector backfills.

• GPU: NVIDIA T4 with 16 GB VRAM is the baseline. Higher-end accelerators (A4000, A4500, A10, A10G, L4) with 16–24 GB VRAM are also supported.
• Driver: install the NVIDIA GPU operator (or the OpenShift NVIDIA GPU operator) so GPU nodes expose the nvidia.com/gpu resource.
• Query-time inference: hosted independently (on-premise or external). See Overview.

For proof-of-concept or staging clusters where CPU-only embedding is acceptable, set embedder.useGPU=false in the Lumen Helm values and omit the GPU node pool.

Networking

Atolio runs in a single VPC/VNet/VLAN with public and private subnets across multiple failure domains.

• CIDR: any non-overlapping /20+ range works.
• Subnets: at least /22 (1,024 IPs) per private subnet to avoid IPv4 exhaustion as pod counts grow.
• Failure domains: at least three, for HA across racks, hypervisors, or zones.
• Ingress: any ingress controller works (NGINX, HAProxy, OpenShift Router, F5 BIG-IP). Inbound HTTPS is required for webhook-driven connectors.
• Egress: outbound HTTPS to source-system APIs, the LLM endpoint, and the container registry. Airgapped deployments host the LLM and registry internally.

Scaling Guidance

Sizing increases with:

• Document count and size: more or larger documents (e.g. PDFs, design files) → more Vespa content nodes or higher per-node memory.
• Embedding dimensionality: higher-dimensional models pressure Vespa content memory. Plan on 128 GB+ per content node for high-dimensional models.
• Concurrent users / query rate: scale Vespa container nodes and Lumen services horizontally.
• Connector throughput: heavy connectors (Slack, GitHub at scale, large Drive estates) increase ingest CPU and GPU embedding throughput.
• HA posture: for stricter SLAs, run 3+ Vespa content nodes and 3+ Lumen services pods.

For larger deployments, Atolio’s deployment team can run a pre-deployment content scan against an expected connector to estimate document inventory and recommend node count, memory tier, and disk size before provisioning.

Customizing the Defaults

Every node count and disk size is exposed as a Helm value in the Vespa and Lumen charts. Commonly tuned settings cover:

• Lumen services replica count and resource requests.
• Vespa content node count and content disk size.
• Vespa container node count.
• Embedding services replica count, GPU toggle, and resource requests.
• OpenShift-specific knobs: security context constraints, image pull secrets, storage class overrides.

If you need a configuration outside the documented defaults (for example, an internal procurement standard, a specific Kubernetes distribution, or an unusual storage driver), share the target hardware and platform with your Atolio support team so they can confirm compatibility: architecture, base image, GPU drivers, and CSI driver.

5 - Atolio Managed vs. Customer Managed Deployment Models

Atolio Managed vs. Customer Managed Deployment

Atolio offers two deployment models to best suit your organization’s needs and security requirements. Regardless of which deployment model you choose, your data always remains within your cloud environment and under your complete control. This ensures maximum security and compliance while still allowing you to benefit from Atolio’s expertise and support.

Atolio Managed Deployment

In this model, an Atolio support engineer handles the initial infrastructure deployment on your behalf, working exclusively within your cloud environment. This option is recommended if you want to:

• Accelerate the initial deployment process
• Leverage Atolio’s expertise with the infrastructure stack
• Focus your team’s efforts on configuration and usage rather than deployment

To enable Atolio managed deployment:

1. Have a cloud engineer with sufficient privileges run our support role script for your Cloud Provider
2. The script will create the IAM resource with the necessary permissions and output details
3. Provide these details to your Atolio deployment support team. They will use this delegated role to:
   • Deploy the initial infrastructure
   • Configure core services
   • Validate the deployment
   • Hand off the deployment to your team

After the initial deployment is complete and validated, you can either:

• Remove the Atolio support role to revoke access
• Maintain the role for future support needs
• Transfer infrastructure management responsibilities to your internal team

Customer Managed Deployment

In this model, your organization’s engineers handle the deployment using the documentation we provide and with the guidance of an Atolio deployment engineer via screensharing. This option is recommended if you want to:

• Maintain complete control over the deployment process
• Integrate the deployment into your existing infrastructure-as-code practices
• Have your team gain in-depth knowledge of the infrastructure stack

To proceed with a customer managed deployment:

1. Follow the prerequisites and setup instructions in the section specific to your cloud provider
2. Use the provided Terraform configurations and scripts to deploy the infrastructure
3. Work with Atolio support for guidance and troubleshooting as needed via screensharing

Regardless of the deployment model chosen, Atolio provides ongoing support for the application itself and can assist with infrastructure-related questions and issues. In both models, your data and infrastructure remain securely within your cloud environment, with all processing and storage occurring within your controlled perimeter.

6 - Operations Best Practices

Troubleshooting

Current configuration and service status can be monitored in the admin: https://search.example.com/admin.

With the appropriate Kubernetes context set, port forwarding to particular pods is a common case to query select APIs. For example, you may wish to query the Vespa document cluster directly. You’d do this by port forwarding the container node (with valid AWS profile set in context):

kubectl port-forward -n atolio-db pod/vespa-container-0 8080

Additionally, the Feeder service provides gRPC APIs which are used by various services and tools. As the port name is not sticky, it is recommended to port forward the service:

kubectl port-forward -n atolio-svc service/feeder 8889

To observe the possible APIs, use grpcurl to describe and explore:

grpcurl -plaintext 127.0.0.1:8889 describe

Note there are two namespaces used in an Atolio deployment. They are atolio-svc (for all services) and atolio-db for Vespa (database and search).

Storing Deployment Artifacts

This completes the initial deployment of the Atolio stack. Please make sure to store the following artifacts created by the deployment process in a safe place for future use:

• Initial configuration (config.hcl) which is needed to generate redeploy from scratch (this generates terraform.tfvars)
• Helm value files (values-lumen.yaml and values-vespa.yaml) which are used by Helm to customize your deployment.
• Google credential files (Client OAuth and Directory API keys)

These will be needed to make future changes and provide access to the Atolio stack for maintenance.

Additionally there is a hidden .terraform directory with Terraform internal state that is needed to re-run Terraform without the need for reconfiguration.

Deploying Updates

The Atolio micro services of the Atolio stack (i.e. Marvin, Search UI, Source Connectors, and Feeder) will be updated by Atolio. This is done by pushing updated Docker images to the Docker Repositories (ECR) hosted by Atolio.

Atolio, under normal circumstances, will not replace pushed images. We follow a typical major/minor/patch versioning model and any changes, including hot fixes, will be pushed under their relevant version.

This means that to update services, simply amend lumenImageTag in both values-lumen.yaml and values-vespa.yaml files with the desired version. If using the image tag for a lumen-infra release, then you do not need to update this value.