Operations Best Practices

This document provides some additional best practices for operating and performing maintenance on an Atolio deployment.

Troubleshooting

Current configuration and service status can be monitored in the admin: https://search.example.com/admin.

With the appropriate Kubernetes context set, port forwarding to particular pods is a common case to query select APIs. For example, you may wish to query the Vespa document cluster directly. You’d do this by port forwarding the container node (with valid AWS profile set in context):

kubectl port-forward -n atolio-db pod/vespa-container-0 8080

Additionally, the Feeder service provides gRPC APIs which are used by various services and tools. As the port name is not sticky, it is recommended to port forward the service:

kubectl port-forward -n atolio-svc service/feeder 8889

To observe the possible APIs, use grpcurl to describe and explore:

grpcurl -plaintext 127.0.0.1:8889 describe

Note there are two namespaces used in an Atolio deployment. They are atolio-svc (for all services) and atolio-db for Vespa (database and search).

Storing Deployment Artifacts

This completes the initial deployment of the Atolio stack. Please make sure to store the following artifacts created by the deployment process in a safe place for future use:

  • Deployment specific Terraform settings (terraform.tfvars and values.yaml)
  • Initial configuration (config.hcl) which is needed to generate redeploy from scratch (this generates terraform.tfvars)
  • Google credential files (Client OAuth and Directory API keys)

These will be needed to make future changes and provide access to the Atolio stack for maintenance.

Additionally there is a hidden .terraform directory with Terraform internal state that is needed to re-run Terraform without the need for reconfiguration.

Deploying Updates

The Atolio micro services of the Atolio stack (i.e. Marvin, Search UI, Source Fleet, and Feeder) will be updated by Atolio. This is done by pushing updated Docker images to the Docker Repositories (ECR) hosted by Atolio.

Atolio, under normal circumstances, will not replace pushed images. We follow a typical major/minor/patch versioning model and any changes, including hot fixes, will be pushed under their relevant version.

This means that to update services, simply amend lumenImageTag in both values-lumen.yaml and values-vespa.yaml files with the desired version. If using the image tag for a lumen-infra release, then you do not need to update this value.