Configuring Sources

Note: Configuration of sources is regularly evolving and curretly requires access to the kubernetes custer. This document covers responsibilities shared by both the Deployment Engineer and Admin.

Atolio can be configured to index content across a variety of sources. This document describes the steps required in both the source systems and within Atolio:

1. First, follow the steps required in the source system (links provided below).
2. Then, provide the necessary configuration to your Deployment Engineer in order to setup the Atolio connector in your environment as outlined in Atolio Configuration.

Atolio Configuration

All source configuration in Atolio is currently performed by your Deployment Engineer. It is accomplished using atolioctl which is a terminal based UI (TUI) that allows your Deployment Engineer to configure all available sources. In the future, this interface will be a browser-based IT Administrator function.

Installing atolioctl

Atolio provides an installer script within the lumen-infra repository that will automatically grab the latest version of atolioctl and install it locally.

Note: The installer script requires that you have curl or wget installed:

• curl
• wget

1. Navigate to the lumen-infra repository and into the scripts directory

cd lumen-infra/deploy/terraform/scripts

2. Run the installer script

./get-atolioctl.sh

3. From there, you should be able to run the tool: atolioctl --help

You can re-run this script to download the latest version of atolioctl or run the script with the --version flag to install a specific version of atolioctl, ex:

./get-atolioctl.sh --version 4.10.2

Deploying Connectors

Prior to configuring a source, the connector (ie the deployment in which the source will run on) must be deployed. Atolio uses the concept of instances to support one or more instances of a source (eg multiple tenants), and the creation of these deployments is managed by modifying values.yaml. Specify connectors like so:

connectors:
  microsoft:
    default:
      log:
        level: "info"
  okta:
    default:
      log:
        level: "info"
  jiracloud:
    default:
      log:
        level: "info"

Connectors are deployed under a default instance. This is an identifier which can be changed as desired, and multiple connectors can be deployed under different instances. Once the instance is set, it should be treated immutable. Modifying this value will result in loss of data and a need to re-index under the new instance. This is due to the instance being part of every resource identifier ingested by Atolio.

  jiracloud:
    internal:
      log:
        level: "info"
    external:
      log:
        level: "info"

To provision the updates into the cluster, you’d upgrade the lumen Helm chart by running create-infra.sh (recommended) or:

helm upgrade lumen-chart ../../k8s/charts/lumen --values values.yaml --values values-templated.yaml -n atolio-svc

Configuring Sources

All source configuration is accomplished through atolioctl which can be obtained through the Releases page. This is a terminal based UI (TUI) which allows an administrator to configure all available sources. In the future, this interface will be browser based.

This section details how to configure each source, including how to obtain the necessary JWT.

First an API token needs to be created for use with atolioctl. This token can be generated as follows using the config.hcl file that was used to run the create-infra.sh script:

export JWT_TOKEN=$(atolioctl connector create-jwt --raw --config-dir . "atolio:*:*:*")

Feeder exposed on Load Balancer

To run configuration, provide the configure command to atolioctl. Make sure to use the correct API host name for your deployment. Once loaded, you should see a collection of configuration options and sources in your terminal:

atolioctl configure --jwt-token-sdk ${JWT_TOKEN} --feeder-address feed.search.example.com:443

Port Forwarding (Feeder not exposed on Load Balancer)

To run configuration, port-forward 8889 from service/feeder and provide the configure command to atolioctl. Ensure the correct AWS profile is in context. Once loaded, you should see a collection of configuration options and sources in your terminal:

kubectl port-forward -n atolio-svc service/feeder 8889
atolioctl configure --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls

Note that for the time being all atolioctl commands below (unless noted otherwise) will require the port forwarding command.

Configuration Tips

To obtain source-specific information, head to source configuration and find the source you’re interested in configuring. Some tips when running atolioctl:

• Run the application in a maximized window to ensure full visibility of all information.
• Press ? to toggle the help menu / key map. You will typically use up/down arrows and <TAB> to navigate the menus.
• No data is persisted until Review & Save is selected and validation has completed. A report will be shown detailing which sources were successfully configured and which not.

Pre-configuration and YAML export/import

Configuration of sources can be done without a functional stack and atolioctl can output the configuration into a YAML file, which can be further hand-modified and imported when the stack is deployed.

Prior to an initial deployment, pre-configuring sources can expedite the initial deployment procedures and time on call. To configure sources before the stack is available, you can pass the --config-file option:

atolioctl configure --config-file atolio.yaml

The file does not need to exist prior to running this command. You’ll be presented with the TUI and will be able to select and configure sources from the registry. Once configured, you can Review & Save per typical procedure and atolioctl will output an atolio.yaml file with the (valid) configuration persisted within the filesystem.

As a reminder, the stack is not yet running at this point. Configuration of most sources does not require an operational stack as the validation is simply making a call to the source API and there is no expectation of the source’s connector running yet. There are some exceptions (in particular Jira, Confluence, and BitBucket)

Once desired sources have been configured and the stack deployed, you can import the pre-configured file. Note at this point it’s expected Feeder is operational and port-forwarding to the service address can occur:

atolioctl configure import atolio.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls
Writing key: /lumen/connectors/github/default/github
Writing key: /lumen/connectors/linear/default/linear
Writing key: /lumen/connectors/notion/default/notion
Writing key: /lumen/connectors/slack/default/slack
Writing key: /lumen/connectors/web/default/web
Writing key: /lumen/connectors/bitbucket/default/bitbucket

Similarly, source configuration of the stack can be exported to a local file. This can be useful if wishing to view the entire configuration in a human-readable YAML file, or perhaps store a backup of the configuration in a vault like 1Password. This file will be in the same format as the one imported in the prior step, allowing a loop of export, modify, import:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 atolio.yaml --disable-feeder-tls

Both export and import commands can also combined into a single edit command that exports the current source configuration into a text editor (specified by the EDITOR environment variable) allowing manual edits. Upon saving/closing the editor, the configuration will be imported. This command is invoked a follows:

atolioctl configure edit --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls

Common Fields: Resources

Atolio supports the ability to avoid certain content from being indexed at all. The example below shows how to specifically exclude a Jira project (with the project key “SP”).

When configuring the Jira source, enter the following in the Resources field:

project:
    excluded:
        - "SP" # The Secret Project

The excluded keyword contains a list of Jira project keys to be excluded from indexing. Similarly you can also use the included keyword to achieve the opposite (in that case only those resources that are specifically listed will be indexed, nothing else). Note that the list entries (e.g. SP in the example above) are case sensitive.

In the absence of both included (opt-in) or excluded (opt-out), everything that the credentials (e.g. API key) have access to will be indexed. Note that this is a layer on top of the permissions model provided to the source. Anything added to included will only be indexed if the underlying credentials grant the indexing job access.

This feature is supported by a majority of Atolio connectors, see each connectors respective documentation for details and scope of inclusions / exclusions.

Source Configuration