Configuring Sources

• 1: Atlassian Bitbucket Connector
• 2: Atlassian Confluence Connector
• 3: Atlassian Connector for Jira & Confluence Data Center
• 4: Atlassian Jira Connector
• 5: GitHub Connector
• 6: GitLab Connector
• 7: Google Connector
• 8: Linear Connector
• 9: Microsoft Connector
• 9.1: OneDrive Connector Setup
• 9.2: Outlook Connector Setup
• 9.3: SharePoint Connector Setup
• 9.4: Teams Connector Setup
• 10: Notion Connector
• 11: Okta Connector
• 12: Salesforce Connector
• 13: ServiceNow Connector
• 14: Slack Connector
• 15: Staffbase Connector
• 16: Web Connector
• 17: Zendesk Connector

Note: Configuration of sources is regularly evolving and curretly requires access to the kubernetes custer. This document covers responsibilities shared by both the Deployment Engineer and Admin.

Atolio can be configured to index content across a variety of sources. This document describes the steps required in both the source systems and within Atolio:

1. First, follow the steps required in the source system (links provided below).
2. Then, provide the necessary configuration to your Deployment Engineer in order to setup the Atolio connector in your environment as outlined in Atolio Configuration.

Atolio Configuration

All source configuration in Atolio is currently performed by your Deployment Engineer. It is accomplished using atolioctl which is a terminal based UI (TUI) that allows your Deployment Engineer to configure all available sources. In the future, this interface will be a browser-based IT Administrator function.

Installing atolioctl

Atolio provides an installer script within the lumen-infra repository that will automatically grab the latest version of atolioctl and install it locally.

Note: The installer script requires that you have curl or wget installed:

• curl
• wget

1. Navigate to the lumen-infra repository and into the scripts directory

cd lumen-infra/deploy/terraform/scripts

2. Run the installer script

./get-atolioctl.sh

3. From there, you should be able to run the tool: atolioctl --help

You can re-run this script to download the latest version of atolioctl or run the script with the --version flag to install a specific version of atolioctl, ex:

./get-atolioctl.sh --version 4.10.2

Deploying Connectors

Prior to configuring a source, the connector (ie the deployment in which the source will run on) must be deployed. Atolio uses the concept of instances to support one or more instances of a source (eg multiple tenants), and the creation of these deployments is managed by modifying values.yaml. Specify connectors like so:

connectors:
  microsoft:
    default:
      log:
        level: "info"
  okta:
    default:
      log:
        level: "info"
  jiracloud:
    default:
      log:
        level: "info"

Connectors are deployed under a default instance. This is an identifier which can be changed as desired, and multiple connectors can be deployed under different instances. Once the instance is set, it should be treated immutable. Modifying this value will result in loss of data and a need to re-index under the new instance. This is due to the instance being part of every resource identifier ingested by Atolio.

  jiracloud:
    internal:
      log:
        level: "info"
    external:
      log:
        level: "info"

To provision the updates into the cluster, you’d upgrade the lumen Helm chart by running create-infra.sh (recommended) or:

helm upgrade lumen-chart ../../k8s/charts/lumen --values values.yaml --values values-templated.yaml -n atolio-svc

Configuring Sources

All source configuration is accomplished through atolioctl which can be obtained through the Releases page. This is a terminal based UI (TUI) which allows an administrator to configure all available sources. In the future, this interface will be browser based.

This section details how to configure each source, including how to obtain the necessary JWT.

First an API token needs to be created for use with atolioctl. This token can be generated as follows using the config.hcl file that was used to run the create-infra.sh script:

export JWT_TOKEN=$(atolioctl connector create-jwt --raw --config-dir . "atolio:*:*:*")

Feeder exposed on Load Balancer

To run configuration, provide the configure command to atolioctl. Make sure to use the correct API host name for your deployment. Once loaded, you should see a collection of configuration options and sources in your terminal:

atolioctl configure --jwt-token-sdk ${JWT_TOKEN} --feeder-address feed.search.example.com:443

Port Forwarding (Feeder not exposed on Load Balancer)

To run configuration, port-forward 8889 from service/feeder and provide the configure command to atolioctl. Ensure the correct AWS profile is in context. Once loaded, you should see a collection of configuration options and sources in your terminal:

kubectl port-forward -n atolio-svc service/feeder 8889
atolioctl configure --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls

Note that for the time being all atolioctl commands below (unless noted otherwise) will require the port forwarding command.

Configuration Tips

To obtain source-specific information, head to source configuration and find the source you’re interested in configuring. Some tips when running atolioctl:

• Run the application in a maximized window to ensure full visibility of all information.
• Press ? to toggle the help menu / key map. You will typically use up/down arrows and <TAB> to navigate the menus.
• No data is persisted until Review & Save is selected and validation has completed. A report will be shown detailing which sources were successfully configured and which not.

Pre-configuration and YAML export/import

Configuration of sources can be done without a functional stack and atolioctl can output the configuration into a YAML file, which can be further hand-modified and imported when the stack is deployed.

Prior to an initial deployment, pre-configuring sources can expedite the initial deployment procedures and time on call. To configure sources before the stack is available, you can pass the --config-file option:

atolioctl configure --config-file atolio.yaml

The file does not need to exist prior to running this command. You’ll be presented with the TUI and will be able to select and configure sources from the registry. Once configured, you can Review & Save per typical procedure and atolioctl will output an atolio.yaml file with the (valid) configuration persisted within the filesystem.

As a reminder, the stack is not yet running at this point. Configuration of most sources does not require an operational stack as the validation is simply making a call to the source API and there is no expectation of the source’s connector running yet. There are some exceptions (in particular Jira, Confluence, and BitBucket)

Once desired sources have been configured and the stack deployed, you can import the pre-configured file. Note at this point it’s expected Feeder is operational and port-forwarding to the service address can occur:

atolioctl configure import atolio.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls
Writing key: /lumen/connectors/github/default/github
Writing key: /lumen/connectors/linear/default/linear
Writing key: /lumen/connectors/notion/default/notion
Writing key: /lumen/connectors/slack/default/slack
Writing key: /lumen/connectors/web/default/web
Writing key: /lumen/connectors/bitbucket/default/bitbucket

Similarly, source configuration of the stack can be exported to a local file. This can be useful if wishing to view the entire configuration in a human-readable YAML file, or perhaps store a backup of the configuration in a vault like 1Password. This file will be in the same format as the one imported in the prior step, allowing a loop of export, modify, import:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 atolio.yaml --disable-feeder-tls

Both export and import commands can also combined into a single edit command that exports the current source configuration into a text editor (specified by the EDITOR environment variable) allowing manual edits. Upon saving/closing the editor, the configuration will be imported. This command is invoked a follows:

atolioctl configure edit --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 --disable-feeder-tls

Common Fields: Resources

Atolio supports the ability to avoid certain content from being indexed at all. The example below shows how to specifically exclude a Jira project (with the project key “SP”).

When configuring the Jira source, enter the following in the Resources field:

project:
    excluded:
        - "SP" # The Secret Project

The excluded keyword contains a list of Jira project keys to be excluded from indexing. Similarly you can also use the included keyword to achieve the opposite (in that case only those resources that are specifically listed will be indexed, nothing else). Note that the list entries (e.g. SP in the example above) are case sensitive.

In the absence of both included (opt-in) or excluded (opt-out), everything that the credentials (e.g. API key) have access to will be indexed. Note that this is a layer on top of the permissions model provided to the source. Anything added to included will only be indexed if the underlying credentials grant the indexing job access.

This feature is supported by a majority of Atolio connectors, see each connectors respective documentation for details and scope of inclusions / exclusions.

Source Configuration

1 - Atlassian Bitbucket Connector

The Atlassian Bitbucket connector indexes repositories, projects, pull requests, commits, and comments across your Bitbucket instance.

The connector supports Atlassian Bitbucket Cloud.

Installing the Bitbucket App Descriptor

In order to target your specific Atolio deployment, the Atlassian Bitbucket connector requires a private listing to be installed instead of the Atlassian marketplace version. This can be disabled again after the connect app is installed.

In your Bitbucket instance:

1. Under Settings -> Workplace settings, navigate to Installed apps under the APPS AND FEATURES section.
2. After clicking Enable development mode, click the link for Install app from URL.

3. Specify the URL provided by your Deployment Engineer. It will be similar to the following with your chosen DNS name:

• https://relay.search.yourdomain.com/connectors/bitbucketcloud/default/install/atlassian-connect.json
• Note: if you have multiple Bitbucket instances, please make sure to provide the correct instance name in the URL above (i.e. default in the example).

Once the Atolio Connect app has been installed, the configuration will be stored securely in your Atolio deployment. You can now disable the development mode as needed.

2 - Atlassian Confluence Connector

The Atlassian Confluence connector indexes spaces, pages, and comments across your Confluence instance.

The connector supports Atlassian Confluence Cloud.

Installing the Confluence App Descriptor

In order to target your specific Atolio deployment, the Atlassian Confluence connector requires a private listing to be installed instead of the Atlassian marketplace version, as explained here. This can be disabled again after the connect app is installed.

To start, within your Confluence instance:

1. Navigate to the application management page by clicking Apps -> Manage your apps.
2. Click Settings at the bottom, then enable private listings and development mode before clicking Apply.

3. The Upload app link will now be enabled.

4. Specify the URL provided by your Deployment Engineer. It will be similar to the following with your chosen DNS name:

• https://relay.search.yourdomain.com/connectors/confluencecloud/default/install/atlassian-connect.json
• Note: if you have multiple Confluence instances, please make sure to provide the correct instance name in the URL above (i.e. default in the example).

Once the Atolio Connect app has been installed, the configuration will be stored securely in your Atolio deployment. You can now disable the development mode and private listings as needed.

(Optional) Scope Confluence indexing

If you want fine-grained control over which Confluence spaces should be indexed, provide a list of inclusions or exclusions to your Deployment Engineer: they can use the includes or excludes feature in Atolio configuration to control the relevant spaces that should be indexed.

3 - Atlassian Connector for Jira & Confluence Data Center

The Atlassian Connector for Jira & Confluence Data Center indexes:

• Projects, issues, and comments across your Jira instance.
• Spaces, pages, and comments across your Confluence instance.
• Users and groups in Jira & Confluence for the purpose of understanding permissions.

The apps have been created with Atlassian SDK and designed to be installed and work for:

• Confluence Server/DC v8.2.3 or later.
• Jira Server/DC v9.10.0 or later.

Installing the App

First, check if you can upload apps by following these steps:

1. Navigate to Administration (gear icon) and click on Manage apps.
2. Under ATLASSIAN MARKETPLACE, click on Manage apps.
3. Look for the Upload app link on the right.
4. If you are unable to upload plugins on your instance, be sure to enable this option on your instance:

• Jira Data Center: https://confluence.atlassian.com/jirakb/how-to-re-enable-plugin-upload-1364557898.html.
• Confluence Data Center: https://confluence.atlassian.com/doc/managing-system-and-marketplace-apps-25788666.html.

1. Click on Upload app.
2. Download the plugin file, e.g. https://atolio.s3.us-west-2.amazonaws.com/jira-dc-connector-0.0.0.3.jar.
3. Provide the file for installation and click Upload.
4. Ensure that the plugin is installed.

Note: you may choose to disable plugin uploads after this step, following the instructions above.

Provide Configuration

Before your configure the plugin, you will need to work with your Deployment Engineer to generate an authentication token for the Jira DC or Confluence DC connector instance. To generate the necessary token, have them run the following command with the appropriate arguments:

atolioctl connector create-jwt "atolio:jiradc:default:jiradc" --jwt-secret-sdk <secret> --jwt-audience-sdk https://search.yourdomain.com --jwt-issuer-sdk https://search.yourdomain.com

The issuer and audience are based on the domain name as specified in config.hcl (i.e. the Terraform lumen_domain_name variable with the URL scheme https:// prefixed).

If you have the values-lumen.yaml and values-lumen-templated.yaml locally available you can run the following command instead (the directory is e.g. deploy/terraform/aws):

atolioctl connector create-jwt --config-dir your-config-dir "atolio:jiradc:default:jiradc"

You can now configure the plugin to point to your Atolio deployment:

1. Under the app, click on Configure.
2. Under “Feeder Settings”, provide the following:

• Host - the feeder address, which will be the feed subdomain with your chosen DNS name, e.g. feed.search.yourdomain.com.
• Port - the port to communicate on, e.g. 443.
• JWT Token - the token generated by your Deployment Engineer.

3. (Optional) Under “Project Settings”, you can choose to index all projects or specify certain projects.
4. Under “Backfiller Status”, click Start Process.

4 - Atlassian Jira Connector

The Atlassian Jira connector indexes projects, issues, and comments across your Jira instance.

The connector supports Atlassian Jira Cloud.

Installing the Jira App Descriptor

In order to target your specific Atolio deployment, the Atlassian Jira connector requires a private listing to be installed instead of the Atlassian marketplace version, as explained here. This can be disabled again after the connect app is installed.

To start, within your Jira instance:

1. Navigate to the application management page by clicking Apps -> Manage your apps.
2. Click Settings at the bottom, then enable private listings and development mode before clicking Apply.

3. The Upload app link will now be enabled.

4. Specify the URL provided by your Deployment Engineer. It will be similar to the following with your chosen DNS name:

• https://relay.search.yourdomain.com/connectors/jiracloud/default/install/atlassian-connect.json
• Note: if you have multiple Jira instances, please make sure to provide the correct instance name in the URL above (i.e. default in the example).

Once the Atolio Connect app has been installed, the configuration will be stored securely in your Atolio deployment. You can now disable the development mode and private listings as needed.

(Optional) Scope Jira indexing

If you want fine-grained control over which Jira projects should be indexed, provide a list of inclusions or exclusions to your Deployment Engineer: they can use the includes or excludes feature in Atolio configuration to control the relevant projects that should be indexed.

5 - GitHub Connector

The GitHub connector enables the indexing of pull requests, commits, comments, and markdown in your GitHub repositories.

Create Access Token

To install the GitHub connector, use a GitHub account with admin access to the organization (requires read access to teams). Using this account, create a personal access token (currently Classic only) according to these instructions. You will need to ensure repo and user scopes are set (as shown below):

Note: for GitHub Enterprise Server installations, you would need to enable the site_admin scope too.

(Optional) Create GitHub App

Note: GitHub App flow does not work with GitHub Enterprise Server installations. You would need to use a personal access token like above

Instead of using a personal access token, a GitHub app can be created to install the GitHub connector. This provide better rate limits, especially if your org is on Enterprise Cloud. You can create a GitHub app according to these instructions.

You will need to ensure the following app permissions are set:

Repository Permissions:

Organization Permissions:

Setup Webhooks

If live indexing of GitHub content is desired, you will want to setup Webhooks. Go to Settings->Webhooks->Add Webhook. After authentication, you can now setup the webhook with the following configs:

The following events are indexed at the repository level:

• Collaborator add, remove, or changed
• Commit comments
• Issues
• Issue comments
• Pull requests
• Pull requests reviews
• Pull requests reviews comments
• Pushes
• Repositories

The following events are indexed at the organization level:

• Team

Please manually check the boxes for each of the events above.

Warning: Setting up webhooks at the organization level will send webhook events for all repositories within the organization. If this is not desireable, please make changes to the configuration to only process and index specific repositories.

Provide Configuration

Provide these configuration values to your Deployment Engineer:

• OrgName should be the identifier for your org (e.g. atolio)
• ApplicationID is your GitHub app ID, can be found at your GitHub app settings page. This field only affects GitHub and GitHub Enterprise Cloud installations with no effects to GitHub Enterprise Server installations
• InstallationID is created on the first installation of the GitHub app on your org, can be found at the end of the URL for your installation: https://github.com/organizations/{ORG}/settings/installations/{INSTALLATION_ID}. This field only affects GitHub and GitHub Enterprise Cloud installations with no effects to GitHub Enterprise Server installations
• SamlAuthenticationLevel is where you enabled SAML authentication, 2 options: enterprise or org
• EnterpriseBaseURL is the hostname of your GitHub Enterprise Server instance
• EnterpriseSlug is your GitHub Enterprise name

Note: If you have many GitHub repositories, it may be worth using the includes or excludes feature in Atolio configuration to control the relevant repositories that should be indexed.

The GitHub connector provides two types of inclusion/exclusions in the resources field of the configuration as illustrated in the example below:

resources:
    repository:
        included:
            - lumen-infra
    file:
        included:
            - .md

The repository name lumen-infra in this example is the repo part in the repository’s URL (which has the following generic format: https://github.com/repos/{owner}/{repo}). The example above will only index the lumen-infra repository.

File filtering is done of the basis of file extension names (including the preceding dot .). The example above will configure the connector to index all markdown files. Note that by default file filtering is completely disabled as it is a relatively time consuming process.

Provide Secrets

Provide these secrets to your Deployment Engineer:

• WebhookSecret is the webhook secret previously created
• Token is the personal access token
• AppPrivateKey is the GitHub App private key in PEM format

(Optional) Configure GitHub user mappings

If employees have brought pre-existing GitHub accounts to their company’s GitHub organization, the identity resolver cannot automatically map them to company user identities. For this reason, the GitHub configuration contains a user mapping table. For example, the entries below map two GitHub user IDs to the user’s username (i.e. email address) in the organization:

12345678:  user1@example.com # "user1"
87654321:  user2@example.com # "user2"

The numerical GitHub ID can easily be obtained using the following: https://api.github.com/users/{USERNAME} in a browser where {USERNAME} is replaced with the GitHub username which can be found in the user’s profile (under the full name printed below the user’s avatar). The ID is in the id field of the JSON response. Adding an entry like this for every user will allow Atolio to link the GitHub user IDs to other sources.

6 - GitLab Connector

The GitLab connector enables indexing of merge requests, commits, and comments across your GitLab projects. Atolio supports GitLab Server (minimum version v16.5-ee) and GitLab Cloud.

Create Access Token

As an admin, perform the following steps:

• Login as admin user
• Go to https://gitlab.company.com/-/user_settings/personal_access_tokens
• Click “Add New Token”
• Give it a name and expiration date, set scope as read_api
• Copy generated token

Setup Webhooks

If live indexing of GitLab content is desired, you will want to setup Webhooks. If you have GitLab Premium you can set up webhooks for groups. If you don’t have GitLab Premium, you need to set up a webhook for each project separately.

• Go to Settings/Webhooks
• Set URL to https://relay.{search.company.com}/connectors/gitlab/{instance}/callback/gitlab
• Set a strong “secret token”
• Check “Comments”, “Issue Events” and “Merge Request Events” under Triggers
• Keep “Enable SSL verification” checked

You also need to set up system hooks for the GitLab instance. This is done by an admin user and is required for getting live project updates.

• Go to Admin Area/System Hooks
• Set URL to the same as the webhook URL
• Set secret token as the same as the webhook secret token
• Keep “Enable SSL verification” checked

Provide Configuration

With the Access Token and Webhook Secret in hand, provide these configuration values to your Deployment Engineer:

• BaseURL is left blank when using gitlab.com. For self-hosted installations specify the base URL for your instance, e.g. gitlab.example.com
• WebhookSecret is the webhook secret previously created
• Token is the personal access token

Note: If you have many GitLab projects, it may be worth using the includes or excludes feature in Atolio configuration to control the relevant projects that should be indexed.

An example inclusion/exclusion configuration will look like the following:

resources:
  group:
    included:
      - examplegroup # Index "examplegroup" from GitLab

GitLab SaaS (gitlab.com) specific configuration

If you’re using the SaaS GitLab product hosted at gitlab.com instead of a self-hosted installation you’ll need to follow the above steps to provide an inclusion filter for at least one group you’d like to index. This is necessary to prevent the connector from attempting to index the entirety of the public SaaS GitLab deployment.

Configuring GitLab user mappings

If people are using their company emails in GitLab, the identity resolver cannot automatically map them to company user identities. For this reason the GitLab configuration contains a user mapping table. For example, the entries:

'12345678':  user1@example.com
'87654321':  user2@example.com

Map two GitLab user IDs to the user’s username (i.e. email address) in the organization. The numerical GitLab ID can easily be obtained using the following URL in a browser where USERNAME is replaced with the GitLab username. The ID is in the id field of the JSON response. On the command this can be obtained by using curl and jq as follows (using the user ID USERNAME in this example):

curl 'https://gitlab.company.com/api/v4/users?access_token=ACCESS_TOKEN&search=USERNAME' | jq '.[0].id'

Adding an entry like this for every user will allow Atolio to link the GitLab user IDs to other sources.

7 - Google Connector

The Google connector enables the indexing of:

• Documents, Sheets, Slides, and other Files in Google Drive
• Email in Gmail
• Users and Groups as metadata in your Google Workspace to enable user mapping / permissions

Content and updates are streamed as changes are detected and processed in your Google Workspace.

Create Terraform Service Account for Google

Terraform will need a service account to make changes to your GCP project. We will grant it permissions which will essentially allow it to create, read, and delete all project resources. As such, you should keep the credentials for this account safe and not reuse them. The Terraform scripts create a much more restricted service account for normal operation.

1. Visit the IAM Service Accounts page https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously to set up the Terraform service account.
2. Click CREATE SERVICE ACCOUNT
3. Choose a name. While it can be anything, the name “Atolio Terraform” is recommended.
4. Select CREATE AND CONTINUE.
5. Grant the following roles: “Owner”, “Security Center Admin”, “Project IAM Admin” (without conditions).
6. Click DONE. Skip step 5, you will not need to grant other users access to this service account.

Now the service account is created we can create API keys:

1. Navigate to https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously.
2. Click the service user that was created in the previous step.
3. Select the KEYS tab click the ADD KEY pull-down menu
4. Select Create new key and use JSON for key type. Click CREATE.

This will create another JSON file with the API key and related information. This file will be needed by your Deployment Engineer (in their deploy/terraform directory).

In order for Terraform to programmatically make modifications, you need to manually enable the Google Cloud Resource Manager API.

1. Navigate to https://console.cloud.google.com/apis/library
2. Search for “Cloud Resource Manager API”
3. Select “Cloud Resource Manager API”. Click ENABLE.

The project will use additional APIs, but now that this API is available to Terraform, the Atolio deployment scripts will use Terraform to enable those APIs automatically.

Grant Service Account Permissions

The Google Workspace domain needs to grant permission to the integration’s default service account to perform operations. Unfortunately Terraform can’t automate this, so it has to be performed manually.

Your Deployment Engineer will need to make note of the unique ID for the default App Engine service account and the list of scopes. Be aware that this ID is of the default service account, typically named App Engine default service account. It is not the Service Account you’ve previously created.

And the list of scopes are:

https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/admin.directory.group.readonly,https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/admin.directory.domain.readonly,https://www.googleapis.com/auth/drive.activity.readonly

We now need to set up Domain-wide Delegation:

1. Navigate to the Google Admin console: https://admin.google.com/ac/owl. An account with admin access is required.
2. Scroll down and click “MANAGE DOMAIN WIDE DELEGATION”.
3. Click Add new and enter your service account client ID (obtained from your Deployment Engineer in the previous step). No overwrite.
4. Enter the value for google_service_account_id as obtained from your Deployment Engineer.
5. Enter the list of scopes as listed above (as a single comma-delimited string).
6. Click AUTHORIZE.

Now Atolio is authorized to retrieve Google Workspace data.

Installing the Gmail / Google Drive Connector

Note: For this step, your deployment engineer will be required.

The Gmail and Google Drive connectors depend on a shared Google Workspace configuration section. Google Workspace requires three properties to be set during configuration:

• ProjectID must reference the project created in GCP. The id, not the name, must be specified.
• ServiceUser is currently a reference to a super user or administrator within your organization. For example, the primary IT email. This may change in the future.
• ServiceAccountKey is the base64-encoded string as obtained during initial install of the Google source.

To obtain the service account key, your deployment engineer will run the following Terraform command:

terraform output -raw google_connector_service_account_key

And optionally:

• Domains is a comma-delimited list of domains applicable to this deployment.
• Parallelism is an optimization to increase the concurrency with respect to indexing Google data. Default is 32, so if set, should be higher than this value.

Once the Google source is configured, the Google Drive and Gmail sources can be configured with the same values.

8 - Linear Connector

The Linear connector indexes teams, projects, issues, and comments across your Linear workspace.

Note: Atolio currently uses an API key for this connector, but will support a published Integration in a future release.

Create the Linear API key & Webhooks

As an administrator:

1. Navigate to Workspace Settings by selecting your tenant and choosing Preferences.
2. Under My Account, click on API.

To create an API key, complete the following steps: 3. Under Personal API keys, provide a Label in the text box, e.g. Atolio. 4. Click on Create new API key. 5. Copy the value of the key to securely share with your Deployment Engineer.

To create a Webhook, complete the following steps: 6. Under Webhooks, click on Create new webhook. 7. Provide a Label in the text box, e.g. Atolio. 8. For URL, specify the URL provided by your Deployment Engineer. It will be similar to the following with your chosen DNS name:

• https://relay.search.yourdomain.com/connectors/linear/default/callback/linear
• Note: if you have multiple Linear instances, please make sure to provide the correct instance name in the URL above (i.e. default in the example).

9. Under Data change events, ensure the following are checked:
   • Issues
   • Projects
   • Comments
10. Copy the value of the Signing secret to securely share with your Deployment Engineer.
11. Click on Create webhook to finalize the creation of this webhook.

Provide Configuration

Once the API key and Webhook have been created, provide these configuration values to your Deployment Engineer:

• UrlKey is the name of your Linear workspace.
• Token is the personal access token.
• WebhookSecret is the webhook secret.

9 - Microsoft Connector

The Microsoft connector supports:

• Entra ID - for resolving user mappings across apps as an Identity Provider
• SharePoint - for indexing sites, docs, spreadsheets, slides, and other files
• Teams - for indexing channels, chat, and messages
• OneDrive - for indexing docs, spreadsheets, slides, and other files
• Outlook - for indexing emails and calendar events

Installing the Microsoft Connector

First, you must create a new App Registration within Microsoft Azure. If you are using Entra ID as an identity provider, then you already created an app registration that can be reused for API access. If you have not done this step, refer to Entra ID - Create New App Registration.

With the app registration created, click on API Permissions and add a permission:

• Select Microsoft Graph
• Type of permission is Application Permissions
• Search for and add the following:
  • Application.Read.All
  • Group.Read.All
  • GroupMember.Read.All
  • User.Read.All
• If necessary, add the additional permissions that are necessary if you are using the SharePoint connector, Teams connector, OneDrive connector, or Outlook connector.
• Once added, select Grant admin consent for ... where ... is your tenant name. This link is beside the Add a permission link.

Provide Configuration

Provide the following values to your Deployment Engineer for them to configure in Atolio:

• ClientCert is the base 64-encoded content of the .PFX file obtained in Azure AD - Create New App Registration
• ClientCertPassword is the certificate’s private key password, if required
• TenantId is the Directory (tenant) ID the app registration belongs to
• ClientId is the Application (client) ID

9.1 - OneDrive Connector Setup

Index OneDrive files by granting the necessary API Permissions in your App Registration.

Grant OneDrive API Permissions

After following the steps in Entra ID - Create New App Registration to create a new App Registration, you will need to add additional API permissions to index OneDrive resources.

In addition to the Entra ID permissions listed in the Microsoft Connector for the Microsoft Graph, you will also need:

• Under Microsoft Graph API:
  • Files.Read.All
• Once added, select Grant admin consent for ... where ... is your tenant name. This link is beside the Add a permission link.

Determine Indexing Scope

If you have many drives, it may be worth using the includes feature explained in the Common Fields: Resources section to index only the relevant drives.

An example inclusion/exclusion configuration will look like the following:

resources:
  group:
    excluded:
      - secret-group@example.com # Ignore drives whose sole owner is "secret-group"
    included:
      - public-group@example.com # Index drives accessible by "public-group"
  user:
    excluded:
      - secret-user@example.com # Ignore drives whose sole owner is "secret-user"

9.2 - Outlook Connector Setup

Index Outlook emails and events by granting the necessary API Permissions in your App Registration.

Grant Outlook API Permissions

After following the steps in Entra ID - Create New App Registration to create a new App Registration, you will need to add additional API permissions to index Outlook resources.

In addition to the Entra ID permissions listed in the Microsoft Connector for the Microsoft Graph, you will also need:

• Under Microsoft Graph API:
  • Mail.Read
  • Calendars.Read
  • MailboxSettings.Read
  • Domain.Read.All
• Once added, select Grant admin consent for ... where ... is your tenant name. This link is beside the Add a permission link.

Determine Indexing Scope

If you have many mailboxes, it may be worth using the includes feature explained in the Common Fields: Resources section to index only the relevant mailboxes.

An example inclusion/exclusion configuration will look like the following:

resources:
  user:
    excluded:
      - secret-user@example.com  # Ignore mailboxes of secret-user@example.com

Manually Specify Email Address Domains

If email address domains of your organization are already listed in MS Graph API /domains query, you can skip this step.

For Atolio to identify contributors of Outlook resources correctly, a list of email address domains must be known. If users in your organization have email addresses like user1@some-domain.com and user2@other-domain.com, this list should consist of some-domain.com, and other-domain.com.

You can manually specify an email address domain list as part of your configuration in the following format:

known-domains:
  - some-domain.com
  - other-domain.com

9.3 - SharePoint Connector Setup

Index SharePoint sites, pages, and files by granting the necessary API Permissions in your App Registration.

Grant SharePoint API Permissions

After following the steps in Entra ID - Create New App Registration to create a new App Registration, you will need to add additional API permissions to index SharePoint resources.

In addition to the Entra ID permissions listed in the Microsoft Connector for the Microsoft Graph, you will also need:

• Under Microsoft Graph API:
  • Files.Read.All
  • Sites.Read.All
• Once added, select Grant admin consent for ... where ... is your tenant name. This link is beside the Add a permission link.

Determine Indexing Scope

If you have many SharePoint sites, it may be worth using the includes feature explained in the Common Fields: Resources section to index only the relevant sites.

9.4 - Teams Connector Setup

Index Teams channels, chat, and messages by granting the necessary API Permissions in your App Registration.

Grant API Permissions for Microsoft Teams

After following the steps in Entra ID - Create New App Registration to create a new App Registration, you will need to add additional API permissions to index Teams resources.

In addition to the Entra ID permissions listed in the Microsoft Connector for the Microsoft Graph, you will also need:

• Under Microsoft Graph API:
  • Channel.ReadBasic.All
  • ChannelMember.Read.All
  • ChannelMessage.Read.All
  • Chat.Read.All
  • ChatMember.Read.All
  • ChatMessage.Read.All
  • Team.ReadBasic.All
  • TeamMember.Read.All
• Once added, select Grant admin consent for ... where ... is your tenant name. This link is beside the Add a permission link.

Determine Indexing Scope

If you have many teams, it may be worth using the includes feature explained in the Common Fields: Resources section to index only the relevant teams.

Limitations

• Only first 1000 replies to a channel post will be indexed, due to limitations with the Microsoft API.

10 - Notion Connector

The Notion connector indexes databases, pages, and comments across your Notion workspace.

Create the Notion integration

As an administrator, you can either follow the steps below or navigate directly to My Integrations:

1. Click on Settings & members.
2. Under Workspace, click on Connections.
3. Click the link for Develop or manage integrations.

Create a new Internal integration by following the steps below:

1. Choose the associated workspace and give it an appropriate name, e.g. Atolio.
2. (Optional) Use the Atolio logo here, if needed.
3. Under Secrets, copy the Internal Integration Secret to share with your Deployment Engineer.
4. Under Capabilities:

• Only “Read content” and “Read comments” are necessary. The other permissions can be removed.
• Ensure that “Read user information including email addresses” is enabled.
• Click Save changes.

Once the integration has been created, provide the secret copied above to your Deployment Engineer for configuration in your Atolio deployment.

11 - Okta Connector

The Okta connector provides identity mappings for each user based on the apps that they use.

Note: As part of Atolio Configuration, this connector should be selected as Identity Provider if it used to resolve app-user mappings.

Creating an API Token

You may have already performed some of the necessary steps in Configuring Authentication. Atolio also requires an API token created using these these instructions or by simply going to https://example.okta.com/admin/access/api/tokens.

Provide the following configuration values to your Deployment Engineer in order to complete these steps:

• Hostname, the host name for your Okta instance (e.g. example.okta.com)
• Token, the API Token created above

12 - Salesforce Connector

The Salesforce connector enables the indexing of Salesforce objects, such as Accounts and Opportunities, using the Salesforce REST API by creating a Connected App in your Salesforce instance.

Setup Connected App

Note: The setup assumes that you are using the lightning experience of Salesforce.

As a best practice, we recommend creating a new user account or identifying an existing user account to be used for integrations. The profile for this user account will be given access to the Connected App in a later step.

1. As an administrator, navigate to Setup.
2. Under “Platform Tools”, go to “Apps” -> “App Manager”.
3. Create a “New Connected App”.

• Choose a Connected App Name and API Name, e.g. Atolio.
• Choose an email for Contact Email.
• (Optional) For Logo Image URL and/or Icon URL, you can use the Atolio logo here.

4. Under API (Enable OAuth Settings), select Enable OAuth Settings.

• For Callback URL, enter a valid URL (e.g. https:/) - Atolio does not use this login flow for authentication.
• For Selected OAuth Scopes, add the “Manage user data via APIs (api)” scope.
• Click the Enable Client Credentials Flow and select OK on the prompt below.

5. Click “Save”, then “Continue” to create the Connected App.
6. When your Connected App has been provisioned, go back to “App Manager” and choose “Manage” from the actions drop-down for your Atolio connected app.
7. Click on “Edit Policies”.

• Under “OAuth Policies”, for Permitted Users select the “Admin approved users are pre-authorized” option.
• Under “Client Credentials Flow”, set the Run As to the desired user mentioned above.

Next, you will add the “Connected App Access” to the profile of the user you selected above.

1. From Setup, navigate to “Users” -> “Users” and find your desired integration user. Click on their “Profile” to open the Profile page.
2. Click “Edit”, then click the checkbox to enable your new Connected App for this user profile.
3. Save those changes.

Provide Configuration

To obtain the necessary client credentials:

1. Go back to “App Manager” and choose “View” from the actions drop-down for your Atolio connected app.
2. Under “API (Enable OAuth Settings)”, click the “Manage Consumer Details” button for the “Consumer Key and Secret”.
3. Copy the “Consumer Key” and “Consumer Secret” to be shared with your Deployment Engineer.

Provide the following values to your Deployment Engineer for them to configure in Atolio:

• base_url is the URL of your Salesforce tenant, e.g. https://yourcompany.my.salesforce.com.
• client_id is the “Consumer Key” of the connected app you created.
• client_secret is the “Consumer Secret” of the connected app you created.

13 - ServiceNow Connector

The ServiceNow connector will index:

• IT Service Management (ITSM) tasks - such as incidents, service requests, problems - including work notes and comments.
• Knowledge bases (KBs) and KB articles.

The ServiceNow connector supports all releases from ServiceNow San Diego to the latest.

Note: If you need support for additional modules (e.g. Security Operations) with fields that aren’t indexed in the base tasks, contact your Atolio support team.

Install ServiceNow Update Set

The Atolio ServiceNow plug-in is available as an update set (XML file). The update set will be provided by your Deployment Engineer (from the Atolio GitHub repository) or your Atolio support team. From ServiceNow documentation, follow the steps below to Commit an update set.

1. Elevate privileges to the security_admin role:

2. Navigate to System Update Sets > Retrieved Update Sets.
3. Click the link to Import Update Set from XML. Upload the XML file provided previously.
4. The application is now available as a retrieved update set with state Loaded. Click on Atolio Connector to open the update set.
5. If necessary, skip or accept updates to resolve any problems.
6. Click Commit Update Set to install the application.

Note: You can view the Commit log by navigating to System Update Sets > Update log and filtering to the Atolio Connector Update set.

Setup Application Credentials

First, create an application user by:

1. Navigating to System Security > Users and Groups > Users.
2. For User ID, choose an appropriate name, e.g. atolio_connector. Click Submit.
3. In the new user record that you just created:

• Under Roles, click Edit... and add x_penti_atolio.atolio_connector to the Roles List before clicking Save.
• Click Set Password to generate a password for this application user.

Next, enable the connector API by:

1. Under the Atolio Connector application, navigate to Integration - Atolio Connector > Administration > Properties.
2. If necessary, enable the connector’s API and integration with ServiceNow Knowledge Management.

Provide Configuration

Provide the following values to your Deployment Engineer for them to configure in Atolio:

• Instance URL is the URL of your ServiceNow tenant.
• Username is the User ID of the application user you created, e.g. atolio_connector.
• Password is the password of the application user you created.

14 - Slack Connector

The Slack connector indexes messages across public channels, private channels, and threads. It will only index messages where the Atolio bot has been invited.

Create Slack App

In order to target your specific Atolio deployment, the Slack connector for Atolio requires a custom Slack app created from a manifest.

First, create the Slack app according the following instructions:

1. In the Slack app dashboard (https://api.slack.com/apps), click on “Create New App”.
2. Select “From an app manifest” when prompted.
3. Select the workspace where the app should be installed.
4. Select the “YAML” option and provide the app manifest by editing the Atolio Slack app manifest template located at https://relay.search.yourdomain.com/connectors/slack/default/install/manifest.yml.
5. Click on “Create”. Once the app is created, open the “Basic Information” page of your new app - it can be accessed from the Slack app dashboard (https://api.slack.com/apps).
6. Locate the “App Credentials” heading and store the client ID, client secret, and signing secret in a safe place. These will be used to configure Slack shortly.
7. (Optional) To provide an app icon, locate the “Display Information” heading and upload the app icon provided here.

Note: it may be required to distribute the app. This can be done by clicking Manage Distribution in the Slack app dashboard.

Provide Configuration

Once the Slack app has been created, provide these configuration values to your Deployment Engineer:

• DefaultTeamID is the Slack team ID for your workspace (e.g. T012ABCDEG3)
• AutoJoinMessage is a message shared by the Atolio integration when it auto joins a channel (which is a separate setting), e.g. This public channel is searchable with Atolio: %URL
• ClientId, ClientSecret, and SigningSecret are the secrets obtained in the previous step.
• Token is a secret which will be populated by the next step. You may leave this blank.
• ConversationTypes are the types of conversations being indexed. If empty, this defaults to public_channel,private_channel,mpim

Once configured, you will finalize the installation of the Slack connector by visiting the following URL (make sure to use your own DNS name for the host part of the URL and include the correct instance name): https://relay.search.yourdomain.com/connectors/slack/default/install/setup. Also make sure the Atolio Web application is authenticated (using your SSO provider) before navigating to this URL to avoid an authentication error. The URL https://relay.search.yourdomain.com/install/connectors/slack/default/install/oauth/callback must be listed as a Redirect URLin the Slack app manifest. For example, with Okta this was done as part of Okta - Create OAuth Client.

In order for Slack conversations to be indexed, the Atolio Slack Bot needs to be added to the conversation (i.e. public or private channel). By default, this needs to be done manually, but the bot can also be configured to auto-join public channels through configuration. If a bot is added to a channel which is later filtered out by configuration, auto-leave can be configured similarly. There is also the option of sending a message to channels which the bot auto-joins. The default behavior is not to send a message on auto-join, but this can be changed by setting the AutoJoinMessage field as described above. Messages are formatted using markdown, and the deployment-specific URL can be included with the special verb %URL.

The Slack Team ID can for instance be copied from the URL after logging into your company’s Slack Web app. For instance navigate to https://app.slack.com/ and note down the team ID in the URL that you get redirected to (e.g. T012ABCDEG3 in https://app.slack.com/client/T012ABCDEG3/.../...).

15 - Staffbase Connector

The Staffbase connector enables indexing of various resources from your Staffbase platform, including Users, Campaigns, Posts, and Comments. This integration allows you to search and access Staffbase content directly through Atolio.

Set Up Administrator API Key and Retrieve Base URL

To set up the Staffbase connector, you’ll need to create an API key with administrator privileges. Follow these steps:

1. Log in to your Staffbase account as an administrator.

2. Navigate to the Admin Panel:

   a. Click your user profile at the top right, then click ‘Studio’ in the drop-down menu.

   

   b. Click the gear icon at the top right corner.

   

   c. In the new page, you will see a column on the left. Click the ‘API Access’ tab.

   

3. Click the “Create New API Key” button.

4. Provide a descriptive name for the key, such as “Atolio Connector”.

5. Under “Access”, select “Read-only” to ensure full access to all resources.

6. Click “Create”.

7. Copy and securely store the generated API key. Note that you won’t be able to view this key again, so make sure to save it in a secure location.

8. On the same page, you’ll also see the base URL for your instance:

Provide Configuration

After creating the API key, provide the following information to your Deployment Engineer:

• API Key: The administrator API key you just generated.
• Staffbase Instance URL: The URL of your Staffbase instance (e.g., https://yourcompany.staffbase.com).

16 - Web Connector

The Web connector enables the indexing of HTML web pages.

Enable Access

The Web connector indexes unauthenticated web page content which can be accessed from your Atolio deployment.

First, determine where your website is hosted. The Web connector can crawl HTML content by either:

• Navigating URLs from a configured baseURL.
• Browsing an Amazon S3 bucket from a configured url.

In order to ensure that the content is accessible in your Atolio deployment, ensure that your Deployment Engineer is aware of the necessary network or AWS permissions as part of the infrastructure deployment.

Note: You may also need to ensure that your HTML content is server-side rendered (SSR) in order to index the complete website content.

(Optional) Provide Filters

Additional filters can be configured by your Deployment Engineer in order to:

• Include or exclude pages, e.g. particular URLs that match a given regex.
• Remove extraneous content, e.g. table of contents.

17 - Zendesk Connector

Zendesk connector enables the indexing of tickets, and help center articles.

Prerequisites

The following configuration should be set on your Zendesk instance, by the account owner or an admin user, before configuring the Zendesk connector.

• Go to Zendesk admin panel
  • https://{your-zendesk-subdomain}.zendesk.com/admin/apps-integrations/apis/zendesk-api/settings
• Enable “API access”
  • If you don’t see such an option, it means it’s already enabled.
• Enable “Token access”
• Create a new API token
  • If you already have an API token other integrations, it’s still advised to create a new one for the Zendesk connector, and provide a descriptive name such as atolio-connector-token. Note the token value since you cannot retrieve it after its creation. The token will be used to configure the connector.
• Create a new admin user
  • If you already have an admin user for other integrations, it’s still advised to create a new admin user for the Zendesk connector, and provide a descriptive username such as atolio-connector-admin-user@<your-org-domain>. The username will be used to configure the connector.

After the configuration, your API settings page should look like the following:

Provide Configuration

Provide the following configuration and answers to your Deployment Engineer:

• Your Zendesk subdomain
• API token, ideally the one you newly created under the name atolio-connector-token
• Admin user, ideally the one you newly created under the name atolio-connector-admin-user@<your-org-domain>
• Whether you want “Help Center” content to be indexed or not
  • “Help Center” is an optional feature of Zendesk for hosting user-generated content (e.g., articles). If you use this feature and want your Help Center content to be indexed, mention this to your Deployment Engineer for them to configure it appropriately.
  • If you’re not sure if your organization uses “Help Center” functionality, try visiting https://{your-zendesk-subdomain}.zendesk.com/knowledge. A 404 error means your org is not using it.

Deployment engineer will also generate a “webhook secret”, and add it to the configuration. Please do not alter the configuration of any “webhook” or “trigger” starting with the word lumen. These are configured by the connector automatically, to index content changes in real-time.

Content Filtering

Atolio supports the ability to avoid certain content from being indexed at all. You can provide inclusions or exclusions with respect to each resource type.

• Users can be filtered w.r.t. their emails.
• Groups can be filtered w.r.t. their names.
• Organizations can be filtered w.r.t. their names.
• Tickets can be filtered w.r.t. their statuses (e.g., “open”, “closed”)

For Help center content:

• User Segments can be filtered w.r.t. their names.
• Articles can be filtered w.r.t. their section IDs.
  • Section ID is a number that can be identified by executing the following command:
    • curl -L -u "{admin-user}/token:{admin-token}" https://{zendesk-subdomain}.zendesk.com/api/v2/help_center/articles/{article-id}.json | jq ".article.section_id"
    • Replace {admin-user}, {admin-token}, {zendesk-subdomain}, and {article-id} values appropriately.

An example of inclusions/exclusions file would be as follows:

resources:
    user:
        excluded:
            - super_secret_user@example.com
    group:
        included:
            - A group name
            - Another group name
    organization:
        included:
            - An organization name
    user_segment:
        included:
            - A segment name
    ticket:
        excluded: 
            - closed  # ticket status
    article:
        excluded:
            - "12345" # section id

If you need content filtering, provide the inclusions & exclusions similar to the file above to your Deployment Engineer.