This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Google Connector

Index documents, sheets, slides, files, sites and email across your Google Workspace (Google Drive, Gmail and Google Sites).

The Google connector enables the indexing of:

  • Documents, Sheets, Slides, and other Files in Google Drive
  • Email in Gmail
  • Pages in Google Sites
  • Users and Groups as metadata in your Google Workspace to enable user mapping / permissions

Content and updates are streamed as changes are detected and processed in your Google Workspace.

Step 0: Create Terraform Service Account for Google

Note: Depending on installation being BYOK, Managed, or otherwise this step may be skipped. Consult with an Atolio Support Engineer for any clarification.

Terraform will need a service account to make changes to your GCP project. We will grant it permissions which will essentially allow it to create, read, and delete all project resources. As such, you should keep the credentials for this account safe and not reuse them. The Terraform scripts create a much more restricted service account for normal operation.

  1. Visit the IAM Service Accounts page https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously to set up the Terraform service account.
  2. Click CREATE SERVICE ACCOUNT
  3. Choose a name. While it can be anything, the name “Atolio Terraform” is recommended.
  4. Select CREATE AND CONTINUE.
  5. Grant the following roles: “Owner”, “Security Center Admin”, “Project IAM Admin” (without conditions).
  6. Click DONE. Skip step 5, you will not need to grant other users access to this service account.

Now the service account is created we can create API keys:

  1. Navigate to https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously.
  2. Click the service user that was created in the previous step.
  3. Select the KEYS tab click the ADD KEY pull-down menu
  4. Select Create new key and use JSON for key type. Click CREATE.

This will create another JSON file with the API key and related information. This file will be needed by your Deployment Engineer (in their deploy/terraform directory).

In order for Terraform to programmatically make modifications, you need to manually enable the Google Cloud Resource Manager API.

  1. Navigate to https://console.cloud.google.com/apis/library
  2. Search for “Cloud Resource Manager API”
  3. Select “Cloud Resource Manager API”. Click ENABLE.

The project will use additional APIs, but now that this API is available to Terraform, the Atolio deployment scripts will use Terraform to enable those APIs automatically.

Step 1: Grant Service Account Permissions

The Google Workspace domain needs to grant permission to the integration’s default service account to perform operations. Unfortunately Terraform can’t automate this, so it has to be performed manually.

Your Deployment Engineer will need to make note of the unique ID for the default App Engine service account and the list of scopes. Be aware that this ID is of the default service account, typically named App Engine default service account. It is not the Service Account you’ve previously created.

And the list of scopes are:

https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/admin.directory.group.readonly,https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/admin.directory.domain.readonly,https://www.googleapis.com/auth/drive.activity.readonly

We now need to set up Domain-wide Delegation:

  1. Navigate to the Google Admin console: https://admin.google.com/ac/owl. An account with admin access is required.
  2. Scroll down and click “MANAGE DOMAIN WIDE DELEGATION”.
  3. Click Add new and enter your service account client ID (obtained from your Deployment Engineer in the previous step). No overwrite.
  4. Enter the value for google_service_account_id as obtained from your Deployment Engineer.
  5. Enter the list of scopes as listed above (as a single comma-delimited string).
  6. Click AUTHORIZE.

Now Atolio is authorized to retrieve Google Workspace data.

Step 2: Deploy the Connector

Before the source can be configured, the connector pod must be running in the cluster. Connectors are deployed by enabling them in values-lumen.yaml. See Deploying Connectors for the full details, including how instances work.

connectors:
  google:
    default:
      enabled: true

default is the instance identifier. Treat it as immutable once set, because the instance is part of every resource identifier the connector ingests. Apply the change by upgrading the lumen Helm chart with create-infra.sh (recommended) or helm upgrade as described in the sources overview.

Step 3: Configure the Source

Source configuration is applied with atolioctl using a YAML file. If you have not already installed atolioctl, follow Configuring Sources first. The commands below also need an API token in JWT_TOKEN and the address the feeder server’s gRPC endpoint, both described next.

atolioctl Prerequisites

JWT_TOKEN holds the API token atolioctl uses to authenticate to the feeder. Generate one from the deployment’s Helm values directory, which holds the values-lumen.yaml and values-lumen-templated.yaml files produced by create-infra.sh. The token is signed with the deployment’s jwtSecretKey, and the atolio:*:*:* match pattern grants access across all connectors, instances, and resources:

export JWT_TOKEN=$(atolioctl connector create-jwt --raw --config-dir . "atolio:*:*:*")

--feeder-address is the gRPC address of the feeder server that is used to read and write source configuration through. The standard path is to port-forward the in-cluster service/feeder and point atolioctl at it on localhost:

# Run K8s port forward to feeder as background to re-use same shell
kubectl port-forward -n atolio-svc service/feeder 8889 &

With that port-forward running in current terminal or another, the examples below use --feeder-address :8889. The empty host means localhost, and because the port is not 443 the connection is made in plaintext, which is what the port-forward exposes. If the feeder is instead published on a load balancer, pass that address with port 443, for example --feeder-address feed.search.example.com:443, which connects over TLS.

Edit and Apply the Configuration

Source configuration is stored as a single document that covers every configured connector. The supported workflow is to export the current configuration, edit it, then import it back. Always start from a fresh export so you do not overwrite a change made elsewhere.

Export the current configuration to a local file:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 lumen-connectors.yaml

Add a Google entry alongside the connectors already in the file, substituting the values you collected in Step 1:

connectors:
  # ...existing connectors left in place...
  google:
    default:
      google:
        connector: google
        instance: default
        source: google
        common:
          enabled: true
          identity-provider: true      # This connector is the identity provider
          cron-spec: 0 7 * * *         # Re-sync users on this schedule
        secrets:
          service_account_key:
            value: "<GOOGLE_SERVICE_ACCOUNT_B64>"
        configuration:
          parallelism: 0
          project-id: "<GOOGLE_PROJECT_ID>"
          service-user: "<GOOGLE_SERVICE_USER>"

Exported entries also carry a rendezvous-completed flag and a status block reporting each source’s current state and last validation result. Those fields are managed by Atolio. Leave them untouched on existing entries, and omit them when adding a new entry by hand as shown above.

Import the edited file to write the configuration back into the stack:

atolioctl configure import lumen-connectors.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889

Import writes the configuration into the stack after checking that the YAML is well formed. It does not contact Google. The connector validates the credentials and connection itself when it starts and picks up the configuration, and it reads users after that. If something is wrong, the source reports as misconfigured and the connector pod logs the error. See Troubleshooting for the specific messages.

Configuration Schema

In the exported file, every source lives at connectors.<connector>.<instance>.<source>, so the Google source sits at connectors.google.default.google.

GoogleSourceConfiguration

Google Workspace connector configuration.

FieldTypeRequiredDefaultDescription
connectorstringyesN/AParent connector to reference (can be self)
instancestringyesN/AMulti-state config variable used for certain connectors, often default
sourcestringyesN/ASub-connector reference (can be self), often object key
commonCommonyesN/ACommon connector settings.
secretsSecretsyesN/ASecret credentials for authentication.
configurationConfigurationyesN/AConnector-specific configuration.

Common

Common settings shared across all connectors.

FieldTypeRequiredDefaultDescription
enabledbooleanyesN/ASet to true to run the source.
identity-providerbooleanyesN/ASet to true so Atolio uses this connector to resolve logins to users.
cron-specstringnoN/ACron expression for periodic re-sync of users (e.g., 0 7 * * *). Omit to sync only on demand.

Secrets

Sensitive configuration values for authentication.

FieldTypeRequiredDefaultDescription
service_account_keySecretValueyesN/AGoogle service account key in Base64 to authenticate to Google Workspace.

SecretValue

A reference to a secret value.

FieldTypeRequiredDefaultDescription
valuestringyesN/AThe secret value (typically Base64-encoded).

Configuration

Google Workspace connector-specific configuration.

FieldTypeRequiredDefaultDescription
domainsstringno""Comma-separated list of allowed customer domains. If empty, all domains are considered.
project-idstringyesN/AGCP project ID under which the integration is operating.
service-userstringyesN/AEmail address of an administrator within the Google Workspace account.
parallelismintegeryes0Number of users to process in parallel. Set to 0 for automatic parallelism based on system resources.

1 - Gmail Connector Setup

Provide necessary API permissions to index Gmail content.

Installing the Gmail Connector

Note: For this step, your deployment engineer will be required.

The Gmail and Google Drive connectors depend on a shared Google Workspace configuration section. Google Workspace requires three properties to be set during configuration:

  • ProjectID must reference the project created in GCP. The id, not the name, must be specified.
  • ServiceUser is currently a reference to a super user or administrator within your organization. For example, the primary IT email. This may change in the future.
  • ServiceAccountKey is the base64-encoded string as obtained during initial install of the Google source.

To obtain the service account key, your deployment engineer will run the following Terraform command:

terraform output -raw google_connector_service_account_key

And optionally:

  • Domains is a comma-delimited list of domains applicable to this deployment.
  • Parallelism is an optimization to increase the concurrency with respect to indexing Google data. Default is 32, so if set, should be higher than this value.

Once the Google source is configured, the Google Drive and Gmail sources can be configured with the same values.

Configuration

Source configuration is applied with atolioctl using a YAML file. If you have not already installed atolioctl, follow Configuring Sources first. The commands below also need an API token in JWT_TOKEN and the address the feeder server’s gRPC endpoint, both described next.

Edit and Apply the Configuration

Source configuration is stored as a single document that covers every configured connector. The supported workflow is to export the current configuration, edit it, then import it back. Always start from a fresh export so you do not overwrite a change made elsewhere.

Export the current configuration to a local file:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 lumen-connectors.yaml

Add a Google entry alongside the connectors already in the file, substituting the values you collected in Step 1:

connectors:
  # ...existing connectors left in place...
  google:
    default:
      # ...existing Google sub-connectors left in place...
      gdrive:
        connector: google              # Reference to parent connector, usually Google
        instance: default
        source: gmail
        common:
          enabled: true
          identity-provider: false     # This connector is a workspace tool, not IDP
        secrets:
          service_account_key:
            value: "<GOOGLE_SERVICE_ACCOUNT_B64>"
        configuration:
          parallelism: 0
          drive-contrib-hydration-method: activity
          project-id: "<GOOGLE_PROJECT_ID>"
          service-user: "<GOOGLE_SERVICE_USER>"

Exported entries also carry a rendezvous-completed flag and a status block reporting each source’s current state and last validation result. Those fields are managed by Atolio. Leave them untouched on existing entries, and omit them when adding a new entry by hand as shown above.

Import the edited file to write the configuration back into the stack:

atolioctl configure import lumen-connectors.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889

Import writes the configuration into the stack after checking that the YAML is well formed. It does not contact Google. The connector validates the credentials and connection itself when it starts and picks up the configuration, and it reads users after that. If something is wrong, the source reports as misconfigured and the connector pod logs the error. See Troubleshooting for the specific messages.

Configuration Schema

In the exported file, every source lives at connectors.<connector>.<instance>.<source>, so the Gmail source sits at connectors.google.default.gmail.

GmailSourceConfiguration

Gmail connector configuration.

FieldTypeRequiredDefaultDescription
connectorstringyesN/AParent connector to reference (can be self)
instancestringyesN/AMulti-state config variable used for certain connectors, often default
sourcestringyesN/ASub-connector reference (can be self), often object key
commonCommonyesN/ACommon connector settings.
secretsSecretsyesN/ASecret credentials for authentication.
configurationGmailConfigurationyesN/AGmail-specific configuration.

Common

Common settings shared across all connectors.

FieldTypeRequiredDefaultDescription
enabledbooleanyesN/ASet to true to run the source.
identity-providerbooleanyesN/ASet to true so Atolio uses this connector to resolve logins to users.
cron-specstringnoN/ACron expression for periodic re-sync of users (e.g., 0 7 * * *). Omit to sync only on demand.

Secrets

Sensitive configuration values for authentication.

FieldTypeRequiredDefaultDescription
service_account_keySecretValueyesN/AGoogle service account key in Base64 to authenticate to Google Workspace.

SecretValue

A reference to a secret value.

FieldTypeRequiredDefaultDescription
valuestringyesN/AThe secret value (typically Base64-encoded).

GmailConfiguration

Gmail-specific connector configuration.

FieldTypeRequiredDefaultDescription
domainsstringno""Comma-separated list of allowed customer domains. If empty, all domains are considered.
project-idstringyesN/AGCP project ID under which the integration is operating.
service-userstringyesN/AEmail address of an administrator within the Google Workspace account.
parallelismintegeryes0Number of mailboxes to process in parallel. Set to 0 for automatic parallelism based on system resources.
drive-contrib-hydration-methodstringnonullOverride method to populate contributor files. Can be "permission", "revision", or "activity".

2 - Google Drive Connector Setup

Provide necessary API permissions to index Google Drive contents.

Installing the Google Drive Connector

Note: For this step, your deployment engineer will be required.

The Gmail and Google Drive connectors depend on a shared Google Workspace configuration section. Google Workspace requires three properties to be set during configuration:

  • ProjectID must reference the project created in GCP. The id, not the name, must be specified.
  • ServiceUser is currently a reference to a super user or administrator within your organization. For example, the primary IT email. This may change in the future.
  • ServiceAccountKey is the base64-encoded string as obtained during initial install of the Google source.

To obtain the service account key, your deployment engineer will run the following Terraform command:

terraform output -raw google_connector_service_account_key

And optionally:

  • Domains is a comma-delimited list of domains applicable to this deployment.
  • Parallelism is an optimization to increase the concurrency with respect to indexing Google data. Default is 32, so if set, should be higher than this value.

Once the Google source is configured, the Google Drive and Gmail sources can be configured with the same values.

Configuration

Source configuration is applied with atolioctl using a YAML file. If you have not already installed atolioctl, follow Configuring Sources first. The commands below also need an API token in JWT_TOKEN and the address the feeder server’s gRPC endpoint, both described next.

Edit and Apply the Configuration

Source configuration is stored as a single document that covers every configured connector. The supported workflow is to export the current configuration, edit it, then import it back. Always start from a fresh export so you do not overwrite a change made elsewhere.

Export the current configuration to a local file:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 lumen-connectors.yaml

Add a Google entry alongside the connectors already in the file, substituting the values you collected in Step 1:

connectors:
  # ...existing connectors left in place...
  google:
    default:
      # ...existing Google sub-connectors left in place...
      gdrive:
        connector: google              # Reference to parent connector, usually Google
        instance: default
        source: gdrive
        common:
          enabled: true
          identity-provider: false     # This connector is a workspace tool, not IDP
        secrets:
          service_account_key:
            value: "<GOOGLE_SERVICE_ACCOUNT_B64>"
        configuration:
          parallelism: 0
          drive-contrib-hydration-method: activity
          project-id: "<GOOGLE_PROJECT_ID>"
          service-user: "<GOOGLE_SERVICE_USER>"

Exported entries also carry a rendezvous-completed flag and a status block reporting each source’s current state and last validation result. Those fields are managed by Atolio. Leave them untouched on existing entries, and omit them when adding a new entry by hand as shown above.

Import the edited file to write the configuration back into the stack:

atolioctl configure import lumen-connectors.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889

Import writes the configuration into the stack after checking that the YAML is well formed. It does not contact Google. The connector validates the credentials and connection itself when it starts and picks up the configuration, and it reads users after that. If something is wrong, the source reports as misconfigured and the connector pod logs the error. See Troubleshooting for the specific messages.

Configuration Schema

In the exported file, every source lives at connectors.<connector>.<instance>.<source>, so the Google Drive source sits at connectors.google.default.gdrive.

GoogleDriveSourceConfiguration

Google Drive connector configuration.

FieldTypeRequiredDefaultDescription
connectorstringyesN/AParent connector to reference (can be self)
instancestringyesN/AMulti-state config variable used for certain connectors, often default
sourcestringyesN/ASub-connector reference (can be self), often object key
commonCommonyesN/ACommon connector settings.
secretsSecretsyesN/ASecret credentials for authentication.
configurationGDriveConfigurationyesN/AGoogle Drive-specific configuration.

Common

Common settings shared across all connectors.

FieldTypeRequiredDefaultDescription
enabledbooleanyesN/ASet to true to run the source.
identity-providerbooleanyesN/ASet to true so Atolio uses this connector to resolve logins to users.
cron-specstringnoN/ACron expression for periodic re-sync of users (e.g., 0 7 * * *). Omit to sync only on demand.

Secrets

Sensitive configuration values for authentication.

FieldTypeRequiredDefaultDescription
service_account_keySecretValueyesN/AGoogle service account key in Base64 to authenticate to Google Workspace.

SecretValue

A reference to a secret value.

FieldTypeRequiredDefaultDescription
valuestringyesN/AThe secret value (typically Base64-encoded).

GDriveConfiguration

Google Drive-specific connector configuration.

FieldTypeRequiredDefaultDescription
domainsstringno""Comma-separated list of allowed customer domains. If empty, all domains are considered.
project-idstringyesN/AGCP project ID under which the integration is operating.
service-userstringyesN/AEmail address of an administrator within the Google Workspace account.
parallelismintegeryes0Number of files to process in parallel. Set to 0 for automatic parallelism based on system resources.
drive-contrib-hydration-methodstringnonullOverride method to populate contributor files. Can be "permission", "revision", or "activity".

3 - Google Sites Connector Setup

Provide necessary API permissions to index Google Sites.

Installing the Google Sites Connector

Note: For this step, your deployment engineer will be required.

Prerequisites

Google Sites Connector requires you to enable Google Vault for your organization. Google Vault enrollment details are available in the official documentation

You will need a Service Account as described in Grant Service Account Permissions section. In addition to the scopes listed there, you must grant the following scopes to the Service Account:

- https://www.googleapis.com/auth/ediscovery
- https://www.googleapis.com/auth/devstorage.read_only

Vault API access must be manually enabled by the GCP project associated with the Service Account. This can be done via GCP API Console, by following the steps in the official documentation.

Configuration

Source configuration is applied with atolioctl using a YAML file. If you have not already installed atolioctl, follow Configuring Sources first. The commands below also need an API token in JWT_TOKEN and the address the feeder server’s gRPC endpoint, both described next.

Edit and Apply the Configuration

Source configuration is stored as a single document that covers every configured connector. The supported workflow is to export the current configuration, edit it, then import it back. Always start from a fresh export so you do not overwrite a change made elsewhere.

Export the current configuration to a local file:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 lumen-connectors.yaml

Add a Google entry alongside the connectors already in the file, substituting the values you collected in Step 1:

connectors:
  # ...existing connectors left in place...
  google:
    default:
      # ...existing Google sub-connectors left in place...
      gsites:
        connector: google              # Reference to parent connector, usually Google
        instance: default
        source: gsites
        common:
          enabled: true
          identity-provider: false     # This connector is a workspace tool, not IDP
        secrets:
          service_account_key:
            value: "<GOOGLE_SERVICE_ACCOUNT_B64>"
        configuration:
          service-user: "<GOOGLE_SERVICE_USER>"
          site-urls:
            - https://sites.google.com/your-org/your-site
          site-configs:
            https://sites.google.com/your-org/your-site
              # If you have a custom domain for your Google Site, specify it here.
              top-level-url: https://your-custom-domain.com/path

              # If you want to configure deeplinks for each page, 
              # you can do it by providing a "page name" -> "URL suffix" map here.
              # 
              # For a page titled "Home", the deeplink would be:
              # https://your-custom-domain.com/path/home
              page-name-to-url-suffix:
                Home: /home
                About: /about
              
              # If you have setup "Restricted" access for your site, specify it here. 
              # Default is `false`.
              restricted: true

If you do not prefer to have a static configuration for deeplinks, you can also provide a “url path” on each Google Site page. Simply add a text in the form of urlpath:[/relative-path-to-your-page] in the body of each page, and Atolio will index the page with the user-provided deeplink.

Exported entries also carry a rendezvous-completed flag and a status block reporting each source’s current state and last validation result. Those fields are managed by Atolio. Leave them untouched on existing entries, and omit them when adding a new entry by hand as shown above.

Import the edited file to write the configuration back into the stack:

atolioctl configure import lumen-connectors.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889

Import writes the configuration into the stack after checking that the YAML is well formed. It does not contact Google. The connector validates the credentials and connection itself when it starts and picks up the configuration, and it reads users after that. If something is wrong, the source reports as misconfigured and the connector pod logs the error. See Troubleshooting for the specific messages.

Configuration Schema

In the exported file, every source lives at connectors.<connector>.<instance>.<source>, so the Google Sites source sits at connectors.google.default.gsites.

GoogleSitesSourceConfiguration

Google Sites connector configuration.

FieldTypeRequiredDefaultDescription
connectorstringyesN/AParent connector to reference (can be self)
instancestringyesN/AMulti-state config variable used for certain connectors, often default
sourcestringyesN/ASub-connector reference (can be self), often object key
commonCommonyesN/ACommon connector settings.
secretsSecretsyesN/ASecret credentials for authentication.
configurationGSitesConfigurationyesN/AGoogle Sites-specific configuration.

Common

Common settings shared across all connectors.

FieldTypeRequiredDefaultDescription
enabledbooleanyesN/ASet to true to run the source.
identity-providerbooleanyesN/ASet to true so Atolio uses this connector to resolve logins to users.
cron-specstringno0 7 * * *Cron expression for periodic re-sync of users (e.g., 0 7 * * *). Omit to sync only on demand.

Secrets

Sensitive configuration values for authentication.

FieldTypeRequiredDefaultDescription
service_account_keySecretValueyesN/AGoogle service account key in Base64 to authenticate to Google Workspace.

SecretValue

A reference to a secret value.

FieldTypeRequiredDefaultDescription
valuestringyesN/AThe secret value (typically Base64-encoded).

GSitesConfiguration

Google Sites-specific connector configuration.

FieldTypeRequiredDefaultDescription
domainsstringno""Comma-separated list of allowed customer domains. If empty, all domains are considered.
service-userstringyesN/AEmail address of an administrator within the Google Workspace account.
site-urlsstring[]yesN/AList of top-level Google Site URLs to index (e.g., https://sites.google.com/your-org/your-site).
site-configsmap[string]SiteConfignoN/AOptional map of site-specific configuration indexed by site URL.

SiteConfig

Per-site configuration for Google Sites.

FieldTypeRequiredDefaultDescription
top-level-urlstringnoN/ACustom domain URL for the site if using a custom domain.
page-name-to-url-suffixmap[string]stringnoN/AMap of page names to URL suffixes for generating deeplinks (e.g., Home: /home).
restrictedbooleannofalseSet to true if the site has restricted access enabled.