Google Connector

The Google connector enables the indexing of:

• Documents, Sheets, Slides, and other Files in Google Drive
• Email in Gmail
• Pages in Google Sites
• Users and Groups as metadata in your Google Workspace to enable user mapping / permissions

Content and updates are streamed as changes are detected and processed in your Google Workspace.

Step 0: Create Terraform Service Account for Google

Note: Depending on installation being BYOK, Managed, or otherwise this step may be skipped. Consult with an Atolio Support Engineer for any clarification.

Terraform will need a service account to make changes to your GCP project. We will grant it permissions which will essentially allow it to create, read, and delete all project resources. As such, you should keep the credentials for this account safe and not reuse them. The Terraform scripts create a much more restricted service account for normal operation.

1. Visit the IAM Service Accounts page https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously to set up the Terraform service account.
2. Click CREATE SERVICE ACCOUNT
3. Choose a name. While it can be anything, the name “Atolio Terraform” is recommended.
4. Select CREATE AND CONTINUE.
5. Grant the following roles: “Owner”, “Security Center Admin”, “Project IAM Admin” (without conditions).
6. Click DONE. Skip step 5, you will not need to grant other users access to this service account.

Now the service account is created we can create API keys:

1. Navigate to https://console.cloud.google.com/iam-admin/serviceaccounts and select the project created previously.
2. Click the service user that was created in the previous step.
3. Select the KEYS tab click the ADD KEY pull-down menu
4. Select Create new key and use JSON for key type. Click CREATE.

This will create another JSON file with the API key and related information. This file will be needed by your Deployment Engineer (in their deploy/terraform directory).

In order for Terraform to programmatically make modifications, you need to manually enable the Google Cloud Resource Manager API.

1. Navigate to https://console.cloud.google.com/apis/library
2. Search for “Cloud Resource Manager API”
3. Select “Cloud Resource Manager API”. Click ENABLE.

The project will use additional APIs, but now that this API is available to Terraform, the Atolio deployment scripts will use Terraform to enable those APIs automatically.

Step 1: Grant Service Account Permissions

The Google Workspace domain needs to grant permission to the integration’s default service account to perform operations. Unfortunately Terraform can’t automate this, so it has to be performed manually.

Your Deployment Engineer will need to make note of the unique ID for the default App Engine service account and the list of scopes. Be aware that this ID is of the default service account, typically named App Engine default service account. It is not the Service Account you’ve previously created.

And the list of scopes are:

https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/admin.directory.user.readonly,https://www.googleapis.com/auth/admin.directory.group.readonly,https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/admin.directory.domain.readonly,https://www.googleapis.com/auth/drive.activity.readonly

We now need to set up Domain-wide Delegation:

1. Navigate to the Google Admin console: https://admin.google.com/ac/owl. An account with admin access is required.
2. Scroll down and click “MANAGE DOMAIN WIDE DELEGATION”.
3. Click Add new and enter your service account client ID (obtained from your Deployment Engineer in the previous step). No overwrite.
4. Enter the value for google_service_account_id as obtained from your Deployment Engineer.
5. Enter the list of scopes as listed above (as a single comma-delimited string).
6. Click AUTHORIZE.

Now Atolio is authorized to retrieve Google Workspace data.

Step 2: Deploy the Connector

Before the source can be configured, the connector pod must be running in the cluster. Connectors are deployed by enabling them in values-lumen.yaml. See Deploying Connectors for the full details, including how instances work.

connectors:
  google:
    default:
      enabled: true

default is the instance identifier. Treat it as immutable once set, because the instance is part of every resource identifier the connector ingests. Apply the change by upgrading the lumen Helm chart with create-infra.sh (recommended) or helm upgrade as described in the sources overview.

Step 3: Configure the Source

Source configuration is applied with atolioctl using a YAML file. If you have not already installed atolioctl, follow Configuring Sources first. The commands below also need an API token in JWT_TOKEN and the address the feeder server’s gRPC endpoint, both described next.

atolioctl Prerequisites

JWT_TOKEN holds the API token atolioctl uses to authenticate to the feeder. Generate one from the deployment’s Helm values directory, which holds the values-lumen.yaml and values-lumen-templated.yaml files produced by create-infra.sh. The token is signed with the deployment’s jwtSecretKey, and the atolio:*:*:* match pattern grants access across all connectors, instances, and resources:

export JWT_TOKEN=$(atolioctl connector create-jwt --raw --config-dir . "atolio:*:*:*")

--feeder-address is the gRPC address of the feeder server that is used to read and write source configuration through. The standard path is to port-forward the in-cluster service/feeder and point atolioctl at it on localhost:

# Run K8s port forward to feeder as background to re-use same shell
kubectl port-forward -n atolio-svc service/feeder 8889 &

With that port-forward running in current terminal or another, the examples below use --feeder-address :8889. The empty host means localhost, and because the port is not 443 the connection is made in plaintext, which is what the port-forward exposes. If the feeder is instead published on a load balancer, pass that address with port 443, for example --feeder-address feed.search.example.com:443, which connects over TLS.

Edit and Apply the Configuration

Source configuration is stored as a single document that covers every configured connector. The supported workflow is to export the current configuration, edit it, then import it back. Always start from a fresh export so you do not overwrite a change made elsewhere.

Export the current configuration to a local file:

atolioctl configure export --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889 lumen-connectors.yaml

Add a Google entry alongside the connectors already in the file, substituting the values you collected in Step 1:

connectors:
  # ...existing connectors left in place...
  google:
    default:
      google:
        connector: google
        instance: default
        source: google
        common:
          enabled: true
          identity-provider: true      # This connector is the identity provider
          cron-spec: 0 7 * * *         # Re-sync users on this schedule
        secrets:
          service_account_key:
            value: "<GOOGLE_SERVICE_ACCOUNT_B64>"
        configuration:
          parallelism: 0
          project-id: "<GOOGLE_PROJECT_ID>"
          service-user: "<GOOGLE_SERVICE_USER>"

Exported entries also carry a rendezvous-completed flag and a status block reporting each source’s current state and last validation result. Those fields are managed by Atolio. Leave them untouched on existing entries, and omit them when adding a new entry by hand as shown above.

Import the edited file to write the configuration back into the stack:

atolioctl configure import lumen-connectors.yaml --jwt-token-sdk ${JWT_TOKEN} --feeder-address :8889

Import writes the configuration into the stack after checking that the YAML is well formed. It does not contact Google. The connector validates the credentials and connection itself when it starts and picks up the configuration, and it reads users after that. If something is wrong, the source reports as misconfigured and the connector pod logs the error. See Troubleshooting for the specific messages.

Configuration Schema

In the exported file, every source lives at connectors.<connector>.<instance>.<source>, so the Google source sits at connectors.google.default.google.

GoogleSourceConfiguration

Google Workspace connector configuration.

Common

Common settings shared across all connectors.

Secrets

Sensitive configuration values for authentication.

SecretValue

A reference to a secret value.

Configuration

Google Workspace connector-specific configuration.