Web Connector

Index internal or external websites by crawling HTML pages and content.

The Web connector enables the indexing of HTML web pages.

Enable Access

The Web connector indexes unauthenticated web page content which can be accessed from your Atolio deployment.

First, determine where your website is hosted. The Web connector can crawl HTML content by either:

Navigating a website, starting from a configured url.
Browsing an Amazon S3 bucket, from a configured url of the form s3://bucket/prefix.

Both modes require a baseURL as well. It bounds the crawl to that site, and it is what page and file URLs are built from — in S3 mode it is prepended to bucket keys to form the address each indexed document links to. A source configured without a baseURL is reported as misconfigured and does not index.

In order to ensure that the content is accessible in your Atolio deployment, ensure that your Deployment Engineer is aware of the necessary network or AWS permissions as part of the infrastructure deployment.

Note: You may also need to ensure that your HTML content is server-side rendered (SSR) in order to index the complete website content.

Required Configuration Values

Enter these values in the Atolio Admin console (v4.24 and later), or send them to Atolio Support (your Deployment Engineer).

Value	Where to find it	Example / format	Secret
Starting URL	`Enable Access`, where your website is hosted	`https://intranet.example.com` or `s3://bucket/prefix`	No
Base URL	`Enable Access`, the site the crawl is bounded to and indexed links are built from (required)	`https://intranet.example.com`	No
Network or AWS access details (only if the site is private or hosted in S3)	Your network or AWS administrator, so the infrastructure deployment can reach the content	Allowed network ranges, or S3 bucket permissions	No

Configuration Example

Add this block to connector-values.yaml. Replace each <PLACEHOLDER> with the matching value from the table above, and remove any line marked “Only if” that doesn’t apply.

connectors:
  web:
    default:
      web:
        connector: web
        instance: default
        source: web
        common:
          enabled: true
          identity-provider: false
          cron-spec: "0 7 * * *"  # The site is only crawled on this schedule
        secrets: {}
        configuration:
          url: "<STARTING_URL>"
          base-url: "<BASE_URL>"

(Optional) Provide Filters

Additional filters can be configured by your Deployment Engineer in order to:

Include or exclude pages, e.g. particular URLs that match a given regex.
Remove extraneous content, e.g. table of contents.