OLAB Assist Sign in
For site owners

OLABAssistBot, our crawler

If you found OLABAssistBot in your server logs: it is the crawler of OLAB Assist. It reads public pages to check how well a website can be found by search engines and AI assistants.

How it identifies itself

Every request carries this user agent:

OLABAssistBot/2.0 (+https://olabassist.com/bot)

When it reads pages that AI assistants cite as sources, it sends the same name inside a browser-style user agent: Mozilla/5.0 (compatible; OLABAssistBot/2.0 (+https://olabassist.com/bot)).

When it visits

  • Someone runs a free analysis of a website on our homepage.
  • A customer monitors their website with us: a scan every day, and scans they start themselves.
  • A customer of Market Gap Radar compares their website with competitors' websites.
  • An AI assistant cites a page in an answer about a customer's topic. Each such page is read at most once a day.

What it reads

  • The start page, /robots.txt, /llms.txt and /sitemap.xml.
  • For a monitored website, a limited number of other pages from its sitemap or homepage links (between 10 and 250 per scan, depending on the plan).
  • Only public pages. It never signs in, never submits forms and does not download images, PDFs or videos.
  • It stops reading a response after 1.5 MB and waits at most 8 seconds for each one. It opens at most 10 pages of one website at a time.

Checking access for AI search crawlers

Part of the analysis is whether AI search crawlers can reach a website. For this, each analysis requests the start page once with the user agent of each of these crawlers: OAI-SearchBot, PerplexityBot and Claude-SearchBot. These requests come from our servers, not from those companies, and they only read the start page.

How to stop it

OLABAssistBot follows robots.txt when it opens the other pages of a website. To keep it out of them, add this to your robots.txt:

User-agent: OLABAssistBot
Disallow: /

Rules for User-agent: * apply when OLABAssistBot is not named. The start page and the files listed above are still read once per analysis, because the analysis is about them.

Where requests come from

Our servers run on Cloudflare's network, so requests come from Cloudflare addresses.

Questions

Write to [email protected].