Crawl Your Website as an AI Agent Knowledge Source

That's what crawling a website as a knowledge source does. You give Macha a URL, it fetches the pages, strips them down to clean text, chunks and embeds that text, and from then on your agents can search it on every live ticket. No copy-pasting your shipping policy into a prompt. No re-uploading a PDF every time you change your return window. The agent answers from the same page your customers can read — and when the page changes, you re-crawl and it's current again.

This is a how-to. By the end you'll know exactly how to crawl a site into Macha, how to scope it so you don't ingest junk, what the real limits and trade-offs are, and when crawling is the wrong tool and a live connector is better. Macha is an AI agent layer that sits on top of the helpdesk you already use — Zendesk, Freshdesk, Gorgias, Front — so everything here feeds agents that resolve, triage, and route tickets inside your existing workflow.

What a website knowledge source actually is

In Macha, a knowledge source (a "data source") is any body of content your agents can search. There are three kinds:

• Uploaded files — PDFs, spreadsheets, and documents. Small files get injected directly into the agent's prompt; larger files are chunked, embedded, and searched on demand.
• Crawled websites — a URL (or a single page) that Macha fetches and indexes.
• Live connector sources — Google Docs, Notion, and Confluence pages that stay in sync with the original because Macha reads the latest content at query time.

A crawl sits in the middle. Unlike a live connector, a crawl is a point-in-time snapshot — Macha indexes the page as it existed when you crawled it, and it stays that way until you re-crawl. That's the single most important thing to understand before you start, and we'll come back to it in the watch-outs.

How crawling-to-knowledge works under the hood

Every credible crawler-for-AI works the same way, and it helps to know the shape of it so the settings make sense.

A crawler starts from a seed URL and follows links outward. How far it follows them is the scope. Then for each page it keeps, it does the part that matters for an AI agent: it throws away the navigation, ads, and sidebars and keeps the actual content as clean text. Modern crawlers built for LLMs — Firecrawl, Spider, Crawl4AI — all converge on returning Markdown or structured text rather than raw HTML, precisely because an agent reasons better over clean prose than over a wall of <div>s. That cleaned text is then chunked into passages, each passage is turned into a vector embedding, and those embeddings are stored so the agent can retrieve the most relevant chunks for a given question. This retrieval-augmented pattern is the standard way to ground an LLM in your own content.

Scope is where people get into trouble, so it's worth being precise. Amazon's Bedrock Web Crawler — a good public reference for how a production crawler behaves — offers three scope levels that map onto the choice every crawler forces you to make (AWS docs):

The lesson under that table: broad scope is a trap. Bedrock's own docs warn against crawling large sites without filters because it "will take a very long time" and pulls in pages you never wanted in the agent's head. The narrower you scope, the higher the signal — and the fewer credits and knowledge-source slots you burn.

How to crawl a site into Macha, step by step

1. Open Knowledge Sources and add a website

From the dashboard, go to your knowledge sources and create a new one. Macha gives you the three options above; choose to crawl a website (or, for a single page, the webpage option). You'll see your existing sources listed here, so you can tell at a glance what your agents already know.

2. Paste the most specific URL you can

Don't reflexively paste your homepage. Paste the narrowest URL that contains the answers you want. If your refund logic lives at help.yourstore.com/returns, seed that — not yourstore.com. A tighter seed URL is the cheapest, most effective quality control you have. For a single high-value page (one detailed FAQ, one policy page), use the single-webpage option instead of a full crawl.

3. Let it fetch, parse, and index

Macha fetches the page(s), extracts the readable content, and runs it through the same chunking-and-embedding pipeline used for uploaded files. One practical detail from Macha's changelog worth knowing: the crawler's per-page size limit was raised from 512KB to 2MB (April 2, 2026), which fixed ingestion failures on content-heavy pages like large e-commerce product and category pages. If you tried to crawl a heavy store page in the past and it choked, it works now.

4. Attach the source to your agents — and scope it

A crawled source does nothing until an agent can use it. Open the agent you want to give knowledge to and attach the source. Each source row has an inline "All documents" / "Selected" toggle, so you can hand an agent the whole crawl or just the specific pages relevant to its job — your WISMO agent might only need the shipping and tracking pages, not your entire knowledge base.

That's it. The next time a ticket comes in, the agent searches the crawled content, retrieves the relevant chunks, and grounds its reply in your actual page — with the source available so you can see where an answer came from.

What to crawl first (and what to skip)

Not all of your site is good agent food. A quick triage:

Crawl these:

• Your help center / knowledge base — it's literally written to answer support questions.
• Policy pages — returns, refunds, shipping, warranty, privacy. High-stakes, frequently asked, and you want the agent quoting the canonical wording.
• Product docs and FAQs — the "how do I…" content.
• Pricing and plan pages — as long as they're current.

Skip these:

• Your blog, mostly. Marketing posts are persuasive, not precise, and they age badly. A blog claim from 2023 in your agent's knowledge is a liability.
• Login-gated or dynamic pages — most crawlers only reliably index static, publicly reachable HTML. Account dashboards and JS-heavy app screens won't crawl cleanly.
• Anything that changes hourly — stock levels, live order status, prices that move. That's not knowledge, it's data, and it belongs in a tool, not a crawl (more on that next).

The honest watch-outs

Crawling is the right tool a lot of the time. It is not the right tool all of the time, and a how-to that pretends otherwise isn't worth reading.

A crawl is a snapshot, not a live feed. This is the big one. When you crawl a page, Macha indexes it as it is now. Change the page tomorrow and the agent still knows yesterday's version until you re-crawl. Tools that "auto-detect site changes" exist in the market, but the safe mental model is: treat a crawl as a copy you own and are responsible for refreshing. Put a recurring reminder on your highest-churn pages — policies especially. If a page genuinely changes constantly and accuracy is critical, use a live connector (Notion, Confluence, Google Docs) instead, because those read the latest content at query time and never go stale.

Knowledge isn't the same as action. A crawl can tell an agent what your return policy says. It cannot look up whether order #4821 is eligible — that's a live lookup against your store, which is a connected tool's job (Shopify, Stripe, your helpdesk). The pattern that works: crawl the policy so the agent knows the rules, connect the tool so the agent can apply them to a specific customer. Crawling the wrong thing — trying to use a snapshot for data that needs to be live — is the most common mistake.

Garbage in, confident garbage out. A crawler indexes whatever's on the page, including the outdated FAQ nobody's touched since 2022 and the contradictory answer two clicks away. Crawl a tidy, curated help center and you get a tidy agent. Crawl your entire sprawling site and you've taught the agent your contradictions. Scope tightly; review what came in.

Only crawl what you're allowed to. Crawl your own properties, or sites you have explicit authorization to crawl. Production crawlers respect robots.txt directives by standard (RFC 9309), and you should too — both as a courtesy and because some of your own pages are disallowed for good reasons.

Mind your limits. Each crawled site counts as one of your plan's knowledge sources — Macha plans include a set number (Starter, Professional, and Enterprise differ; see pricing). That's another reason to consolidate around a few well-scoped, high-value crawls rather than a dozen sprawling ones.

Where crawling fits in a Macha setup

Think in three layers. Crawled and uploaded knowledge is your reference library — the policies and docs the agent reasons over. Live connectors are your always-current documents — the wiki pages and runbooks that change weekly. Tools are your hands — the Shopify/Stripe/helpdesk actions that fetch a specific customer's data and do things. A good agent uses all three: it knows the policy (crawl), reads the latest runbook (connector), and looks up the actual order (tool).

There's also a neat closed loop worth knowing about. Macha Studies can run an AI analysis across thousands of past tickets and then push the results back as a knowledge source — each analyzed row becomes an indexed document. So your knowledge sources don't have to be limited to what you wrote on your website; they can include patterns Macha extracted from how your team actually resolved tickets. Crawl what you published, and layer on what you learned.

FAQ

How many pages can Macha crawl from one URL? A crawl follows links from your seed URL within the scope you set; the per-page content limit is 2MB (raised from 512KB in April 2026) so even heavy pages ingest cleanly. The practical guidance is to seed a narrow, specific URL rather than a whole domain — tighter scope means higher-quality knowledge and fewer wasted credits.

Does a crawled page stay up to date automatically? No. A crawl is a point-in-time snapshot. When the source page changes, re-crawl it to refresh the index. For content that changes constantly, use a live connector (Notion, Confluence, Google Docs), which reads the latest version at query time.

Can I crawl just one page instead of a whole site? Yes — use the single-webpage option to fetch and index one URL. It's ideal for a single detailed FAQ or policy page you want the agent to quote precisely.

Do crawled sources work with my existing helpdesk? Yes. Macha sits on top of Zendesk, Freshdesk, Gorgias, and Front. The crawled knowledge feeds agents that work inside your existing helpdesk — you don't change tools.

Will crawling let the agent take actions, like issuing a refund? No. Crawling supplies knowledge (what the policy says). Taking an action (issuing the refund, checking an order) is a connected tool's job. Use both together: knowledge to know the rule, tools to act on it.

Is there a limit on how many sites I can crawl? Each crawled site is one knowledge source, and plans include a set number of sources. See the pricing page for current limits per plan.

Start crawling

Pick your single most-asked-about section — your returns policy, your shipping FAQ, your getting-started docs — and crawl that one URL first. Attach it to an agent, scope it to that agent's job, and watch the next relevant ticket get answered from your own words. Then expand. Start a 7-day free trial, no credit card required, connect your helpdesk, and add your first website knowledge source — the full walkthrough lives in the data sources docs.

Written by Abbas (Customer Support & AI, Macha) · Reviewed by Ankeet Guha (Co-founder & CTO) · Published 2026-06-24 · Last updated 2026-06-24.