Shopify
Stripe
Slack
Notion
Google Workspace
Confluence
How Do You Index a Website or Webpage for Your AI Agent?
Macha turns your website into agent knowledge two ways: Add Website crawls a site and indexes the first 200 pages it finds, and Add Webpage indexes a single URL. Neither updates on its own, so you resync the source when your content changes.
Key takeaways
- Macha indexes web content for an agent two ways: Add Website crawls and indexes a whole site, while Add Webpage indexes one URL.
- Add Website has a system limit of 200 pages, so on a large site point the crawl at the section that matters most, such as the docs subdomain.
- The Add Website form takes a name, a URL and a Crawl Depth of up to 50.
- Macha does not watch an indexed website for changes; the Resync button on the source pulls the latest version and clears out pages that were removed.
- An indexed site only informs agents it is linked to, so after indexing you point the agent at the whole source or at specific pages.
To index a website for a Macha agent, open Sources and use Add Website to crawl the whole site (Macha adds the first 200 pages it finds) or Add Webpage to index one URL. Then link the source to your agent, and hit Resync whenever the site changes.
| Option | What it indexes | Limit | Use it for |
|---|---|---|---|
| Add Website | Every page Macha finds by following links from the URL you give it | First 200 pages; Crawl Depth up to 50 | Your docs, help center or whole site |
| Add Webpage | One URL | A single page | One blog post, FAQ or product page |
Watch it
Should you add a whole website or a single webpage?
From the Sources page, you can point Macha at the web two ways:
Add Website crawls the entire site. Macha follows the links it finds, discovers every page, and indexes all of them. Use it when you want the agent to have access to your whole website: all your documentation, every page. There's a system limit: it adds the first 200 pages it finds.
Add Webpage indexes a single URL. Use it when you only need one specific page in the agent's knowledge, such as a single blog post, one FAQ or one product page.
The choice is only about scope: the whole site, or one page.
What happens when you add a website?
Hit Add Source on a website and Macha starts crawling. As it goes, it lists the pages it's discovering, so you can watch it find your content in real time. In the example from the video, it discovers 160 pages across the site and indexes them. Once done, the whole site is searchable knowledge for any agent you link it to.
Remember the 200-page cap: if your site is larger, Macha indexes the first 200 pages it finds. For a big site, point the crawl at the section that matters most (your docs subdomain, say) rather than the entire marketing site, so the 200 pages are the right ones.
Which option fits your content?
| You want the agent to know… | Use |
|---|---|
| Your entire site / all your docs | Add Website (crawl, up to 200 pages) |
| One specific page | Add Webpage (single URL) |
| A site bigger than 200 pages | Add Website pointed at the most important section |
How do you keep an indexed website up to date?
Like uploaded files, an indexed website is a snapshot: Macha indexes the pages as they are when you add them, and it doesn't watch your site for changes. When you update content, hit the Resync button on the source to pull the latest version. A resync also clears out pages you've removed from the site. (For content that changes constantly, consider reading it live instead. See knowledge source vs. tool.)
How does an agent start using the indexed site?
Indexing adds the site to your sources; an agent only uses sources it's linked to. Point your agent at the source, either the whole thing or specific pages, and it references your site content when answering. (See connecting a knowledge base to an agent.)
Frequently asked questions
What's the difference between Add Website and Add Webpage? Add Website crawls and indexes the whole site (up to 200 pages); Add Webpage indexes a single URL.
How many pages can it index? Up to the first 200 pages it discovers on a site.
What if my site is bigger than 200 pages? Point the crawl at the most important section (like your docs) so the 200 pages are the right ones.
Does it stay up to date? Not on its own. Macha doesn't monitor your site, so hit Resync on the source after you change content; the resync also drops pages that no longer exist. For truly dynamic content, read it live with a tool.
Can I see what it indexed? Yes. During the crawl, Macha lists the pages it discovers.
Index your docs: crawl your help site and let your agent answer from it. $50 of free usage, no credit card required. Start free.
