탐색으로 건너뛰기

How do I list a site's URLs? Does Map find every page?

Use POST /v1/web/map from a public starting page to list URLs inside a site. Map does not guarantee every page on a domain: each request returns at most 100 deduplicated URLs and follows links only to max_depth (1–3, default 1).

Scope a crawl with depth and path filters

Start from a structured entry page (such as a help-center index) and narrow the output before spending credits on unrelated sections:

  • limit: 1–100 (default 50). Billing is $0.001 per returned URL; a call that returns zero URLs costs nothing and adds a note when a page builds links with JavaScript or links only externally.
  • max_depth: 1–3 hops from the starting url (default 1).
  • select_paths and exclude_paths: Up to 10 path regular expressions each (up to 200 characters per pattern; invalid regex returns 400 bad_request). include_external defaults to false.
curl -sS https://api.beatapi.io/v1/web/map \
-H "Authorization: Bearer $BEATAPI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/help","select_paths":["/help/.*"],"max_depth":2,"limit":50}'

Verify completeness and read target pages

Record the starting url, max_depth, and path filters alongside the URL inventory, group discovered URLs by topic, and pass selected pages to POST /v1/web/read to inspect page content. When a section looks incomplete, compare the Map output against the site’s sitemap.xml or official directory.

Before running a request, inspect the contract for free: data:web.search, data:web.read, data:web.map.

Back to Web Search questions.