How crawling and indexing work

The URL still has to pass HTTP(S), default-port, DNS, and public-address checks. Private IP ranges, internal hostnames, credentials in URLs, and unsafe redirect hops are rejected.

1. Refleks reads robots.txt for the crawler user-agent and collects any sitemap declarations it contains.
2. It checks same-host /sitemap.xml and /sitemap_index.xml as additional candidates. A sitemap index can contribute same-host child sitemaps.
3. It queues the URL you entered first, then eligible sitemap entries. After pages are fetched, same-host links from those pages extend the queue.
4. The queue stops at the active plan’s per-site page limit. Discovery itself is also bounded to a multiple of that target, with a hard safety ceiling, so a link-heavy page cannot create an unbounded crawl.

Sitemaps and links are additive discovery sources. There is no public include-list editor in the current dashboard; use a focused starting URL, a focused sitemap, canonical URLs, and robots rules to shape the crawl.

Redirects are revalidated and must stay on the original hostname. The widget’s origin checks are a separate concern: they normalize only the www. prefix for installation requests, not for crawl discovery.

The crawler removes hashes and common tracking parameters when normalizing URLs. It sorts query parameters for deduplication and can collapse known presentation-state parameters when the parameter-free URL is already queued or visited.

If an important heading, product list, price, or navigation item appears only after client-side code runs, confirm that browser rendering is enabled for the deployment. Otherwise publish the content in the server response or use the dashboard’s manual text import where appropriate.

• Refleks identifies itself with a RefleksCrawler/1.0 user-agent and chooses the most specific matching user-agent group that it can parse.
• Matching Disallow rules win by path specificity; an equally specific Allow wins the tie.
• Sitemap declarations are read from robots.txt, but a sitemap URL is still normalized, safety-checked, and restricted to the crawl hostname.
• If robots.txt cannot be fetched, there are no parsed robots rules to apply. URL safety, response checks, and all other exclusions still apply, and an individual page can still be recorded as a fetch failure.

Refleks looks for same-host hreflang alternates on the root document. Filtering turns on only when the metadata identifies a primary language and at least two distinct foreign root patterns; nested locale paths also need repeated sitemap evidence. When enabled, alternate-language URLs are recorded as skipped and the primary-language URL set is preferred.

A lone /de path or an ambiguous language tag is not enough to classify a site. Refleks does not promise a complete translation index or automatic language detection for every page.

1. A refresh fetches the current crawl scope and prepares a new run. Each page receives a content hash, and the indexing key also includes its title, URL, embedding model/dimensions, and indexing version.
2. When the existing page’s indexing key and chunks are unchanged, Refleks can copy the existing chunks instead of regenerating embeddings. Changed content, titles, URLs, or indexing configuration are re-embedded.
3. The new pages and chunks are staged under a new run. The previous active snapshot remains queryable until the new run is complete and consistent.
4. If the crawl has failed page ingestion, loses pages because of transient fetch failures, finds no indexable pages, or exceeds a crawl budget, the incomplete run is discarded and the existing snapshot is preserved.

Re-crawls are bounded by your plan’s daily crawl cap and a same-site cooldown. See refresh limits for the account behavior.

The crawl job exposes a queued, fetching, indexing, finished, or failed stage with discovered, processed, fetched, indexed-page, and chunk counts. The dashboard’s crawl report also lists recent jobs, indexed URLs, skipped URLs, the latest diagnostics, and the last successful crawl time.

• Extraction favors visible headings, paragraphs, lists, and tables from main/article/role=main content and removes navigation, forms, hidden elements, media, and common cookie/banner/popup blocks.
• Pages with less than 200 characters of extracted text are skipped. A page can be technically reachable and still have too little content to index.
• Retrieval uses the active snapshot and returns page URLs with matching chunks when available. Similarity and a source URL are evidence for review, not a factual guarantee.
• AI answers may still be incomplete or wrong. Keep important policies, prices, and dates authoritative on the website and review the linked source.