How Web Crawlers Actually Work (And What Auditing My Own Site Taught Me)

If you've ever wondered how Google "knows" your site exists, or why a page you just edited still shows old text in search results — this post is for you. No prior knowledge assumed. By the end, you'll also see a real bug I found and fixed on this exact site while writing it.

The one-sentence version

A crawler is a program that visits a page, reads it, follows its links to find more pages, and remembers what it saw — so a search engine can answer questions without visiting the live internet every time you search.

That's it. Everything else in this post is detail on how that's done at scale, and why it sometimes goes wrong.

The loop, step by step

Every crawler — from a hobby script to Googlebot — runs some version of this loop:

1. Frontier — a queue of URLs waiting to be visited. New crawls start from a handful of seed URLs (often your sitemap.xml) and grow from there as links are discovered.
2. robots.txt check — before fetching anything, the crawler checks whether it's allowed to. This file lives at yoursite.com/robots.txt and is a set of rules, not a lock — well-behaved crawlers honor it, nothing enforces it.
3. Fetch — the crawler downloads the page, one request at a time per site, so it doesn't overwhelm your server.
4. Changed since last visit? — this is the step most beginners don't know exists, and it's the one this whole post is really about. Instead of re-reading every page every time, crawlers compare a fingerprint (a short hash of the content) against what they saw last time. Same fingerprint → skip, for free. Different → keep going.
5. Extract — pull out the text, the <title>, the meta description, and every link on the page. The links go back into the Frontier.
6. Store + Index — the content gets saved somewhere searchable. This is the step that actually makes your page show up in search results.

Steps 1–6 repeat forever, for every page, on every site the crawler knows about. The only thing that changes over time is how often a given page gets re-visited — and that's decided by a signal you control.

The signal most developers skip: lastmod

Your sitemap.xml can carry a <lastmod> timestamp on every URL:

<url>
  <loc>https://example.com/pricing</loc>
  <lastmod>2026-09-08T12:00:00.000Z</lastmod>
</url>

This is how you tell a crawler "this page actually changed, come back sooner" — without it, every page looks equally stale, and a real content update can sit unnoticed for a long time.

It's a small tag. It's also the exact thing I found missing on my own portfolio a few days ago.

The audit: what I actually found on my own site

I'd recently rewritten my homepage's title and description — new positioning, new copy. Weeks later, Google's search result still showed the old title. Not a typo, not a caching glitch on my end — the actual indexed snapshot was outdated.

Checking sitemap.xml explained why:

<url>
  <loc>https://www.mjubayer.dev</loc>
  <changefreq>weekly</changefreq>
  <priority>1.0</priority>
</url>

No <lastmod> at all. Every static page — home, about, projects, contact — had the same gap. From a crawler's point of view, nothing had a reason to be re-checked ahead of schedule. The content changed; the signal saying so never got sent.

The fix, and the trap hiding inside it

The obvious fix — "just add <lastmod> using each page's git commit date" — turned out to be wrong in a specific, useful way. My homepage's real <title> and <meta description> don't live in the page file at all; they come from a shared data file imported across every route. A page-file-only lastmod would have stayed frozen even while the indexed title changed underneath it — the exact bug I was trying to fix, just moved one level deeper.

The actual fix: a route's lastmod is the most recent commit touching anything that changes what a crawler sees on that route — the page file, its components, and the shared data file. That's a more honest definition of "this page changed," and it's the one that actually closes the gap.

A bug that had nothing to do with crawling

While shipping the new homepage copy, the text rendered — then turned invisible. Same background color, same text color, black on black. The cause: a Tailwind color I'd named base silently collided with Tailwind's built-in text-base utility (which controls font size, not color) — two rules fighting for the same class name, and the newer one won.

Nothing about this was crawler-related. It's here because it's the same lesson twice in one afternoon: verify against the real output, not against what the code is supposed to do. I only caught the stale index by reading the actual generated sitemap.xml, and only caught the invisible text by reading the actual computed CSS — not by trusting either one to be correct because the code "looked right."

A short checklist for your own site

If you want to check your own site for the same class of problem:

• Does sitemap.xml exist, and does every URL have a <lastmod>? Open it in a browser and look — no tool needed.
• Does robots.txt accidentally block something it shouldn't? A stray Disallow: / blocks your entire site.
• Does your indexed title/description in Google match your live page? Search site:yoursite.com and compare.
• If a page's <title> comes from a shared file, does your lastmod logic know about that file? This is the one that's easy to miss.

None of this requires deep SEO knowledge — it requires reading the actual output your system produces, the same way you'd debug any other silent failure.

Takeaway

Crawlers aren't mysterious. They're a loop: fetch, check if it changed, extract, store, repeat — and the only lever most developers actually control is how clearly they signal what changed and when. Get that one signal right, and the rest of the loop takes care of itself.