Open a crawler, point it at the homepage, and let it run. In a few minutes you have an inventory of every URL the crawler could reach, with status codes, canonicals, titles, and internal links attached. That inventory is the starting point of a technical audit, but on its own it answers only half the question. It tells you what is on the site. It does not tell you what Google is doing with it. A complete audit needs both views, and it needs a way to turn a pile of raw data into a ranked list of things to fix.
Three tools cover that ground for most sites: a desktop crawler that produces the inventory, an analysis layer that interprets it, and Google Search Console for Google’s own perspective. Screaming Frog, Sitebulb, and Search Console fill those roles. This is not the only stack; Botify, Oncrawl, ContentKing, and similar platforms cover comparable territory, and enterprise sites often need them. But the desktop crawler tier plus Search Console handles most audit work, and Search Console is non-negotiable because nothing else shows what Google actually indexed.
What each tool is for #
The division of labor is the reason to run all three rather than picking one.
Screaming Frog SEO Spider is the crawler. It walks the site the way a search engine would and returns a URL inventory: status codes, redirects, canonical tags, meta data, heading tags, internal links, images, and any structured data it finds. The free tier crawls up to 500 URLs per crawl, which is enough for a small site or a sample; a paid licence removes that cap and unlocks JavaScript rendering, custom extraction, and API integrations. Its output is raw data, exportable to a spreadsheet.
Sitebulb runs a similar crawl but adds an interpretation layer on top. It flags specific issues, ranks them by importance, and draws the site’s link structure as a visual map. Where Screaming Frog hands you the data, Sitebulb hands you an opinion about what matters most.
Google Search Console is the only source for what Google actually does with the site. It reports which URLs are indexed, how Googlebot is crawling, how pages perform in search, and whether any manual actions are in force. A crawler can tell you a page is technically fine; only Search Console tells you Google chose not to index it anyway.
Starting with Screaming Frog #
A first crawl produces the baseline inventory. A few configuration choices change what that crawl sees, so set them before you run it.
Switch the user agent from the default to Googlebot Smartphone so the crawl matches Google’s primary crawler, which matters under mobile-first indexing. Turn on JavaScript rendering (a paid-licence feature) for sites that build content client-side, or the crawl will miss what Googlebot renders. Leave “respect robots.txt” on except when the point of the audit is to see what robots.txt blocks. Use custom extraction with XPath or CSS selectors to pull specific fields, such as author names, publish dates, or schema content, out of every page.
Then work through the first-pass reports:
- Response codes. Sort by status. Find 4xx and 5xx pages, redirect chains, and pages returning a code they should not.
- Page titles and meta descriptions. Filter for missing, duplicate, too long, and too short. Each is a distinct fix.
- H1 tags. Missing, duplicated across pages, or multiple on a single page.
- Canonicals. Pages with none, pages pointing elsewhere, pages sending conflicting signals.
- Internal links. Orphan pages with no inbound internal links, pages with a single inbound link, and the most-linked pages on the site.
- Images. Missing alt text, oversized files.
- Hreflang. On international sites, validate the tag implementation.
Bulk export moves any of these into a spreadsheet, and that exported crawl becomes the foundation for most of the work that follows.
Moving to Sitebulb #
Sitebulb crawls much like Screaming Frog does, then adds the layer Screaming Frog leaves to you: judgment about priority.
Its hint system checks each crawl against a large library of issues, groups them by category such as indexability, internal linking, content, and performance, and tags each with an importance level. Every hint lists the URLs that trigger it and explains why it matters. Crawl maps visualize the internal link structure, which makes orphan pages, isolated sections, and lopsided link distribution easier to spot than they are in a spreadsheet. Comparison reports set two crawls side by side to show what changed: new errors, resolved issues, added and removed pages. The tool can also pull Search Console data in to combine crawl findings with performance data.
When you read the hints, weight the critical ones (broken or actively harmful) first, then the high-priority ones that reliably affect ranking or indexation, then indexability conflicts such as a noindex on a page that should be indexed. Teams that want raw data can stop at Screaming Frog; teams that value the prioritized list and the visualizations add Sitebulb. Many workflows use both, Screaming Frog for bulk operations and extraction, Sitebulb for the ranked issue list.
Search Console: the third source #
Search Console shows Google’s side of the relationship, and several reports carry most of the diagnostic weight.
The Page indexing report (formerly Index Coverage) classifies every URL Google knows about: indexed, crawled but currently not indexed, discovered but currently not crawled, page with redirect, excluded by noindex, blocked by robots.txt, soft 404, not found, and server error. The classifications point at pipeline problems. “Discovered but currently not crawled” means Google knows about the URLs but has not fetched them, which on a large or frequently updated site can signal crawl budget pressure. “Crawled but currently not indexed” means Google fetched the page and declined to index it, which tends to be a quality signal. Soft 404 points at thin or empty content.
The Crawl Stats report shows Googlebot’s activity: requests per day, bytes downloaded, average response time, and a breakdown by file type, response code, and purpose. On sites without server log access, it is the primary window into crawl behavior.
The Performance report carries clicks, impressions, click-through rate, and position, sliced by query, page, country, device, and appearance. It separates pages that are indexed but not ranking from pages that rank but earn no clicks, each a different problem.
The Enhancement reports show how Google interprets structured data: which pages carry each schema type, and the errors and warnings against them. These catch schema problems the crawlers cannot, because they reflect Google’s parsing rather than the raw markup. Finally, the Manual actions report shows whether Google’s spam team has penalized the site. Check it first when rankings drop; a manual action explains problems no crawl would surface.
Running the audit #
A full audit on a healthy site moves through a predictable sequence. Crawl the site with the crawler configured to match how Googlebot accesses it. Export the Page indexing report and cross-reference the two URL lists: URLs Google knows about that the crawl never found (a discovery problem, often orphan pages), URLs the crawl found that Google has not indexed (a quality or technical block), and URLs in both lists carrying different status (a canonicalization question). Diagnose each discrepancy against its usual causes, rank the issues by impact rather than by count, and for each priority issue write down the change, who makes it, and how to confirm the fix. Then set up monitoring: scheduled re-crawls, Search Console alerts on new errors, and a dashboard combining crawl and Search Console data so trends stay visible.
A handful of findings recur across most audits: redirect chains where the destination should be one hop away; mixed signals like a noindex page that also sits in the sitemap and is named as canonical; orphan pages left behind when a linking module was removed; soft 404s returning 200 with “no results” content; canonical conflicts across tag, sitemap, and internal links; hreflang errors like missing return tags; schema whose required properties are absent or which describes content the page does not show; and images missing alt text. The fixes are usually straightforward. The identification is the work, and the tooling exists to support it.
Cadence should match the site. A stable site can run an annual deep audit with quarterly check-ins. High-velocity sites, large e-commerce catalogs, and news publishers need monthly or continuous monitoring, with synthetic checks on critical templates and alerts that fire within minutes of a status-code change. Sites in the first weeks after a migration warrant daily attention, because that is when undetected issues accumulate fastest. The right rhythm is whatever catches material problems inside their useful diagnostic window.
FAQ #
Do I need both Screaming Frog and Sitebulb?
No. Either one crawls a site. Run both when you want Screaming Frog’s bulk export and custom extraction alongside Sitebulb’s prioritized hints and visual crawl maps. If you only need raw data, one crawler is enough.
Can the free Screaming Frog tier handle a real audit?
For sites under 500 URLs, or for sampling a section of a larger site, yes. The 500-URL cap and the lack of JavaScript rendering, saved crawls, and API integrations make the paid licence necessary for full-site audits of anything larger.
Why would a page be crawled but not indexed?
Google fetched it and decided it did not merit indexing, usually a quality signal such as thin, duplicate, or low-value content. It is a content and relevance question, not a technical block, and the Page indexing report labels it distinctly from “discovered but currently not crawled,” which points instead at crawl capacity.
An audit is less a project than a rhythm. The tools change, new versions ship, features move, platforms rise and fall, but the discipline underneath stays the same: crawl to see what is there, compare it to what Google sees, diagnose the gap, and fix what matters most first. Teams that run that loop on a steady cadence catch problems while they are still cheap. Teams that wait until something breaks pay more and move slower. Start with a single baseline crawl and one pass through the Search Console reports; that alone will surface the first round of fixes worth making.