When you create your first project in Semrush, Site Audit is automatically configured and launched for the domain you provided. Once the initial crawl is complete, you can view the results in Site Audit from the side panel or the home page.
For any subsequent projects you create, you need to configure Site Audit manually. Create a new project from Site Audit and follow the prompts to configure and launch your audit campaign.
If your Site Audit isn't running, see Troubleshooting Site Audit.
General settings
The first step of the setup wizard is General settings. From here, click Start audit to run an audit with the default settings, or customize your audit settings first. You can change your settings and re-run your audit at any time to crawl a more specific area of your site.
Scope
To crawl a specific domain, subdomain, or subfolder, enter it in the Scope field. If you enter a domain, you can also select the Crawl all subdomains checkbox.
By default, Site Audit crawls the root domain, including all available subdomains and subfolders of your site. To limit the crawl, specify a subdomain or subfolder as the crawl scope and clear the Crawl all subdomains checkbox.
For example, to audit only your blog, set the crawl scope to sr09.balochseotools.com/_blog or sr09.balochseotools.com/blog/, depending on whether it's implemented as a subdomain or a subfolder.
Limit per audit
Next, enter how many pages to crawl per audit in the Limit per audit field. Base this number on your subscription and how often you re-audit your site.
- With Pro SEO Toolkit, you can crawl up to 100,000 pages per month and 20,000 pages per audit
- With Guru SEO Toolkit, you can crawl up to 300,000 pages per month and 20,000 pages per audit
- With Business SEO Toolkit, you can crawl up to 1 million pages per month and 100,000 pages per audit
Pages to crawl
The Pages to crawl setting determines how SiteAuditBot finds pages to audit. In addition to setting the crawl source, you can include or exclude specific subfolders and URL parameters in the Allow/disallow rules and URL parameter rules steps.
There are 4 crawl sources: Website, Robots.txt sitemap, Sitemap by URL, and Import URLs from file.
- Website: Site Audit crawls your site using a breadth-first search algorithm, following the links in your page code — starting from the homepage. This finds the pages most accessible from your homepage, which aren't always your most important ones. To prioritize by importance instead, use a sitemap as your crawl source.
- Robots.txt sitemap: Site Audit crawls only the URLs found in the sitemap linked in your robots.txt file
- Sitemap by URL: Site Audit crawls the sitemap you specify. Enter the sitemap URL directly instead of relying on the link in your robots.txt file. Site Audit uses 1 sitemap URL at a time — if your site has several sitemaps, use Import URLs from file instead.
- Import URLs from file: Site Audit crawls a specific set of pages you upload. Format your file as a .csv or .txt with 1 URL per line, then upload it from your computer. Use this option to check specific pages and conserve crawl budget. After upload, the wizard shows how many URLs it detected so you can confirm the file was read correctly.
Schedule
Select how often Site Audit automatically audits your site:
- Weekly (choose any day of the week)
- Daily
- Once
You can re-run the audit at any time.
Once you've configured your settings, click Start audit.
Advanced setup and configuration
Steps 2 to 5 of the setup wizard are optional. Use them to control which user agent crawls your site, how fast it crawls, which subfolders and URL parameters to include, and how to reach pages behind restrictions. If the default settings work for you, skip ahead and start your audit.
User agent
Here you choose the user agent Site Audit uses to crawl your site. Your options are SiteAuditBot (Desktop), SiteAuditBot (Mobile), and OpenAI-Search — use OpenAI-Search to check whether your site is crawlable by OpenAI's search bot.
By default, Site Audit crawls with SiteAuditBot (Mobile), which audits your site the way Google's mobile crawler would. You can switch to SiteAuditBot (Desktop) at any time.
When you change the user agent, the string below the field updates to match. You can copy this string into a cURL request to test the user agent yourself.
Crawl delay
You have 3 options for setting a crawl delay: Minimum, 1 URL per 2 seconds, and Respect robots.txt.
Minimum: SiteAuditBot crawls at its normal rate, waiting around 1 second before crawling the next page. This is the default and the fastest option.
1 URL per 2 seconds: SiteAuditBot slows to 1 URL every 2 seconds. Use this if the crawl slows your site and you don't have a crawl delay directive in your robots.txt file. Your audit takes longer to complete, but causes fewer speed issues for visitors during the crawl.
Respect robots.txt: SiteAuditBot follows the crawl delay specified in your robots.txt file. A crawl delay looks like this:
Crawl-delay: 20
JS rendering
Turn this on to have SiteAuditBot execute your JavaScript files and see the same content your visitors do. This gives you more accurate crawl results and a clearer picture of your site's health.
With JS rendering off, Site Audit reads only your HTML — faster and lighter on your site, but less accurate on JavaScript-heavy pages.
Note: JS rendering is available with Guru or Business SEO Toolkit subscriptions.
Allow/disallow rules
Use this step to crawl or block specific subfolders. You can specify multiple subfolders at once.
Enter everything in the URL after the TLD. For example, to crawl http://www.example.com/shoes/mens/, enter /shoes/mens/ in Allow subfolders.
To skip a subfolder, enter its path in Disallow subfolders. For example, to crawl the men's shoes category but skip the hiking boots subcategory at https://example.com/shoes/mens/hiking-boots/, enter /shoes/mens/hiking-boots/ in Disallow subfolders.
Important: Include the trailing slash. If you enter /shoes instead of /shoes/, Site Audit skips everything in the /shoes/ subfolder and every URL starting with /shoes, such as www.example.com/shoes-men.
URL parameter rules
URL parameters, also known as query strings, are elements of a URL that fall outside the hierarchical path structure. They're appended to the end of a URL and pass instructions to the web browser.
A URL parameter consists of a ?, the parameter name, and =. For example, ?page=3 could indicate the third page of a paginated series.
Use the URL parameter rules step to specify the parameters your site uses so Site Audit strips them from URLs while crawling. Without this, a crawler that encounters 2 versions of a page — one with a parameter and one without — may crawl both and waste your crawl budget.
Enter each parameter in Parameters to ignore.
For example, entering page strips every value of that parameter — ?page=1, ?page=2, and so on. Site Audit then treats /shoes and /shoes/?page=1 as a single URL and crawls the page once.
Common parameters include pagination, language, and subcategory. These are especially common on sites with large product or content catalogs. UTM parameters, used to track clicks and traffic from marketing campaigns, are another frequent case.
To change these settings on an existing campaign, open the settings gear and select Audit settings, then adjust the same options described above.
Restrictions to bypass
To audit a site in pre-production or hidden by basic access authentication, you have 3 options.
Bypass disallow rules in robots.txt and meta robots tag
Upload the .txt file Semrush provides to the root folder of your site. This verifies your ownership and allows SiteAuditBot to crawl pages blocked by a disallow directive in your robots.txt file or by a robots meta tag in your page's <head> section.
Crawl with my credentials
Enter the username and password you use to access the hidden part of your site. SiteAuditBot uses these credentials to reach password-protected areas. Recommended for non-public sites and sites under development.
Crawl with Web Bot Auth signature
Some hosting platforms, such as Shopify, block unknown bots by default for security or performance reasons. A Web Bot Auth signature lets SiteAuditBot identify itself and prove it's authorized to access your site. To configure it:
- Select the Crawl with Web Bot Auth signature checkbox
- In Signature Agent, enter the agent URL, for example https://shopify.com
- In Signature Input, enter the input string your host requires
- In Signature, enter the encrypted signature your platform provides
Troubleshooting
If you see the Auditing domain has failed error, your server is likely blocking SiteAuditBot. Follow the Site Audit troubleshooting steps to allowlist the bot.
You can also download the log file generated when a crawl fails and send it to your webmaster, who can use it to identify why SiteAuditBot is blocked.
Connect Google Analytics and Site Audit
Once you've completed the setup wizard, you can connect your Google Analytics account to surface issues affecting your top-viewed pages.
Back to Semrush Toolkits category