How to fix Shopify crawl issues in Screaming Frog and Sitebulb using Web Bot Auth

If you’ve ever tried crawling a Shopify site at scale, you’ll know how quickly those dreaded 429 Too Many Requests errors can derail your audit. For years, it has been one of the most common frustrations for SEOs working with Shopify stores, and this issue has been worsening in recent months. Good news, though. Shopify has now rolled out a feature called Web Bot Auth that allows you to securely authorise crawlers such as Screaming Frog, Sitebulb, JetOctopus, or OnCrawl. Once set up correctly, it puts a stop to the 429s and lets you fly through audits without wasting hours rerunning failed crawls.

I’ve recently configured this for multiple Shopify stores, and I have estimated that it will save us easily 10 hours a month. Below, I’ll walk you through exactly how it works and how to set it up step by step in Screaming Frog and Sitebulb.

What is Shopify Web Bot Auth?

Shopify introduced Web Bot Auth to give merchants more control over automated tools accessing their sites. It works by generating HTTP message signatures that you then add as headers in your crawler. Shopify checks these headers to verify your requests are authorised, and in return, you don’t get blocked or throttled.

Key features worth knowing

• Secure authentication: cryptographically signed headers unique to your store.
• Expiration control: signatures last a maximum of 3 months; after that, you need to regenerate.
• Domain scoped: each signature only works on the domain you choose.
• Easy to manage: all signatures are visible in Shopify Admin under Online Store → Preferences.

This mechanism is designed specifically for accessibility testing, SEO audits, automated scripts, and data analysis. In short, it’s built for us.

Step 1: Create your signature in Shopify

1. Go to Online Store → Preferences in Shopify Admin.
2. Scroll down to Crawler access.
3. Click Create signature.
4. Give it a name (e.g. “SEO Audit – Screaming Frog”).
5. Pick the domain you want it to apply to.
6. Set the expiry (max 3 months).
7. Copy the three generated values:
   • Signature
   • Signature-Input
   • Signature-Agent (always “https://shopify.com”)

Step 2: Configure Screaming Frog

Now head into Screaming Frog and paste those headers in.

1. Go to Configuration → HTTP Header.
2. Add the three values exactly as given by Shopify:
   • Header: Signature → Value: your signature
   • Header: Signature-Input → Value: your signature-input
   • Header: Signature-Agent → Value: “https://shopify.com”

Save this configuration and run your crawl. You’ll see the difference immediately – no more stalling out after a few hundred pages.

Step 3: Configure Sitebulb

Sitebulb supports custom headers, too. You just need to add them when setting up your project.

1. Start a new audit in Sitebulb.
2. Go to Crawler Settings → Custom HTTP Headers.
3. Enter the same three headers and values:
   • HTTP Header 1: Signature
   • HTTP Header 2: Signature-Input
   • HTTP Header 3: Signature-Agent

Tips and considerations

• Expiry dates: remember that signatures expire. Put a reminder in your calendar to regenerate and update your crawler configs every 3 months.
• Domain-specific: each signature only works for the domain you picked in Shopify. If you’re working across multiple storefronts, generate a signature for each one.
• Security: treat these keys like passwords. Don’t share them around in plain text or in unsecured docs.
• Preview themes: you can also use Web Bot Auth on preview themes, which is handy for testing before launch. Just request the preview link from the client or the devs and crawl using your authorised headers.

Links to official docs

• Shopify Web Bot Auth overview: Shopify support docs
• Screaming Frog custom HTTP headers: Screaming Frog guide
• Sitebulb custom headers: Sitebulb support

Web Bot Auth has turned Shopify crawling into a speedy breeze

This small feature makes a huge difference if you regularly crawl Shopify stores. Instead of burning hours restarting failed crawls or working with incomplete exports, you can run clean, reliable audits every time.

In our experience, Web Bot Auth has turned Shopify crawling from a constant headache into a smooth, predictable process. If you’re hitting 429 errors, get this set up and thank yourself later.