If you’re honing your technical SEO skills and finding that tools such as Screaming Frog are taking too long to complete the crawl. Or the resource requirement is hogging down your computer! Why don’t you invest in a budget Window Server VPS? Seriously, if you have a paid license of Screaming Frog (or SiteBulb) run them in the cloud.
Let’s start. First, you will need a Windows VPS from a cloud provider. I use a provider called Contabo, as you can get a super cheap Windows Cloud VPS with 16 GB RAM, 6 cores, and 400 GB of disk space for around 20 Euros a month. Once ordered, you usually get the access details within the hour.
To access Windows Server, I’m using a Remote Desktop Connection. Depending on your OS, there are alternatives to Remote Desktop. TeamViewer and TightVNC are popular alternatives. For this tutorial, I shall be using the good old, tried, assuming you now have a Windows Server VPS ready, let us begin. Login into your VPS via Internet Explorer and download Chrome. IE is not very user-friendly and the strict
Security settings can be frustrating, so I recommend getting Chrome first. A quick tip: download Chrome on your parent machine, copy the installation file to the VPS, and run the installation. That way, you will circumvent the annoying strict IE security settings. From Chrome, you can go ahead to the Screaming Frog download page and download it.
Download Screaming Frog SEO Spider
Run the Screaming Frog Windows Installer
Fantastic – You are now running Screaming Frog in the Cloud
Recap
If you are familiar with Screaming Frog (SF) go ahead activate your license. You can now use all the great features you enjoy with SF.
Why Windows over Linux?
With Windows Server in the cloud running Screaming Frog you can fully utilise all of the great SF features, for example pausing your crawls and updating the excludes list on the fly. Try doing that on the fly using Linux command lines (CLI)? It is simply not intuitive. Other features are easy to operate such as crawl scheduling, API calls to Search Console, Google Analytics and AHREFS. Again, try doing that on CLI.
Another positive you’ll notice is that you can schedule all your crawls without the worry of someone powering down your office machine. You have the added bonus of logging in from anywhere, on the move to grab and analyse data.
Benefit over CLI
You can use your Screaming Frog license on multiple machines, so no worry of having to fork out for additional licenses. If this post has been of some use to you then feel free to leave a comment below. Happy to take some questions on setup and configuration. Lastly, if you’re a Unix command line purest thinking of putting Screaming Frog in the cloud on Linux, be warned setup is easy enough. However, day to day usage is difficult. Pausing and updating exclusions on the fly can be a royal pain in the backside. As for scheduling crawls ‘good luck’. Using remote desktop is far easier and less stressful. VMWare is doable if you’re on an Apple machine.
Final notes:
Depending on what size of VPS and RAM resources you have available, ensure you tweak SF configuration settings to allow for crawl scheduling. Windows Server will usually require around 2GB RAM to run efficiently. The rest you can allocate for SF.
I generally create a folder on the Windows Server desktop for crawl scheduling, with sub-folders for the websites I’m crawling. Larger sites as you probably know can use up to 1GB HDD space per crawl, so ensure you choose a cloud provider with enough space to store your crawls. Personally, I copy recent crawls to my parent machine and delete older crawls from the VPS to save HDD space.
I would also recommend after you have set up and configured the scheduled crawls, to connect to the VPS to see if everything is running fine. Just like on your local machine, crawls can pause if resources are low. One last recommendation, whitelist the cloud server IP with the sites your crawling. Certainly if it is for clients sites if you are scheduling crawls for competitor sites. Slow down the crawl rate, you are less likely to get booted by their firewalls.
Thanks for reading, feel free to share this article.
Let’s start. First, you will need a Windows VPS from a cloud provider. I use a provider called Contabo, as you can get a super cheap Windows Cloud VPS with 16 GB RAM, 6 cores, and 400 GB of disk space for around 20 Euros a month. Once ordered, you usually get the access details within the hour.
To access Windows Server, I’m using a Remote Desktop Connection. Depending on your OS, there are alternatives to Remote Desktop. TeamViewer and TightVNC are popular alternatives. For this tutorial, I shall be using the good old, tried, assuming you now have a Windows Server VPS ready, let us begin. Login into your VPS via Internet Explorer and download Chrome. IE is not very user-friendly and the strict
Security settings can be frustrating, so I recommend getting Chrome first. A quick tip: download Chrome on your parent machine, copy the installation file to the VPS, and run the installation. That way, you will circumvent the annoying strict IE security settings. From Chrome, you can go ahead to the Screaming Frog download page and download it.
Download Screaming Frog SEO Spider
Run the Screaming Frog Windows Installer
Fantastic – You are now running Screaming Frog in the Cloud
Recap
- Order a Windows Server VPS with a Cloud Hosting Provider
- Login into the VPS using Remote Desktop Connection
- Forget using Internet Explorer, download and install Chrome Browser
- From Chrome, download and install Screaming Frog
If you are familiar with Screaming Frog (SF) go ahead activate your license. You can now use all the great features you enjoy with SF.
Why Windows over Linux?
With Windows Server in the cloud running Screaming Frog you can fully utilise all of the great SF features, for example pausing your crawls and updating the excludes list on the fly. Try doing that on the fly using Linux command lines (CLI)? It is simply not intuitive. Other features are easy to operate such as crawl scheduling, API calls to Search Console, Google Analytics and AHREFS. Again, try doing that on CLI.
Another positive you’ll notice is that you can schedule all your crawls without the worry of someone powering down your office machine. You have the added bonus of logging in from anywhere, on the move to grab and analyse data.
Benefit over CLI
- Use excludes and includes on the fly
- Quick API Access
- Crawl scheduling
- Easy custom extraction
You can use your Screaming Frog license on multiple machines, so no worry of having to fork out for additional licenses. If this post has been of some use to you then feel free to leave a comment below. Happy to take some questions on setup and configuration. Lastly, if you’re a Unix command line purest thinking of putting Screaming Frog in the cloud on Linux, be warned setup is easy enough. However, day to day usage is difficult. Pausing and updating exclusions on the fly can be a royal pain in the backside. As for scheduling crawls ‘good luck’. Using remote desktop is far easier and less stressful. VMWare is doable if you’re on an Apple machine.
Final notes:
Depending on what size of VPS and RAM resources you have available, ensure you tweak SF configuration settings to allow for crawl scheduling. Windows Server will usually require around 2GB RAM to run efficiently. The rest you can allocate for SF.
I generally create a folder on the Windows Server desktop for crawl scheduling, with sub-folders for the websites I’m crawling. Larger sites as you probably know can use up to 1GB HDD space per crawl, so ensure you choose a cloud provider with enough space to store your crawls. Personally, I copy recent crawls to my parent machine and delete older crawls from the VPS to save HDD space.
I would also recommend after you have set up and configured the scheduled crawls, to connect to the VPS to see if everything is running fine. Just like on your local machine, crawls can pause if resources are low. One last recommendation, whitelist the cloud server IP with the sites your crawling. Certainly if it is for clients sites if you are scheduling crawls for competitor sites. Slow down the crawl rate, you are less likely to get booted by their firewalls.
Thanks for reading, feel free to share this article.
Last edited:
