• Home
  • Run Screaming Frog in the Cloud on Windows Server

    If you’re honing your technical SEO skills and finding that tools such as Screaming Frog are taking too long to complete the crawl. Or the resource requirement is hogging down your computer! Why don’t you invest in a budget Window Server VPS? Seriously, if you have a paid license of Screaming Frog (or SiteBulb) run them in the cloud.

    Let’s start, firstly you will need a Windows VPS from a cloud provider. I use a provider called Contabo as you can get a super cheap Windows Cloud VPS with 16gb Ram, 6 cores and 400gb of disk space. Expect to pay around 20 Euros a month. Once ordered you usually get the access details within the hour.
    To access Windows Server, I’m using Remote Desktop Connection. Depending on what OS you are using there are alternatives to Remote Desktop. TeamViewer and TightVNC are popular alternatives. For this tutorial I shall be using good-old tried and tested Remote Desktop.

    Assuming you now have a Windows Server VPS ready, let us begin. Login into your VPS, via Internet Explorer download Chrome. IE is not very user-friendly and the strict
    sf-vps-7-770x515.png


    security settings can be frustrating, so that is why I recommend getting Chrome first. Quick tip – download Chrome on your parent machine, copy the installation file to the VPS and run the installation. That way you will circumvent the annoying strict IE security settings. From Chrome, you can go ahead to the Screaming Frog download page and download it.

    Download Screaming Frog SEO Spider
    sf-vps-1.png

    sf-vps-2.png

    Run the Screaming Frog Windows Installer
    sf-vps-3.png

    sf-vps-4.png


    Fantastic – You are now running Screamin Frog in the Cloud
    sf-vps-5.png


    Recap
    • Order a Windows Server VPS with a Cloud Hosting Provider
    • Login into the VPS using Remote Desktop Connection
    • Forget using Internet Explorer, download and install Chrome Browser
    • From Chrome, download and install Screaming Frog
    Using Windows Server is much like using a regular version of Windows OS, albeit with tighter security settings. Once you have downloaded Screaming Frog, go ahead and run the installer. You should now see Screaming Frog in the Start Menu. Go ahead start the program up. Depending on what version of Windows server you are running, you may need to download some additional libraries such as .net and Java.
    If you are familiar with Screaming Frog (SF) go ahead activate your license. You can now use all the great features you enjoy with SF.

    Why Windows over Linux?
    With Windows Server in the cloud running Screaming Frog you can fully utilise all of the great SF features, for example pausing your crawls and updating the excludes list on the fly. Try doing that on the fly using Linux command lines (CLI)? It is simply not intuitive. Other features are easy to operate such as crawl scheduling, API calls to Search Console, Google Analytics and AHREFS. Again, try doing that on CLI.

    sf-vps-6.png


    Another positive you’ll notice is that you can schedule all your crawls without the worry of someone powering down your office machine. You have the added bonus of logging in from anywhere, on the move to grab and analyse data.

    Benefit over CLI
    • Use excludes and includes on the fly
    • Quick API Access
    • Crawl scheduling
    • Easy custom extraction
    The cloud VPS providers I have recently been using are Vultr, Digital Ocean and Contabo. They all allow you to quickly deploy a cloud server and tweak the settings to your needs. Digital Ocean and Vultr are more expensive and faster, Contabo is slightly slower but has far better burstable Ram.

    You can use your Screaming Frog license on multiple machines, so no worry of having to fork out for additional licenses. If this post has been of some use to you then feel free to leave a comment below. Happy to take some questions on setup and configuration. Lastly, if you’re a Unix command line purest thinking of putting Screaming Frog in the cloud on Linux, be warned setup is easy enough. However, day to day usage is difficult. Pausing and updating exclusions on the fly can be a royal pain in the backside. As for scheduling crawls ‘good luck’. Using remote desktop is far easier and less stressful. VMWare is doable if you’re on an Apple machine.

    Final notes:
    Depending on what size of VPS and RAM resources you have available, ensure you tweak SF configuration settings to allow for crawl scheduling. Windows Server will usually require around 2GB RAM to run efficiently. The rest you can allocate for SF.

    I generally create a folder on the Windows Server desktop for crawl scheduling, with sub-folders for the websites I’m crawling. Larger sites as you probably know can use up to 1GB HDD space per crawl, so ensure you choose a cloud provider with enough space to store your crawls. Personally, I copy recent crawls to my parent machine and delete older crawls from the VPS to save HDD space.

    I would also recommend after you have set up and configured the scheduled crawls, to connect to the VPS to see if everything is running fine. Just like on your local machine, crawls can pause if resources are low. One last recommendation, whitelist the cloud server IP with the sites your crawling. Certainly if it is for clients sites if you are scheduling crawls for competitor sites. Slow down the crawl rate, you are less likely to get booted by their firewalls.

    Thanks for reading, feel free to share this article.