How To Find Out Your Entire Website's PageSpeed Insights Score Using Screaming Frog Crawler

Screaming Frog continues to be the most handy SEO tool I can think of. If the existing features weren’t enough, I came across their blog post on how to crawl an entire website to bulk crawl PageSpeed Insights scores for individual pages. Here’s the link link to source blog post: https://www.screamingfrog.co.uk/scaling-pagespeed-insights/

PageSpeed Insights tool only allows you to plug a single page in order to get scores.

This obviously presents a challenge. Say, you’re preparing for a website pitch and need to compare SEO performance for brand vs competitors. You can plug in the homepages of the client vs competitors and compare the scores for the homepage. However, this only addresses the homepage. What about category/sub-category/product pages? These could be equally [if not more important for your website]

I’ll summarize the instructions given in the Screaming Frog blog post.

1. Append “https://developers.google.com/speed/pagespeed/insights/?url=” to all your website URLs to convert them to: https://developers.google.com/speed/pagespeed/insights/?url=http://analyticslog.com/. Repeat this for all pages of your website.

2. Switch Screaming Frog to List mode

3. Turn on JavaScript rendering [Config > Spider > Rendering]

4. Increase AJAX timeout from 5 sec to 10 sec

5. Limit max threads to 1 and Max URLs to 0.1 - 0.5 per sec [Config > Speed]

6. Setup custom extracts . This would be extracting the values from the value in the XPath [Config > Custom > Extract]

   Mobile optimiztion score: (//div[@class="speed-report-card"]//span[1])[1]

   Desktop optimization score: (//div[@class="speed-report-card"]//span[1])[2]
   Mobile FCP: (//div[@class="speed-report-card left"]//span[1])[1]

   Mobile DCL: (//div[@class="speed-report-card left"]//span[2])[1]

   Desktop FCP: (//div[@class="speed-report-card left"]//span[1])[2]

   Desktop DCL: (//div[@class="speed-report-card left"]//span[2])[2]

7. Hit Run [and now patiently wait]

Here, FCP stands for First Contentful Paint. Quoting from Google Devs:

FCP is the point when the browser renders the first bit of content from the DOM, which may be text, an image, SVG, or even a  element.

https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics?utm_source=GDB&utm_campaign=GDB2018Jan#first_paint_and_first_contentful_paint

While DCL is Document Content Loaded - as in when the HTML content is loaded and parsed.

Once the website has been crawled and scores have been generated, you can now compare the distribution of scores across your client + competitors:

1. Mean PageSpeed score for your website [all pages] vs competitors [all pages]

2. Median PageSpeed score [probably a better metric]

3. Quartile distribution of scores

4. Box plot to find out what are the PageSpeed scores within which 50% of the page lie, highest/lowest values and if there are any outliers. Check this blog post If you’d like to generate box plots R Studio.

5. Based on URL structure, find out the key category pages and quickly pull up the scores.

6. Show the distribution of values rather than a single score to bring attention of required SEO changes

Here are the results for my website.

Immediately post crawling, I can see that FCP/DCL metrics are missing for my blog [across all pages, desktop and mobile]. Why is this so?

These two metrics are generated from Chrome User Experience report.

Quote:

The median value for each metric (FCP or DCL) is compared to all the pages monitored by the [Chrome User Experience] CrUX report. Each metric is assigned a category, depending on where it falls in the distribution:

• Fast: The median value of the metric is in the fastest third of all page loads.

• Slow: The median value of the metric is in the slowest third of all page loads.

• Average: The median value of the metric is in the middle third of all page loads

Source: https://developers.google.com/speed/docs/insights/about

https://developers.google.com/web/tools/chrome-user-experience-report/

TL;DR: The bigger your website, the more likely it is to have FCP and DCL metrics available.

Putting these into a pivot > Chart quickly shows you a histogram.

The mean score comes out to be 76. As the distribution is skewed, the median is a better metric. This comes to 83 [for mobile optimization score].

Depending on the number of sites you’re crawling, you could put very easily highlight the key values and quickly draw attention to SEO. Nothing gets more attention of your clients than these two topics: Customers and Competitors.

Using this feature in Screaming Frog adds immense value to your SEO audit as it shows the depth of homework done in understanding the website.