Paul Calvano

Serving Static Content with Cloud Storage? Don’t Forget the CDN!

2026-02-09T04:00:00+00:00

Many websites use cloud storage as part of how they deliver static content to end users. However using these services without a CDN in front of them has the potential to negatively impact performance. You might be surprised by how many sites do just that - about 8.5% of websites that use a CDN for their primary content, based on HTTP Archive data from January 2026!

Background

Content Delivery Networks (CDNs) are an essential component of website delivery, especially when it comes to web performance. According to the 2025 Web Almanac, approximately 70% of popular websites use them. CDNs enable users to connect to servers that are geographically close to them (and at lower latencies), provide distributed caching to offload backend/origin servers and provide other services such as image optimization, edge compute and security. Some CDNs offer cloud storage services, but pretty much all of them can sit in front of another cloud provider’s storage when it is configured for web delivery.

Many websites with a cloud backend/origin host their static content on cloud storage services such as Google Cloud Storage, Amazon S3, Oracle Cloud ObjectStorage and Azure Blob Storage. However it’s important to understand that these services operate as backends and do not automatically provide CDN functionality. Put another way: your content is only behind a CDN if you configure it to be!

In numerous web performance audits over the years, I’ve found cloud storage hostnames being used to deliver static content to end users. During my Performance Mistakes talk in November 2024, I shared that there were 580K websites serving content directly from Amazon S3. This has the potential to negatively impact performance, since those resources are generally served from the locations they are hosted in and not a CDN.

I queried the HTTP Archive to see how common this still is and found a few surprising statistics -

Overall 5.8% of websites serve at least 1 request directly from cloud storage!
8.53% of websites that use a CDN for delivering their primary content, serve at least 1 request directly from cloud storage!
The number of websites serving content directly from Amazon S3 is now 629K - which is an increase of 8.5% since November 2024!

Cloud Storage is not distributed like CDNs

When you configure cloud storage services, you usually have to define which region your content will be hosted in. This is essentially where your static content will live. When you deliver that content directly via a cloud storage solution, then it will fetch the asset the same as it would if it was hosted on a webserver.

For example, below is a request that was delivered from a European news site. I’m browsing it from the northeast US. The hostname s3.eu-central-1.amazonaws.com indicates that this content is being delivered from Amazon’s S3 region in Frankfurt, and I can confirm the same via a traceroute.

When I examine this in Chrome DevTools, I can see very high TCP and TLS connection times for this request!

What type of content is delivered directly via Cloud Storage

In this analysis, I’ve searched for 4 different popular cloud storage providers based on their documented URL structures for web delivery. For Amazon S3 I’m looking for hostnames ending in amazonaws.com with s3 optionally within the hostname. For Google Cloud Storage, I’m looking for a subdomain of storage.googleapis.com. For Azure, a subdomain of windows.net and for Oracle a subdomain of either oraclecloud.com or customer-oci.com.

The graph below illustrates the amount of requests delivered by these services across all measured sites, and it’s grouped by content type. Amazon S3 is by far the most commonly used cloud storage service when it comes to this type of delivery, followed by Google Cloud Storage.

If we look at the same data by the percentage of requests to each cloud storage service, we can see that regardless of the service almost 75% of content delivered from cloud storage are scripts and images. This type of content is best served to users via a CDN rather than a centralized storage solution.

Delivery of JSON content is also common. While JSON can be dynamically generated, if it’s being served from a cloud storage then it is likely cacheable. HTML delivery is less common, except for Oracle Cloud where it represents 11.8% of cloud storage requests!

Caching and Compression

One thing that may not be apparent when configuring cloud storage for delivery of web content is that compression and downstream caching are often not enabled by default.

Based on the HTTP Archive, a majority of compressible content types are not being served compressed when delivered directly from Cloud Storage. This is a critical performance mistake, and is often overlooked because most web servers and CDNs do this by default!

Most content being delivered from cloud storage should be cacheable unless personalized. However the percentage of cloud storage services including Cache-Control headers is incredibly low (with the exception of Google Cloud Storage). That means that clients will have to frequently make requests to these cloud storage services.

Service	Cacheable Requests	Non-Cacheable Requests	% Cacheable	% Non Cacheable
Amazon S3	736,950	2,587,320	22.2%	77.8%
Google Cloud Storage	969,660	259,629	78.9%	21.1%
Azure Blob Storage	102,857	376,291	21.5%	78.5%
Oracle Cloud Object Storage	6,763	27,877	19.5%	80.5%

Digging a bit deeper we can see that images, JS, JSON and CSS account for most of the non-cacheable content delivered via Amazon S3. These asset types are often delivered with cache-control headers allowing caching from Google Cloud Storage.

	Amazon S3		Google Cloud Storage		Azure Blob Storage		Oracle Cloud Object Storage
type	cacheable	not-cacheable	cacheable	not-cacheable	cacheable	not-cacheable	cacheable	not-cacheable
image	561,592	1,743,876	489,387	131,988	84,912	284,536	4,668	18,132
script	91,890	272,434	370,127	45,621	5,378	26,061	1,658	2,599
json	41,046	201,543	48,534	27,917	303	6,083	263	1,126
css	26,926	103,007	30,893	1,340	2,919	18,666	141	863
xml	40	85,778	193	11,909	9	9,186		33
other	2,111	82,043	4,623	2,170	408	14,539		903
font	13,002	33,652	24,619	1,344	8,242	8,511	30	31
video	191	37,699	441	19,435	680	5,183		79
text	148	16,395	386	4,245	2	2,840		37
html		4,350	20	10,382		541		4,071
audio	3	6,514	431	3,278	4	138	3	3
wasm	1	29	6			7

Here’s an example from a popular movie theater chain in the US. This content appears to be loaded by a 3rd party (Unbounce), and that third party is configured to load images directly from S3. There are no cache headers present, which means that the browser will heuristically cache the resources. On this particular page, this third party’s S3 content accounted for over 9MB of content - none of them containing a cache-control header! Beyond that there are a few opportunities for image optimization that could be applied.

In an example from another movie theater’s website, we can see render-blocking CSS and JS loaded from Amazon S3, with no cache-control header to indicate caching, and no compression. This is perhaps the worst case scenario - where you have content critical to the rendering of your website that is loaded slowly from a centralized location, with unnecessary large payloads and then unpredictable caching (or no caching) due to a missing cache-control header.

Cloud Storage vs CDN Delivery Costs

It’s also worth evaluating how much delivering traffic directly from these cloud storage solutions is costing. Depending on your contracts with the providers, you may find that delivering directly via cloud storage is more expensive compared to CDN delivery - especially if you are not caching or compressing the content! If that is the case, you can get a double-win by saving money while improving performance!

Conclusion

It’s incredible that 8.5% of websites that utilize a CDN are delivering content to users directly from cloud storage services. Discovering and fixing these could provide a quick performance boost. When you audit your website’s performance, if you notice cloud storage hostnames then you should definitely investigate how they got there, and move that content behind your CDNs.

HTTP Archive queries

This section provides some details on how this analysis was performed, including SQL queries. Please be warned that some of the SQL queries process a significant amount of bytes - which can be very expensive to run.

What Percentage of CDN delivered sites have request for Cloud Storage hosted assets

This query counts the number of sites that contain at least 1 request for an asset delivered directly via a Cloud Storage service (without a CDN).


 SELECT
    IF(JSON_VALUE(p.summary.cdn) IS NOT NULL AND NOT JSON_VALUE(p.summary.cdn) = "",true, false) AS usesCDN,
    CASE
       WHEN NET.HOST(url) LIKE "%.amazonaws.com" THEN "Amazon S3"
       WHEN NET.HOST(url) LIKE "%storage.googleapis.com" THEN "Google Cloud Storage"
       WHEN NET.HOST(url) LIKE "%.windows.net" THEN "Azure Blob Storage"
       WHEN NET.HOST(url) LIKE "%.oraclecloud.com" THEN "Oracle Cloud Object Storage"
       WHEN NET.HOST(url) LIKE "%.customer-oci.com" THEN "Oracle Cloud Object Storage"
       ELSE "Unknown"
    END AS CloudStorage,
    COUNT(DISTINCT p.page) AS sites
FROM `httparchive.crawl.requests` AS r
INNER JOIN `httparchive.crawl.pages` AS p
ON r.page = p.page
WHERE
   p.date = "2026-01-01" AND r.date = "2026-01-01"
   AND p.client = "mobile" AND r.client = "mobile"
   AND p.is_root_page = TRUE AND r.is_root_page = TRUE
   AND (
     NET.HOST(url) LIKE "%.amazonaws.com"
     OR NET.HOST(url) LIKE "%storage.googleapis.com"
     OR NET.HOST(url) LIKE "%.windows.net"
     OR NET.HOST(url) LIKE "%.oraclecloud.com"
     OR NET.HOST(url) LIKE "%.customer-oci.com"
   )
GROUP BY 1,2

Caching and Compression of Cloud Storage Delivered Assets

This query counts the number of requests being delivered cached and compressed by content type.


 
SELECT
  IF(JSON_VALUE(p.summary.cdn) IS NOT NULL AND NOT JSON_VALUE(p.summary.cdn) = "",true, false) AS usesCDN,
  CASE
    WHEN NET.HOST(url) LIKE "%.amazonaws.com" THEN "Amazon S3"
    WHEN NET.HOST(url) LIKE "%storage.googleapis.com" THEN "Google Cloud Storage"
    WHEN NET.HOST(url) LIKE "%.windows.net" THEN "Azure Blob Storage"
    WHEN NET.HOST(url) LIKE "%.oraclecloud.com" THEN "Oracle Cloud Object Storage"
    WHEN NET.HOST(url) LIKE "%.customer-oci.com" THEN "Oracle Cloud Object Storage"
    ELSE "Unknown"
  END AS CloudStorage,
  JSON_VALUE(r.summary.type) AS type,
  CASE JSON_VALUE(r.payload._contentEncoding)
    WHEN 'gzip' THEN 'Gzip'
    WHEN 'br' THEN 'Brotli'
    WHEN 'zstd' THEN 'zStandard'
    ELSE 'No compression'
  END AS compression_type,
  IF(SAFE_CAST(JSON_VALUE(r.payload._cache_time) AS INT64) > 0, "cacheable", "not-cacheable") AS cacheable,
  COUNT(DISTINCT p.page) AS sites,
  COUNT(*) AS requests
FROM `httparchive.crawl.requests` AS r
INNER JOIN `httparchive.crawl.pages` AS p
ON r.page = p.page
WHERE
  p.date = "2026-01-01" AND r.date = "2026-01-01"
  AND p.client = "mobile" AND r.client = "mobile"
  AND p.is_root_page = TRUE AND r.is_root_page = TRUE
  AND (
      NET.HOST(url) LIKE "%.amazonaws.com"
      OR NET.HOST(url) LIKE "%storage.googleapis.com"
      OR NET.HOST(url) LIKE "%.windows.net"
      OR NET.HOST(url) LIKE "%.oraclecloud.com"
      OR NET.HOST(url) LIKE "%.customer-oci.com"
  )
GROUP BY 1,2,3,4,5

Examples of Sites that load content from cloud storage services

Detailed examples of sites that are loading content from cloud storage services.


 SELECT
   p.rank,
   r.page AS page,
   JSON_VALUE(p.summary.cdn) AS pageCDN,
   NET.HOST(r.url) AS hostname,
   COUNT(*) AS requests,
   SUM(CAST(JSON_VALUE(r.summary.respBodySize) AS INT64)/1024/1024) AS responseMB,
   STRING_AGG(DISTINCT JSON_VALUE(r.summary.type)) AS types,
   FROM `httparchive.crawl.requests` AS r
INNER JOIN `httparchive.crawl.pages` AS p
ON r.page = p.page
WHERE
   p.date = "2026-01-01" AND r.date = "2026-01-01"
   AND p.client = "mobile" AND r.client = "mobile"
   AND p.is_root_page = TRUE AND r.is_root_page = TRUE
   AND p.rank <= 1000 AND r.rank <= 1000
   AND (
     NET.HOST(url) LIKE "%.amazonaws.com"
     OR NET.HOST(url) LIKE "%storage.googleapis.com"
     OR NET.HOST(url) LIKE "%.windows.net"
     OR NET.HOST(url) LIKE "%.oraclecloud.com"
     OR NET.HOST(url) LIKE "%.customer-oci.com"
   )
GROUP BY 1,2,3,4
ORDER BY 6 DESC

Third Parties and Single Points of Failure

2025-12-29T04:00:00+00:00

You’ve heard it many times - third party content can easily cause an otherwise well performing website to become sluggish and slow. And depending on how this content is loaded, it can also introduce single points of failure (SPOFs). When a large cloud provider or content delivery network (CDN) experiences a disruption, their impacts are felt across the world and often triggers headlines about the many websites that were affected. However, there are numerous secondary impacts triggered by third party content, which can be disruptive even to companies that don’t use the affected provider.

In this blog post I’ll discuss some of the performance and availability risks associated with third party content and how you can test for these single points of failure (SPOFs) on your websites. I’ll use HTTP Archive data to explore how many sites could be at risk and RUM Archive to get a sense of how prone to slowdowns some of these third parties are!

Third Party Failure = SPOF

A third-party single point of failure (SPOF) can occur when a website depends on an external service for critical or render-blocking resources. If the third party content fails to load, then the page load may be stalled until the request times out. For example, in the two filmstrips (taken from a WebPageTest private instance) you can see how a failure of a third-party affects the user experience. The SPOF measurement in this example simulates what a client would see if a render-blocking third party, such as a consent management service, became unavailable. In this case, the user would essentially see a blank screen until the third party request times out.

Most recently, people have been noticing SPOFs due to outages experienced by large cloud providers and CDNs. For example, a major CDN recently experienced a few high profile outages and many of their customers were impacted. However, many sites that don’t use their services directly were also affected due to third party content that was being delivered through them. That could have been avoided by self hosting content that is critical to loading of the website.

This risk isn’t just isolated to CDNs, nor is it a new issue. For example, many years ago websites started adding Facebook like buttons to their pages via synchronously loaded scripts. When Facebook experienced an outage way back in 2012, the failed third party requests slowed down a large number of websites and browsers were stalled waiting for a script to load. Facebook fixed this a long time ago, and even introduced a non-blocking script loader pattern for other websites to adopt. Additionally, the like button is being deprecated in February 2026. They plan to return an empty response to avoid issues on sites that will inevitably forget to remove them!

There have been other occurrences where a third party stops providing their service or is compromised as well. Using the HTTP Archive, we can see how many websites are still referencing this old content. For example:

RawGit announced it was shutting down in 2018. Their webpage recommends alternatives, but based on the November 2025 HTTP Archive data, there are over 25K websites still requesting content from it!
In February 2024, the Polyfill.io service was sold to a third party and concerns were raised immediately about what that meant for sites using the service. By June 2024 it was used to deliver malicious content via a supply chain attack. Based on HTTP Archive data, I can see references to this third party on 161K websites in February 2024, and then it gradually decreased from 115K to 109K through June 2024! All occurrences of its use stopped after July 2024 – possibly due to intervention by browser vendors and their domain registrar taking down the domain name.

Non-web applications can also experience issues from third party failures as well. For example, a while back one of Etsy’s CDNs experienced an incident and I needed to fail over our traffic to our other CDN. During the failover, I discovered that part of the process for this CDN failover actually relied on a third party which had a dependency on the same CDN I needed to route away from! To avoid a similar occurrence, a break-glass failover mechanism was created to bypass all dependencies on CDNs during a failover event.

But I Don’t Use <Provider>?

When you add a third party to your website, you are telling the browser to load their content as if it were your own. Depending on how that content is loaded, it could block the page from rendering anything to the screen. If a render-blocking third party happens to be served by a provider that is experiencing a performance degradation or outage, then the site’s performance may be significantly affected.

Any third party content added to your website carries risks, and should be added with care and tested thoroughly. However, extra care should be taken when it comes to render-blocking requests. This isn’t new advice either - Steve Souders first wrote about this in 2010! There have been PerfPlanet articles about this dating back to 2011 (just search for SPOF)! A simple recommendation remains relevant 15 years later: avoid third party single points of failure, and test/monitor your pages to ensure that none are introduced.

How Prevalent are Third Party SPOFs Today?

I’m raising a lot of concern about a problem that has been well-known and talked about for over 15 years. You might wonder how much of an issue this can still be on today’s web. According to the December 2025 HTTP Archive dataset, a shocking 67.7% (of 15.78 million websites) request at least one render-blocking third party! Note: This classification is based on request URLs that are from a different domain name than the page URLs. This does not include subdomains of the page URLs host, which means the actual numbers may be higher!)

Additionally, 60% (9.4 million) of sites load at least one render-blocking third party through a different CDN from their primary content! Even more shocking is that there are over 1 million websites loading render-blocking content via a third party that does not use a CDN at all!

Sites with Render Blocking Third Parties - HTTP Archive December 2025
Category	Number of Sites	% of Sites
Number of Sites	15,780,490
Sites with a Render Blocking Third Party	10,683,209	67.70%
Sites with a Render Blocking Third Party on a different CDN	9,444,516	59.85%
Sites with a Render Blocking Third Party not using a CDN	1,075,294	6.81%

If we focus on the websites that have a render-blocking third party hosted on a different CDN, you can see that the request types skew towards CSS and JavaScript (with many sites loading both of these types of requests).

When we break this down by hostname and content type, you can see that Google Fonts is one of the larger sources of render-blocking third parties. There are also other font providers like Typekit and Fontawesome as well. Since CSS is generally render-blocking, including third party CSS for font loading may introduce a SPOF risk!!

Additionally, there are hundreds of thousands of sites that are utilizing third parties to load libraries onto their sites. For example, cdnjs.cloudflare.com, cdn.jsdelivr.net, ajax.googleapis.com, code.jquery.com, maxcdn.bootstrapcdn.com, and unpkg.com. If you find yourself using any of these to deliver content that is critical to the loading of your website, I highly encourage you to read Harry Robert’s excellent writeup about why you should self host your static assets.

Websites w/ Third Party Content - HTTP Archive December 2025
hostname	css	script	other	html	text
fonts.googleapis.com	5,837,138			22
www.gstatic.com	1,277,310	1,273,686
cdnjs.cloudflare.com	675,493	390,541	9		115
cdn.jsdelivr.net	660,322	351,682	20	34	6
www.youtube.com	772,925	39,762	5
ajax.googleapis.com	57,354	702,890
code.jquery.com	67,244	368,220
maxcdn.bootstrapcdn.com	332,198	49,794
use.fontawesome.com	315,930	21,300		6
tpc.googlesyndication.com	173	299,810
use.typekit.net	251,913	43,365		6	71
static.xx.fbcdn.net	147,085	136,605
p.typekit.net	272,158
static1.squarespace.com	236,853	6,458
definitions.sqspcdn.com	204,076	37,235
unpkg.com	127,853	84,910			13,738
pagead2.googlesyndication.com	435	203,536		204
www.google.com	326	186,648	146	14,379
assets.squarespace.com	135,575	41,561
c0.wp.com	85,064	90,345
stackpath.bootstrapcdn.com	101,810	27,460
kit.fontawesome.com	4,497	101,627	14
maps.googleapis.com		98,646
a.amxrtb.com		97,849
cdn-cookieyes.com		91,836		10
consent.cookiebot.com		79,041
t1.daumcdn.net	22,278	54,285

Third Party Slowdowns

While major cloud provider failures are not a frequent occurrence, slowdowns and microoutages absolutely are. If your content is delivered by a third party then the performance of that content is out of your control. Self hosting where possible will put you in control of the delivery of this content and reduce the risk of a slowdown triggered by a third party. If you have to load any content from a third party, you can (and should) test for single points of failure and monitor their performance.

Some Real User Monitoring (RUM) services have the ability to collect resource timing data, which makes it possible to monitor third party content across all your site’s visitors. You can also monitor via synthetic measurements as well - which may be helpful in case payload sizes drastically increase. At Etsy I use Speedcurve’s performance budgets to detect shifts in quantifiable metrics (such as requests, sizes, etc). I often correlate alerts with our RUM data to determine if a third party change resulted in a slowdown, and also add custom metrics for third parties that may present a SPOF risk. In the example below, you can see that the number of blocking scripts and stylesheets are consistently low - but the budget is set so that any shift from the high-water mark will trigger an alert.

Speedcurve also provides the ability to track individual third parties as part of their performance budgets - which can be helpful if you have identified a third party that you want to track over time.

I’m also using Catchpoint to trend third party domains over time. In the graph below, I’m aggregating results from a large number of Chrome synthetic measurements, grouping the results by hostname, and excluding first party domains. This type of reporting provides the ability to monitor all discovered third parties over time; collecting insights into their performance, availability, requests counts and payload sizes. In the example below you can see that one of the third parties experienced a large outage on November 25th, and another experienced a smaller outage on December 5th. Fortunately these were not render blocking and did not impact the user experience.

The RUM Archive is also an interesting project for analyzing third party performance data. This dataset comes from approximately 100 of Akamai’s customers mPulse data. While this will not capture as many third parties as we can observe in the HTTP Archive, it does provide the ability to look at their real world performance data! When I combined these two data sources for render blocking third parties, I found 15 third parties that had at least 1 million RUM measurements. The table below breaks down their DNS, TCP and TTFB times. Many of these third parties have relatively fast DNS times, but there’s a lot of room for improvement when it comes to TCP connection times and TTFB.

While the table above illustrates the p75 timings, we can also look at other percentiles as well to get a larger picture of the overall performance impact of these third parties. The graph below illustrates the inter-quartile range for TTFB, with the bars representing the p25 through p75 - essentially 50% of all measurements. The whiskers represent the p5 and p95. A few third parties stand out for having very poor TTFB, while CookieLaw and Critero seem to have the fastest and most consistent performance.

How to Detect Render-Blocking Third Parties

When you run a test in WebPageTest, you can see a yellow circle with an X in the waterfall next to each request that is render-blocking. A similar visual is used in the legacy UI if you are using a WebPageTest private instance. If you notice any render blocking third party domains, then make a note of them so that you can run SPOF tests.

There’s also my Third Party Analyzer tool, which will highlight SPOF risks based on resources that are render blocking and load before FCP. This works with any WebpageTest private instance tests, Catchpoint WPT shared URLs, or Speedcurve tests. You can read more about this tool here.

If you run a measurement with DebugBear, you can see a “Blocking” indicator next to every resource that is render-blocking . They also illustrate the priority of each request as well as the domain name. Similar to the previous examples, if you see render-blocking content for a third party domain name, then make a note of them so that you can run SPOF tests.

And lastly, you can also see requests via a Lighthouse test. As with the other examples, just look at the list of requests and determine which of these are third parties.

How to Test for Third Party SPOFs

The most common way of testing for third party single points of failure is to identify which content is render-blocking, and then to test what would happen if that content fails. For years, the easiest way of testing for SPOFs was via WebPageTest as it had a SPOF simulation feature. Today, there are a handful of other methods of testing for them and below I’ll show a few examples.

WebPageTest Private Instance

In my earlier example I used a WebPageTest private instance, which has a feature that simulates a SPOF. When you add a hostname to the list, it will run two tests - one without any changes and the other with the hostname routed to a server that silently drops the request (simulating a failure). The results of this test show two filmstrips, and you can easily determine whether the page rendering is stalled when the third party failed to load.

Catchpoint WebPageTest

There is also a SPOF feature in the public WebPagetest instance hosted by Catchpoint. At the time of writing this, the SPOF feature is not currently working. Once the feature is fixed, it should work the same as the above example. Until then you can still use WebPageTest to simulate a third party failure by using a script that overrides DNS for the third party, pointing it to [blackhole.webagetest.org](http://blackhole.webagetest.org). This emulates how the SPOF feature in WebPageTest works.

Hosts File Entries

You can use a hosts file to override DNS and route one or more third parties to blackhole.webpagetest.org. In order to do this, look up the IP address of the blackhole hostname. At the time of writing, it resolved to 3.219.212.117.

Next you can update your /etc/hosts file (Mac) or C:\Windows\System32\drivers\etc\hosts file (Windows). You can add one or more third party hostnames. Once the browser picks up the new hosts file entries, you’ll be able to test for failure by simply browsing to the site you are testing.

3.219.212.117 cdn.cookielaw.org cdn.optimizely.com

Chrome DevTools

Chrome DevTools has recently added an individual request throttling feature in Chrome. This feature should be enabled with Chrome 144, but if you are using an earlier version (or Chrome Canary) then you can enable an experimental flag to allow individual request throttling. You can find this flag at chrome://flags/#devtools-individual-request-throttling. DebugBear shared a great blog post about how to use this feature here.

Once enabled, you’ll be able to throttle a request or domain by selecting a network profile for the content. The default options are Block, Fast 4G, Slow 4G and 3G.

To simulate a single point of failure, I wanted something even slower than 3G. So in the network throttling profile section, I defined a profile that adds 10 seconds of latency. When you refresh the page with this profile, you can experience the site in your browser as if the third party was struggling to load the content.

Try configuring the network throttling for a render blocking third party, and then go to the performance panel in DevTools and capture a trace with filmstrips. If you see a long gap in page rendering, then you know you have a SPOF! Here’s an example from a popular US ecommerce site I found in the HTTP Archive. It’s loading a JavaScript request for a consent management service at cdn-ukwest.onetrust.com. If I slow that down by 10 seconds and measure the page load, I can see that the FCP is also delayed by the same. Simply adding the async script tag for this third party eliminates the risk of a SPOF!

Conclusion

Third Party Single Points of Failure have been talked about for 15 years now. The problem is well understood, and there’s lots of guidance and testing approaches around this. However, that hasn’t stopped 67% of websites from adding third parties in this way!

Cloud provider and CDN outages happen, and it’s unfortunate for all involved when they do. However, someone else’s cloud provider outage shouldn’t take down your website. I highly recommend auditing your third party domains to ensure that a slowdown or failure will not result in a disruption of service on your websites. Beyond that, as a general preventive measure - try to self host as much of your render-blocking content as possible.

This article represents my own views and opinions and not those of Etsy.

Originally published at https://calendar.perfplanet.com/2025/third-parties-and-single-points-of-failure/

HTTP Archive queries

Number of sites containing render blocking third parties

This query counts the number of sites that contain a render blocking third party, and also whether they are using a different CDN (or no CDN)!


   SELECT
    "Number of Sites" AS category,
    COUNT(DISTINCT page) AS sites
  FROM
    `httparchive.crawl.pages` 
  WHERE
    date = "2025-12-01"
    AND is_root_page = TRUE 
    AND client = "mobile" 
  

UNION ALL

 SELECT
    "Sites with a Render Blocking Third Party" AS category,
    COUNT(DISTINCT page) AS sites
  FROM
    `httparchive.crawl.requests`
  WHERE
    date = "2025-12-01"
    AND is_root_page = TRUE 
    AND client = "mobile" 
    AND NET.REG_DOMAIN(page) != NET.REG_DOMAIN(url)
    AND JSON_VALUE(payload._renderBlocking) = "blocking"

UNION ALL


 SELECT
    "Sites with a Render Blocking Third Party on a different CDN" AS category,
    COUNT(DISTINCT p.page) AS sites
  FROM
    `httparchive.crawl.requests` AS r
    INNER JOIN `httparchive.crawl.pages`  AS p
    ON r.page = p.page
  WHERE
    r.date = "2025-12-01" AND p.date = "2025-12-01"
    AND r.is_root_page = TRUE AND p.is_root_page = TRUE
    AND r.client = "mobile" AND p.client = "mobile"
    AND JSON_VALUE(r.payload._renderBlocking) = "blocking"
    AND NET.REG_DOMAIN(p.page) != NET.REG_DOMAIN(r.url)
    AND JSON_VALUE(p.summary.cdn) != JSON_VALUE(r.payload._cdn_provider)

UNION ALL

SELECT
    "Sites with a Render Blocking Third Party not using a CDN" AS category,
    COUNT(DISTINCT p.page) AS sites
  FROM
    `httparchive.crawl.requests` AS r
    INNER JOIN `httparchive.crawl.pages`  AS p
    ON r.page = p.page
  WHERE
    r.date = "2025-12-01" AND p.date = "2025-12-01"
    AND r.is_root_page = TRUE AND p.is_root_page = TRUE
    AND r.client = "mobile" AND p.client = "mobile"
    AND JSON_VALUE(r.payload._renderBlocking) = "blocking"
    AND NET.REG_DOMAIN(p.page) != NET.REG_DOMAIN(r.url)
    AND (
        JSON_VALUE(r.payload._cdn_provider) IS NULL
        OR JSON_VALUE(r.payload._cdn_provider) = "")

Third Party Render Blocking Requests by Content Type

This summarizes the request types that are used for render blocking third party requests.


 SELECT
    r.type AS requestType,
    COUNT(DISTINCT p.page) AS sites,
    COUNT(*) AS requests
  FROM
    `httparchive.crawl.requests` AS r
    INNER JOIN `httparchive.crawl.pages`  AS p
    ON r.page = p.page
  WHERE
    r.date = "2025-12-01" AND p.date = "2025-12-01"
    AND r.is_root_page = TRUE AND p.is_root_page = TRUE
    AND r.client = "mobile" AND p.client = "mobile"
    AND JSON_VALUE(r.payload._renderBlocking) = "blocking"
    AND NET.REG_DOMAIN(p.page) != NET.REG_DOMAIN(r.url)
    AND JSON_VALUE(p.summary.cdn) != JSON_VALUE(r.payload._cdn_provider)
    AND is_main_document = FALSE
GROUP BY 1
ORDER BY 2 DESC

Third Party Hostnames w/ Render Blocking Content

This query breaks down popular third party hostnames that are being used for render blocking content, whith a focus on requests that use a different CDN from the primary website.



 SELECT
    NET.HOST(url) as hostname,
    r.type AS requestType,
    COUNT(DISTINCT p.page) AS sites,
    COUNT(*) AS requests
  FROM
    `httparchive.crawl.requests` AS r
    INNER JOIN `httparchive.crawl.pages`  AS p
    ON r.page = p.page
  WHERE
    r.date = "2025-12-01" AND p.date = "2025-12-01"
    -- root pages
    AND r.is_root_page = TRUE AND p.is_root_page = TRUE
    -- mobile
    AND r.client = "mobile" AND p.client = "mobile"
    -- render blocking requests
    AND JSON_VALUE(r.payload._renderBlocking) = "blocking"
    -- with a different domain name from the page
    AND NET.REG_DOMAIN(p.page) != NET.REG_DOMAIN(r.url)
    -- and a different CDN
    AND JSON_VALUE(p.summary.cdn) != JSON_VALUE(r.payload._cdn_provider)
GROUP BY 1,2
ORDER BY 3 DESC

RawGit usage


 
 SELECT
    COUNT(DISTINCT page)
  FROM
    `httparchive.crawl.requests` AS r    
  WHERE
    date = "2025-11-01"
    AND is_root_page = TRUE
    AND client = "mobile" 
  AND NET.HOST(url) LIKE "%rawgit.com"

Polyfill.io Usage


 
 SELECT
    date,
    COUNT(DISTINCT page)
  FROM
    `httparchive.crawl.requests` AS r
    
  WHERE
    date BETWEEN "2024-01-01" AND "2025-01-01"
    AND is_root_page = TRUE
    AND client = "mobile" 
  AND NET.HOST(url) LIKE "%polyfill.io"
GROUP BY date
ORDER BY date

RUM Archive stats for Render Blocking Third Parties


 SELECT 
  NET.HOST(url) AS hostname, 
  SUM(fetches) AS freq,

  -- DNS
  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(DNSHISTOGRAM),
    [0.75],
    10,
    false
  )) / 1000 AS dns_p75,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(DNSHISTOGRAM),
    [0.95],
    10,
    false
  )) / 1000 AS dns_p95,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(DNSHISTOGRAM),
    [0.99],
    10,
    false
  )) / 1000 AS dns_p99,

  -- TCP
  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TCPHISTOGRAM),
    [0.75],
    10,
    false
  )) / 1000 AS tcp_p75,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TCPHISTOGRAM),
    [0.95],
    10,
    false
  )) / 1000 AS tcp_p95,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TCPHISTOGRAM),
    [0.99],
    10,
    false
  )) / 1000 AS tcp_p99,

  -- TLS
  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TLSHISTOGRAM),
    [0.75],
    10,
    false
  )) / 1000 AS tls_p75,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TLSHISTOGRAM),
    [0.95],
    10,
    false
  )) / 1000 AS tls_p95,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TLSHISTOGRAM),
    [0.99],
    10,
    false
  )) / 1000 AS tls_p99,

  -- TTFB
  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TTFBHISTOGRAM),
    [0.75],
    10,
    false
  )) / 1000 AS ttfb_p75,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TTFBHISTOGRAM),
    [0.95],
    10,
    false
  )) / 1000 AS ttfb_p95,

  PARSE_NUMERIC(`akamai-mpulse-rumarchive.rumarchive.PERCENTILE_APPROX`(
    ARRAY_AGG(TTFBHISTOGRAM),
    [0.99],
    10,
    false
  )) / 1000 AS ttfb_p99

FROM `akamai-mpulse-rumarchive.rumarchive.rumarchive_resources` 
WHERE date = "2025-12-20" 
  AND NET.HOST(url) IN (
    SELECT
        NET.HOST(url) as hostname,
      FROM
        `httparchive.crawl.requests` AS r
        INNER JOIN `httparchive.crawl.pages`  AS p
        ON r.page = p.page
      WHERE
        r.date = "2025-12-01" AND p.date = "2025-12-01"
        -- root pages
        AND r.is_root_page = TRUE AND p.is_root_page = TRUE
        -- mobile
        AND r.client = "mobile" AND p.client = "mobile"
        -- render blocking requests
        AND JSON_VALUE(r.payload._renderBlocking) = "blocking"
        -- with a different domain name from the page
        AND NET.REG_DOMAIN(p.page) != NET.REG_DOMAIN(r.url)
        -- and a different CDN
        AND JSON_VALUE(p.summary.cdn) != JSON_VALUE(r.payload._cdn_provider)
    GROUP BY 1
    ORDER BY COUNT(DISTINCT p.page) DESC
    )
GROUP BY 1
HAVING SUM(fetches) >= 1000000
ORDER BY 2 DESC;

AI Bots and Robots.txt

2025-08-21T04:00:00+00:00

There’s been a lot of discussion lately around AI crawlers and bots, which are used to train LLMs and/or fetch content on behalf of their users. In the past few weeks I’ve seen blog posts about the amount of traffic from these crawlers, techniques and products to control how and what they can crawl, reports of misbehaving crawlers and more. Ironically, there’s even AI based services to mitigate AI crawler bots! Given how much interest there is, I thought I’d try and explore some HTTP Archive data to see how sites are using robots.txt to state their preferences on AI crawling.

Robots.txt

A robots.txt file is located at the root of an origin, and provides instructions for how bots should interact with a website. This practice began in 1994 and quickly became a de facto standard. Years later, Google introduced the Robots Exclusion Protocol, which was standardized in 2022.

A simple example of a robots.txt directive is below. This directive tells a User-Agent with the string GPTBot that it is not permitted to crawl the site.

User-Agent: GPTBot
Disallow: /

It’s important to note that robots.txt does not restrict access to bots by itself, as adherence to it is voluntary. But analyzing the content of these files can provide some insight on the overall sentiment towards AI bots.

How Many Sites use Robots.txt Files?

The HTTP Archive collects page details from millions of websites each month, and uses a custom metric to fetch a robots.txt file from each site. The data from July 2025 shows that 94% of 12 million websites have a robots.txt file containing at least 1 directive. The HTTP Archive’s Web Almanac has an entire chapter on SEO, containing more details around the contents of robots.txt - and is definitely worth a read.

	Sites	% of Sites
Serves a Robots.txt file	12,155,217	94.12%
No Robots.txt	759,409	5.88%

Some bots choose to identify themselves in the User-Agent string of an HTTP request. Others attempt to hide what they are doing, which is one of the reasons why many sites utilize services to block unwanted traffic. This research will focus on robots.txt files, which tell us whether site owners would like to restrict AI bots based on their advertised User-Agent strings.

Many AI services publish the User-Agent strings for this purpose, and also provide guidance on how they adhere to robots.txt directives. Additionally it’s commone for a service to advertise multiple User-Agents since they can be used for different purposes (crawling, responding to user input, etc).

For example

ChatGPT advertises themselves as ChatGPT-User, GPTBot, and OAI-SearchBot
Anthropic advertises themselves as ClaudeBot, Claude-User, and Claude-SearchBot
Apple advertises themselves as Applebot-Extended

There are many more AI agents, with more showing up all the time - and unfortunately not all of them respect the robots.txt directives. For this research, I’ve used a list of AI Bots from the AI Robots.txt Github repository to determine which robots.txt entries are targeted towards the various AI services.

User-Agents Referenced in Robots.txt

The most popular User-Agent referenced in robots.txt files is simply a wildcard *. In fact 97.4% of robots.txt files have at least 1 directive using this wildcard, often to allow and/or disallow bots to part or all of their site’s content to all bots. Typically the next most frequent group of User-Agents are for search bots (googlebot, bingbot, etc) and SEO bots (mj12bot, ahrefsbot, semrushbot, etc).

Over the past few years, User-Agents for AI Bots have been added to many sites’ robots.txt files. As of July 2025, AI Bots top the list of User Agents referenced across popular sites. In fact almost 21% of the top 1000 websites have rules for ChatGPT’s “GPTBot” in their robots.txt file. There’s an interesting pattern shift around site popularity, with a greater percentage of popular sites having AI bot directives vs a large percentage of SEO bot directives on less popular sites.

The table below breaks this out by site popularity (using Google’s CrUX rank). In the darker shaded areas of this table, you can see many references to bots operated by popular AI services - ChatGPT, Claude, Google, Perplexity, Anthropic, etc.

When Did Sites Start Adding AI Crawlers to robots.txt

In August 2023, ChatGPT added documentation about its crawler, including instructions on how site owners can block them. Shortly afterwards, articles with instructions on how to block ChatGPT started appearing, it was discussed in Hacker News and there were also claims that some sites were scrambling to block AI agents. While the claims may have sounded sensational, the numbers supported it. In August 2023 the number of sites that included rules for GPTBot in robots.txt files went from 0 to almost 125k sites! A month later it was 299k sites. By November GPTBot was referenced on 578k websites! That’s a massive increase in a short period of time.

In the tables below you can see the number of websites referencing specific User-Agents in their robots.txt files month to month. Claudebot first appeared in December 2023 on just 2,382 sites (increasing to 30k within 4 months) and PerplexityBot appeared in January 2024 with just 157 sites (increasing to 31k in April 2024). These were not picked up as quickly as GPTBot, which may have been due to limited public awareness of the rise of these models.

Throughout 2024 you can see that more and more sites include directives for the AI bots. Apple’s crawler was revealed in May 2024, and news reports about it started showing up in June. By September there was almost 262k sites including it in their robots.txt files. Perplexity and Claude also started appearing in over 100k site’s robot.txt in May 2024. And a handful of new ones started appearing as they gained popularity.

That brings us to 2025 where you can see that ChatGPT, Claude, Facebook and others appear in the robots.txt files of over 560k sites! A few newer bots started showing up as well - belonging to Meta, DuckDuckGo, and Quora. This might be due to some services and platforms updating robots.txt files automatically, but it could also be due to more awareness as there have been frequent articles about AI crawlers and bots throughout 2025.

You can explore this data more in this interactive visualization. There are many more User-Agents than what I listed here, and if you scroll down you might spot some newer ones.

To Allow or Not to Allow…

So far we’ve looked at how many websites are including directives for various User-Agents, and we’ve observed substantial growth in the references to AI crawlers and bots. But we’ve also observed a difference in how sites are using them based on popularity. Now let’s look at the types of rules being applied to them.

It’s not uncommon for rules to exist both allowing a bot and disallowing certain paths. So the presence of both could indicate that site owners want to allow those bots to access portions of their sites. For example on Vimeo’s robots.txt file, you can find

# Block Open AI - Allowing specific marketing/informational paths
User-agent: GPTBot
Disallow: /
Allow: /features/
Allow: /solutions/
Allow: /enterprise/
Allow: /integrations/
Allow: /blog/
Allow: /create/

Some sites look to restrict AI traffic, such as this excerpt from Healthline’s robots.txt file -

User-agent: GPTBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

This graph below shows AI Crawler directives for popular websites. From this data we can see that it’s very common for popular websites to attempt to restrict access to AI bots. However, they are not uniform in the bots that they choose to disallow - most of the time only including directives for the most popular AI services.

If we look at this across all sites a new pattern emerges. We can see a lot more sites that have both allow and disallow directives for AI agents. This may be due to large platforms automatically updating these agents in their customer’s robots.txt file. For example, Squarespace and Cloudflare both have solutions that appear to automatically add AI crawlers to a site’s robots.txt file.

You can explore this data more in these interacitive visualizations:

Conclusion

AI crawlers and bots have become a significant source of synthetic traffic for many sites, and awareness has been growing over the last few years. More and more bots are being introduced which makes it a challenge to keep up. It’s interesting to see how trends in news cause major shifts in website strategy across the web. Often new technologies or features are adopted at a gradual rate. The appearance of the AI bot user agents in so many websites over a short period of time reflects the general sentiment that site owners have towards the scraping of their content for training models.

However, there isn’t uniformity in how AI bots are being treated across the web - especially when compared to the month to month consistency of the SEO and Search agents. The most popular AI services bots appear in more robots.txt files because they have been noticed, and newer ones aren’t picked up as quickly. One can interpret this to show that although the intent to manage how these bots crawl websites is clear, the effectiveness for an individual site is limited to how well each site maintains the list of bots and crawlers.

HTTP Archive queries

Number of sites containing robots.txt files

This query counts the number of websites that contain a robots.txt file. In order to ensure that we are counting an actual robots.txt file and not error pages, this query only counts robots.txt files that return an HTTP 200 status code and contains one of the following directives: allow, disallow, crawl_delay, noindex, sitemap or user_agent.


SELECT
 sites,
 sites_with_robots_txt,
 ROUND(sites_with_robots_txt / sites,4) AS pct_sites_with_robots_txt
FROM (
 SELECT
 COUNT(*) AS sites,
 COUNTIF(CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.allow") AS INT64)
   + CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.disallow") AS INT64)
   + CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.crawl_delay") AS INT64)
   + CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.noindex") AS INT64)
   + CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.sitemap") AS INT64) 
   + CAST(JSON_VALUE(custom_metrics.robots_txt.record_counts.by_type, "$.user_agent") AS INT64) > 0)
   AS sites_with_robots_txt
 FROM `httparchive.crawl.pages` AS pages
 WHERE date = "2025-07-01"
 AND client = "mobile"
 AND CAST(JSON_VALUE(custom_metrics.robots_txt, "$.status") AS INT64) = 200
 AND is_root_page = true
)

Percent of Sites with User-Agents Appearing in robots.txt Files, by popularity rank

The HTTP Archive stores robots.txt information in a custom metrics object. In this SQL script, we’re UNNESTing each user-agent and then searching for it’s stats within the custom metric.


CREATE TEMP FUNCTION GetByAgent(json STRING, agent STRING)
RETURNS STRING
LANGUAGE js AS r"""
 try {
   const obj = JSON.parse(json || '{}');
   const byua = (((obj || {}).record_counts || {}).by_useragent) || {};
   const body = byua[agent] || byua[String(agent).toLowerCase()] || byua[String(agent).toUpperCase()];
   return body ? JSON.stringify(body) : null;
 } catch (e) { return null; }
""";


WITH robots_txt_ua AS (
 SELECT
   rank,
   page,
   agent,
   GetByAgent(TO_JSON_STRING(custom_metrics.robots_txt), agent) AS agent_obj
 FROM `httparchive.crawl.pages`,
 UNNEST(
   REGEXP_EXTRACT_ALL(
     TO_JSON_STRING(JSON_QUERY(custom_metrics.robots_txt, '$.record_counts.by_useragent')),
     r'"([^"]+)":\{'
   )
 ) AS agent
 WHERE date = "2025-07-01"
   AND client = "mobile"
   AND is_root_page = TRUE
),
robots_txt_rule_counts_by_ua AS (
 SELECT
   rank,
   page,
   LOWER(agent) AS agent,
   agent_obj,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.allow') AS INT64)        AS allow_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.crawl_delay') AS INT64)  AS crawl_delay_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.disallow') AS INT64)     AS disallow_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.noindex') AS INT64)      AS noindex_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.other') AS INT64)        AS other_cnt
 FROM robots_txt_ua
),
pages_in_rank_group AS (
 SELECT
   rank,
   COUNT(DISTINCT page) AS total_sites
 FROM `httparchive.crawl.pages`
 WHERE
   date = "2025-07-01"
   AND client = "mobile"
   AND is_root_page = TRUE
GROUP BY 1
ORDER BY 1
)

SELECT
 robots_txt_rule_counts_by_ua.rank,
 agent,
 total_sites,
 COUNT(DISTINCT page) AS sites
FROM robots_txt_rule_counts_by_ua
LEFT JOIN pages_in_rank_group
ON robots_txt_rule_counts_by_ua.rank = pages_in_rank_group.rank
GROUP BY 1,2,3
ORDER BY 1,4 DESC

2023-2025 Trend of Sites with User-Agents Appearing in robots.txt Files

This query builds on top of the previous one, but runs against multiple dates.

Warning: As of July 2025, this SQL query processes approximately 330GB of data. Running multiple queries like this can be costly.


CREATE TEMP FUNCTION GetByAgent(json STRING, agent STRING)
RETURNS STRING
LANGUAGE js AS r"""
 try {
   const obj = JSON.parse(json || '{}');
   const byua = (((obj || {}).record_counts || {}).by_useragent) || {};
   const body = byua[agent] || byua[String(agent).toLowerCase()] || byua[String(agent).toUpperCase()];
   return body ? JSON.stringify(body) : null;
 } catch (e) { return null; }
""";


WITH robots_txt_ua AS (
 SELECT
   date,
   page,
   rank,
   agent,
   GetByAgent(TO_JSON_STRING(custom_metrics.robots_txt), agent) AS agent_obj
 FROM `httparchive.crawl.pages`,
 UNNEST(
   REGEXP_EXTRACT_ALL(
     TO_JSON_STRING(JSON_QUERY(custom_metrics.robots_txt, '$.record_counts.by_useragent')),
     r'"([^"]+)":\{'
   )
 ) AS agent
 WHERE
   date >= "2023-01-01"
   AND client = "mobile"
   AND is_root_page = TRUE
)

SELECT
 date,
 agent,
 COUNT(DISTINCT page) AS sites
FROM robots_txt_rule_counts_by_ua
GROUP BY 1,2
HAVING COUNT(DISTINCT page) > 100
ORDER BY 1,3 DESC

AI Bot Directives

This query uses the [AI Robots.txt Github repository](https://github.com/ai-robots-txt/ai.robots.txt){:target="_blank"} to classify User-Agents by AI service. Is then counts the number of sites that include different directives and combinations of directives in their robots.txt files


CREATE TEMP FUNCTION GetByAgent(json STRING, agent STRING)
RETURNS STRING
LANGUAGE js AS r"""
 try {
   const obj = JSON.parse(json || '{}');
   const byua = (((obj || {}).record_counts || {}).by_useragent) || {};
   const body = byua[agent] || byua[String(agent).toLowerCase()] || byua[String(agent).toUpperCase()];
   return body ? JSON.stringify(body) : null;
 } catch (e) { return null; }
""";


WITH bots AS (
 SELECT 'AddSearchBot' AS name,'Unclear at this time.' AS operator,'Unclear at this time.' AS respect_robotstxt,'AI Search Crawlers' AS `function`
 UNION ALL SELECT 'AI2Bot','Ai2','Yes','Content is used to train open language models.'
 UNION ALL SELECT 'Ai2Bot-Dolma','Ai2','Yes','Content is used to train open language models.'
 UNION ALL SELECT 'aiHitBot','aiHit','Yes','A massive, artificial intelligence/machine learning, automated system.'
 UNION ALL SELECT 'Amazonbot','Amazon','Yes','Service improvement and enabling answers for Alexa users.'
 UNION ALL SELECT 'Andibot','Andi','Unclear at this time','Search engine using generative AI, AI Search Assistant'
 UNION ALL SELECT 'anthropic-ai','Anthropic','Unclear at this time.','Scrapes data to train Anthropics AI products.'
 UNION ALL SELECT 'Applebot','Apple','Unclear at this time.','AI Search Crawlers'
 UNION ALL SELECT 'Applebot-Extended','Apple','Yes','Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.'
 UNION ALL SELECT 'Awario','Awario','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'bedrockbot','Amazon','Yes','Data scraping for custom AI applications.'
 UNION ALL SELECT 'bigsur.ai','Big Sur AI','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'Brightbot 1.0','Browsing.ai','Unclear at this time.','LLM/AI training.'
 UNION ALL SELECT 'Bytespider','ByteDance','No','LLM training.'
 UNION ALL SELECT 'CCBot','Common Crawl Foundation','Yes','Provides open crawl dataset, used for many purposes, including Machine Learning/AI.'
 UNION ALL SELECT 'ChatGPT Agent','OpenAI','Yes','AI Agents'
 UNION ALL SELECT 'ChatGPT-User','OpenAI','Yes','Takes action based on user prompts.'
 UNION ALL SELECT 'Claude-SearchBot','Anthropic','Yes','Claude-SearchBot navigates the web to improve search result quality.'
 UNION ALL SELECT 'Claude-User','Anthropic','Yes','Claude-User supports Claude AI users by fetching pages for questions.'
 UNION ALL SELECT 'Claude-Web','Anthropic','Unclear at this time.','Undocumented AI Agents'
 UNION ALL SELECT 'ClaudeBot','Anthropic','Yes','Scrapes data to train Anthropics AI products.'
 UNION ALL SELECT 'CloudVertexBot','Unclear at this time.','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'cohere-ai','Cohere','Unclear at this time.','Retrieves data to provide responses to user-initiated prompts.'
 UNION ALL SELECT 'cohere-training-data-crawler','Cohere','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'Cotoyogi','ROIS','Yes','AI LLM Scraper.'
 UNION ALL SELECT 'Crawlspace','Crawlspace','Yes','Scrapes data'
 UNION ALL SELECT 'Datenbank Crawler','Datenbank','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'Devin','Devin AI','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'Diffbot','Diffbot','At the discretion of Diffbot users.','Aggregates structured web data for monitoring and AI model training.'
 UNION ALL SELECT 'DuckAssistBot','Unclear at this time.','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'Echobot Bot','Echobox','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'EchoboxBot','Echobox','Unclear at this time.','Data collection to support AI-powered products.'
 UNION ALL SELECT 'FacebookBot','Meta/Facebook','Yes','Training language models'
 UNION ALL SELECT 'facebookexternalhit','Meta/Facebook','No','Ostensibly only for sharing, but likely used as an AI crawler as well'
 UNION ALL SELECT 'Factset_spyderbot','Factset','Unclear at this time.','AI model training.'
 UNION ALL SELECT 'FirecrawlAgent','Firecrawl','Yes','AI scraper and LLM training'
 UNION ALL SELECT 'FriendlyCrawler','Unknown','Yes','We are using the data from the crawler to build datasets for machine learning experiments.'
 UNION ALL SELECT 'Gemini-Deep-Research','Google','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'Google-CloudVertexBot','Google','Yes','Build and manage AI models for businesses employing Vertex AI'
 UNION ALL SELECT 'Google-Extended','Google','Yes','LLM training.'
 UNION ALL SELECT 'GoogleAgent-Mariner','Google','Unclear at this time.','AI Agents'
 UNION ALL SELECT 'GoogleOther','Google','Yes','Scrapes data.'
 UNION ALL SELECT 'GoogleOther-Image','Google','Yes','Scrapes data.'
 UNION ALL SELECT 'GoogleOther-Video','Google','Yes','Scrapes data.'
 UNION ALL SELECT 'GPTBot','OpenAI','Yes','Scrapes data to train OpenAIs products.'
 UNION ALL SELECT 'iaskspider/2.0','iAsk','No','Crawls sites to provide answers to user queries.'
 UNION ALL SELECT 'ICC-Crawler','NICT','Yes','Scrapes data to train and support AI technologies.'
 UNION ALL SELECT 'ImagesiftBot','ImageSift','Yes','Scrapes the internet for publicly available images.'
 UNION ALL SELECT 'img2dataset','img2dataset','Unclear at this time.','Scrapes images for use in LLMs.'
 UNION ALL SELECT 'ISSCyberRiskCrawler','ISS-Corporate','No','Scrapes data to train machine learning models.'
 UNION ALL SELECT 'Kangaroo Bot','Unclear at this time.','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'LinerBot','Unclear at this time.','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'meta-externalagent','Meta/Facebook','Yes','Used to train models and improve products.'
 UNION ALL SELECT 'meta-externalfetcher','Meta/Facebook','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'MistralAI-User','Mistral','Unclear at this time.','AI Assistants'
 UNION ALL SELECT 'MistralAI-User/1.0','Mistral AI','Yes','Takes action based on user prompts.'
 UNION ALL SELECT 'MyCentralAIScraperBot','Unclear at this time.','Unclear at this time.','AI data scraper'
 UNION ALL SELECT 'netEstate Imprint Crawler','netEstate','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'NovaAct','Unclear at this time.','Unclear at this time.','AI Agents'
 UNION ALL SELECT 'OAI-SearchBot','OpenAI','Yes','Search result generation.'
 UNION ALL SELECT 'omgili','Webz.io','Yes','Data is sold.'
 UNION ALL SELECT 'omgilibot','Webz.io','Yes','Data is sold.'
 UNION ALL SELECT 'OpenAI','OpenAI','Yes','Unclear at this time.'
 UNION ALL SELECT 'Operator','Unclear at this time.','Unclear at this time.','AI Agents'
 UNION ALL SELECT 'PanguBot','Huawei','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'Panscient','Panscient','Yes','Data collection and analysis using machine learning and AI.'
 UNION ALL SELECT 'panscient.com','Panscient','Yes','Data collection and analysis using machine learning and AI.'
 UNION ALL SELECT 'Perplexity-User','Perplexity','No','Used to answer queries at the request of users.'
 UNION ALL SELECT 'PerplexityBot','Perplexity','Yes','Search result generation.'
 UNION ALL SELECT 'PetalBot','Huawei','Yes','Used to provide recommendations in Hauwei assistant and AI search services.'
 UNION ALL SELECT 'PhindBot','phind','Unclear at this time.','AI-enhanced search engine.'
 UNION ALL SELECT 'Poseidon Research Crawler','Poseidon Research','Unclear at this time.','AI research crawler'
 UNION ALL SELECT 'QualifiedBot','Qualified','Unclear at this time.','Company offers AI agents and other related products.'
 UNION ALL SELECT 'QuillBot','QuillBot','Unclear at this time.','Company offers AI detection, writing tools and other services.'
 UNION ALL SELECT 'quillbot.com','QuillBot','Unclear at this time.','Company offers AI detection, writing tools and other services.'
 UNION ALL SELECT 'SBIntuitionsBot','SB Intuitions','Yes','Uses data gathered in AI development and information analysis.'
 UNION ALL SELECT 'Scrapy','Zyte','Unclear at this time.','Scrapes data for a variety of uses including training AI.'
 UNION ALL SELECT 'SemrushBot-OCOB','Semrush','Yes','Crawls your site for ContentShake AI tool.'
 UNION ALL SELECT 'SemrushBot-SWA','Semrush','Yes','Checks URLs on your site for SEO Writing Assistant.'
 UNION ALL SELECT 'Sidetrade indexer bot','Sidetrade','Unclear at this time.','Extracts data for a variety of uses including training AI.'
 UNION ALL SELECT 'Thinkbot','Thinkbot','No','Insights on AI integration and automation.'
 UNION ALL SELECT 'TikTokSpider','ByteDance','Unclear at this time.','LLM training.'
 UNION ALL SELECT 'Timpibot','Timpi','Unclear at this time.','Scrapes data for use in training LLMs.'
 UNION ALL SELECT 'VelenPublicWebCrawler','Velen Crawler','Yes','Scrapes data for business data sets and machine learning models.'
 UNION ALL SELECT 'WARDBot','WEBSPARK','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'Webzio-Extended','Unclear at this time.','Unclear at this time.','AI Data Scrapers'
 UNION ALL SELECT 'wpbot','QuantumCloud','Unclear at this time.','Live chat support and lead generation.'
 UNION ALL SELECT 'YaK','Meltwater','Unclear at this time.','AI-enabled consumer intelligence'
 UNION ALL SELECT 'YandexAdditional','Yandex','Yes','Scrapes/analyzes data for the YandexGPT LLM.'
 UNION ALL SELECT 'YandexAdditionalBot','Yandex','Yes','Scrapes/analyzes data for the YandexGPT LLM.'
 UNION ALL SELECT 'YouBot','You','Yes','Scrapes data for search engine and LLMs.'
),
robots_txt_ua AS (
 SELECT
   page,
   agent,
   GetByAgent(TO_JSON_STRING(custom_metrics.robots_txt), agent) AS agent_obj
 FROM `httparchive.crawl.pages`,
 UNNEST(
   REGEXP_EXTRACT_ALL(
     TO_JSON_STRING(JSON_QUERY(custom_metrics.robots_txt, '$.record_counts.by_useragent')),
     r'"([^"]+)":\{'
   )
 ) AS agent
 WHERE date = "2025-07-01"
   AND client = "mobile"
   AND is_root_page = TRUE
),
robots_txt_rule_counts_by_ua AS (
 SELECT
   page,
   LOWER(agent) AS agent,
   agent_obj,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.allow') AS INT64)        AS allow_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.crawl_delay') AS INT64)  AS crawl_delay_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.disallow') AS INT64)     AS disallow_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.noindex') AS INT64)      AS noindex_cnt,
   SAFE_CAST(JSON_VALUE(agent_obj, '$.other') AS INT64)        AS other_cnt
 FROM robots_txt_ua
)




SELECT
 agent,
 operator,
 respect_robotstxt,
 SUM(freq) AS sites,
 SUM(IF(directives="both allow and disallow", freq, NULL)) AS both,
 SUM(IF(directives="allow", freq, NULL)) AS allow,
 SUM(IF(directives="disallow", freq, NULL)) AS disallow,
 SUM(IF(directives="crawl_delay", freq, NULL)) AS crawl_delay,
 SUM(IF(directives="noindex", freq, NULL)) AS noindex
FROM (
 SELECT
   agent,
   operator,
   respect_robotstxt,
   CASE
     WHEN allow_cnt > 0 AND disallow_cnt > 0 THEN "both allow and disallow"
     WHEN allow_cnt > 0 THEN "allow"
     WHEN disallow_cnt > 0 THEN "disallow"
     WHEN crawl_delay_cnt > 0 THEN "crawl_delay"
     WHEN noindex_cnt > 0 THEN "noindex"
     ELSE "other"
   END as directives,
   COUNT(DISTINCT page) AS freq,
 FROM robots_txt_rule_counts_by_ua r
 JOIN bots b
 ON LOWER(r.agent) = LOWER(b.name)
 GROUP BY 1,2,3,4
 ORDER BY 5 DESC
)
GROUP BY 1,2,3
ORDER BY 4 DESC

Discovering Third Party Performance Risks

2024-09-03T04:00:00+00:00

It likely comes as no surprise that third party content can be a significant contributor to slow loading websites and poor user experience. As performance engineers, we often need to find ways to balance requirements for their features with the strain that they can put on user experience. Unfortunately, for many sites this becomes a reaction to slowdowns and failures detected in production.

I thought it might be interesting to attempt to identify third parties that could pose a performance risk, so that they could be proactively analyzed. That led to building a tool called Third Party Explorer, which leverages WebpageTest data to help analyze a third party’s impact on a page load. The idea behind this tool is that some of the insights already collected during a WebPageTest measurement may enable you to prioritize a list of domains to evaluate proactively.

Why Are Third Parties Slow?

According to the Web Almanac, 94% of websites utilize at least one third party domain, and the median popular site uses 43 third parties. Not all third parties impact performance in the same way, but all it takes is one poorly configured third party. Often when someone discovers third party issues, it’s due to availability, loading performance, and interactivity delays caused by them.

When I think about how or whether a third party’s content may impact performance, I usually consider the following:

When is the third party loaded?
- Is it render blocking?
- Does it load before important rendering metrics, such as First Contentful Paint or Largest Contentful Paint?
- Are there gaps in loading first party content that correlate to the third party?
- Are there gaps in loading other third party content that correlate to the third party?
How is it delivered?
- How much content is served to the client?
- What type of content is served to the client?
- Is it using a Content Delivery Network to deliver resources?
- Is it allowing the browser to cache static resources?
- Are its resources compressed adequately?
What is it doing?
- How much CPU time is used?
- Does it result in excessive long tasks?
- Does it result in excessive requests?

If you go down that list and answer these questions for a particular domain, you are likely to start forming an opinion of whether that third party could be a performance risk. It’s important to note that your analysis does not stop here, but rather it’s just beginning. If you suspect that a particular third party is a performance risk, testing, validating and ongoing monitoring should come after discovery.

Analyzing Third Party Performance in WebPageTest

When you run a measurement via WebPageTest, the existing reports can help you quickly identify some important third parties to analyze. For example, the graph below is a “Connection View”, which I find helpful when looking for gaps that correlate to the loading of a particular third party. Dark shaded bars also indicate data transfer, so it may be easy to spot third parties that have an excessive amount of darker shades in this view. Looking at this example, there are no gaps in the loading of first party content, most of the third party content is loading after LCP, and there are a few third parties that appear to be loading a lot of content.

If this type of analysis is unfamiliar, then I highly recommend checking out Matt Hobb’s comprehensive guide to WebPageTest!

There are other interesting reports in WebPageTest that can be useful for analyzing third party content. For example, the “Domains Breakdown” report shows a summary of domains loaded on the site, as well as their request and byte count. The “Performance Optimization Overview” report shows a report card for each request, indicating whether a few performance best practices are met. And the “Opportunities & Experiments” section provides a comprehensive summary of some performance issues that were identified, with the option of running experiments using WebPageTest Pro.

Third Party Explorer

While WebPageTest provides a great deal of insight into third parties, sometimes it can be challenging to get a sense of which third parties to focus on. I wanted to build on top of the Domain Breakdown feature by creating a dashboard view that enables you to dive deeper. Fortunately all of this data exists in WebPageTest measurement results, so no further instrumentation was needed.

After navigating to the Third Party Explorer tool, enter a WebPageTest test result URL and click submit. This will download and parse the results from your WebPageTest measurement.

Once the test results are parsed, you can see a summary of the domains, including potential performance and SPOF risks. Note that I emphasize “potential” since you’ll really need to test your site to determine whether they are truly an issue.

Scroll down a little further, and you’ll see a list of domains. For each domain you can see the number of KB or requests loaded, and filter the results by content type. Each column is sortable. In the table you can see a lot of information (you may need to scroll to the right), such as:

Whether the domain is using a CDN
Number of requests and total bytes transferred
Render blocking content
A summary of third party requests or bytes between specific time ranges, such as before FCP, between FCP and LCP, etc.
A summary of requests that have not been compressed
Requests grouped by cache TTLs
Requests grouped by CPU overhead

The two checkboxes in this table are for you to use. They are prepopulated with some suggestions based on the results, and alllow you to make a note of which domains to follow up on during your analysis. So you can check and uncheck domains during your analysis, and then review which ones to followup on later.

For example, sorting through these results, I can see that:

Google Tag Manager and TikTok both have high CPU execution times, and load a lot of scripts between LCP and Page Load.
The domain cdn.pdst.fm is loading 22KB of JavaScript, which is not compressed.
The domain js.adsrvr.org is render blocking, but loads towards the end of the HTML body.

While it seems like we’re in pretty good shape, there’s still a few things that we could improve on here. When I started building this tool there were a few more third parties in this list - some with significant performance issues such as inadequate compression levels, large JavaScript payloads that were not cached on a CDN, and cacheable content delivered via S3 instead of a CDN. Fortunately the third parties acted on feedback I shared with them and addressed the issues. I’ve started using this tool to proactively assess the performance impact of third parties prior to integration.

What are Perf Risks and SPOF Risks?

In the domain summary, you’ll find two checkboxes next to each domain, labeled “Perf Risk” and SPOF Risk”. If you find this checked for a domain, it doesn’t necessary mean that it’s a problem - but that it’s worth reviewing the domain to determine whether it is. I used a simple rubric based on my own experiences/opinions for determining which domains to label as a performance risk or a single point of failure risk:

You’ll find SPOF risk checked if the domain has at least one render blocking request.

You’ll find Perf risk checked if:

Overall
- A domain has at least one render blocking request
- If a domain has more than 10KB of text based content that is not compressed with gzip, brotli or zstd
- If a domain is delivering at more than 20KB requests (excluding XHRs) without a CDN
Before LCP
- A domain loads more than 30KB of content
- A domain loads any requests (excluding XHRs) that are not cacheable, have no cache policy or have a TTL of 0s
- A domain’s scripts utilizes over 30ms or more CPU time (compile + evaluate + execute)
Between LCP and Page Load
- A domain loads more than 50KB of content
- A domain loads requests (excluding XHRs) that are not cacheable, have no cache policy or have a TTL of 0s
- A domain’s scripts utilizes over 50ms or more CPU time (compile + evaluate + execute)
After Page Load
- A domain loads more than 100KB of content
- A domain’s scripts utilizes over 100ms or more CPU time (compile + evaluate + execute)

Feel free to use the checkboxes to select or deselect domains that you feel are not applicable to your site. And if you have ideas on a better rubric for a third party peformance risks, I’m open to suggestions!

Bringing it Back to WebPageTest for a Deeper Analysis

Now that we’ve reviewed how to use the tool, go run a WebPageTest measurement and see what you can find. Once you have some third parties to investigate, there’s a few things you can try:

At the bottom of the Third Party Explorer tool you will find sections for “SPOF Evaluation” and “Performance Evaluation”. If you use the checkboxes in the tool to mark up which third parties you are concerned about, then you can use these sections to launch a SPOF test via WebPageTest or to populate a list of domains to block. This will enable you to run WebPageTest measurements to see what your user experience would be like if the third party failed or if it was removed.

If you subscribe to WebPageTest Pro, check out the list of “Opportunities & Experiments” and try to see what optimizing a particular third party might do. From there you can experiment with blocking third parties, running them as first party content, etc.

Conclusion

Third party performance is no doubt a significant cause of poor user experience on the web. While it can be challenging to identify poorly performing third parties proactively, attempting to do so during the evaluation stage of implementing one can prove to be mutually beneficial for both you and the third party.

There’s no shortage of great tools out there that will help you identify when a third party is slowing down your site. The goal of this article was to help you proactively assess whether a third party meets a criteria that merits some further review and analysis.

Third Parties and Certificate Revocations

2024-08-04T04:00:00+00:00

On Monday July 29th, DigiCert announced the need to revoke a large number of certificates due to a bug in domain validation. The CA/B Forum’s strict requirements to revoke these certificates within 24 hours resulted in a pretty busy Monday and Tuesday for a lot of folks. For some others, the deadline was moved to August 3rd due to exceptional circumstances. What remained a mystery was how many sites and third parties would be affected, how many would be prepared in time and what the impact of a mass revocation might look like across the web. In this blog post we’ll use the HTTP Archive to explore the impact.

Which hostnames were affected?

In a bugzilla post, DigiCert shared a list of 86k serial numbers for certificates that were impacted and needed to be revoked. The list did not include hostnames, so it wasn’t easy to see which domains would be affected from the files alone. The HTTP Archive contains details for every certificate it encounters, so I was able write some queries to correlate the list of serial numbers with certificate used across 16 million websites. If you are interested in the queries used to do this analysis, you’ll find them at the end of this post.

In my analysis, I found 13,823 of the certificate serial numbers on publicly available web pages during last month’s HTTP Archive crawl. Many of these were first party resources, but a few hundred hostnames belonged to popular third parties. Overall I found that 1,241,943 websites would have been impacted by this revocation in some way, meaning they either made a first or third party request for a resource that used at least one of the affected certificates! Here’s a list of some of the more popular domains that were affected. The list contains an apex domain, the number of sites requesting resources from it, and the number of impacted subdomains (containing certificates needing to be revoked).

Third Party with Certificates Affected by DigiCert Recovations
Domain	Number of Sites	Number of Hostnames
yahoo.com	467,827	49
rubiconproject.com	387,241	20
fontawesome.com	299,959	10
pinterest.com	281,145	57
taboola.com	133,309	43
pinimg.com	91,573	14
ib-ibi.com	60,557	1
snapchat.com	49,390	15
advertising.com	41,815	1
datadoghq-browser-agent.com	35,404	1
sift.com	13,962	1
scdn.co	10,949	1
usonar.jp	7,402	4
sojern.com	7,371	4
olark.com	6,966	1

Looking the most popular third party domains impacted by this revocation, you can see that many of them reissued their certificates on July 30th based on the validity dates. The initial deadline to reissue certificates was July 30th 19:30 UTC.

Third Party Hostnames Affected by DigiCert Revocation
Host	ValidFrom	ValidTo
pixel.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
pr-bh.ybp.yahoo.com	Jul 30 00:00:00 2024 GMT	Jan 22 23:59:59 2025 GMT
ups.analytics.yahoo.com	Jul 30 00:00:00 2024 GMT	Jan 22 23:59:59 2025 GMT
kit.fontawesome.com	Jul 30 00:00:00 2024 GMT	Jan 27 23:59:59 2025 GMT
token.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
eus.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
pixel-us-east.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
secure-assets.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
ct.pinterest.com	Aug 2 00:00:00 2024 GMT	Aug 7 23:59:59 2025 GMT
cms.analytics.yahoo.com	Jul 30 00:00:00 2024 GMT	Jan 22 23:59:59 2025 GMT
fastlane.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
ka-p.fontawesome.com	Jul 30 00:00:00 2024 GMT	Jan 27 23:59:59 2025 GMT
log.pinterest.com	Jul 30 00:00:00 2024 GMT	Aug 7 23:59:59 2024 GMT
s.pinimg.com	Jul 30 00:00:00 2024 GMT	Aug 7 23:59:59 2024 GMT
assets.pinterest.com	Jul 30 00:00:00 2024 GMT	Aug 7 23:59:59 2024 GMT
sync.taboola.com	Jul 30 00:00:00 2024 GMT	Dec 31 23:59:59 2024 GMT
prebid-server.rubiconproject.com	Jul 30 00:00:00 2024 GMT	Apr 3 23:59:59 2025 GMT
global.ib-ibi.com	Jul 30 00:00:00 2024 GMT	Apr 2 23:59:59 2025 GMT
cdn.taboola.com	Jul 30 00:00:00 2024 GMT	Dec 31 23:59:59 2024 GMT

After the final deadline of August 3rd, 2024 19:30 UTC had passed, I ran an test against a list of the 13,823 hostames I discovered. I found that 78.51% of them had reissued their certificates prior to the initial deadline or switched to a using certificates not subject to revocation. Another 5.3% of the hostnames reissued their certificates during the extension. However 9.3% of hostnames - 1,291 - had failed to get their certificates reissued and were revoked on Aug 3rd. Since then, 156 hostnames reissued - but there are still 1,135 (8.21%) hostnames delivering a revoked certificate!

Validity Start Dates for Affected Certificates
Certificate Validity Date	Certificates	Percent of Certificates
Jul 29 2024	121	0.88%
Jul 30 2024	9,680	70.03%
Jul 31 2024	977	7.07%
Aug 1 2024	426	3.08%
Aug 2 2024	267	1.93%
Aug 3 2024	75	0.54%
Aug 4 2024	90	0.65%
Using Different Cert	1,052	7.61%
Not Updated	1,135	8.21%

When the certificates were revoked, there were only 2 major third parties that were affected. One was a security service used by approximately 14k sites. Another was a live chat system that was used by a ~200 sites. Fortunately the failure of those third parties did not impact functionality of the sites, and they have since reissued their certificates.

Monitoring for third party failures

Often we don’t have the liberty of advance notice of impending failures - such as the recent Crowdstrike outages, CDN failures, and other major internet platform incidents. In this case, we had at least 24 hours notice that a massive certificate revocation event would occur (and then a few additional days after the extension).

Using the HTTP Archive data I could see that none of the third parties used by my employer were impacted. However to be absolutely certain I configured a Catchpoint dashboard to monitor for third party availability issues. This dashboard displays the % availability for each third party host, the number of failures for each third party, and some load time metrics. The idea was that if a particular third party we use experienced an issue, we’d be able to identify it quickly.

The dashboard was created by using a line chart broken down by host. Enabling “host data” allowed me to chart some host metrics such as availability and number of failures, as well as exclude first party content. You can see some more details on how to do this in this blog post from Catchpoint.

You may ask why not use real user monitoring (RUM) data for this? RUM can give you timing information on third party requests and additional metrics if the third party sets the Timing-Allow-Origin header. It’s great for detecting performance issues related to their party content. However, detecting failures in loading third party resources is not as easy since a failure simply won’t show up in resource timing data.

Preparing for Third Party Failures

When a popular third party fails or degrades, sometimes you’ll read about it in the news, especially if it breaks functionality on a large number of websites. Far too often organizations handle third party performance/failure risks reactively. There’s a few things you can do to prepare though.

Identify third party single poin of failures (SPOFs).
- WebPageTest’s SPOF feature is great for this!
Identify third party performance risks.
- Test to see what happens when you block or remove their resources. (WebPageTest or Chrome)
Identify third parties that might impact functionality on your site.
- Test to see what happens when you block or remove their resources.
Monitor third party performance and availability.
- A combination of RUM for performance and Synthetic measurements for availability can be helpful here.

I’ve also been working on a tool that will help identify potential third parties that are worth investigating for performance or single point of failure risks. Hoping to share that with you all very soon!

Conclusion

While 86k certificates may not sound like a huge amount compared to the scale of the web, the way those certificates were used across some very popular third parties could have impacted over a million websites. There’s been a lot of negativity about DigiCert regarding this, but I have a lot of empathy for what they’ve been dealing with this past week. It was no doubt frustrating for folks to frantically update certificates. This could have been incredibly disruptive to a large part of the web due to third party failures, but it wasn’t.

HTTP Archive queries

This section provides some details on how this analysis was performed, including SQL queries and commands for testing the certificates. Please be warned that some of the SQL queries process a signicant amount of bytes - which can be very expensive to run (particularly the first one).

Extract certificate serial numbers from HTTP Archive

The list of affected certificates that DigiCert provided included the serial numbers, but not a hostname. In order to identify the hostnames, this query was written to collect serial numbers from all DigiCert certificates found in the HTTP Archive requests table. The approach involved base64 decoding the certificate, converting it to bytes and extracting the substring where the serial number exists. This is a bit of a hack, but it worked!

Warning: this query processes 13 TB of data, which is much higher than the 1 TB free tier. The results for it have been saved in another BigQuery table for analysis: `httparchive.scratchspace.2024_07_01_cert_serials`.



CREATE TEMPORARY FUNCTION extractCertHex(cert_block STRING)
RETURNS STRING
LANGUAGE js AS '''
   // Extract the Base64 content between the certificate tags
   const base64Match = cert_block.match(/-----BEGIN CERTIFICATE-----\\s*([A-Za-z0-9+/=\\s]+)\\s*-----END CERTIFICATE-----/);
   if (!base64Match) {
       return 'Invalid Certificate Block';
   }
   const base64Content = base64Match[1].replace(/\\s+/g, '');

   // Base64 decode
   function base64ToBytes(base64) {
       const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
       let bytes = [];
       let buffer = 0, bits = 0;

       for (let i = 0; i < base64.length; i++) {
           if (base64[i] === '=') break;
           const val = chars.indexOf(base64[i]);
           buffer = (buffer << 6) | val;
           bits += 6;
           if (bits >= 8) {
               bits -= 8;
               bytes.push((buffer >> bits) & 0xFF);
           }
       }

       return bytes;
   }

   // Decode the Base64 content
   const cert_der = base64ToBytes(base64Content);

   // Convert DER to hexadecimal
   let cert_hex = '';
   for (let i = 0; i < cert_der.length; i++) {
       const hex = cert_der[i].toString(16).padStart(2, '0');
       cert_hex += hex;
   }

   return cert_hex;
''';

SELECT
 DISTINCT NET.HOST(url) AS host,
 SUBSTR(extractCertHex(JSON_EXTRACT_SCALAR(payload, "$._certificates[0]")),31,32) AS serial_num
FROM 
   `httparchive.requests.2024_07_01_mobile`
WHERE
 JSON_EXTRACT_SCALAR(payload, "$._securityDetails.issuer") LIKE "%DigiCert%"
 AND JSON_EXTRACT_SCALAR(payload, "$._certificates[0]") IS NOT NULL

Identify hostnames subject to revocation

In order to identify the hostnames subject to revocation, I uploaded a copy of the revocation serial numbers to a table: `httparchive.scratchspace.digicert_revocation_20240730`. Then I performed a simple `INNER JOIN` on the output from the previous query to identify hostnames that had a serial number in the revocation list.


SELECT DISTINCT 
   host, 
   d.serial AS serial
FROM 
   `httparchive.scratchspace.2024_07_01_cert_serials` ha
INNER JOIN 
   `httparchive.scratchspace.digicert_revocation_20240730` as d
ON ha.serial_num = d.serial
WHERE 
   d.serial IS NOT NULL

Summarize popular third party domains that had hostnames impacted by the revocation

This query summarized domain names from the requests table by the number of sites loading a resource from it, and the number of hostnames that appeared in DigiCert's list of revoked hostnames. The previous query is used in the `IN()` clause of this query.


SELECT 
   NET.REG_DOMAIN(url) domain, 
   COUNT(DISTINCT page) sites, 
   COUNT(DISTINCT NET.HOST(url)) hostnames
FROM 
   `httparchive.all.requests` AS r
WHERE 
  date = "2024-07-01"
  AND client = "mobile"
  AND is_root_page = true
  AND NET.HOST(url) IN (
    SELECT DISTINCT host
    FROM `httparchive.scratchspace.2024_07_01_cert_serials` ha
    INNER JOIN `httparchive.scratchspace.digicert_revocation_20240730` as d 
    ON ha.serial_num = d.serial
    WHERE d.serial IS NOT NULL    
  )
GROUP BY 1
ORDER BY 2 DESC

Summarize popular third party hostnames that had hostnames impacted by the revocation

This query is similar to the previous one, but summarizes by hostname instead of domain name.


SELECT 
   NET.HOST(url) hostname, 
   COUNT(DISTINCT page) sites
FROM 
   `httparchive.all.requests` AS r
WHERE 
  date = "2024-07-01"
  AND client = "mobile"
  AND is_root_page = true
  AND NET.HOST(url) IN (
    SELECT DISTINCT host
    FROM `httparchive.scratchspace.2024_07_01_cert_serials` ha
    INNER JOIN `httparchive.scratchspace.digicert_revocation_20240730` as d 
    ON ha.serial_num = d.serial
    WHERE d.serial IS NOT NULL    
  )
GROUP BY 1
ORDER BY 2 DESC

Bash Script to test certificates for affected hosts

This script will loop through a list of hostnames and extract the validity dates and serial numbers for each certificate, timing out after 3 seconds if the host is unresponsive.


for i in $(cat all_hosts.txt); do 
  timeout 3 echo | openssl s_client -connect "$i:443" 2>/dev/null | 
  openssl x509 -noout -startdate -enddate -serial 2>/dev/null | 
  awk -F= -v host="$i" '
    /^notBefore/ { start = $2 } 
    /^notAfter/  { end = $2 } 
    /^serial/    { serial = $2 } 
    END { 
      print host "," start "," end "," serial 
    }'
done > all_hosts_checked_20240803_1930UTC.csv

Choosing Between gzip, Brotli and zStandard Compression

2024-03-19T04:00:00+00:00

HTTP compression is a mechanism that allows a web server to deliver text based content using less bytes, and it’s been supported on the web for a very long time. In fact the first web browser to support gzip compression was NCSA Mosaic v2.1 way back in 1993! The web has obviously come a long way since then, but today pretty much every web server and browser still supports gzip compression.

In recent years, new and innovative compression methods have gained browser support. One in particular that has achieved widespread adoption is Brotli. First supported in Chrome 50 (2016), it was supported by all modern browsers a year later. Brotli is able to compress files much smaller than gzip, albeit with a higher computational overhead. Based on HTTP Archive data from January 2024, Brotli is actually used more than gzip for JavaScript and CSS!

Facebook’s zStandard compression is another promising new method which aims to serve smaller payloads compared to gzip while also being faster. zStandard was added to the IANA’s list of HTTP content encodings with an identifier of “zstd”, and support for it was added to Chrome in version 123, which was released this month. Facebook recently shared some benchmarks that show it performing signifcantly faster than gzip.

Beyond this, there’s also shared dictionaries for Brotli and zStandard, which have the potential to significantly reduce byte sizes.

While it may take some time for browsers, web servers and CDNs to catch up, it’s worth pondering which compression method is right for your content. A few years ago I wrote a blog post about Brotli compression as well as a tool to help you determine how Brotli could compress your content relative to gzip. I’ve updated this tool to include zStandard compression as well as show the relative latency incurred at each compression level. You can find the new/updated tool at https://tools.paulcalvano.com/compression-tester/

How HTTP Compression Works

When a client makes an HTTP request, it includes an Accept-Encoding header to advertise the compression encodings it can support. The web server then selects one of the advertised encodings that it also supports and serves a compressed response with a Content-Encoding header to indicate which compression was used.

In the example below, the client advertised support for gzip, Brotli, and Deflate compression. The server returned a gzip compressed response containing a text/html document.

    GET / HTTP/2
    Host: httparchive.org
    Accept-Encoding: gzip, deflate, br

    HTTP/2 200
    content-type: text/html; charset=utf-8
    content-encoding: gzip

If a client sends multiple encodings in its Accept-Encoding header, then the server will have to choose one. For example, if I send an HTTP request to Facebook’s homepage and advertise support for gzip, Brotli and zStandard - their server chooses to deliver the response via zStandard.

    GET / HTTP/2
    Host: www.facebook.com
    accept-encoding: gzip, deflate, br, zstd
    
    HTTP/2 200 
    content-encoding: zstd
    content-type: text/html; charset="utf-8"

Gzip Compression

Gzip is fundamentally supported by all web servers, browsers and intermediaries (CDNs, proxies, etc), mostly by default. Despite how easy it is to serve content gzip compressed, there are a few things to keep in mind:

Most web servers and CDNs default to gzip compression level 6, which is a reasonable default. Some servers (ie, NGINX) default to gzip level 1, which usually results in faster compression times but results in a larger file. Make sure to check your compression levels!
Many CDNs can gzip compress resources for you, which is helpful if you missed something on your origin server. Some CDNs enable this by default.
Some CDNs may decompress and recompress content for you if they need to inspect or manipulate the contents. This may have an impact on your time to first byte (TTFB) especially if the content that needs to be recompressed is large.

Brotli Compression

Brotli compression was created by Google and is supported across all major web browsers. At its highest compression level files can often be reduced 15-25% more than gzip. Higher compression levels come at a significant latency cost though.

Many popular web servers support Brotli via modules. For example, Apache has a mod_brotli module which defaults to level 5. NGINX has a module called ngx_brotli which defaults to level 6. Brotli is supported by a variety of CDNs, albeit in slightly different ways. It’s best to understand how your specific CDN handles this compression algorithm. For example:

Most CDNs have the ability to serve Brotli compressed payloads from an origin server that supports Brotli.
- This is done by varying the content by the Content-Encoding response header.
- Some CDNs do this by default, others require additional steps.
Some CDNs can Brotli compress responses even if an origin server does not support Brotli.
- Usually they fetch the result via gzip from your origin.
- Some CDNs can perform on the fly Brotli compression for static and dynamic, usually at specific compression levels they define.
- Some CDNs can pre-compress static content at higher compression levels

While the highest compression levels might be ideal for static content, you’ll want to be careful with dynamic content to avoid impacting your TTFB. Additionally if your CDN offers dynamic Brotli compression, then you may want to determine if the byte savings are worth the latency overhead from decompressing and re-compressing the response at the edge. In some cases it may be best to Brotli compress dynamic content from the origin, or stick with Gzip if your origin doesn’t support Brotli.

zStandard

zStandard is a newer compression method developed by Facebook. It was designed to compress at ratios similar to gzip compression, but with faster compression and decompression speeds. Historically its usage has been mostly filesystem related, however Chrome is the first browser to support it as of March 2024. While most CDNs do not have support for zStandard yet, many could feasibly vary the cache key for origin compressed responses similar to the way they support Brotli.

When examining the top 10k sites in the HTTP Archive, zStandard compression usage appears to be mostly confined to sites owned by Meta such as www.facebook.com, www.instagram.com, www.messenger.com, www.whatsapp.com, etc) and Netflix. The Meta sites are delivering zStandard encoded content to Chrome browsers. Netflix’s appears to default to gzip compression, possibly implementing zStandard as a test.

Compression Tester

A few years ago I wrote a compression tool that was designed to help you determine whether gzip is sufficient for your content, or if Brotli could provide a reduction in payload sizes. I’ve released a new version of this tool that includes zStandard compression as well as compression times for each individual test.

You can find the new/updated tool at https://tools.paulcalvano.com/compression-tester/

When we’re looking at these results, a few things you’ll want to keep in mind are:

If your content is dynamic / non-cacheable, then it will be very sensitive to any additional latency you add. Higher Brotli and zStandard compression levels can bec computationally expensive, so it’s best to use a more moderate compression level.
If your content is static / cacheable, then you may want to use higher compression levels.
Compression settings are typically set on a server and not per request. If you do not have flexibility with setting this at a more granular level, you should go with the lowest common denominator required for all of your content.

Let’s dive deeper into a few examples:

Example: Facebook

When browsing Facebook, the largest JavaScript resource I saw was 2.37 MB uncompressed. It was delivered to my browser as a 514 KB zStandard compressed response. When testing this file via the Compression Tester tool, it was served a 645 KB gzip and 526 KB Brotli response.

Comparing the compression test results to the delivered responses, we can see that the server likely compressed in the middle range. For example it delivered a 526 KB Brotli payload (presumably level 5), but it could have delivered Brotli level 11 . This would come at a higher computational cost though - which may have been a factor in their selection. zStandard also appears to be delivering a smaller file, but based on this test the computational overhead is more than double what gzip level 6 costs.

For this response, Brotli level 9 would provide the best compression ratio with a CPU overhead similar to zStandard 12. If it’s possible to precompress payloads, then the highest compression levels would reduce the payloads further. However Brotli appears to outperform zStandard in both compression ratio and compression time up until level 9 for this response.

The base HTML page for Facebook, it is 62 KB uncompressed. They support gzip, Brotli and zStandard - and it seems to be compressed at the lowest compression levels. While compression level 1 is often undesired, in this case there doesn’t appear to be much of an advantage of applying higher levels of compression due to limited byte savings. Additionally for Facebook’s HTML all 3 compression algorithms produce a similar size payload - but Brotli and zStandard both compress their HTML faster than gzip.

Sandals.com Homepage

Out of the top 10 thousand websites, Sandals has the largest HTML payload - clocking in at almost 7.9MB (delivered as a 804 KB gzip compressed payload). Additionally they have a 9.4MB script bundle (gzip compressed to 2.5 MB). In the waterfall graph below you can see the impact that these large payloads are having on the experience. Reducing them solves only part of the problem, as that’s still a huge amount of content for the browser to parse, evaluate, and execute. But let’s see how these compression algorithms do.

Sandals appears to be gzip compressing at the highest possible compression level. If we assume that this page is dynamically generated and try to stay within the relative compression times:

Brotli level 9 or zStandard level 15 would result in approximately 75% smaller payload with faster compression times compared to gzip 9.
Brotli level 5 appears to be a good tradeoff between compression ratio and time (209 KB, and 73% faster to compress)
In this particular example, zStandard is producing a larger payload compared to Brotli.

Sandal’s is also using Cloudflare’s CDN which supports Brotli compression, so enabling this could be a quick performance win for them.

Compression Levels vs Compression Times

When considering a compression level, it’s important to balance the time it takes to compress a payload with the estimated byte savings. For example, utilizing the maximum compression level will often produce the smallest payload, but will do so at a higher computational cost. Likewise, the lowest compression levels are often really fast to compress, but might not be as effective.

I tested the base HTML of the top 10 thousand websites’ HTML pages and their largest first party request. The results below show that the majority of gzip compression seems to fall within the estimated range of 4-6 (likely 6 since that is a common default). However ~30% of sites are utilizing gzip level 1, which often provides inadequate compression. The majority of Brotli compression appears to be level 4. However there’s ~25% of sites delivering Brotli level 1. And finally the majority of Brotli level 11 usage seems to be for static content.

The table below details a few sites that are serving their HTML using either gzip or Brotli level 1. Using such a low compression level will likely result in larger payload. In many of these examples, the Brotli compressed payload is actually larger than gzip. Fortunately some of these sites are leveraging services that will automatically deliver gzip because of the byte discrepancy - but they could still benefit from increasing the compression level.

	Content Length Delivered (KB)
url	Uncompressed	gzip	Brotli
https://www.epicurious.com	3877	393	684
https://www.anker.com	3273	628	232
https://www.bonappetit.com	1889	230	348
https://www.allure.com	1817	239	372
https://www.gq.com	1665	226	340
https://www.newyorker.com	1638	252	342
https://www.vanityfair.com	1593	231	339
https://seekingalpha.com	1453	279
https://www.vogue.com	1446	204	312
https://www.cntraveler.com	1422	205	311

The table below shows the results of applying gzip level 6, brotli level 5 and zStandard level 12 against the base HTML on these pages A few observations:

Gzip level 6 reduces most of the gzipped payloads by 25-30% compared to the size delivered via gzip level 1.
Brotli level 5 was able to reduce their sizes by almost 75% compared to gzip level 1. In many cases the compression time overhead is comparable to gzip level 6 - but this varies.
zStandard level 12 was able to provide similar compression levels to brotli level 5 while maintaining compression times similar to gzip level 6.

Based on these examples, real time zStandard compression seems to provide a slight advantage over Brotli - achieving the same sizes with faster compression times.

	>After applying higher compression levels			Compression Time (seconds)
url	gzip (L6)	Brotli (L5)	zStd (L12)	gzip (lL6)	Brotli L5)	zStd (L12)
https://www.epicurious.com	271	145	145	0.083	0.093	0.068
https://www.anker.com	412	219	219	0.099	0.081	0.088
https://www.bonappetit.com	169	113	114	0.044	0.061	0.054
https://www.allure.com	179	135	135	0.051	0.067	0.043
https://www.gq.com	168	118	119	0.049	0.053	0.050
https://www.newyorker.com	191	127	128	0.053	0.071	0.043
https://www.vanityfair.com	173	128	128	0.058	0.098	0.042
https://seekingalpha.com	227	171	174	0.049	0.073	0.079
https://www.vogue.com	152	115	115	0.053	0.075	0.047
https://www.cntraveler.com	155	117	117	0.052	0.052	0.048

Cacheable objects can often be compressed at higher levels - especially if they are able to be precompressed. When evaluating the largest first party objects hosted on the top 10 thousand sites, I found that Brotli level 5 and zStandard level 12 resulted in similar file sizes and compression times - much like the results above. However when evaluating Brotli compression level 11 vs zStandard 19, the smallest files are almost always generated by Brotli level 11. zStandard’s compression times are 4x faster than Brotli 11 though. So if you have the ability to precompress your objects, Brotli 11 may still be preferred. If not, then zStandard may be the better option.

	% File size reduction compared to gzip level 6
	br5	zstd12	br11	zstd19
Average	8.85%	9.07%	19.18%	14.11%
p50	6.99%	7.33%	17.53%	12.40%
p75	10.25%	10.94%	22.21%	17.10%
p90	15.29%	15.74%	27.21%	22.40%

Conclusion

HTTP compression is an incredibly important feature, and has long been overlooked due to the universal support of gzip compression across all web servers and browsers. It’s great to see innovation in this space, and the addition of another compression encoding along with the possibility of shared dictionary compression in the future.

The research I’ve shared in this article also shows that for many sites Brotli will provide better compression for static content. zStandard could potentially provide some benefits for dynamic content due to its faster compression speeds. Additionally:

A surprising amount of sites are using low level gzip compression, and should consider increasing their compression levels.
For dynamic content
- Brotli level 5 usually result in smaller payloads, at similar or slightly slower compression times.
- zStandard level 12 often produces similar payloads to Brotli level 5, with compression times faster than gzip and Brotli.
For static content
- Brotli level 11 produces the smallest payloads
- zStandard is able to apply their highest compression levels much faster than Brotli, but the payloads are still smaller with Brotli.

Of course your mileage will vary and there’s really no universal answer. It’s worth running some tests on your site to see whether your content would benefit, and which compression levels to consider. And then experimenting with RUM data to evaluate whether the approach you decide on is successful. I hope that tool I created helps you get started on this analysis for your site!

Identifying Font Subsetting Opportunities with Web Font Analyzer

2024-02-16T14:00:00+00:00

10 years ago, custom web fonts were a niche feature used by ~10% of websites. Today they are used by over 83% of websites! Fonts are generally loaded as a high priority resource, and some sites use techniques such as preload and early hints to get them to load as quickly as possible. Custom web fonts are important to many sites, since rendering with a specific typography is often preferred from a design perspective. However, this can easily become a performance issue when a large amount of fonts are loaded.

In this article, we’ll explore some potential issues around font loading and the performance benefits of a lesser used feature - font subsetting. We’ll look at HTTP Archive data to understand how prevalent the issue is, and then examine a few case studies. And finally, I created a new tool - Web Font Analyzer - that may help you explore whether font subsetting is something to consider for your site.

Background

There’s been a lot written about web fonts over the years. In fact the HTTP Archive’s Web Almanac has an entire chapter dedicated to this topic. A few years ago Zach Leatherman wrote a fantastic checklist on font loading strategies, which included using preload to load fonts earlier. Way back in 2016 the CSS font-display attribute was introduced, and today it is supported on all modern browsers and used by almost a third of websites! Variable fonts are heavily used by Google Fonts which are widely used across the web. Barry Pollard wrote a great article on self hosting Google Fonts. And just last month Stoyan Stefanov wrote an article about google font sizes.

The font-display feature was a major step forward in font loading performance, as it gave developers control over whether to prioritize rendering or typography. Using font-display:swap would avoid a rendering delay by painting text using a system font and then swapping the actual font after it’s loaded.

Font optimization strategies are great, but when combined the results can be confusing. For example, preloading fonts is a great way to get them to load earlier, but using font-display:swap at the same time may result in a less effective use of bandwidth early in the page load. It’s a good idea to understand exactly how your fonts are loading, how they are being used and what they contain.

Font sizes across popular sites

Using the HTTP Archive we can explore font usage across millions of websites. For the purpose of this analysis, we’ll focus on the top million sites. As of January 2024, 81.5% of the top million sites are utilizing at least one custom web font. Usage varies widely, with the average site loading 238 KB of fonts.

Depending on the font loading strategy used, fonts may be delivered at different parts of the page loading lifecycle. For the purpose of this analysis, I’m going to break this up into 4 parts:

Before FCP - Means that the fonts finished downloading before the First Contentful Paint was measured. This could indicate that a font was render blocking, fetched with a high priority or preloaded.
FCP to LCP - Means that the font was loaded in between the First and Largest Contentful Paint. These fonts were loaded while other resources critical to the user experience were loading
LCP to onLoad - The fonts were loaded after the Largest Contentful Paint but before the onLoad event.
After onLoad - This could indicate that the font was either delayed or not used by the DOM until much later on.

Out of the top million sites that are loading fonts, 63.6% are loading at least 1 custom web font prior to FCP. I would expect this to be high, especially considering how often preloading fonts is recommended.

However, 25% of these sites are loading more than 75 KB of fonts before the FCP, and over 4500 sites are loading more than 500 KB of fonts during this time! Regardless of the benefits of the font loading strategies applied - I’d say that there is likely some waste occurring.

Glyphs vs Characters on Pages

A font is essentially a typographical representation of a character. One can display the same text with different fonts and they will appear differently on a web page - however the unicode value for the character will always be the same. For example, the space character is 0032, the values ABC are 0065, 0066 and 0067 respectively across numerous fonts.

Some font packages are designed to display icons, and others are designed for text. However the one thing that is not always apparent to developers is how many glyphs are included in each font package. For example, a popular Google font called “Material Icons” is used on 114K websites. It contains 2229 glyphs and adds 128 KB to websites using it. It’s very unlikely that sites are making use of all these glyphs.

The most popular Google font is Open Sans, and it’s used by over 1.5 million websites. You can use a tool like FontDrop to explore the contents of your fonts, and you might be surprised! This font contains 280 glyphs and adds 43 KB to pages using it. Loading a few fonts like this can really add up.

So how does that compare with the rendered HTML of a page? Using the HTTP Archive, I was able to write a query that extracts and summarizes the visible glyphs in rendered HTML pages for the top 10K sites. We can then compare this to the minimum and maximum number of glyphs in a page’s custom web fonts. The results might surprise you!.

The median popular site contains 3 fonts, totalling 95 KB. The rendered HTML has 101 glyphs, while the smallest font has 248 glyphs.
At every percentile, the number of glyphs in the smallest font has 2-3x the number of glyphs compared to the rendered HTML.

	Font Glyphs vs Rendered HTML
	Font Count	Font Weight	Visibly Glyphs	Min Font Glyphs	Max Font Glyphs
p50	3	95 KB	101	248	524
p75	5	193 KB	124	444	861
p95	9	542 KB	221	901	2229
p99	15	1001 KB	631	1530	3248

How to Test Your Page

There’s a few tools in this area that provide some insight into font usage:

FontDrop provides a useful way to explore the contents of your fonts. Simply drop the font file onto their UI and it will show you the metadata and all the glyphs contained in it.
YellowLabs has a font audit that will tell you if you have unused glyphs and summarize it by character sets. This can also be useful when deciding to subset fonts.

As part of this research, I created a tool named “Web Font Analyzer” that will leverage results from a WebPageTest measurement to show you a summary of the glyphs used on a page, when they are loaded relative to some performance metrics and how many glyphs are supported by each font. I’ve also attempted to provide some guidance on how you can use this information in the tool.

You can find the tool at https://tools.paulcalvano.com/wpt-font-analysis/.

Using the HTTP Archive we can identify sites that exhibit some of these issues, and there are quite a few. Let’s look at a few examples using this tool alongside WebPageTest.

Example - Mayoclinic

After running a WebPageTest measurement for Mayoclinic’s homepage, I copy and pasted the URL of the test results into the font analyzer tool. The tool will fetch the measurement data and create a summary of the fonts that were loaded.

The page summary section tells you how many fonts you are loading, and how large they are. It also provides a summary of the number of visible characters in the rendered HTML. In this example, there were 366 KB of fonts loaded on a page that contained only 85 visible glyphs. You can also click on “show glyphs” to see a summary of the actual glyphs in the rendered HTML.

Next we can see where these fonts are loading. The majority of these fonts are loading prior to the First Contentful Paint.And all of them are loading prior to the Largest Contentful Paint. It’s very likely that these font assets are competing with other resources for bandwidth.

If we examine the summary of fonts used on the page, we can see that many of them contain more than 500 glyphs!

In the WebPageTest measurement we can see 8 font files loading immediately after the HTML.

In the HTTP response header for the base page, there is a Link header with preloads for 5 font files. Of these preloads, 4 are for fonts hosted on www.mayoclinic.com, and the other is design.mayloclinic.com. While the preloaded fonts are downloading, the HTML initiates 4 preloads for the font files on www.mayoclinic.com (however those are already in flight). Then the fonts.css file loads, which references fonts from design.mayoclinic.com. During this page load, approximately 700ms was spent loading fonts, half of which were unused. Each of these font files were ~42 KB and contained > 500 glyphs. Meanwhile the page contained less than 100 glyphs in the rendered HTML.

Recommendations:

Update CSS to use the fonts from www.mayoclinc.com.
Subset the font files to a latin character set to reduce their weight by up to 90%.
Continue preloading the (much smaller) font files.

Example - Kia

You can see another example on Kia.com US homepage. The page weight is 21.5 MB, and 3 MB of that are font files. There are 85 visible glyphs on the page, and expanding the glyphs we can see that there are two korean glyphs (codepoints 54620 and 44544) that are used to render the “Korean” language switch link in the footer.

Prior to LCP, there was almost 7 MB of content loaded, including all of the fonts. Font weight accounts for 14% of overall page weight, but a whopping 42% of bytes loaded prior to LCP! The fonts are almost certainly competing for bandwidth.

When looking at the individual font assets loaded, we can see that these are not using font-display:swap, and that 3 of them contain 15,190 glyphs! It also appears that all of the KiaSignature fonts are delivered as WOFF files (converting them to WOFF2 would reduce some of the bytes). Their icons font is also delivered as a TTF file, and not gzip compressed.

The fonts are referenced in clientlib-base.css and they are not being preloaded. At the start of the waterfall we can see the first-party CSS and JS loading, but then clientlib-base.js gets interrupted by the higher priority KiaSignatureRegular.woff font file. This font is 965 KB, which delays the JavaScript and ultimately the first contentful paint. Additionally, the font is only cached for 1 day - so repeat visitors will need to download the fonts again.

Further down the waterfall we can see numerous PNG images. However they are being delayed by 2.1 MB of fonts. At the same time, a 2 MB hero image loaded via Adobe Experience Cloud (Scene7) is fetched and used as the poster image for a hero video.

While fonts are not the only factor affecting the performance of this page, they are clearly holding back the FCP and LCP by competing with other resources for bandwidth. Applying some easy to implement performance optimizations on the images such as lazy loading, using optimal image formats, and better cache directives will help - but ultimately the font loading will still cause delays - so subsetting them would be ideal.

Recommendations:

Subset the KiaSignature fonts for regional websites so that they are using only the necessary glyphs. For example on the US site, using latin + extended latin + the specific KR characters needed.
Convert WOFF files to WOFF2
Enable gzip compression for kia-icons.ttf. This would reduce the file to 104 KB (45% smaller). Subset the font to reduce the size even further.
Fonts are only cached on the browser for 1 day. TTL should be increased.
Not font related, but consider image and video compression, alternate image formats,and lazy loading images.

How to Subset Fonts

In some of the examples above, font subsetting would be an excellent optimization. However based on the data from the HTTP Archive, it doesn’t seem that this is being used very often. Zach Leatherman wrote a great tool for subsetting fonts, called Glyphhanger. You can also use the fonttools command line library to subset your fonts.

To use fonttools, you need to install the library.

sudo apt-get install fonttools

Once installed, you’ll have the pyfmtfont application which can be used to subset fonts. You can see some examples of its usage in this blog post.

When I was creating the WPT Font Analyzer tool, I started using a fontawesome font to render a checkbox, an exclamation point and an information circle icon. This resulted in a 124 KB font, which was many times the size of the rest of the tool! I was able to reduce this to a small 1 KB font by running the following command to create a subsetted font with the glyphs I needed.

pyftsubset fa-solid-900.woff2 \
        --unicodes=U+f05a+f058+f071 \
        --flavor=woff2 \
        --output-file=fa-solid-900-subset.woff2

Let’s try this against the 3 Kia fonts we saw from the previous example. In this example, I’m subsetting

Basic latin characters (02-7E)
Copyright symbol (A9)
Registered Trademark symbol (AE)
Trademark Symbol (2122)
Double Quotes (201C-201D)
Korean Hangul syllables (D55D, ADC0)

pyftsubset KiaSignatureRegular.woff \
          --flavor=woff2 \
          --unicodes="U+0020-007E,U+00A9,U+00AE,U+2122,U+201C-021D,U+D55D,U+ADC0" \
          --output-file=KiaSignatureRegular_subset.woff2

The resulting file is 10 KB for each of the KiaSignature fonts. So in this example 3 MB of font weight could be reduced to 30 KB! You can download this subsetted font here and examine it in FontDrop.

Conclusion

Fonts can be challenging to support from a web performance perspective, especially as their placement on modern web pages occurs at the intersection of design and web operations. Over the years there have been some innovative approaches to font loading, designed to limit the performance overhead of them. There’s also been a lot of great research in the web performance industry on this topic and best practices published. It’s always worth evaluating the end user experience to ensure that the tools and optimizations put in place are having the desired impact on user experience. I’m hopeful that the Web Font Analyzer tool I created adds another lens for you to evaluate font loading through.

Many thanks to Barry Pollard and Tim Vereecke for reviewing this.

Queries

Interested in seeing some of the HTTP Archive Queries behind this analysis? Here’s a few of the queries I used. Please be aware that some of these queries will exceed the free tier quota - so be careful when running them! (You can read more on how to minimize query costs at har.fyi.)

Percent of sites using fonts by rank

This query will calculate the percentage of sites that are using at least 1 custom web font, and group the results by rank.


SELECT 
  rank, 
  IF(SAFE_CAST(JSON_EXTRACT(summary, "$.reqFont") AS INT64) >0,"Fonts", "No Fonts") as f,
  COUNT(*),
  COUNT(0) / SUM(COUNT(0)) OVER (PARTITION BY rank) AS pct

FROM `httparchive.all.pages`
WHERE 
  date = "2024-01-01"
  AND is_root_page = true
  AND client = "mobile"
GROUP BY rank,f
ORDER BY rank ASC

Average font weight across top million sites

This query will calculate the average, median and p75 font weight of sites with a rank <= 1 million, which are using at least 1 custom web font.


SELECT 
  COUNT(*) AS freq,
  ROUND(AVG(SAFE_CAST(JSON_EXTRACT(summary, "$.bytesFont") AS INT64)),2) AS avgFontSize,
  ROUND(APPROX_QUANTILES(SAFE_CAST(JSON_EXTRACT(summary, "$.bytesFont") AS INT64), 100)[SAFE_ORDINAL(50)],2) p50FontSIze,
  ROUND(APPROX_QUANTILES(SAFE_CAST(JSON_EXTRACT(summary, "$.bytesFont") AS INT64), 100)[SAFE_ORDINAL(75)],2) p75FontSIze
FROM `httparchive.all.pages`
WHERE 
  date = "2024-01-01"
  AND is_root_page = true
  AND client = "mobile"
  AND rank <= 1000000
  AND SAFE_CAST(JSON_EXTRACT(summary, "$.reqFont") AS INT64) > 0

When do fonts start loading?

This query will run against the top 100K sites to identify the FCP, LCP and onLoad time for each measurement. It JOINs the requests table to search for the earliest start time for a font download. Then it groups the results by time interval - such as "Before FCP", "Between FCP and LCP", etc. This query processes ~1TB of data.


CREATE TEMP FUNCTION GetLcpTime(json_data STRING)
RETURNS INT64
LANGUAGE js AS """
  var data = JSON.parse(json_data);

  if (data && Array.isArray(data)) {
    for (var i = 0; i < data.length; i++) {
      if (data[i] && data[i].name === 'LargestContentfulPaint') {
        return data[i].time;
      }
    }
  }
  
  return null;
""";

SELECT 
CASE
    WHEN FCP IS null THEN 'error - no FCP'
    WHEN LCP IS null THEN 'error - no LCP'
    WHEN onLoad IS null THEN 'error - no onLoad'
    WHEN firstFontStartTime IS null THEN 'error - no firstFontStartTime'
    WHEN firstFontStartTime < FCP THEN "Before FCP"
    WHEN firstFontStartTime BETWEEN FCP AND LCP THEN "Between FCP and LCP"
    WHEN firstFontStartTime BETWEEN LCP AND onLoad THEN "Between LCP and onLoad"
    WHEN firstFontStartTime >  onLoad THEN "After onLoad"
    ELSE 'Unhandled Error'
  END AS test,
  COUNT(*)
FROM (
  SELECT 
    p.page,
    SAFE_CAST(JSON_EXTRACT(p.payload, "$._firstContentfulPaint") AS INT64) AS FCP,
    getLcpTime(JSON_EXTRACT(p.payload, "$._chromeUserTiming")) AS LCP,
    SAFE_CAST(JSON_EXTRACT(p.summary, "$.onLoad") AS INT64) AS onLoad,
    MIN(SAFE_CAST(JSON_EXTRACT(r.payload, "$._all_start") AS INT64)) AS firstFontStartTime,
  FROM `httparchive.all.pages` AS p
  INNER JOIN `httparchive.all.requests` AS r
  ON CAST(JSON_EXTRACT(p.summary, "$.pageid") AS INT64) = CAST(JSON_EXTRACT(r.summary, "$.pageid") AS INT64)
  WHERE 
    p.date = "2024-01-01" AND r.date = "2023-11-01"
    AND p.is_root_page = true AND r.is_root_page = true
    AND p.client = "mobile" AND r.client = "mobile"
    AND rank <= 100000
    AND r.type = "font"
    AND SAFE_CAST(JSON_EXTRACT(p.summary, "$.reqFont") AS INT64) > 0
  GROUP BY 1,2,3,4
)
GROUP BY 1
ORDER BY 2 DESC

How many font bytes are downloaded before FCP?

This query will run against the top 1 million sites to aggregate the number of bytes loaded before FCP. This query processes ~1.3TB of data.



SELECT 
  COUNT(*),
  ROUND(APPROX_QUANTILES(fontBytesBeforeFCP, 100)[SAFE_ORDINAL(50)],2) p50FontSize,
  ROUND(APPROX_QUANTILES(fontBytesBeforeFCP, 100)[SAFE_ORDINAL(75)],2) p75FontSize,
  ROUND(APPROX_QUANTILES(fontBytesBeforeFCP, 100)[SAFE_ORDINAL(95)],2) p95FontSize,
  ROUND(APPROX_QUANTILES(fontBytesBeforeFCP, 100)[SAFE_ORDINAL(99)],2) p99FontSize,
FROM (
  SELECT 
    p.page,
    SUM(SAFE_CAST(JSON_EXTRACT(r.summary, "$.respSize") AS INT64)) AS fontBytesBeforeFCP,
  FROM `httparchive.all.pages` AS p
  INNER JOIN `httparchive.all.requests` AS r
  ON CAST(JSON_EXTRACT(p.summary, "$.pageid") AS INT64) = CAST(JSON_EXTRACT(r.summary, "$.pageid") AS INT64)
  WHERE 
    p.date = "2023-11-01" AND r.date = "2023-11-01"
    AND p.is_root_page = true AND r.is_root_page = true
    AND p.client = "mobile" AND r.client = "mobile"
    AND rank <= 1000000
    AND r.type = "font"
    AND SAFE_CAST(JSON_EXTRACT(p.summary, "$.reqFont") AS INT64) > 0
    AND SAFE_CAST(JSON_EXTRACT(r.payload, "$._all_start") AS INT64) < SAFE_CAST(JSON_EXTRACT(p.payload, "$._firstContentfulPaint") AS INT64)
  GROUP BY 1
)

Font glyphs vs rendered HTML

This query provides a list of the top 10K web sites, individual font URLs, the number of glyphs in the font, and the number of visible glyphs in the rendered HTML. It also provides a link to the WebPageTest results from the HTTP Archive run for further analysis. This query processes almost 4 TB of data. I stored the results of the query in a scratchspace table in the HTTP Archive `httparchive.scratchspace.2024_01_01_font_glyphs_top100k`


CREATE TEMPORARY FUNCTION  CountVisibleGlyphs(html STRING)
RETURNS INT64
LANGUAGE js
AS """
  var extractedText = '';
  if (html) {
    // Remove HTML tags and keep only text content
    extractedText = html.replace(/<[^>]+>/g, '');

    // Remove extra spaces and newlines
    extractedText = extractedText.replace(/\\s+/g, ' ');

    // Remove leading and trailing spaces
    extractedText = extractedText.trim();

    // Count unique characters
    var uniqueChars = new Set(extractedText.split('')).size;
    return uniqueChars;
  } else {
    return 0; // Handle cases with empty HTML content
  }
""";

SELECT 
  pages.page, 
  rank,
  fontRequests.url,
  CAST(JSON_EXTRACT_SCALAR(fontRequests.summary, "$.respBodySize") AS INT64) AS font_size,
  CAST(JSON_EXTRACT_SCALAR(fontRequests.payload, "$._font_details.counts.num_glyphs") AS INT64) AS glyphs,
  CountVisibleGlyphs(htmlRequests.response_body) AS visibleGlyphs,
  CONCAT("https://webpagetest.httparchive.org/result/", wptid, "/") AS webpagetest, 

FROM `httparchive.all.pages` AS pages
INNER JOIN `httparchive.all.requests` AS fontRequests
  ON CAST(JSON_EXTRACT(pages.summary, "$.pageid") AS INT64) = CAST(JSON_EXTRACT(fontRequests.summary, "$.pageid") AS INT64)
INNER JOIN `httparchive.all.requests` AS htmlRequests
  ON CAST(JSON_EXTRACT(pages.summary, "$.pageid") AS INT64) = CAST(JSON_EXTRACT(htmlRequests.summary, "$.pageid") AS INT64)

WHERE 
  pages.date = "2024-01-01" 
  AND fontRequests.date = "2024-01-01"
  AND htmlRequests.date = "2024-01-01"
    
  -- mobile
  AND pages.client="mobile" 
  AND fontRequests.client="mobile"
  AND htmlRequests.client="mobile"
  
  -- root pages
  AND pages.is_root_page = true
  AND fontRequests.is_root_page = true
  AND htmlRequests.is_root_page = true

  -- font requests and HTML request
  AND fontRequests.type = "font"
  AND htmlRequests.is_main_document = true

  AND rank <= 10000

Font glyphs vs rendered HTML - analysis

This query uses the results from the previous table to compared the minimum and maximum number of glyphs in a font to the glyphs in the rendered HTML. Since this query uses the saved results from the previous query, it process a very small amount of data (~1 MB)


WITH fontStats AS (
  SELECT page, 
  COUNT(*) AS fonts,
  MIN(CAST(font_size AS INT64)) AS minSize, 
  MAX(CAST(font_size AS INT64)) AS maxSize, 
  SUM(CAST(font_size AS INT64)) AS totalSize, 
  min(CAST(glyphs AS INT64)) AS minGlyphs, 
  MAX(CAST(glyphs AS INT64)) AS maxGlyphs, 
  AVG(visibleGlyphs) AS visibleGlyphs 
FROM `httparchive.scratchspace.2024_01_01_font_glyphs_top100k` 
GROUP BY 1
)

SELECT 
  ROUND(APPROX_QUANTILES(fonts, 100)[SAFE_ORDINAL(50)],2) p50FontCount,
  ROUND(APPROX_QUANTILES(fonts, 100)[SAFE_ORDINAL(75)],2) p75FontCount,
  ROUND(APPROX_QUANTILES(fonts, 100)[SAFE_ORDINAL(95)],2) p95FontCount,
  ROUND(APPROX_QUANTILES(fonts, 100)[SAFE_ORDINAL(99)],2) p99FontCount,
  ROUND(APPROX_QUANTILES(totalSize, 100)[SAFE_ORDINAL(50)],2) p50FontWeight,
  ROUND(APPROX_QUANTILES(totalSize, 100)[SAFE_ORDINAL(75)],2) p75FontWeight,
  ROUND(APPROX_QUANTILES(totalSize, 100)[SAFE_ORDINAL(95)],2) p95FontWeight,
  ROUND(APPROX_QUANTILES(totalSize, 100)[SAFE_ORDINAL(99)],2) p99FontWeight,  
  ROUND(APPROX_QUANTILES(visibleGlyphs, 100)[SAFE_ORDINAL(50)],2) p50VisibleGlyphs,
  ROUND(APPROX_QUANTILES(visibleGlyphs, 100)[SAFE_ORDINAL(75)],2) p75VisibleGlyphs,
  ROUND(APPROX_QUANTILES(visibleGlyphs, 100)[SAFE_ORDINAL(95)],2) p95VisibleGlyphs,
  ROUND(APPROX_QUANTILES(visibleGlyphs, 100)[SAFE_ORDINAL(99)],2) p99VisibleGlyphs,
  ROUND(APPROX_QUANTILES(minGlyphs, 100)[SAFE_ORDINAL(50)],2) p50MinGlyphs,
  ROUND(APPROX_QUANTILES(minGlyphs, 100)[SAFE_ORDINAL(75)],2) p75MinGlyphs,
  ROUND(APPROX_QUANTILES(minGlyphs, 100)[SAFE_ORDINAL(95)],2) p95MinGlyphs,
  ROUND(APPROX_QUANTILES(minGlyphs, 100)[SAFE_ORDINAL(99)],2) p99MinGlyphs,  
  ROUND(APPROX_QUANTILES(maxGlyphs, 100)[SAFE_ORDINAL(50)],2) p50MaxGlyphs,
  ROUND(APPROX_QUANTILES(maxGlyphs, 100)[SAFE_ORDINAL(75)],2) p75MaxGlyphs,
  ROUND(APPROX_QUANTILES(maxGlyphs, 100)[SAFE_ORDINAL(95)],2) p95MaxGlyphs,
  ROUND(APPROX_QUANTILES(maxGlyphs, 100)[SAFE_ORDINAL(99)],2) p99MaxGlyphs,   
FROM fontStats

Internet Explorer’s Decline in Usage in 2021

2022-01-31T14:00:00+00:00

In May 2021 Microsoft announced that it would be officially retiring Internet Explorer in favor of the Chromium based Microsoft Edge. Usage for the legacy browser had been very low over the past few years, although many websites have still maintained polyfills for the older browser. In fact a number of my clients have recently told me that supporting IE 11 is required by their business, and is still a consideration when adding new features to their sites. I’m sure that the official retirement of Internet Explorer will help numerous organizations embrace modern web features and move on from some expensive polyfills. The official retirement date is June 15, 2022.

A few years ago Philip Walton wrote an excellent blog post about loading polyfills only when needed. His strategy focused on optimizing the experience for users on modern browsers, and loading polyfills based on browser support for required features (or lack thereof). One thing I really like about this approach is that he recommends creating separate bundles for modern browsers. This avoids unnecessary CPU load on modern browsers, since they won’t have to parse, compile and evaluate the polyfill JavaScript.

More recently, Alan Davalos published a blog post where he made the argument that the baseline for web development in 2022 should change as a result of this. He provided numerous examples of websites that have officially dropped support for the legacy browser in 2021.

This made me curious about the current traffic levels for Internet Explorer, and how that has changed over the past year. Looking at Akamai’s mPulse data, which measures real user performance data for users across all browsers, I can see a steady decline since March 2021. Most of the Internet Explorer traffic was from the IE 11 browser, with earlier browsers being a small fraction (< 0.01% of pages).

Looking at this data daily, you can see that the decline in Internet Explorer usage was linear throughout the year. It also seems that the decline accelerated after the announcement in May 2021. Usage dropped even further in November 2021.

The zig-zag pattern in the daily traffic indicates that the majority of users utilizing IE 11 are doing so during the Monday-Friday work week, with traffic dropping by two thirds on the weekends. If business users are the primary source of Internet Explorer usage, then they could be at risk of 0-day exploits once official support ends.

I thought it would be interesting to see which countries had the highest percentage of Internet Explorer traffic during January 2022. Sure enough this varied from country to country. For example, in the United States 0.97% of pages were loaded from an Internet Explorer browser. In Great Britain, it only accounted for 0.29% of pages.

Some countries had a noticeably higher percentage of Internet Explorer traffic - for example South Korea (2.36%), China (1.76%), Hong Kong (1.78%). And then there are some countries with a shockingly high percentage of Internet Explorer traffic: Haiti (26.15%), Belize (9.12%), Jamaica (7.13%), Cambodia (6.8%) and more. The graph below shows a summary Internet Explorer usage per country. You can also view an interactive version of it here.

Internet Explorer is going away, and that is ultimately a good thing for the web. As support officially ends, website owners should consider removing expensive polyfills for the browser and add support for technologies that are supported on modern browsers. However this won’t happen by itself, and sites that have been bundling polyfills into a large monolithic bundle will need to put in the effort to analyze which ones are still needed. Fortunately there are some great resources on bundle analysis, such as Sia Karamalegos’ guide to Lighthouse Treemap and Nolan Lawson’s guide to bundle analysis tools.

Organizations should also look to remove the legacy browsers on their employees’ machines to avoid security risks down the line. Microsoft provides guides and resources to remove legacy Internet Explorer browsers from organizations, which are available here.

Page Visibility: If a tree falls in the forest…

2021-12-31T16:30:00+00:00

If a tree falls in the forest and no one is around, does it make a sound? Likewise, if a web page loads in a background tab then does its load time really matter? As a user, the time it takes for background tabs to load may seem irrelevant since you are unlikely to notice delays. However if you are managing a website and measuring user experience, then it’s important to understand how visibility state can influence the data you are analyzing.

In this blog post we’ll explore the Page Visibility API as well as some data from Akamai’s mPulse to understand the visibility states of real users loading billions of page views, and what it means for web performance.

Page Visibility

The Page Visibility API defines a programmatic way of determining the visibility state of a top-level browsing content, as well as a method of measuring visibility state changes over time. Web developers can use this information to determine whether a page is visible to an end user. It also gives them the ability to scale back the work being performed on a page load. The Page Visibility API is also supported in all modern web browsers.

Using this API is very simple. The attribute document.visibiltyState will return visible or hidden depending on whether the page is visible or not. If you want to see the value changing as you switch tabs, you can also monitor the visibility changes. For example,

console.log(document.visibilityState + ': ' + Date())
document.onvisibilitychange = () => console.log(document.visibilityState + ': ' + Date())

The W3C specification, also provides an example of using this API to decide whether to autoplay a video on page load based on visibility state. It adds an event listener to listen for changes in the visibility state so that the video playback can start automatically when the page is visible.

RUM tools can collect this data as well. For example, mPulse collects the visibility state of a page once the page load has completed and also measures for changes in visibility throughout the page load.

The data in this blog post is based on billions of page views across all sites using mPulse during the month of November 2021.

Visibility States

How often do you right click on a link and load a page in a background tab? Or click on a link on your mobile device and switch applications before it finishes loading? Or lock the screen on your mobile device while waiting for a page to load?

The graph below breaks down visibility states by device type using RUM data from mPulse. The visibility state is measured as soon as the onLoad event is fired. 11.18% of all Desktop page views were loaded in a hidden visibility state. Similarly, 9.59% of Mobile page views were loaded in a hidden visibility state.

Note: Less than 1% of pages were loaded in a prerender state. This feature has been deprecated in the Page Visibility API, supported inconsistently across browsers, and therefore not as useful for this analysis.

Why is Visibility State Important for Web Performance?

Most modern browsers prioritize work being done in the foreground, and as such one would expect that a page loaded in a background tab may be slower. Individually you can examine this behavior by loading a page and looking at it’s navigation timing data. Load that same page in a non-visible tab a few times and look at the difference in timing.

When analyzing the median page load time (onLoad metric) in mPulse, I can see a significant difference in performance based on visibility state. For example, the median time to load a page on Desktop was 2.8 seconds. The median load time for pages in a visible state was 2.7 seconds and the median load time in a hidden state took 4 seconds. The median load times for pages loaded in a hidden visibility state was 32% slower on Desktop and 37% slower on Mobile!

Going back to the tree falling in a forest analogy, does this really matter? I’d say yes and no, for the following reasons:

From a user experience, if the page is not visible then the user is not impacted by the delay.
As performance engineers we look to RUM data to tell us what our users are experiencing. If a large enough percentage of users are loading tabs in the background, then it may impact the metrics we are analyzing. This is exacerbated even further if you are looking at upper percentile stats.

The graph below illustrates the median load times as well as the p75, p95 and p99 based on visibility state. The p95 for all Desktop pages loaded in a visible state was 14.37 seconds. Comparatively, the page load times for hidden states was 37.65 seconds, which is more than twice as slow!.

Taking this one step further, the graph below shows the distribution of load times in a histogram, for both hidden and visible states. As the response times increase, hidden visibility states account for a larger percentage of experiences. At the 95th percentile, 25% of all pages were loaded in a hidden visibility state. (Note: the x axis in this graph ends at 18 seconds, which is around the 95th percentile).

Now let’s explore the upper percentiles. The graph below shows the same data for the slowest 5% of experiences. The percentage of hidden visibility states increases with respect to the load time, eventually approaching 36%. If you are analyzing upper percentiles to tackle some of your long tail performance issues - then not filtering out hidden visibility states will leave a significant amount of noise in your data.

Page visibility by desktop browser

During the month of November 2021, Chrome, Edge, Safari, Firefox and Internet Explorer accounted for 96.3% of all desktop page views. Chrome was the dominant browser, with 63.9% of all page views. However all of these browsers support the page visibility API.

The table below details the distribution of visibility states by Desktop browser. The percentage of hidden visibility states varies widely by browser. This may be influenced by a variety of factors, such as browser UI features (such as tabbed browsing) or by the end user switching between applications on their machine.

Chrome had the highest percentage of hidden visibility state page loads (12.9%) compared to other desktop browsers. Interestingly, Edge (which is now Chromium based) had 7.86% of page loads. Given that legacy Internet Explorer (version 11 and earlier) had only 4.4% hidden visibility states, it’s possible that some users of Edge and Internet Explorer are less likely to utilize the tabbed browsing features.

	% of page loads by visibility state
Browser	hidden	visible
Chrome	12.91%	87.09%
Edge	7.86%	92.14%
Safari	6.86%	92.89%
Firefox	9.25%	90.75%
IE	4.44%	95.56%

It’s worth noting that if you are measuring your site with a synthetic measurement, you are almost always loading a page in a visible state.

Page visibility by mobile browser

Mobile web traffic is split between browser apps, WebViews, and in-app browsers. Mobile Safari is the dominant mobile browser, with 39.2% of page loads. WebViews also represented a significant traffic share, accounting for 17% of all mobile page views (or 20% if we include Chrome Mobile and Firefox Mobile’s iOS apps).

The table below breaks down the visibility states measured by mobile web browsers. Overall, 5.98% of pages on Mobile Safari were loaded in a hidden visibility state, which is comparable to Desktop. However Chrome Mobile had less hidden visibility states (8.88% mobile vs 12.91% desktop). Interestingly, some Chromium based mobile browsers (such as Samsung Internet, MiuiBrowser and Edge Mobile) had a much higher percentage of hidden visibility states - likely due to differences in their UI and user base.

	% of page loads by visibility state
Browser	hidden	visible
Mobile Safari	5.98%	93.99%
Chrome Mobile	8.88%	91.12%
Samsung Internet	12.34%	87.66%
Crosswalk	2.93%	97.07%
Firefox Mobile	2.48%	97.52%
MiuiBrowser	16.43%	83.57%
Opera Mobile	12.45%	87.55%
Yandex Browser	6.16%	93.83%
Edge Mobile	10.10%	89.89%
UC Browser	9.74%	90.25%

When we look at distribution of visibility states across WebView’s and in-app browsers, we can see that WebViews have the highest percentage of hidden visibility states. This could be due to mobile users switching apps or locking their screens before a page is finished loading.

Another interesting observation is that the social media apps Instagram and Snapchat have a higher % of hidden visibility states compared to Facebook and Pinterest. Could this be due to differences in demographics of these platforms?

	% of page loads by visibility state
Browser	hidden	visible
Mobile Safari UI/WKWebView	26.61%	73.20%
Chrome Mobile WebView	12.73%	87.27%
Facebook	2.78%	97.22%
Chrome Mobile iOS	7.77%	92.20%
Instagram	7.16%	92.83%
LINE	1.87%	98.12%
Firefox iOS	6.24%	93.75%
Snapchat	9.46%	90.48%
Android Webkit	4.41%	95.59%
Pinterest	3.69%	96.31%

One thing in common across all browsers - desktop and mobile - is that there are a significant amount of hidden visibility states that are worth accounting for in our performance measurements.

Beyond page load time

When we look at other performance metrics, we can see that times measured for metrics such as Total Blocking Time, Largest Contentful Paint and First Contentful Paint were also impacted by visibility state. The graph below illustrates this for Chrome Desktop pages. The metric that was most impacted was Total Blocking Time, which was almost twice as slow when hidden. This makes sense since the browser is likely not prioritizing the execution of JavaScript on the page, which also explains the 46% increase in page load times

Largest Contentful Paint is one of the Core Web Vitals, which Google is using as a signal for search ranking. The mPulse data shows us that the p75 LCP for a hidden visibility state is 23% slower than when it is visible. At the p95, the LCP is almost twice as slow when hidden.

Google’s Chrome User Experience Report (CrUX) already filters out hidden visibility states, which means that your search ranking will not be impacted by slow non-visible page loads. However the tools you are using to monitor these thresholds may have a blind spot here. Fortunately it’s easy enough to collect visibility state data using the Page Visibility API. For example, in mPulse, the visibility state is a dimension that you can filter in any dashboard. You can also create custom dashboards to track the distribution of experiences based on visibility states if that interests you.

Conclusion

The Page Visibility API is an incredibly useful way of determining the visibility state of a page load. It can be used to provide developers with the ability to fine tune experiences based on visibility state, which can conserve CPU and battery usage. It’s also measurable with RUM, and based on the data from mPulse we can see that page load times are slower across all browsers when the visibility state is hidden.

While this may not matter as much for end user experience, it’s happening at a high enough frequency that it can influence your performance metrics. If you are optimizing for the long tail of web performance, you may want to filter out hidden visibility states.

Originally published at https://calendar.perfplanet.com/2021/page-visibility-if-a-tree-falls-in-the-forest/

What can the HTTP Archive tell us about Largest Contentful Paint?

2021-06-07T16:30:00+00:00

Largest Contentful Paint (LCP) is an important metric that measures when the largest element in the browser’s viewport becomes visible. This could be an image, a background image, a poster image for a video, or even a block of text. The metric is measured with the Largest Contentful Paint API, which is supported in Chromium browsers. Optimizing for this metric is critical to end user experience, since it affects their ability to visualize your content.

Google has promoted this metric as one of the three “Core Web Vitals” that affect user experience on the web. It is also slated to become a search ranking signal over the next few weeks, which has created a lot of awareness about it. The suggested target for a good Largest Contentful Paint is less than 2.5 seconds for at least 75% of page loads.

Source: https://web.dev/lcp/

Some of the recent posts on WPOStats feature interesting case studies about this metric. For example,

Google’s research found that when Core Web Vitals are met, users are 24% less likely to abandon a page before it finishes loading.
Vodafone improved LCP by 31% and saw an 8% increase in sales.
NDTV improved their LCP by 55% and saw a 50% reduction in bounce rate.
Tokopedia improved their LCP by 55% and saw a 23% increase in session duration.

Identifying the Largest Contentful Paint Element

The name of this metric implies that size is used as a proxy for importance. Because of this, you may be wondering specifically which image or text triggered it as well as the percentage of the viewport it consumed. There are a few ways to examine this:

One way to visualize the Largest Contentful Paint is to look at a WebPageTest filmstrip. You’ll be able to see when visual changes occurred (yellow outline) as well as when the Largest Contentful Paint event occurred (red outline).

In Chrome DevTools, you can also click on the LCP indicator in the “Performance” tab to examine the Largest Contentful Paint element in your browser. Using this method you can see and inspect the exact element (image, text, etc) that triggered it.

Lighthouse also has an audit that identifies the Largest Contentful Paint element. If you examine the screenshot below you’ll notice that there is a yellow box around the largest element, as well as an HTML snippet.

How Large is the Largest Contentful Paint?

The HTTP Archive runs Lighthouse audits for approximately 7.2 million websites every month. In the May 2021 dataset, Lighthouse was able to identify an LCP element in 97.35% of the tests. Since we have the ability to query all of these Lighthouse test results, we can analyze the result of the LCP audits and get more insight into what drives this metric across the web.

Using the same boundaries that Lighthouse uses to draw the rectangle around the LCP element, it’s possible to calculate the area of it. In the above example, the product of the LCP image’s height (191) and width (340) was 64,940 pixels. Since the Lighthouse test was run with an emulated Moto G4 user agent with a screen size of 640x360, we can also calculate that this particular LCP image took up 28% of the viewport.

The graph below shows the cumulative distribution of the LCP element as a percentage of screen size. The median LCP element takes up 31% of the screen size! At the 75th percentile the LCP element is nearly twice as large, taking up 59% of the screen size. Additionally 10.6% of sites actually had an LCP element that exceeded the viewport (which is why the y axis doesn’t reach 100%).

The graph below illustrates the same data in a histogram. From this we can see that 4.03% of sites (285,751) had a LCP element that took up 0 pixels. Upon further inspection, the 0 pixel elements appear to have been used in carousels, so by the time the audit completed the LCP element slid out of the viewport.

Node Paths of LCP Elements

Another interesting aspect of the Largest Contentful Paint audit is the nodePath of the element, which shows you where in the DOM this element was. In the example we looked at earlier, the nodePath was: ```

1,HTML,1,BODY,8,DIV,2,SECTION,1,DIV,0,DIV,0,DIV,0,UL,0,LI,0,ARTICLE,1,DIV,0,DIV,0,A,0,IMG

If we look at the last element in the node path, we can get some insight into the type of element that triggered the Largest Contentful Paint. The most common node that triggered the Largest Contentful Paint was <IMG>, which accounted for 42% of all sites. Next was <DIV> at 27% (which could include text or images). The <H1> through <H5> header elements accounted for 7.18% of all Largest Contentful Paints.

LCP Node (last element in path)	Number of Sites	Percent of Sites
IMG	3,067,354	42.12%
DIV	1,981,416	27.21%
P	766,977	10.53%
H1	291,091	4.00%
	192,498	2.64%
SECTION	182,267	2.50%
H2	144,534	1.98%
A	107,501	1.48%
SPAN	85,245	1.17%
HEADER	67,762	0.93%
LI	64,212	0.88%
H3	60,679	0.83%
RS-SBG	51,623	0.71%
TD	48,470	0.67%
H4	19,039	0.26%
VIDEO	15,649	0.21%
ARTICLE	12,860	0.18%
FIGURE	9,208	0.13%
BODY	8,859	0.12%
image	8,077	0.11%
CENTER	7,960	0.11%

The <VIDEO> element only accounted for 0.21% of sites. According to the Web Almanac, the <video> element was used on 0.49% of mobile websites - so from this we can estimate that half of sites loading videos are triggering LCP with video poster images.

Image Weight for the LCP

One of the Lighthouse audits looks for opportunities to preload the Largest Contentful Paint element, and estimates the potential savings in performance. This audit also identifies the URL for the LCP element - which can give us some insights into what type of images are being loaded as a LCP element. In the HTTP Archive data, only 67% of the Lighthouse tests were able to identify a URL for an LCP element. Based on this, we can infer that text nodes are used for the LCP on approximately 33% of sites.

The graph below shows the distribution of sizes for the image element that was associated with the Largest Contentful Paint. The median LCP element size was 80KB. At the 90th percentile, the LCP element size was 512KB. If you have a large LCP image then you should consider optimizing it before you attempt to follow the Lighthouse preload recommendation.

Additionally, 70% of the LCP element images were JPEG and 25% were PNG. Only 3% of sites served a webp as their LCP element.

format	sites	% of Sites
jpg	3,161,991	69.37%
png	1,122,585	24.63%
webp	141,441	3.10%
gif	84,829	1.86%
svg	34,123	0.75%
Other	13,272	0.29%

When we look at the LCP element as a percentage of page weight, we can see that the median LCP element is 4.17% of the total page weight. At the higher percentiles, the LCP elements are larger and also a larger percentage of page weight.

percentile	ImageRequests	ImageKB	TotalKB	LCP as a % of Page Weight
p25	15	422	1,138	3.01%
p50	26	1,142	2,185	4.17%
p75	45	2,692	4,108	5.58%
p95	103	8,008	10,036	8.42%

Since images account for 52% of the median page weight (for the sites that have a LCP image element), we can infer that at the median 8% of page weight is used to render content to 31% of the screen.

How does this change based on Site Popularity?

The HTTP Archive now contains rank groupings, obtained from the Chrome User Experience Report. This can enable us to segment this analysis based on the popularity of sites. The rank grouping indicator buckets sites into the top 1K, 10K, 100K, 1 million and 10 million.

When we look at the Largest Contentful Paint image size based on popularity, it’s interesting to note that the most popular sites tend to be serving smaller images for the LCP element. While there may be numerous reasons for this, I suspect that the more popular sites are investing in image optimization solutions.

Page weight follows the same pattern, with the least popular websites having some of the largest page weights. If we look at the LCP element based on the percentage of page weight, you can see that within the top 100K sites the ratios are very close. In the less popular sites, the LCP element tends to be a much greater percentage of page weight.

rank	p25	p50	p75	p95
Top 1k	1.61%	2.12%	2.85%	5.67%
Top 10k	1.76%	2.27%	3.00%	4.96%
Top 100k	2.07%	2.87%	3.77%	5.78%
Top 1 million	2.53%	3.49%	4.60%	6.95%
Top 10 million	3.11%	4.30%	5.75%	8.65%

We can also make some interesting observations about how popular sites are optimizing their LCP assets. Looking at the various image formats, JPG images are the most common LCP element. Some other formats such as PNG, WebP, GIF and SVG are used more frequently in the more popular sites.

Conclusion

Largest Contentful Paint is an important metric that helps illustrate when a page’s most significant content is rendered to the screen. In reviewing the HTTP Archive data, we can see that this area represents between 30% and 60% of a mobile viewport for a majority of sites.

There are a shocking number of sites that have a LCP element that consumes a large percentage of the viewport and are delivered as large unoptimized images. Site owners should evaluate both what is triggering the Largest Contentful Paint as well as how it is loaded. Optimizing for the Largest Contentful Paint will ensure that the browser has the opportunity to load and render this content as quickly as possible.

If you are interested in seeing some of the SQL queries and raw data used in this analysis, I’ve created a post with all the details in the HTTP Archive discussion forums. You can also see all the data used for these graphs in this Google Sheet.