Correlating Performance Metrics to Page Characteristics

When we talk about web performance measurement, there is a long list of metrics to choose from. As an industry we are converging on metrics that gauge user experience – such as “Time to Interactive” and “Time to Visually Ready”. Other metrics such as onLoad and First Contentful Paint are also widely used and available in most browsers via APIs such as Navigation Timing and Paint Timing. And then there are Speed Index, Start Render, Fully Loaded time and many others, including protocol times (DNS/TCP/TLS) and backend times (TTFB). You are optimizing your sites and have all these measurements at your disposal – so what do you use to evaluate your changes?

Let’s say you have a page that loads in 5 seconds (onLoad) and you make a small improvement that shaves off 100ms during the critical render path. If you measure your improvement with the onLoad metric then you’ll see a mere 2% performance improvement. Such a small percentage might make some question the value in investing your time on such an optimization. However if you look at what it is optimizing – the critical render path – and then choose a metric that gauges it (such as First Contentful Paint), you may see an improvement as high as 10% during the critical render path. Conversely, if you optimize your JavaScript and reduce the processing overhead then Time to Interactive would be the metric to use. Don’t box yourself into using a single metric for everything. It’s best to figure out what you are optimizing for, and then how to measure that.

The HTTP Archive gives us the ability to dig into countless insights about how web pages are built. We can use this data to correlate these page attributes to different performance metrics. For example, a few months ago I was able to correlate page weight to performance and found that onLoad times of larger pages were slower. When I looked at the First Contentful Paint metrics I was not able to see the same correlation. In this article we’ll go a bit deeper and explore some more correlations across a wider set of metrics.

In statistics, the Pearson Correlation Coefficient is a measure of the linear correlation between two variables. The coefficient ranges from -1 to 1, where 1 implies a perfect linear relationship, 0 implies no linear correlation and -1 implies that an inverse correlation is present (ie, Y decreases while X increases).

Google BigQuery has a built in aggregate function that can calculate the pearson correlation coefficient for a set of number pairs. In order to use it, we just need to call the CORR() function and pass it the two columns we want to correlate. For example, correlating Page Weight to Load Time via the HTTP Archive would be as simple as:

We can expand this query to look at the correlation between First Contentful Paint and Page Weight as well.

The output of this shows that the pearson correlation coefficient for onLoad vs Total Page Weight is 0.20. The first contentful paint correlation is 0.03, which is significantly lower and consistent with my earlier observation. The closer the coefficient is to 1, the stronger the relationship between the two variables. Based on this, there is a slight correlation between page weight and onLoad time, but not for first contentful paint.

Now let’s expand this to include other metrics and page characteristics. In the example below I’m going to correlate the following characteristics of requests, page weight, JavaScript loading and more with a set of web performance metrics.

Note: The performance measurements in this analysis are from HTTP Archive and not real user measurement sources such as CrUX or mPulse. We’ll be looking at 1 measurement for each of the ~4 million sites to get a snapshot of the correlation. I strongly recommend measuring your performance improvement via RUM to understand how it impacts user populations at different percentiles.

The query for this includes UNION’ed queries for each metric. The CORR() function is called for each of the attributes mentioned above as well. It’s a rather large query, and you can see the full query here (Note: this will process 38GB of data).

In the table below, the correlation values for each page characteristic are highlighted based on their strength compared to each metric Dark green indicates that there is a strong correlation. The lighter shades indicate that the correlation is weaker, and red indicates a negative correlation.

The results provide a lot of insight into where you may be able to measure some optimizations. For example, factors more likely to affect rendering of pages include the number of CSS requests, amount of JavaScript and CPU overhead. Image weight is measurable to onLoad, while JavaScript impacts are most noticeable at Time To Interactive and Fully Loaded, but is not as impactful to onLoad

Conclusion

Your mileage will vary from site to site, but the results here show a strong correlation between some particular characteristics of page design and key performance metrics. It’s also just as important to understand that you may not see a noticeable impact across all measurements for a particular optimization. Understanding what you are optimizing for, and then how you plan to measure it is critical.

Originally published at https://discuss.httparchive.org/t/correlating-performance-metrics-to-page-characteristics/1548

Mobile Trends during the US Holiday Weekend

Over the past few years we’ve seen a tremendous growth in mobile traffic on the web. Because of this many of the most successful websites have invested in optimizing the experience of users on whatever device they use and however they connect to the internet. With mobile traffic now exceeding desktop, serving a quality mobile experience is more important than ever. During the recent holiday weekend, I was wondering how much retail traffic occurred via mobile or desktop devices. Was there a large shift towards mobile during peak times on Black Friday and Cyber Monday? Did mobile usage spike on specific days, or times of day? And when users are connecting from mobile, are they connecting over cellular networks or WiFi?

During normal day-to-day traffic we see shifts in device usage on the weekends. For example, in a recent study I learned that percentage of mobile traffic globally is 43% during the week but increases to 53% over the weekend. Tablet usage also increases marginally over the weekend as well. In the graph below you can see this trend for October 2018.

In previous blog posts and talks I’ve shared some insights using data from Akamai mPulse. The data I’m using for this analysis is a subset of overall mPulse traffic – specifically the US traffic of more than 50 retail websites. To avoid skewing stats by some larger sites, I’ve also ensured that none of the sites in this dataset account for more than a few percentage points of the total data set. (Note: non-US retail traffic during the US holiday weekend is a topic I may explore in a future analysis as well.)

The graph below illustrates the distribution of pages from Desktop, Mobile and Tablet form factors between Thanksgiving and Cyber Monday 2018. There are a few interesting peaks:

  • Thanksgiving traffic started to increase around 5pm EST and peaked at 9pm.
  • Black Friday traffic was intense from 9am to 10pm EST
  • Sunday evening traffic spiked between 8pm and 10pm EST
  • Cyber Monday traffic was as high as Black Friday for most of the day, and then bursted 30% higher than Black Friday’s peak during the evening

The fluctuations in mobile traffic were particularly interesting to me, so I decided to graph this for each day on a 24 hour axis. The graph below shows the percentage of mobile traffic per hour for each day. There was upwards of 60% mobile traffic during the early mornings, leveling off at 53% during the day. The percentage of mobile traffic during the evening hours of Thanksgiving spiked to 58%. Meanwhile, on Cyber Monday desktop traffic dominated for most of the day. As device usage fluctuates by time of day, it’s important for retailers to focus on providing an optimal experience to all users regardless of how they connect.

Now that we know what types of devices people are using, let’s explore how they accessed the web. The graph below illustrates the distribution of Desktop pages loaded over Cellular, Corporate and other Non-Mobile networks. The percentage of desktop traffic from corporate networks increased significantly during Cyber Monday and to a smaller extent on Black Friday. This indicates that a fair amount of online shopping was done by people while they were at work. We typically see spikes like this in other industries (such as streaming events), especially when major events are occuring during business hours.

When we look at the same data for Mobile devices, we can see an interesting pattern in connectivity. During each day mobile networks accounted for 40% of mobile traffic between the hours of 12pm and 2pm ET.

The Cyber Monday traffic patterns were quite interesting, so I decided to look at them on a per minute level. The graph below shows the relative page views for Desktop, Mobile and Tablet traffic. As we saw earlier, Desktop traffic was strongest during the Cyber Monday business day and then started to decline after 5pm EST as the US east coast business day ended. Mobile traffic quickly took its place. In the evening we saw an increase from both Desktop and Mobile traffic, which resulted in the impressive Cyber Monday evening peak. The periodic drops in traffic are likely due to the struggles that some retailers faced.

Many of these stats are similar to what I’ve seen in previous years, but this is the first time we’re able to see an aggregate view of the holiday traffic patterns like this. I’m interested to see how this changes next year.

One important thing to note: when looking at traffic for specific sites, some of the bursts were much more intense than what we see in aggregate. For example, one retailer I worked with had a timed event on Saturday evening, which was one of the lowest traffic days of the weekend. However this retailer managed to double their traffic within 4 minutes. I was particularly impressed with that retailer, because their response times also improved during this time – likely as a result of excellent holiday preparations!

Based on these stats, the 2018 Holiday shopping season seems off to a great start, and we’re seeing some impressive amounts of traffic – both per site and in aggregate across the retail industry. As with most years, mobile traffic continues to grow and spikes during certain times. However just as important as what devices are accessing your sites, the way that they are connecting matter greatly. Mobile traffic is mostly split between Cellular and WiFi, but that distribution varies based on both the time of day as well as the day of the week. In general we are seeing spikes in utilizations for both all form factors, which highlights the importance of ensuring that you can deliver optimal experiences to each device and regardless of how a user is connecting to your site.

On Becoming a Contributor to the HTTP Archive

The HTTP Archive is an open source project that tracks how the web is built. Twice a month it crawls 1.3 million web pages on desktop and emulated mobile devices, and collects technical information about each of the web pages. That information is then aggregated and made available in curated reports. The raw data is also made available via Google BigQuery, which makes answering interesting questions about the web accessible to anyone with some knowledge of SQL as well as the curiosity to dig in.

Continue reading