1 February 2024

We Need to Talk About FPS Metrics Reporting... (Part 2)



There's a well-known idiom that's often said: "There's lies, damn lies, and then there's statistics...". This implies that the "statistics" in question are another, worse form of lie that is somehow obfuscated from the receiver of the information.

We also have multiple well-known sayings which revolve around the concept of, "you can make the statistics/data say anything you want". It seems readily apparent that people, in general, do not like or trust "the statistics".

I experience this, in my own way, in my day-to-day work. Scientists are currently not the most trusted of individuals - for whatever reason - and one of those reasons, in both cases, is a lack of understanding on the part of the consumer of the results of data analysis, both within and outside of scientific circles.

In the same way people say "science is hard", people say "statistics" is hard... and this is for good reason - though it might not be for the specific reason that might immediately spring to mind!

Statistics is not that difficult once you know what you are doing (at least in my opinion). The difficult part is knowing which statistical test to apply when and where. Yes, the difficulty, as when designing scientific experiments, is understanding the context, limitations and biases of what and how you wish to test.

This is why there are many statistical tests where the number of data points needs to be below or above a certain limit; why it is important to know the relationship between the individual data points and the set as a whole; and how the interpretation of the result of the analysis might be changed based on myriad factors.

Hence, we come to today's topic for discussion: hardware performance testing in games!

Last time, I attempted to communicate the shortfalls and incorrect analysis being performed in the industry at large. Admittedly, I was unsuccessful in many ways and was roundly dismissed by most parties...

Today, I will try a different tack.

29 December 2023

Looking back at 2023 and predictions for 2024...


Hap-pey Burf-yay!
 
 
The introduction to last year's post could almost be copy/pasted into this one. Work was even more intense than last year and I know for a fact that 2024 will be even tougher, so my hopes for spending time writing for this blog look set to be a big loss... But, that doesn't mean that I won't try and address things when I feel I can dedicate the time or perform more comparisons and benchmarks for everyone to digest.
 
2023 was a big year for hardware releases - with most of the current generations of CPU and GPUs being released at some point. Sure, we're getting some minor refreshes from Nvidia next year but, overall, 2024 looks set to be quite boring.
 
With that in mind, my predictions for this coming year were quite hard to pin down - what is left to predict? 

So let's start where we always do: the review of last year's predictions.

12 November 2023

The Performance Uplift of RDNA 3 over RDNA 2...


Yes, this is an RDNA 2 card, I wasn't able to create a new one with RDNA 3 just yet...

Much was made about the performance uplift (or lack thereof) of the initial RDNA3 cards released by AMD. Navi 31 (the RX 7900 XTX and XT) failed to meet expectations - seemingly both internal to AMD and externally (for various reasons) - this lack of performance also extended to the RX 7900 GRE which, despite lower core clocks, still underperformed compared to where it could be calculated that it should be...

Disappointingly, Navi 33 (the RX 7600) performed exactly the same as the equivalent RDNA2 counterpart, the RX 6650 XT, showing that there was zero performance uplift gen-on-gen in that lower tier part...

In the meantime, rumours swirled that Navi 32 was going to be 'fixed'. So, what is the truth of the matter? I intend to investigate a little and get to the bottom of the situation like I did with my Ampere vs Ada Lovelace performance uplift analysis...

2 November 2023

Alan Wake 2 Performance Analysis...


Alan Wake 2 is the new hotness in the games industry. Love it or hate it, you cannot deny the impact that it has had on the general conversation with regards to hardware, software, and game development. While I'm not as on-board with the near universal, unfettered praise for the title as most reviewers appear to be, I do find the discussions surrounding the hardware requirements and performance of that hardware near and dear to my heart.

So, without further ado, let's take a look at how current mid-range hardware performs in this game and whether that actually lines up with the hardware requirements that the developers put out just before the game's release...

27 October 2023

In Defence Of: Older Hardware...

Via Twitter...

The release of the required PC hardware specifications of Alan Wake 2, along with the revelation that RX 5000 series and GTX 10 series cards would not be supported* caused quite a stir in various online fora. I, myself, have not been overly happy with them but it may not be for the typical reasons that proponents of advancing technology(!!) would like to paint. At the same time, despite the hyperbole on both sides of the equation, I think there is room for reasoned discourse on the topic and platforms like Twitter and Reddit don't tend to promote or facilitate that. 

So, here goes...
*They can run the game, just not to their normal relative performance envelope compared to other cards due to the fact that they do not support DX12 Ultimate mesh shaders.