Analysing a graphics card's performance has been debated for years. In fan boy threads across the world wide web, debates have continually thrown up the question - which is best?
How do you measure what is 'best'? Image quality? Driver support? Price? The one area which is generally agreed that can separate two graphics cards is Frames Per Second. This method has been used for years and for the most part is/was the accepted method to measure graphics card performance. Indeed, we use this method ourselves currently. There is however a change afoot whereby FPS is no longer the sole method for measuring how good a GPU is.
The problem with FPS
While frames per second measurement may well be very good at measuring how fast a GPU is, it can as we shall see, give a false representation of gaming experience. Measuring FPS is done by counting the number of frames rendered within a second. Obviously, we are not quick enough to do this ourselves so we commonly will use some form of benchmarking tool, FRAPS for example. This tool will then be used over a length of time for example 60 seconds. Each second's total frame count is then averaged over the 60 second period to give us our average FPS measurement. What the frames per second method of measurement will not show us though is what is commonly referred to as 'micro stutter'.
I think it is safe to assume most gamers will have encountered micro stutter at one time or another, especially those who use a multi-GPU setup. Micro stutter is a term used to describe a situation when smooth gameplay, with regular FPS rendering being suddenly interrupted in a large drop in FPS then immediately returned to normal, then back down and back up again. This 'stuttering' can happen over milliseconds and is more often 'felt' by the gamer rather than directly witnessed due to the minute time scales involved. This micro stuttering will ruin gaming experience.
Imagine the latest graphics card produces an 'average' 100 FPS in your favourite game. You would be thrilled at this new card and after reading the latest reviews you rush out to buy it. Upon playing this game with your new hardware though you are left a little disgruntled. Upon your own testing you confirm that it does indeed grant 100FPS however the gameplay feels 'choppy' compared to your previous card which only churned out 60 FPS. Why is this? Simply put, you may well be encountering micro stuttering.
It is all very well having a GPU that churns out a high average FPS but if that average FPS figure is masking occasional huge drops in FPS then the gaming experience is likely to be less satisfying than a GPU which is capable of rendering frames in a more consistent manner, albeit slower on average. By that I mean having a lower 'overall' FPS average reading.
To put things at the extreme end, say you have a GPU capable of churning out 100FPS. For 59 seconds of a 60 second benchmark, a very low frame time is recorded of 10ms. This means for 59 seconds, your game experience will be like a knife the butter. However for one frame the GPU stalls and creates a massive hike in frametime to 999 milliseconds. In an FPS measurement, this would hardly make a difference in the average FPS measurement as it would be masked by the excellent FPS count however, with frame time measurement it would be immediately noticeable.
Micro stuttering should not be confused with inevitable troughs and peaks in gaming performance by FPS rendering ability. Some scenes, especially in graphical intensive games such as Crysis 3 are more demanding and therefore a drop in FPS will occur. If that drop is significant enough it will be witnessed on screen and can sometimes 'feel' like micro stuttering whereby frame rates are so low that the game 'feels' choppy. This is simply a performance issue and can be resolved by lowering the graphics settings.
Micro stuttering is not a new phenomenon. Like FPS measurement, it is a subject that has been debated for years and is a certainly a common problem with multi - GPU setups such as Crossfire (AMD) and SLI (NVIDIA). Multi GPU setups suffer terribly from micro stuttering simply because of the way graphics are rendered which amplifies an already inherent problem. That isn't to say the problem is also encountered with single GPU solutions for as we will see, regardless of how many graphics cards you have in your PC system, micro stuttering is a problem for everyone to consider.
Frame time analysis is not a new concept either, although few review sites have employed this as a method of measuring GPU performance up until recently. The fantastic work by Scott Wasson at The Tech Report, of whom much of the credit has to be given has however showed us that Frame Time Analysis is just as important as outright FPS performance, if not more so and thus deserves credit for bringing this phenomena back to the forefront of assessing a GPU's performance.