It was not long ago that Samsung had faced some backlash from the tech community when some people who take benchmark scores too seriously figured out that the little Korean smartphone maker was making its flagship devices benchmarks friendly! By benchmark friendly I mean that the phone was programmed to perform at its maximum potential when certain types of benchmarking tools were run. What this resulted in was that the scores of Samsung products in these test results were surpassing that of devices from other manufacturers powered by the same processors and chipsets. This eventually earned these devices a top spot in the benchmarking test wars. While Samsung chose to remain defiant on that issue, the subsequent hue and cry might have made people think that Samsung would not resort to similar tactics again. Well, that doesn’t seem to be the case!
A fresh report by Ars Technica which was reviewing Samsung’s newest flagship phablet – the Galaxy Note 3 states that they are finding similar behaviour with the Note 3 as well. The Note 3 which they tested was the US version which is powered by the Snapdragon 800 SoC from Qualcomm. It is pertinent to note here that the last time Samsung was “caught” doing this, it was the non US, international version of the Galaxy S4 that showed this behaviour. Anyway, as Ars Technica proceeded with the tests, they found that the Note 3 was delivering better than expected benchmarking scores. They also compared its score to that of the LG G2 which also packs in the same processor as that on the Note 3 – but the latter was found at times to be 20% “faster” than the G2 on some benchmarks! How does the same CPU perform differently on two devices? Ars Technica decided to dig deep and figure that out and have presented a deep analysis on how Samsung went about doing this which is better read there.
Now, coming to the moral part of this, I am right there in the middle of the divide. On one side, yes, it amounts to fraud when a manufacturer “fakes” benchmarking results to gain a competitive edge. While most non techie people would not buy a smartphone just based on their benchmarking scores, it should be noted that these scores are part of most detailed reviews on popular websites. Now, a phone that gives a better score is generally considered better – so there is every chance that a customer might be misled to believe that in real world terms the Note 3 is faster than a competing device running the same processor.
On the other hand, isn’t benchmarking all about show casing the potential capacity of a hardware to perform? When did bencmarking become a tool to showcase the “normal” behaviour of a hardware? Remember the PMPO (Peak Momentary Performance Output) wars that we have been so used to when it came to terminologies related to audio? There is no such thing as a PMPO. It was a term used by audio component makers and their copywriters to jazz up the power output of speakers and make it look good on sales brochures. Technically, PMPO was the peak power a set of speakers could handle. The catch is, it doesn’t say for how long it could handle this peak power. Your high powered “10,000 watt” speaker if subjected to 10,000 watts of actual power would run for a few milliseconds before exploding. But hey,10,000- watts PMPO – sounds earth shattering, no? Their actual rating in RMS terms would turn out to be a less than spectacular figure.
Isn’t somewhat similar happening here too? The Qualcomm Snapdragon CAN and DOES run at a higher clock speeds when benchmarks are run – but clocks down to “normal” speeds when apps and games are run- using the potential only when required. So, can Samsung be really questioned for doing this?
Your say!
