Samsung wasn’t alone in cheating on Android benchmark scores; it appears that almost everyone is doing it. Does that make it OK?
Following up from yesterday’s excellent Ars Technica article, Anandtech has a rundown on Android benchmarks, noting that with the exception of Google itself and possibly Motorola, everyone’s tweaking in order to get a few more benchmark points.
Ouch.
The pain point here really shouldn’t be the numbers, because the reality is that the numbers given back by the benchmarks do represent what the individual cores can actually do.
It’s just that it’s representing that data as ordinary usage, whereas it’s absolute peak usage, and if you ran a phone or tablet like that, you’d chop huge chunks out of the battery life. Nobody wants that.
If everyone’s doing it, though, it essentially means that those benchmarks become significantly less useful as a comparative tool, which is largely what I’ve been doing for years anyway.
A benchmark should never be the be-all and end-all of assessing a device, because the actual performance and experience of using that device is what you’ll be faced with day by day. That’s why I tend to write as I do around benchmarks, using them sparingly for comparison, rather than as the only source of useful tech information.
Kudos to Google, though, for not entering into the whole silly race; yet another reason, I’d say, to invest in Nexus hardware if Android is your mobile operating system of choice.
Source: Anandtech
What they need is full disclosure about what is happening. Lying and cheating is not the way to handle this. What they should do is say “This is what you would expect under normal use” but that “this is what the device is capable of if made to work at full tilt”. Both benchmarks are indicative of the ability the smartphone has but in different scenarios.