Posted in: Android, Mobile software

Almost all Android makers cheat on benchmarks

by 207 comments

Cheating on benchmarks – everyone is doing it as it turns out, not just Samsung. AnandTech has done some extensive digging and it turns out that virtually all Android vendors cheat on at least a couple of tests, save for Motorola.

Samsung’s list of benchmarks seems longer than the rest and the company does it across three chipsets – Snapdragon, Exynos and Atom.

Here’s the tablet that AnandTech prepared that shows which device cheats on which test (yes, with that title):

I Can’t Believe I Have to Make This Table
Device SoC Cheats In
    3DM AnTuTu AndEBench Basemark X Geekbench 3 GFXB 2.7 Vellamo
ASUS Padfone Infinity Qualcomm Snapdragon 800 N Y N N N N Y
HTC One Qualcomm Snapdragon 600 Y Y N N N Y Y
HTC One mini Qualcomm Snapdragon 400 Y Y N N N Y Y
LG G2 Qualcomm Snapdragon 800 N Y N N N N Y
Moto RAZR i Intel Atom Z2460 N N N N N N N
Moto X Qualcomm Snapdragon S4 Pro N N N N N N N
Nexus 4 Qualcomm APQ8064 N N N N N N N
Nexus 7 Qualcomm Snapdragon 600 N N N N N N N
Samsung Galaxy S 4 Qualcomm Snapdragon 600 N Y Y N N N Y
Samsung Galaxy Note 3 Qualcomm Snapdragon 800 Y Y Y Y Y N Y
Samsung Galaxy Tab 3 10.1 Intel Atom Z2560 N Y Y N N N N
Samsung Galaxy Note 10.1 (2014 Edition) Samsung Exynos 5420 Y(1.4) Y(1.4) Y(1.4) Y(1.4) Y(1.4) N Y(1.9)
NVIDIA Shield Tegra 4 N N N N N N N

NVIDIA doesn’t make phones but it’s still a pleasant surprise to see the company didn’t boost the Tegra 4 chipset in the Shield handheld console, even though it needs to show a win with the new chipset.

Sony wasn’t included as the company doesn’t send review samples to AnandTech.

Note that this cheating doesn’t actually involve boosting the CPU or GPU speed – all it does is disable the lower clock speeds and sleep modes on all CPU cores. Here’s a graph of what that looks like – when the device detects a benchmark on its list, all CPU cores are locked at their max frequency, while if the benchmark is renamed (so that the device doesn’t detect it), there are natural drops in CPU frequency varying with the load.

There doesn’t seem to be any fiddling going on with the GPU, but he boosted CPU still results in a small (3-10%) boost in scores for 3D benchmarks.

You can read Anand’s extensive article on about benchmark cheats here.

Source

Comments

Rules for posting