A few weeks ago we covered Huawei using a professional DSLR camera in a commercial about their selfie capabilities, and it seems they just can’t stop finding ways to try and trick consumers – this time, its benchmarks they’re manipulating.
Benchmarks are a popular and (until now?) legitimate way of showcasing a device’s performance – whether its CPU, GPU, or system performance benchmarks. The whole point of benchmarks is to give consumers an accurate result of how devices perform under stress.
Unfortunately, it seems Huawei has included a “benchmark detection mechanism” in some of their latest devices. What this mechanism essentially does is push the SoC to a much higher power limit and thermal headroom when it detects that it is being benchmarked in certain apps – thus, resulting in much higher performance scores than users would normally get in daily use. This is not only dishonest, its dangerous for the lifespan of the SoC, and can very quickly overheat a device and reduce its battery life.
Huawei’s defense of this practice can be best summed up as, “everybody else does it too!” – and sadly, they’re actually right. Benchmark cheating is nothing new at all – it was rampant in the PC scene a decade ago, and its been in the mobile scene for a while as well.
The Exynos variant of the Samsung Galaxy S4 was caught doing similar benchmark cheating practices, and the editors for mobile review site AnandTech discovered a large number of other smartphone vendors manipulating benchmark data as well. This led to a number of devices, such as the Galaxy S4, Galaxy Note 3, HTC One, and numerous others being delisted from the 3DMark database.
Though most manufacturers ceased the behavior after being called out, a few continued – for example, OnePlus continued cheating in popular benchmarks such as GeekBench, though has apparently stopped the practice since the OnePlus 5T – which is actually a great device and is highly popular in the modding and development community, so maybe OnePlus felt they didn’t need to manipulate benchmark data.
Sadly, it seems Huawei is picking up where other device vendors have left off, as there are verified reports of Huawei manipulating benchmarks with devices including the Huawei P20, Huawei P20 Pro, and the Honor Play. Basically, the Huawei P20’s performance had regressed when compared to the Huawei Mate 10 Pro – and at the time, Huawei said it was a “firmware issue” – an outright bold lie, as it was actually due to the above-mentioned benchmark detection mechanism, which triggers a vastly superior power limit for the SoC.
Not only is this bad for the SoC itself, as the overall efficiency of the SoC decreases when this is done, because the SoC is being pushed well outside of its normal operating window (imagine overclocking your CPU as high as it can go without any thermal grease). However, this is also bad for the market itself.
For one, it makes the SoC look bad, as if it needs to have the benchmark data manipulated to be worth selling. Second, it casts doubt on the Chinese smartphone market – which is already rife with imitations, clones, and shady practices. Furthermore, Huawei’s benchmark behavior is perhaps the most egregious to be seen yet, as there is truly a massive difference in performance results with the benchmark detection mechanism on compared to when it is off. So much, that AnandTech remarked that the differences in publicly available benchmark results, and their own internal testings, were “absolutely astonishing”.
Huawei has responded to being found out, and as we said earlier, their response was basically “everyone else does it too”. What they literally said (quote from Dr. Wang Chenglu, President of Software at Huawei’s Consumer Business Group) was “others do the same thing, get high scores, and Huawei cannot stay silent. Huawei wants to come together with others in China to find the best verification benchmark for user experience.”
Dr. Wang also pointed at other manufacturers with misleading benchmark numbers, and cited an unnamed popular smartphone manufacturer in China as the biggest culprit. According to Dr. Wang, benchmark cheating has become “common practice in China,” and while Huawei wants to “open up” to consumers, they have trouble when competitors “continually post unrealistic scores.” Huawei is trying to face off against its major Chinese competition, a task which is seemingly made difficult when other device makers produce unrealistic score numbers.
To summarize, Huawei
- Admits they are manipulating benchmark numbers
- Admits everyone else in the Chinese phone market does it
- Has no plans to stop because of #2.
How can anyone trust the Chinese phone market after this kind of stuff? In any case, Huawei did make a statement that the company will “ensure future benchmark data is independently verified by third parties”.
Huawei sent a full response to XDA:
“Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.
When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.
In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to.
Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.”
In any case, true benchmarks ran by AnandTech tested the Huawei P20, P20 Pro, and Honor Play – and all of the devices performed similarly, because they all share the same HiSilicon Kirin 970 SoC – but the real performance of each device is limited by the thermal limits, as each device has a different chassis and cooling design. The Huawei P20 Pro has the best thermals, so it does perform better in a true performance state.
The editor of AnandTech, Andrei Frumusanu, had the following to say:
“In the past we’ve seen vendors actually raise the SoC frequencies, or locking them to their maximum states, raising performance beyond what’s usually available to generic applications. What Huawei instead is doing is boosting benchmark scores by coming at it from the other direction – the benchmarking applications are the only use-cases where the SoC actually performs to its advertised speeds. Meanwhile every other real-world application is throttled to a significant degree below that state due to the thermal limitations of the hardware. What we end up seeing with unthrottled performance is perhaps the ‘true’ form of an unconstrained SoC, although this is completely academic when compared to what users actually experience.”
Their graphs demonstrate that Huawei’s newest phones already reach 3.5-4.4W in their true performance state, while 3.5W TDP is the maximum amount that can be sustained. On the other hand, the phone goes into overdrive with TDP when performing the publicly available benchmarks, with power figures going above 6W and peaking at 8.5W. As noted by AnandTech, these figures quickly trigger an overheating notification on the device, signifying the mismatch of thermal limits with software expectations.
What can be taken from this situation is that the true performance figures aren’t stable as they depend on the phone’s temperature. Huawei doesn’t block the GPU from reaching its peak GPU frequency state. The default behavior is actually a “harsh thermal throttling mechanism […] that will try to maintain significantly lower SoC temperature levels and overall power consumption.”
The phones’ normal mode can reach the same peak power consumption figures during the GPU benchmarks as the ones posted by the unthrottled variants. However, these numbers quickly fall back significantly, and AnandTech notes that the phone throttles down to 2.2W in some cases, which has the effect of significantly reducing performance.
The publication states that the benchmark cheating behavior has seemingly only been introduced in this year’s devices. Phones like the Huawei Mate 9 and the Huawei P10 are not affected, and it seems that only EMUI 8.0 and newer devices are affected. AnandTech was also told by Huawei that this was “purely a software implementation,” which corroborated the publication’s findings.
AnandTech posted GPU performance true figures for the affected Kirin 970-powered phones, and the conclusion is that Huawei is significantly behind its competitors in both GPU performance and efficiency.
The publication also posted power comparison graphs for the Kirin 970 and Kirin 960-powered Huawei devices. The graphs show that Huawei’s power throttling adjustments are in fact better for the user experience as they mitigate the problem of higher power consumption. AnandTech’s testing of the Kirin 960 showed that it had “awful GPU power characteristics,” while the Kirin 970-powered devices have a new strict throttling mechanism to bring down the power consumption and temperatures.
AnandTech also mentioned that the new throttling policy makes sense when considering the fact that both the Kirin 960 and the Kirin 970 show power draws that are much above their sustainable levels for their respective form factors.
Though Huawei hasn’t necessarily done anything wrong by introducing a new throttling mechanism, they’ve certainly crossed into the cheating threshold by the exclusion of popular benchmark applications via a whitelist, which is what is referred to as benchmark cheating in this case.
UL which acquired Futuremark, the company behind PCMark and 3DMark, has delisted the Huawei P20, Huawei P20 Pro, Huawei Nova 3, and the Honor Play from 3DMark.
The company has verified benchmark cheating on the Huawei P20 Pro, Huawei Nova 3, and the Honor Play. On the basis of AnandTech’s testing and reporting, it has also delisted the standard Huawei P20. Users will no longer be able to view the benchmark results of the affected phones as the company does not wish to host cheated benchmark scores.
The company found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app (which is not available to the public), despite the fact that the tests are identical.
In the announcement, UL added that it was happy to see Huawei committing to adopt a more transparent approach in the future. UL’s view is that optional performance modes that can be set by the user are allowed under its current rules as long as they are disabled by default. The company states: “A device must run the benchmark as if it were any other application.”
In conclusion, all Huawei GPU performance benchmarks that were taken using publicly available benchmarks should not be taken as a representation of actual performance.
Huawei reached out to UL to discuss the best practices for benchmark testing, and issued a statement:
“Huawei and UL (creators of 3DMark) have held comprehensive discussions on benchmarking practices this week, and have reached a positive agreement on the next steps in working together.
In the discussion, Huawei explained that its smartphones use an artificial intelligent resource scheduling mechanism. Because different scenarios have different resource needs, the latest Huawei handsets leverage innovative technologies such as artificial intelligence to optimize resource allocation in a way so that the hardware can demonstrate its capabilities to the fullest extent, while fulfilling user demands across all scenarios.
UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.
Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device.
Huawei and UL have also discussed current common benchmark testing methodologies in general. UL and Huawei would like to participate in an industry movement to develop benchmarking standards that best serve the needs of manufacturers, press, and consumers.
To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.”