iOS / iPhone and Android benchmark charts

[LilGuyLee]

Hi,

Apologies now if I’m missing something obvious but I'm a bit confused about the relative scores between the Android system and the IoS. I've been looking at your mobile CPU benchmarks, while trying to get a feeling for the differences between the various system on chip solutions out there for work, and something seems to be inflating the IoS numbers.

For example, compare the “Apple iPad 4 (WiFi)” with the “Samsung Nexus 10”:
- Ipad memory is DDR2 compared to Nexus DDR3. The relative scores of the memory (2848 vs 3477) seem to be in line with expectations.
- Both devices use ARM A15s (if Wikipedia for the A6X specs and Exynos 5250can be believed) with the IPad clocked at 1.4GHz as compared to the Nexus at 1.7GHz.
On the face of it, I'd expect the Nexus to provide CPU scores of 1.1 to 1.5 times that of of the Ipad (memory and clock speed), but the CPU scores are 7,976 for the Nexus and 26,430(?!) for the IPad.

Am I missing something? Everything seems to imply the mobile benchmark is comparable between Apple/Android but that's counter intuitive to the result of 3 times more in favour of the A6x even though it has worse memory results and a slower clock speed.

Obviously the OS is different, but are there fundamental differences in the code of the two (one mainly Java the other mainly ARM native)? Are the tests being skewed by OS update calls to the screen, or background GPU rendering cycle stealing CPU memory bandwidth or something?

As I said, I’m confused…

[David (PassMark)]

The use of Java (the JVM) slows down the Android devices compared to using native code on the iPhone. The improvement can vary, but x2 to x3 speed improvements can be had with native code.

The real killer however is the performance doing encryption. On iOS (V4.3 and above) AES encryption, the industry standard, is hardware accelerated. This is the same as on modern PCs. So the iPhone gets 20x the performance of Android.

Android Nexus 10 - AES Encryption 4,579 KB/Sec.
iPad4 - AES Encryption 101,125 KB/Sec

These two items combined means iPad / IPhone scores way better.

[LilGuyLee]

David,

Thanks for the reply. Confusion cleared up. I was worried that there was something non-obvious in the way SoCs were put together which could cause major performance issues.

I’d actually pre-empted your reply, grabbed a Galaxy S2 from someone, installed your app then checked out what dev kits you’d used with “Addons Detector”. The lack of any dev kit in there, including NDK, gave it away really (although I didn’t know that about the AES).

As a person who may well have used your mobile benchmarks (I use your site at home lots for comparing PC gaming components) to compare the above two devices, and then was gutted when my shiny new IPad 4 was beaten on frame rate by a friends cheaper Nexus 10 (and don’t they take great pleasure in rubbing your face in it…)

Can I make a suggestion? Rename the current CPU Benchmark to “Java VM Benchmark” and add a new “CPU Benchmark” which is raw NDK processor performance. Also, if the 3d benchmark portion is run from the VM, change it to run under the NDK. Almost every 3d app for Android uses the NDK environment as it’s interface. Really, if I was using the benchmarks I’d want them to reflect actual the performance I’m likely to see, especially if I were trying to make a choice between platforms or jumping ship to the alternative platform.

I’m not sure what to suggest about the AES, I’m a bit shocked that the Android platform doesn’t use hardware encryption/decryption if it’s available. Perhaps remove it from the NDK CPU performance test (how often is it used in games?) - though I'd be tempted to leave it in just to push Android in to supporting it.

Cheers,
Lee

[David (PassMark)]

The aim (for us) in doing this wasn't to make Android look better or worse than the iPad.
So we aren't going to make a collection of changes to make Android come out on top.

Most people aren't going to know what a Java VM is. Then we would tell them it is the standard way that Android applications are developed, to which is their response would of course be, then why don't you just call it an Android benchmark.'

From the android.com site,
"...most apps do not need the Android NDK"
" In general, you should only use the NDK if it is essential to your app"

The main problems we saw with the NDK was that,
1) It wasn't the recommended development method.
2) You introduce dependencies on the CPU architecture. So your native code might run on ARM, but not on Android devices with Intel chips. We wanted to run the benchmark on every current and future Android device.

My opinion is that picking Java as the development standard was a dumb idea for a device where many of the tasks are very performance critical and CPU power is limited. But that is what they did.

[LilGuyLee]

I think you may have picked up on the wrong motives, I didn’t suggest them to make the one look worse or better than the other. I was just suggesting a way for the mobile benchmark to be more comparable to the two.

With respect to the CPU benchmarks, I agree most people may not know, and most likely will not care about if it’s a VM or native. But then, the headline description of the benchmark could describe it as “mainly used for general purpose applications” and “mainly used to for 3d or highly graphics intensive applications”.

While I understand the compatibility problems when the new Intel architecture arrives, I think we’ll have to agree to disagree on the NDK argument. To me, it’s analogous to the PC, you don’t need an OpenGL or DirectX interface to implement 99% of the (non-game) PC applications on the market, but when you’re testing a PC graphics card you make sure the environment is as close as possible to that a game would use, and you talk to the hardware via the performance interface.

The Android system, from what I can tell, is the same; for 99% of the applications on the market you don’t need the NDK. However it’s provided for those performance applications, like 3d games, which do. I would have thought, if you’re providing a 3d benchmark, you would want it to be representative of the interface which almost all the 3d applications would use? From what I’ve encountered, as a side line of trying to understand the relative performances of SoC solutions, almost every single 3d app in android uses the NDK as it’s interface to the GPU.

[David (PassMark)]

Your argument that NDK is analogous to Directx ignores that fact that NDK isn't required at all -for anything. Where as DirectX is the only option from Microsoft for 3D. A better argument could be made around 2D programming. As you could use DirectX for 2D (e.g. using DirectWrite), but you would only do it under exceptional circumstances.

Secondly we don't hand code low level 3D operations in the JavaVM. We use OpenGL to do most of the 3D work. So the 3D speed should be dependant on the hardware and the OpenGL implementation on the platform. So going native won't help much for 3D rendering performance. (it might help in other game related areas like physics, AI & path finding however, but we aren't doing any of this in the 3D test).