Machine Learning Inference Performance

AIMark 3

AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.

鲁大师 / Master Lu - AIMark 3 - InceptionV3 鲁大师 / Master Lu - AIMark 3 - ResNet34 鲁大师 / Master Lu - AIMark 3 - MobileNet-SSD 鲁大师 / Master Lu - AIMark 3 - DeepLabV3

In AIMark, the Oppo Reno 10x ends up quite adequately amongst the Snapdragon 855 devices who come shipping with the corresponding Qualcomm libraries to make the benchmark work. We do note that other S855 phones have a slight edge here, likely due to their more optimised software stack.

AIBenchmark 3

AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.

We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.

AIBenchmark 3 - NNAPI CPU

We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.

AIBenchmark 3 - 1 - The Life - CPU/FP AIBenchmark 3 - 2 - Zoo - CPU/FP AIBenchmark 3 - 3 - Pioneers - CPU/INT AIBenchmark 3 - 4 - Let's Play - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/INT AIBenchmark 3 - 8 - Blur iT! - CPU/FP

In the CPU accelerated workloads, we see that the Oppo Reno 10x absolutely stands out compared to all other Snapdragon devices, having nothing in common with the rest of the pack in terms of the resulting performance figures. The phone ends up with one performance regression, one test that didn’t complete, but otherwise the performance looks to actually be better than that of what’s available on other Snapdragon 855 phones.

The only explanation that I can offer here is that it looks like rather than using Qualcomm’s NN driver stack, the phone is falling back to the default Android Tensorflow libraries. The interesting thing here is indeed that the phone looks to be performing better with these libraries than that the ones that are shipping with the Qualcomm BSP, something the company should certainly look into.

AIBenchmark 3 - NNAPI INT8

AIBenchmark 3 - 1 - The Life - INT8 AIBenchmark 3 - 2 - Zoo - Int8 AIBenchmark 3 - 3 - Pioneers - INT8 AIBenchmark 3 - 5 - Masterpiece - INT8 AIBenchmark 3 - 6 - Cartoons - INT8

AIBenchmark 3 - NNAPI FP16

AIBenchmark 3 - 1 - The Life - FP16 AIBenchmark 3 - 2 - Zoo - FP16 AIBenchmark 3 - 3 - Pioneers - FP16 AIBenchmark 3 - 5 - Masterpiece - FP16 AIBenchmark 3 - 6 - Cartoons - FP16 AIBenchmark 3 - 9 - Berlin Driving - FP16 AIBenchmark 3 - 10 - WESPE-dn - FP16

AIBenchmark 3 - NNAPI FP32

AIBenchmark 3 - 10 - WESPE-dn - FP32

Unfortunately for the rest of the AI Benchmark tests which make use of NNAPI acceleration for INT8, FP16 and FP32 models, the Reno 10x just didn’t have the correct drivers and thus wasn’t able to perform the tests.

If there’s one thing that’s worse than bad performance, it’s not able to perform a task at all. I’m not sure if the Reno 10x performs any differently in a global firmware version of if this issue is solely a characteristic of the Chinese variant. In any case, I find it quite disappointing for the machine learning ecosystem as Oppo is a major vendor, and the Reno is its flagship device. How are application developers supposed to embrace machine learning when we the software situation still has such enormous draw-backs?

System Performance GPU Performance
POST A COMMENT

40 Comments

View All Comments

  • dullard - Wednesday, September 18, 2019 - link

    I don't think I've ever thought about or ever noticed phone weights. Size and bulk? Yes. Weight? Never. Reply
  • MarcusMo - Wednesday, September 18, 2019 - link

    It’s one of these things that you don’t notice until you pick up a phone that passes the point of being too heavy.
    I agree with the op, I hate the feeling of having a brick in my jacket pocket.
    Reply
  • wr3zzz - Wednesday, September 18, 2019 - link

    That's a good point. If I still had to wear a suit everyday then the weight would be problematic.

    For those that don't have to wear suits, I actually like the slightly heavier phones, though anything more than 200g is still too much even for jean pockets.
    Reply
  • Calista - Wednesday, September 18, 2019 - link

    Fuly agree, I find my Note 8 at 195 gram too heavy. I never felt the same in regard to my S7 Edge at 157 gram. And I find the Nokia 3310 at 85 gram a bit too flimsy instead. Depending on the individual of course, but to me a weight of 120-140 seem perfect. The S4 for example is 130 gram. Not feeling flimsy, still very thin and light. Reply
  • eek2121 - Wednesday, September 18, 2019 - link

    The entire design trend of phones needs a rethink. At least one major OEM should put things in reverse and do things like a *plastic* or *metal* back, bezels on the top and bottom (with stereo speakers for the occasional time when you have to use your phones sound), and a SLIGHTLY smaller screen. As far as a headphone jack, I'd rather just see a dongle in the box TBH. The whole headphone jack thing is overblown. Pumping audio through the USB-Type C port means you get a clean digital signal, making it possible to do things like optical out, etc. Combine that with a built in amp on the phone (along with the 'quad DAC' that some LG phones have) and you get a nice clean signal that can either be converted to a headphone jack or hooked up to a soundbar or surround sound system.

    I'd also like to see micro SD cards return and have a storage manager built into the system that combines internal and external storage. Encrypt it if you want, but as quickly as prices have fallen (I just bought 4 128 GB microSD cards for $15...and they are REAL. Ran a tester on them.), External storage will be the way to go when you run out of space. I'd also like to see an AOSP style release without vendor junkware.

    The problem is I wouldn't pay a premium for those features. Such a phone would be worth about $499 if it had 4 GB of RAM and a Snapdragon 855 combined with a 2560x1440 AMOLED screen.
    Reply
  • Tams80 - Wednesday, September 18, 2019 - link

    I agree with a lot of that, but definitely not the headphone jack port. It should stay, as as a connector it is more durable when in use because it can rotate. We're finding out that many USB C ports are not that much better than microUSB ports in terms of durability.
    Besides, all headphones end up analogue.

    It's not like the USB C port would suddenly lose audio transmission capability.
    Reply
  • danielfranklin - Wednesday, September 18, 2019 - link

    " I want all my groceries on one bag, but i dont want that bag to be heavy! " Reply
  • Tams80 - Wednesday, September 18, 2019 - link

    They could easily dispense of some weight by using plastic. Reply
  • drexnx - Wednesday, September 18, 2019 - link

    all that heftiness and chunk and they still couldn't find room for a 3.5mm jack? Reply
  • ToTTenTranz - Wednesday, September 18, 2019 - link

    "Low-light Conclusion - Terrible

    (...)

    Overall, the Reno 10x does adequately in low-light, which is something great to see as it means that future devices from the vendor will only continue to iterate and improve upon the current results."

    So.. um.. which is it?
    Reply

Log in

Don't have an account? Sign up now