Machine Learning Inference Performance

AIMark 3

AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.

鲁大师 / Master Lu - AIMark 3 - InceptionV3 鲁大师 / Master Lu - AIMark 3 - ResNet34 鲁大师 / Master Lu - AIMark 3 - MobileNet-SSD 鲁大师 / Master Lu - AIMark 3 - DeepLabV3

Unfortunately for the Black Shark 2, the devices lacked the proper drivers to properly run AIMark, and the benchmark repeatedly crashed upon starting the benchmarks. We had the same issue on the OnePlus 7 Pro, pointing out to some software incompatibility.

AIBenchmark 3

AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.

We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.

AIBenchmark 3 - NNAPI CPU

We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.

AIBenchmark 3 - 1 - The Life - CPU/FP AIBenchmark 3 - 2 - Zoo - CPU/FP AIBenchmark 3 - 3 - Pioneers - CPU/INT AIBenchmark 3 - 4 - Let's Play - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/INT AIBenchmark 3 - 8 - Blur iT! - CPU/FP

In AI Benchmark’s CPU workloads, the Black Shark 2 ends up with a bit of a odd spread of scores. In the shorter running benchmarks the phone is getting relatively average inference times, while on the longer running tests for some reason the BS2 falls behind other S855 devices. In fact it looks like the BS2 is landing as amongst the worse off S855 devices in the latter listed tests.

AIBenchmark 3 - NNAPI INT8

AIBenchmark 3 - 1 - The Life - INT8 AIBenchmark 3 - 2 - Zoo - Int8 AIBenchmark 3 - 3 - Pioneers - INT8 AIBenchmark 3 - 5 - Masterpiece - INT8 AIBenchmark 3 - 6 - Cartoons - INT8

AIBenchmark 3 - NNAPI FP16

AIBenchmark 3 - 1 - The Life - FP16 AIBenchmark 3 - 2 - Zoo - FP16 AIBenchmark 3 - 3 - Pioneers - FP16 AIBenchmark 3 - 5 - Masterpiece - FP16 AIBenchmark 3 - 6 - Cartoons - FP16 AIBenchmark 3 - 9 - Berlin Driving - FP16 AIBenchmark 3 - 10 - WESPE-dn - FP16

AIBenchmark 3 - NNAPI FP32

AIBenchmark 3 - 10 - WESPE-dn - FP32

In the INT8, FP16 and FP32 accelerated tests which make use of acceleration blocks such as the Hexagon DSP and the GPU, we see the Black Shark 2 perform very well and in line with other Snapdragon 855 devices.

Overall, the Black Shark 2 is a good performer in the machine learning inferencing benchmarks, but like other devices, it’s not quite the very best in every regard, pointing out that the vendor could have improved upon its performance by keeping the software stack more up to date with what Qualcomm is offering, a widespread issue that I expect to persist over the next years as the ecosystem quickly evolves.

System Performance GPU Performance - Worst of S855
POST A COMMENT

63 Comments

View All Comments

  • s.yu - Friday, September 27, 2019 - link

    No I checked and only the top tier is available in the US, which explains the price there.
    Though I haven't been keeping track of the base tier price elsewhere.
    Reply
  • Tams80 - Thursday, September 26, 2019 - link

    The ROG Phone II at least has a lot more going for it. It has specs that are actually nice to have outside of gaming. A bit like a top-end PC build that isn't all 'gaming' flash and you can enjoy your films in HDR on, FLAC files on, edit videos on, etc. Reply
  • masteiffy - Wednesday, September 25, 2019 - link

    Hello Andrei, when are we getting the new iPhone reviews, I’ve been waiting for the review to decide which to buy. Thanks! Reply
  • Andrei Frumusanu - Wednesday, September 25, 2019 - link

    Next week. Reply
  • imaheadcase - Wednesday, September 25, 2019 - link

    I don't think i know anyone who games on a phone outside of simple games that don't require anything but a low end smartphone. We don't need gaming phones. Reply
  • StormyParis - Wednesday, September 25, 2019 - link

    You're old ;-p and not old enough to have (pre-)teen kids ! Reply
  • PeachNCream - Wednesday, September 25, 2019 - link

    Playing video games is more than just parking it at a desk behind a 1990's-style box. At least 50% of my gaming is now on a phone and some of the highest grossing titles are mobile ones. Reply
  • Wardrive86 - Wednesday, September 25, 2019 - link

    There is a lot of really high end games on android now (Shadowgun, Ark, PUBG, CoD) high end emulation (Dolphin @1080p+, Citra) To each their own though, personally I do 100% of my gaming on a phone. Reply
  • s.yu - Thursday, September 26, 2019 - link

    Sometimes gaming on a phone is fine, but sometimes ports don't work as intended, for example some achievements of RPGs become near impossible on a touchscreen as opposed to a keyboard. Reply
  • PeachNCream - Thursday, September 26, 2019 - link

    I don't think the attainment of an achievement would move the bar at all on what platform I use to play a game. Those things never really struck me as very important so I admit that I'm surprised someone would even mention them as a factor that influences their decision. Reply

Log in

Don't have an account? Sign up now