CPU Tests: Legacy and Web

In order to gather data to compare with older benchmarks, we are still keeping a number of tests under our ‘legacy’ section. This includes all the former major versions of CineBench (R15, R11.5, R10) as well as x264 HD 3.0 and the first very naïve version of 3DPM v2.1. We won’t be transferring the data over from the old testing into Bench, otherwise it would be populated with 200 CPUs with only one data point, so it will fill up as we test more CPUs like the others.

The other section here is our web tests.

Web Tests: Kraken, Octane, and Speedometer

Benchmarking using web tools is always a bit difficult. Browsers change almost daily, and the way the web is used changes even quicker. While there is some scope for advanced computational based benchmarks, most users care about responsiveness, which requires a strong back-end to work quickly to provide on the front-end. The benchmarks we chose for our web tests are essentially industry standards – at least once upon a time.

It should be noted that for each test, the browser is closed and re-opened a new with a fresh cache. We use a fixed Chromium version for our tests with the update capabilities removed to ensure consistency.

Mozilla Kraken 1.1

Kraken is a 2010 benchmark from Mozilla and does a series of JavaScript tests. These tests are a little more involved than previous tests, looking at artificial intelligence, audio manipulation, image manipulation, json parsing, and cryptographic functions. The benchmark starts with an initial download of data for the audio and imaging, and then runs through 10 times giving a timed result.

We loop through the 10-run test four times (so that’s a total of 40 runs), and average the four end-results. The result is given as time to complete the test, and we’re reaching a slow asymptotic limit with regards the highest IPC processors.

(7-1) Kraken 1.1 Web Test

Google Octane 2.0

Our second test is also JavaScript based, but uses a lot more variation of newer JS techniques, such as object-oriented programming, kernel simulation, object creation/destruction, garbage collection, array manipulations, compiler latency and code execution.

Octane was developed after the discontinuation of other tests, with the goal of being more web-like than previous tests. It has been a popular benchmark, making it an obvious target for optimizations in the JavaScript engines. Ultimately it was retired in early 2017 due to this, although it is still widely used as a tool to determine general CPU performance in a number of web tasks.

(7-2) Google Octane 2.0 Web Test

Speedometer 2: JavaScript Frameworks

Our newest web test is Speedometer 2, which is a test over a series of JavaScript frameworks to do three simple things: built a list, enable each item in the list, and remove the list. All the frameworks implement the same visual cues, but obviously apply them from different coding angles.

Our test goes through the list of frameworks, and produces a final score indicative of ‘rpm’, one of the benchmarks internal metrics.

We repeat over the benchmark for a dozen loops, taking the average of the last five.

(7-3) Speedometer 2.0 Web Test

Legacy Tests

(6-5a) x264 HD 3.0 Pass 1(6-5b) x264 HD 3.0 Pass 2(6-4a) 3DPM v1 ST(6-4b) 3DPM v1 MT(6-3a) CineBench R15 ST(6-3b) CineBench R15 MT

CPU Tests: Encoding CPU Tests: Synthetic and SPEC
Comments Locked

210 Comments

View All Comments

  • dullard - Thursday, January 21, 2021 - link

    Yes, it is sad that even well respected PhDs in the field can't seem to understand that TDP is not total consumed power. Never has been, never will be. TDP is simply the minimum power to design your cooling system around.

    I actually think that Intel went in the right direction with Tiger Lake. It will do everyone a service to drop any mention of TDP solely into the fine print of tech documents because so many people misunderstand it.

    Yes, TSMC has a fantastic node right now with lower power that AMD is making good use of. Yes, that makes Intel look bad. Lets clearly state that fact and move on.

    Power usage matters for mobile (battery life), servers (cooling requirements and energy costs), and the mining fad (profits). Power usage does not matter to most desktop users.
  • dullard - Thursday, January 21, 2021 - link

    Also don't forget that we are talking about 12 seconds or 28 seconds of more power, then it drops back down unless the motherboard manufacturer overrides it. The costs to desktop users for those few seconds is fractions of a penny.
  • bji - Thursday, January 21, 2021 - link

    "minimum power to design your cooling system around" makes NO SENSE.

    You don't design any cooling system to handle the "minimum", you design it to handle the "maximum".

    It sounds like you've bought into Intel's convoluted logic for justifying their meaningless TDP ratings?
  • iphonebestgamephone - Thursday, January 21, 2021 - link

    Why are there low end and high end coolers then? Arent the cheap ones for the minimum, in this case 65w?
  • Spunjji - Friday, January 22, 2021 - link

    dullard's comments are, indeed, a post-hoc justification in search of an audience.
  • dullard - Friday, January 22, 2021 - link

    Bji, no, that is not how how engineering works. You need to know the failure limit on the minimum side. If your cooling system cannot consistently cool at least 65W, then your product will fail to meet specifications. That is a very important number for a system designer. Make a 60W cooling system around the 10700 chip and you'll have a disaster.

    You can always cool more than 65W and have more and/or faster turbos. There is no upper limit to how much cooling capability you can use. A 65W cooler will work, a 125W cooler will work, a 5000 W cooler will work. All you get with better cooling is more turbo, more often. That is a selling point, but that is it - a selling point. It is the the 65W number that is the critical design requirement to avoid failures.
  • edzieba - Friday, January 22, 2021 - link

    Minor correction on " Never has been, never will be": TDP and peak package power draw WERE synonymous once, for consumer CPUs, back when a CPU just ran at a single fixed frequency all the time. It's not been true for a very long time, but now persists as a 'widely believed fact'.
    Something being true only in very specific scenarios but being applied generally out of ignorance is pretty common in the 'enthusiast' world: RAM heatsinks (if you're not running DDR2 FBDIMMs they're purely decorative), m.2 heatsinks (cooling the NAND dies is actively harmful, cooling the controller was only necessary for a single model of OEM-only brown-box Samsung drives because nobody had the tool to tell the controller to not run at max power all the time), hugely oversized WC radiators (from the days when rad area was calculated assuming repurposed low-density-high-flow car AC radiators, not current high-density-low-flow radiators), etc.
    Even now "more cores = more better" in the consumer market, despite very few consumer-facing workloads spanning more than a handful of threads (and rarely maxing out more than a single core).
  • dullard - Friday, January 22, 2021 - link

    I'll give you credit there. I should have said "not since turbo" instead of "never has been". Good catch. I wish there was an edit button.
  • Spunjji - Friday, January 22, 2021 - link

    What's really sad is that you apparently prefer to write a long comment trying to dunk on the author, rather than read the article he wrote for you to enjoy *for free*.

    "I actually think that Intel went in the right direction with Tiger Lake"
    You think poorly.

    "Yes, TSMC has a fantastic node right now with lower power that AMD is making good use of. Yes, that makes Intel look bad. Lets clearly state that fact and move on."
    Aaaand there's the motivation for the sour grapes.
  • dullard - Friday, January 22, 2021 - link

    Spunjji, I must assume since you didn't have anything to actually refute what I said, that you have nothing to refute it and instead choose to bash the messenger. Thanks for backing me up!

Log in

Don't have an account? Sign up now