Pianoteq Benchmark Feature Suggestion

That would be amazing! And to bring it to perfection, a platform, where you can upload your results and share it with others, so everyone can compare and see what best fits his needs.
Concerning those Intel NUCs, they are very interesting pieces of technology! jarosujo has set up a headless machine with the DC3217BY (Ivy Bridge processor generation): http://www.forum-pianoteq.com/viewtopic.php?pid=930717 This one is around 150€. It does not have an integrated 3.5mm phone connector output, but I wouldn’t rely on that output anyway. A USB sound card for ~50-100€ (or maybe even less?) pretty certainly does a better job.
But to come back to topic, such a benchmarking system/platform would be a great thing!

Thanks for pointing that out!
I don’t know, whether it indeed would be that expensive – the developers would probably know.
But I suppose, the aim of this thread is to get to know, whether others users would at all be interested in such a feature.

Here's a simple script I made for Linux to produce a benchmark.  Those of you with Windows scripting chops can figure out how to do this under Windows.

#!/usr/bin/perl

# copy this script to your PTQ root directory
# (e.g. below the AMD64/i386 directories)

# File to use:
$FILE = "He_s a Pirate.mid";

# which version to use?
$VER = "./amd64/Pianoteq\\ 4";
#$VER = "./i386/Pianoteq\\ 4";

# start time
$START = time;

# run
system("$VER --play-and-quit --headless --no-audio-prefs --preset \"U4 Daily Practice\" --midi \"$FILE\" --wav /tmp/_delete_me.tmp");

# end time
$END = time;

# remove temp file
unlink("/tmp/_delete_me.tmp");

# calculate score
# since we like higher numbers, we will invert
$TIME = $END - $START;
$SCORE = int(10000/$TIME);

print "The PTQ score for this system is $SCORE\n";

I did not use the idea of monitoring idle time.  I thought about that for a while and realized under Intel CPUs with hyperthreading, you don't know whether it's a real core that's available or a virtual core.  However, the output to wave option does not run using "playing time" so it probably uses as much resources as possible -- hence you can take NUMBER/time as a simple score.

Here are numbers for some machines I have access to.   EDIT: now includes a few desktop machines.  In the 1-core VM tests, PTQ does not seem to comprehend virtualized CPUs and runs even slower when more CPUs are allocated.

i5 M430 (Nehalem-C) @ 2.2ghz 64-bit: 500
i5 2500 (Sandy Bridge 1-core VM) 3.3ghz 64-bit: 476
i5 M430 (Nehalem-C) @ 2.2ghz 32-bit: 476
AMD FX-8120 (Bulldozer) 3.1ghz 64-bit: 434
i3 3227U (Ivy Bridge) @ 1.9ghz 32-bit: 416
AMD Phenom II (Thuban 1-core VM) 3.2ghz 64-bit: 370
AMD Phenom II (Deneb 1-core VM) 3.0ghz 64-bit: 322
i5 M430 (Nehalem-C) @ 1.2ghz 64-bit: 285
i5 M430 (Nehalem-C) @ 1.2ghz 32-bit: 256
Pentium T2300 (Core 2) @ 1.5ghz 32-bit: 217
i3 3227U (Ivy Bridge) @ 800mhz 32-bit: 192
Pentium T2300 (Core 2) @ 800mhz 32-bit: 133
Intel Atom N450 (Pineview) @ 1.66ghz 32-bit: 94

The 250ish threshold for PTQ4 the rough requirement for 200+ polyphony.  The T2300 (~200 score) garbles notes under more complex playback.

---

Oh, and here's a score you guys will be really interested in:

Intel Atom C2750 @ 2.4ghz 64-bit: 333
Intel Atom C2750 @ 1.2ghz 64-bit: 178

What's this?  This is the 8-core server version of the latest Intel Atom CPU.  The only difference is this has 8-cores (using it at work to run firewall VMs) while consumer versions have 2/4 cores.  (Pianoteq only uses 2 cores maximum from monitoring cpu usage.)

So you if want an ultra low power CPU, a Bay Trail NUC/Chromebook would draw 7.5W under full load.  This is a SoC number ... non-SoC CPUs add another 10W-15W because the northbridge/southbridge also draws power.  (See a recent Anandtech review where 17W non-SoC laptops draw 35Wish compared to AMD Kabini SoC laptop drawing UNDER power rating.)  The 333 score puts it solidly above the ~250 requirement ... hence I'd wait to see when the ASUS Baytrail Chromebooks are coming out.  They'll supposedly be available for $200-$250 (USD, no idea about Europe pricing) and at that kind of power usage, you can leave it on all the time -- sleep or no sleep.

Very good idea of creating a database of benchmark scores! Pianoteq should then simply offer to upload the result of the benchmark to their servers after completion. A simple list of CPU's and average score on some page somewhere on the site would be sufficient. Pianoteq seems to me like a well-written and well-optimized application, and most modern CPU's can handle it with ease. However, these low-power systems will make this sort of information valueable again, as some balance right on the edge!

Indeed, jarosujo's review was very helpful! I believe that price was for a second hand unit, all in all you'd probably end up spending more. I can see the use for a good external soundcard if you're into recording or studio work, but for a lot of people modern on-board sound is good enough. It would be for me! It would also help keep costs down. I'm running Pianoteq on two different Realtek on-board sound interfaces now, the latency is low, sound quality good, and no interference. I'm not an audiophile though, and don't see the need for 24 bit 192 kHz etc., especially not for daily practice.
I'm hoping this will make it to market, would be pretty great to strap onto the VPC1 for example...

You're right, everything has a cost. However I don't think it would increase the price of the product by $50. Most of the functionality you'd need is already built into Pianoteq, Modartt would simply need to arrange some specific benchmark routine of existing functions. I would wager a simple alpha version to iterate on could be whipped up in an afternoon. For all we know they already have this in non-production code!
There's a performance index in the options for example. I didn't find any documentation on it, but it could surely be utilised.
Furthermore I don't think it's a good attitude to keep from suggesting features fearing the implicated cost, and encourage others to hold back. Modartt don't have to listen to us, but they might if the things we suggest make sense. Let's leave the business decisions up to them. Keeping the product minimal and hacker friendly seems important to them, I don't think we need to fear feature creep and bloat either.

Great work Mossy! I had no idea Pianoteq had a command line interface like that. I don't have the CLI skills to whip up the same script for Windows, but I did give the command a spin, very interesting.

I'm not sure how well it reflects actual live performance though? Perhaps it's the same thing, I don't know. Running it at different sample rates would also be important I think, but maybe that's possible by loading different --prefs files. And of course polyphony as you mentioned.
It's a bit curious that Pianoteq would garble notes when just producing a wav file, I thought it would take all the time it needed and to produce perfect output on any system. Or perhaps you meant that it does that when playing the midi in real-time on the low-scoring systems?

You sure have a lot of computers! Very useful benchmarks, especially the last ones. Thanks for taking the time to investigate this! The CPU world is pretty confusing, and I'm certainly no expert. Only getting interested now because I want to play the piano with the minimum amount of hassle, power consumption and fan noise.

Yes I'm looking at general benchmarks of that sort, but like you say, it would be useful to really know what you're getting. A Pianoteq specific benchmark would be nice. Thanks for the tip though and for your contributions to the community!

Although this kind of (crude) benchmarking is currently possible, my suggestion still stands. I think it would be very convenient to have a benchmark like this built into Pianoteq, preferably playing live so the user can monitor the sound for defects, together with a database of user-submitted scores. My humble opinion!

Garbled notes when playing realtime.

I originally thought of using different prefs before examining the possible command line options.  If PTQ generated in playback time, we would have to use different preferences/ polyphony in order to measure amount of free CPU available for different scenarios.  But because PTQ uses all the CPU power it can to generate a WAV as quick as possible (different time needed for different computers), the time spent on a complex MIDI is a direct inverse of the CPU power available to run PTQ.

This is my guess on how to run do the above benchmark under Windows by hand:

"Pianoteq 4.exe" --play-and-quit --headless --no-audio-prefs --preset "U4 Daily Practice" --midi "He_s a Pirate.mid" --wav C:\_tmp_delete_me.wav

Use a stopwatch (or smartphone stopwatch wapp) to time how long it takes to run.  Then calculate:

SCORE = 10000 / (time to run)

In order to get a "big number is better" benchmark.

That is a good enough test - on a computer you already own. What I'm proposing would help when buying new equipment.