Intel Bay Trail fully supported?

While Intel greatly improved the floating point performance for Bay Trail (Pianoteq, games, media encoding) over previous generation Atoms, it still is dog slow for integer work.  We bought an 8-core 2.6ghz Bay Trail to replace a 1.6ghz Core 2 Duo for network services & VM hosting ... and it turned out to be twice as slow.  So using a RT kernel with Bay Trail in effect continually interrupts relatively fast floating point operations for Pianoteq with dog slow task management integer operations.

If no other CPU intensive app runs while Pianoteq is, there's no purpose of using a RT kernel.  It would be better to use a regular kernel and increase the priority of Pianoteq so no other process interrupts it.

And what do you think is doing the "task management"? Exactly. In other words: the kernel is the most central part of a OS. It is the thing that makes multi-tasking possible, and it is actually the thing that implements it. The kernel decides when and if a task is run. It has the ultimate power. User-space processes can only ask nicely.

Realtime isn't about efficiency. It isn't even about lower latencies. It is about predictability. This means that in a realtime operating system an application gets the same slices of CPU time in a very regular spacing: you can therefore always calculate timing from CPU cycles of the program, at least up to a guaranteed precision. In other words: a realtime environment "promises" your program to run it very regularly and without waiting periods outside certain boundaries.

A modern multitasking operating system, like Linux, Windows or Mac OS, is no realtime system. That means that the kernel (sic!) can decide to make an application wait for a longer time when something 'more important' has come up, e.g. an IO operation. It also will usually lower an application's priority when it hogs much CPU time, so that background processes at least get a chance. If an application finds that it has nothing more to do, it can tell the kernel 'going to sleep now' and the kernel will immediately reschedule the allotted CPU cycles to other applications. This is called 'preemption'.

All these tricks improve efficiency and throughput. They come, however, at a price, and that is occasional latency. Usually things will be very snappy, but from time to time an application might even 'hang' for maybe 50 or 100ms. (Such long latencies are rather unusual, however.) The user will normally not notice this, so it is a good bargain. Now when you want to do audio processing with exceedingly low latencies, things get tricky. This is the reason you can configure buffer sizes in Pianoteq: there is a certain amount of 'jitter' in the scheduling, and the application has to compensate for it by buffering. This increases latency and decreases load. The smaller the buffer size is, the more often the application has to do IO and the more often the OS has to do switching, effectively increasing the load. And when buffers are too small, you get the telltale audio dropouts.

The Linux RT patches as found e.g. in Ubuntu first and foremost make the Linux kernel itself preemptible. That is, the kernel can actually decide to stop an operation and give control back to applications. This makes things more "even" in terms of timing, improving realtime behaviour (although the Linux kernel will probably never be able to fulfill 'hard' realtime constraints). Another part of the patch is called the low-latency patch which more or less chances the scheduling behaviour: it runs more often.

All the changes in RT kernels mean than the kernel switches more often between applications (and itself), and more evenly. This smoothes timing, making latency spikes rarer and smaller. However, it has a price: the efficiency of that kernel is... bad. Very bad. Which means: if you have lots of computing power to spare and are not IO-limited in what you want to do (and audio processing on a fast computer meets these criteria), you can drastically reduce the minimal latency, although you will pay the price in terms of decreased efficiency and thus increased CPU load.

On a slower system running only a single application however, the RT patch is therefore actually detrimental, as Mossy described. Your best bet in that scenario is: run a normal kernel. Maybe increase buffer size a bit. Live with slightly higher latency. Don't run anything in the background.

Hi,

it was a quote, and obviously I misunderstood it. I meant all system-activities and running processes around the running Pianoteq should be quite the same for both kernels. I have no network activities, no virtual machines (VM), no significant disk I/O (and when, it is a SSD) and no "integer operations"(that I am aware of), that should hog Pianoteq so drastic, as observed. What integer operations are meant? Can I stop them somehow?

Thanks for your profound knowledge, so I quote it here fully again (with some questions):

... in my simple words: To do all those tricks, my computer comes under heavy load, which becomes so high, that all those tricks get worthless . It just gets overloaded and *predictable* stutters

... thanks for coming to practical tips at the end. I will kick off the RT kernel and try to tweak the system with the standard kernel.

A good prove could be, if the named 3.14 RT-kernel runs flawless on system, which have a more powerful CPU than a N2930.

Thank you.

Hello julien,

I have a few more infos for you, eventually helpful:

There are indeed differences in the cpufreq files. On an old Debian wheezy with 3.2.x-Kernel your scaling_cur_freq exists and it seems to contain the second value, that is refreshed in the Perf Window. Example-view: 'CPU frequency 800MHz ... 2001MHz'. This old core2duo has the scaling_driver 'acpi-cpufreq'.

The new Bay Trail celeron with 3.14.x-Kernel on the other hand does not have this file 'scaling_cur_freq'. Maybe due to the new scaling_driver 'intel_pstate'. As far as I know, this is some new Intel powermanagement paradigma. The Perf Window shows always just *one* value. Example-view: 'CPU frequency 2602MHz'. Bay Trail testet with Pianoteq 5 (Trial).

Cheers

It looks like you are right: https://bugzilla.kernel.org/show_bug.cgi?id=57141 . So right now there is no convenient way to read the cpu freq on linux.. scaling_cur_freq does not exist with the intel_pstate governor, and cpuinfo_cur_freq can only be read by root. That's a bit sad.

No, the first one an old core2duo from Lenovo and the second one a new celeron N2930 ("Bay Trail") from Acer.

Sounds plausibel. But even on a weak AMD APU E-350 (1.6 GHz) I had a RT kernel running without any glitches in the past. So it comes unexpected, that a modern Intel Quadcore with 4 x 2.60 GHz (?!) has these massive problems. Ok, thanks for your explanations again, I'll stay tuned ...

Not to mention cache efficiency and pipelining. Modern CPUs rely heavily e.g. on pipelining: they load a whole string of instructions and try to run as many of them in parallel as possible (therefore even inside a single core you find parallel execution). This speeds up things considerably, since processor instructions can be quite different in terms of complexity (and thus many do not need a 'complete' CPU core to themselves). Context switching (i.e. between kernel and application or between applications) effectively kills the pipeline every time, and some CPUs' performance really sucks when this happens too often. Context switching also causes cache misses, making things worse. Then there is the register state that has to be saved/restored.

Long story short: context switching is trouble for the CPU and takes quite a lot of time. Because of all this, multitasking is a tradeoff: more frequent switching is more 'fair' but reduces efficiency, while less switching is more efficient but increases latency. And some CPUs are more sensitive than others.

Here's a simple shell script i wrote to set the governor to all cores at once. You can easily save this to, say, /usr/bin/governor and then chmod +x that file.

Just running "governor" with no arguments will show you the current governor for each core. Running "governor performance" will set all cores to performance.

#!/bin/sh

cd /sys/devices/system/cpu
gov="$1"

if [ "$gov" = "" ]; then
  for core in cpu?; do
    echo -n "Core ${core} set to "
    cat ${core}/cpufreq/scaling_governor
  done
else
  if [ "`whoami`" != "root" ]; then
    echo "Root access is required for this operation. Trying sudo..."
    exec sudo "$0" "$@"
  fi
  for core in cpu?; do
    echo "Setting core ${core} to ${gov}... "
    echo "$gov" > ${core}/cpufreq/scaling_governor
  done
fi

The governors are software modules so this sounds like whoever is building the Debian RT kernels decided not to include the usual range of governors.

On my Bay Trail server running Fedora 3.14.9-200.fc20.x86_64, I get:

> more /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
> conservative userspace powersave ondemand performance

I'm using a C2750.  All the governor options are usable.

Strange though -- it appears those governors SHOULD now be gone in Fedora 20 according to this thread:

* https://ask.fedoraproject.org/en/questi...-governor/

It's possible the Supermicro bios has intel_pstate disabled.

Try adding to your RT kernel boot parameters:

* intel_pstate=disable