FOOD FOR THOUGHT Best Timing Source

[dswartz]

If the /sys/devices stuff shows hpet you have hpet, I would think. Dunno why dahdi isn't using it. What do you get if you do:

module show like timing

in asterisk remote console?

p.s. don't even think of trying to upgrade kernel and glibc in centos5 unless you are prepared for a lot of pain...

 

[malcolmd]

Here's a useful blog post on the matter:

http://www.curiousmentality.co.uk/2...hrtimer-might-not-be-a-high-resolution-timer/

 

[randy7376]

Since current_clocksource shows hpet, the system (and DAHDi) are using hpet.

As I understand it, any software-derived timing source, such as DAHDI, is going to be reliant on the kernel's clock source. I wouldn't be concerned about that aspect of the system as it seems to be just fine. I wouldn't change anything there.

What are the system load values on your server when it's idle? The uptime command will show you the system load values.

 

[luckman212]

Is there any way to confirm that dahdi is using HPET? Because as I mentioned there's no info in the dmesg about it.

My system is very idle most of the time. Here are the numbers:
11:27:46 up 8:09, 1 user, load average: 0.04, 0.08, 0.02

 

[dswartz]

I don't believe the assertion that just because the system is using hpet, dahdi is too. There may or may not be a reason dahdi is not using hpet timer, regardless of whether it is present or not.

 

[luckman212]

I tend to agree with dswartz on this one, especially after reading the post malcolm linked to. What concerns me most is the jitter in my dahdi_test accuracy scores, even on a completely (and I do mean completely) idle system:

These are not good scores. This, combined with the fact that dahdi does not report anything about HPET in the dmesg, lead me to believe that my dahdi is not using HPET. I could be wrong but I think I need to recompile dahdi and change some flags somewhere.

 

[dswartz]

I am puzzled. I would have thought the default options would be correct, but let us know...

 

[luckman212]

Let you know what? :surrenderb:

 

[dswartz]

you said "I think I need to recompile dahdi and change some flags somewhere". If you do that, let us know...

 

[luckman212]

Oh, hehe. Yeah I did that... recompiled but got the same results. Couldn't find any place to change those flags. I looked in /usr/src/dahdi (no configure script in there). The config seems to happen in /usr/src/dahdi/tools but even so, couldn't find any reference to switching on the HPET. I'm completely out of ideas on how to improve the timer on this Revo box. I've tried:

* different kernel timers in /etc/grub.conf: hpet, acpi_pm, tsc
* disable integrated sound card (frees up an IRQ)
* turn off USB in BIOS
* updated BIOS
* comment out all hardware drivers in /etc/dahdi/modules
* updated Asterisk to 1.8.2rc1
* tried adding "divider 10" to kernel boot line in /etc/grub.conf

Pretty much at a loss now. I guess my next move is either:

A) backup my configs, wipe the box and install a non-PiaF setup (just base Ubuntu or something, then add Asterisk + FreePBX manually + reconfigure everything -- yuck

B) buy a Sangoma UT50 and *hope* that it works with the 2.6.18 kernel (even though their website pretty much says it won't work well...) and then double-hope that it's more accurate than the software timer

C) give up and buy a "bigger" system that I can put a PCI card in to provide a HW timer for DAHDI

 

[randy7376]

The kernel option referenced in dahdi_dummy.c - CONFIG_HIGH_RES_TIMERS - does not exist in the 2.6.18 kernel. It exists in 2.6.22 and higher.

Yes, this is dated - I hadn't found anything else that references this, though... From: http://lists.digium.com/pipermail/dahdi-commits/2008-August/000305.html

* ztdummy on i386/x86_64 with kernels >= 2.6.22 can (and should) use
- high resolution timers (CONFIG_HIGH_RES_TIMERS), and (if available),
- the system HPET. This shows as "source: HRTimer". This is
- recommended.

I read that to mean Zaptel will use HPET, if available. I can't see that DAHDi would be that drastically different.

I still contend that DAHDi will use whatever is configured for current_clocksource as it is dependent on that clocksource the way the installed kernel is configured. If it's set for hpet, DAHDi is using hpet.

The article malcolmd linked to implies if the high-res timer/hpet is not available, it will use a low-resolution timer. That doesn't seem to be the case in luckman212's issue as he does have hpet available.

The output regarding the high-resolution timer for DAHDi you posted is from an older version, anyway. The above message may have been removed in later versions. dahdi_dummy.ko is no longer a seperate module and is now a part of dahdi.ko from what I've read. I have the same output for DAHDi that luckman212 has on his system.

Heck, try switching to jiffies or pit and see if dahdi_test improves or worsens.

 

[luckman212]

I tried jiffies + pit. They weren't really any worse than hpet or acpi_pm as far as I could tell.
Once kind of neat thing about pit that I noticed was quite a few dead-on 100.000% accurate hits. Never saw this with the other timers-- they were quite often just a bit under or a bit over the mark. With pit there were a lot of 100's and not a single one went over 100, which the other timers did.

Can someone tell me the basic reason that it would be a fool's errand to try to shoehorn a newer kernel into the pbx that supports these higher resolution timers + timerfd ? Sorry for being so naive about the subject.

 

[dswartz]

It isn't just the kernel. My openvz piaf is running on a host with a 2.6.32 kernel, but can't use timerfd. Why? Because that also requires a 2.8 version of glibc. When you start messing with newer versions of shared libraries and such you are just asking for a complete cluster f***.

 

[luckman212]

I hear what you're saying. Well even without glibc 2.8, a newer kernel would (according to the information that Randy7376 posted above) allow HRtimer to be enabled for dahdi_dummy. This might help, in theory.

 

[randy7376]

nohz=off?

One last thing I can think of since your Aspire Revo U1600 appears to be an Intel Atom-based system...

I have a Toshiba Atom-based netbook that gave me all kinds of performance grief until I added nohz=off to the kernel at boot time. The system was taking about 1-2 minutes to boot without it. It now takes about 20-25 seconds.

Yes, I know, it's a bit of a stretch...

 

[dswartz]

it wouldn't though

I hear what you're saying. Well even without glibc 2.8, a newer kernel would (according to the information that Randy7376 posted above) allow HRtimer to be enabled for dahdi_dummy. This might help, in theory.

apparently the code that accesses those timer features requires glibc2.8. that is what i was getting at. i have a post elsewhere here where i pointed out that running piaf on jroper's ubuntu based container can get timerfd due to the newer glibc but on my centos5 based container, i can't.

 

[luckman212]

Hi Randy, thanks for the tip, I am vaguely aware of those new tickless kernel options. But I thought they were introduced in kernel 2.6.21 and above-- are you sure that it actually had an effect on your system? Guess I will give it a shot--

 

[randy7376]

The Toshiba netbook is running a much later kernel version, but I thought that option had been around longer than 2.6.21. Looking around the web, it would seem you are correct about it being in 2.6.21 and later.

 

[luckman212]

Tested & confirmed ... nohz does nothing on 2.6.18.

 

[Linetux]

First of all, I'm *SO* glad this conversation is happening here... (Stewart, are you reading?)

Second, the UT50/51 works fine with 2.6.18. Some interesting results from a few sources FYI:

Box #1: Sangoma UT50 running on 2.6.18-92.1.22.el5, Intel Xeon X3210

$ zttest
Opened pseudo zap interface, measuring accuracy...
99.901855% 99.999512% 99.999512% 99.999527% 99.999527% 99.901756% 99.999512%
99.999512% 99.999512% 99.999527% 99.901871% 99.999413% 99.999512% 99.999512% 99.999527%
99.999527% 99.901756% 99.999512% 99.999512% 99.999512% 99.999527% 99.901871% 99.999413%
99.999512% 99.999512% 99.999512% 99.901871% 99.999527% 99.999413% 99.999512% 99.999512%
99.999527% 99.999527% 99.999413% 99.999512% 99.999512% 99.999512% 99.901871% 99.999527%
99.999413% 99.999512% 99.999512% 99.901855% 99.999527% 99.999527% 99.999413% 99.999512%
99.901855% 99.999527% 99.999527% 99.999413% 99.999512% 99.999512% 99.901855% 99.999527%
99.999527% 99.999413% 99.999512% 99.901855% 99.999512% 99.999527% 99.999527% 99.999413%
99.901855% 99.999512% 99.999512% 99.999527% 99.999527% 99.999413% 99.999512% 99.999512%
99.999527% 99.999527% 99.999413% 99.901855% 99.999512% 99.999512% 99.999527% 99.999527%
99.901756% 99.999512% 99.999512% 99.999512% 99.999527% 99.999527% 99.999413% 99.999512%
99.999512% 99.999512% 99.999527% 99.901871% 99.999413% 99.999512% 99.999512% 99.999512%
99.901871% 99.999527% 99.999413% 99.999512% 99.999512% 99.901871% 99.999527% 99.999413%
99.999512% 99.999512% 99.999512% 99.999527% 99.999527% 99.999413% 99.999512% 99.999512%
99.901855% 99.999527% 99.999527% 99.999413% 99.999512% 99.901855% 99.999512% 99.999527%
99.999527% 99.999413% 99.901855% 99.999512% 99.999512% 99.999527% 99.999527% 99.999413%
99.901855% 99.999512% 99.999512% 99.999527% 99.999527% 99.901756% 99.999512% 99.999512%
99.999512% 99.999527% 99.901871% 99.999413% 99.999512% 99.999512% 99.999512% 99.999527%
--- Results after 143 passes ---
Best: 100.000 -- Worst: 99.902 -- Average: 99.983793, Difference: 100.016208

Box #2 - Sangoma PRI, 2.6.18-164.6.1.el5, Intel Atom 330


$ zttest
Opened pseudo zap interface, measuring accuracy...
99.997070% 99.988480% 99.993843% 99.994141% 99.993744% 99.994621% 99.995506%
99.992775% 99.991402% 99.994438% 99.993660% 99.994331% 99.994438% 99.993355% 99.994629%
99.993553% 99.993164% 99.994835% 99.993553% 99.993454% 99.994141% 99.994820% 99.993065%
99.994240% 99.991997% 99.993752% 99.994240% 99.994728% 99.993553% 99.995018% 99.993256%
99.993462% 99.992485% 99.994728% 99.993660% 99.993942% 99.994339% 99.993942% 99.994240%
99.994141% 99.992973% 99.993744% 99.992874% 99.994431% 99.993553% 99.994530% 99.994331%
99.994621% 99.992577% 99.992966% 99.994141% 99.996483% 99.989258% 99.993950% 99.993752%
99.993454% 99.992867% 99.993652% 99.994537% 99.992973% 99.993752% 99.994232% 99.995117%
99.992874% 99.990822% 99.994820% 99.994339% 99.992966% 99.994049% 99.994827% 99.993744%
99.994049% 99.992958% 99.993950% 99.995026% 99.994034% 99.993752% 99.993942% 99.994728%
99.992767% 99.991501% 99.993942% 99.994438% 99.994728% 99.993073% 99.993744% 99.995117%
99.993553% 99.992386% 99.994331% 99.992966% 99.994240% 99.993065% 99.994141% 99.994438%
99.993942% 99.992683% 99.993660% 99.995117% 99.994530% 99.993851% 99.994141% 99.994240%
99.993851% 99.992386% 99.994431% 99.992775% 99.994919% 99.993553% 99.994820% 99.994331%
99.993164% 99.992973% 99.994347% 99.993660% 99.993752% 99.994431% 99.994537% 99.993744%
99.994431% 99.993462% 99.992874% 99.993362% 99.994629% 99.993858% 99.994141% 99.994431%
99.993256% 99.993065% 99.993454% 99.995216% 99.993652% 99.993942% 99.993851% 99.995216%
99.994629% 99.991600% 99.994827% 99.993851% 99.993462% 99.994629% 99.994339% 99.994431%
99.993652% 99.992874% 99.994049% 99.993744% 99.994133% 99.994728% 99.993164% 99.994919%
99.992180% 99.993065% 99.993752% 99.994133% 99.995216% 99.994041%
--- Results after 157 passes ---
Best: 99.997 -- Worst: 99.988 -- Average: 99.993815, Difference: 99.993815

Third box: Xorcom PRI (USB), 2.6.18-128.1.10.el5, Xeon

$zttest
Opened pseudo zap interface, measuring accuracy...
99.997856% 99.976463% 99.993263% 99.998817% 99.991119% 99.993362% 99.992767%
99.995995% 99.987602% 99.989548% 99.995026% 99.990326% 99.991508% 99.991310% 99.987793%
99.998924% 99.984566% 99.987404% 99.990723% 99.990631% 99.996094% 99.989845% 99.990631%
99.998726% 99.989349% 99.990334% 99.977837% 99.990822% 99.996582% 99.998047% 99.984177%
99.990723% 99.993172% 99.983986% 99.991104% 99.990334% 99.990723% 99.996880% 99.990913%
99.990921% 99.989059% 99.997177% 99.989845% 99.990822% 99.996582% 99.986336% 99.998238%
99.990723% 99.987015% 99.999992% 99.993942% 99.990723% 99.996681% 99.996979% 99.991997%
99.991989% 99.987801% 99.992470% 99.997261% 99.991608% 99.990532% 99.987404% 99.997162%
99.990044% 99.984566% 99.986427% 99.991302% 99.997459% 99.990623% 99.990234% 99.991013%
99.997757% 99.987305% 99.991013% 99.991798% 99.997948% 99.990822% 99.989738% 99.989929%
99.996780% 99.988182% 99.991112% 99.990532% 99.998245% 99.991791% 99.989647% 99.990227%
99.997261% 99.988472% 99.990532% 99.990341% 99.996201% 99.989647% 99.992088% 99.989738%
99.990234% 99.999710% 99.983582% 99.999413% 99.994919% 99.991402% 99.999031% 99.992676%
99.999603% 99.991798% 99.981155% 99.998154% 99.991508% 99.986809% 99.997368% 99.975098%
99.997948% 99.994621% 99.974899% 99.991989% 99.999619% 99.990623% 99.991508% 99.996880%
99.991600% 99.986725% 99.997078% 99.989449% 99.990913% 99.991013% 99.998627% 99.990433%
99.991310% 99.988380% 99.990814% 99.997856% 99.989357% 99.991791% 99.989838% 99.997269%
99.990532% 99.987602% 99.996689% 99.978317% 99.997650% 99.988571% 99.997948% 99.979393%
99.989449% 99.999908% 99.989655% 99.990913% 99.996773% 99.990631% 99.994133% 99.999512%
99.993942% 99.988869% 99.991798% 99.991402% 99.995712% 99.974907% 99.997650% 99.988571%
99.993553% 99.989738% 99.986435% 99.986328% 99.990334% 99.996582% 99.989845% 99.990913%
99.991119% 99.999222% 99.989838%
--- Results after 170 passes ---
Best: 100.000 -- Worst: 99.975 -- Average: 99.991675, Difference: 99.993066

You can note that the UT50 gives the worst timing results of any of the examples, but curiously it's a predictable 'low' score. I have a feeling that's due to the kernel timing source running on the box.

Also note that the post from tzafrir that was referenced is one of Xorcom's engineers. He knows a thing or two about this subject as well.