FOOD FOR THOUGHT Best Timing Source

[luckman212]

nievz,
questions:

1) what hardware are you running your PIAF on?
2) what version of the Sangoma driver are you running?
3) can you paste output of each of the following commands:
dmesg | grep -i voicetime
dmesg | grep -i dahdi
cat /proc/dahdi/1
/etc/init.d/dahdi status

 

[nievz]

Thanks!

Kernel Version 2.6.18-164.11.1.el5 (SMP)
Asterisk 1.6.0.26-FONCORE-r78
Trixbox 2.8.0.4
Distro Name CentOS release 5.5 (Final)
Processors 1
Model AMD Opteron(tm) Processor 254
CPU Speed 2.81 GHz

PCI Configuration:
Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI
(2x) Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet
Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE
ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC
PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI
(4x) PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge
(4x) PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC
RAID bus controller: Compaq Computer Corporation Smart Array 64xx
System peripheral: Compaq Computer Corporation Integrated Lights Out Processor
System peripheral: Compaq Computer Corporation Integrated Lights Out Controller
(2x) USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB OHCI
VGA compatible controller: ATI Technologies Inc Rage XL
------------------------------

I'm actually running Trixbox using driver wanpipe-voicetime-1.0.13

# dmesg | grep -i voicetime
wanpipe_voicetime: Loading WANPIPE VoiceTime (USB) Driver - v1.0.9
wanpipe_voicetime: Probing WANPIPE VoiceTime (USB) device on 2 ...
[<f8bc16b0>] wud_probe+0x3c8/0x517 [wanpipe_voicetime]
[<f8bc1700>] wud_probe+0x418/0x517 [wanpipe_voicetime]
[<f8802033>] init_module+0x33/0x57 [wanpipe_voicetime]
usbcore: registered new driver wanpipe_voicetime


# dmesg | grep -i dahdi
dahdi: Telephony Interface Registered on major 196
dahdi: Version: 2.3.0.1
dahdi_transcode: Loaded.
INFO-xpp: FEATURE: with sync_tick() from DAHDI
BUG: warning at /usr/src/dahdi-linux-2.3.0.1/drivers/dahdi/dahdi-base.c:5866/dahdi_register() (Tainted: G )
[<f8b7343d>] dahdi_register+0x43/0x297 [dahdi]
dahdi: Registered tone zone 0 (United States / North America)

# cat /proc/dahdi/1
Span 1: WANVTIME/1 "WANVTIME/1 (source: wanpipe_voicetime) 1" (MASTER)

#/etc/init.d/dahdi status
### Span 1: WANVTIME/1 "WANVTIME/1 (source: wanpipe_voicetime) 1" (MASTER)

 

[luckman212]

Hmm I really am not sure. Just a couple of things that stand out there:

a) BUG: warning at /usr/src/dahdi-linux-2.3.0.1/drivers/dahdi/dahdi-base.c:5866/dahdi_register() (Tainted: G )
Not sure exactly what this means-- are you by any chance running the "hacked" DAHDI version that patches in OSLEC the echo canceler? I think I remember seeing something like this once before in a case like that.

b) 2.3.0.1 is also a fairly old version of dahdi... have you considering trying out the current 2.5.0.2 release?

 

[nievz]

No change in performance after 2.5.0.2 install

I've installed 2.5.0.2 and my results are actually worst. This is a freshly rebooted server:

# dmesg | grep -i dahdi
dahdi: Telephony Interface Registered on major 196
dahdi: Version: 2.5.0.2
dahdi_transcode: Loaded.
INFO-xpp: FEATURE: with sync_tick() from DAHDI

# cat /proc/dahdi/1
Span 1: WANVTIME/1 "WANVTIME/1 (source: wanpipe_voicetime) 1" (MASTER)

# /etc/init.d/dahdi status
### Span 1: WANVTIME/1 "WANVTIME/1 (source: wanpipe_voicetime) 1" (MASTER)

# dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.980% 99.976% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.923% 99.978% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.924% 99.979%
99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.979% 99.979% 99.979% 99.979% 99.979% 99.923% 99.979% 99.979%
99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.979% 99.979% 5.594% 99.923% 99.979% 99.979% 99.979% 99.979%
99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.979% 99.923% 99.979% 99.979% 99.979% 99.979% 99.979% 99.979%
99.979% 99.979% 99.979% 99.979%
--- Results after 76 passes ---
Best: 99.980 -- Worst: 5.594 -- Average: 98.733401, Difference: 98.743490

#

Thoughts?

[UPDATE] After things settled down, I'm back to the same result. The driver swap didn't affect the result from when I was using the old drivers. My current result:

]# dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.986% 99.983% 99.984% 99.985% 99.985% 99.985% 99.985% 99.984%
99.984% 99.919% 99.985% 99.985% 99.984% 99.985% 99.985% 99.985%
99.985% 99.984% 99.985% 99.984% 99.984% 99.985% 99.984% 99.984%
99.918% 99.984% 99.985% 99.984% 99.984% 99.984% 99.984% 99.985%
99.984% 99.984% 99.984% 99.984% 99.985% 99.984% 99.918% 99.984%
99.984% 99.984% 99.983% 99.986% 99.983% 99.984% 99.984% 99.983%
99.984% 99.984% 99.984% 99.984% 99.919% 99.984% 99.984% 99.984%
99.984% 99.983% 99.987% 99.992% 99.984% 99.988% 99.987% 99.988%
99.987% 99.987% 99.911% 99.984% 99.988% 99.988% 99.988% 99.988%
99.987% 99.987% 99.988% 99.987% 99.987% 99.987% 99.987%
--- Results after 79 passes ---
Best: 99.992 -- Worst: 99.911 -- Average: 99.980791, Difference: 99.991271

 

[luckman212]

nievz:
sorry, can you post some more output:

1) cat /proc/interrupts
2) cat /sys/devices/system/clocksource/clocksource0/current_clocksource
3) lspci -vb >/tmp/pcidevs.txt
attach the /tmp/pcidevs.txt file to your reply (it is probably too large to copy/paste)

 

[nievz]

# cat /proc/interrupts
CPU0
0: 31185066 IO-APIC-edge timer
1: 3 IO-APIC-edge i8042
6: 6 IO-APIC-edge floppy
8: 1 IO-APIC-edge rtc
9: 1 IO-APIC-level acpi
12: 114 IO-APIC-edge i8042
14: 278294 IO-APIC-edge ide0
169: 31146540 IO-APIC-level ohci_hcd:usb1, ohci_hcd:usb2
177: 32178 IO-APIC-level cciss0
201: 6023174 IO-APIC-level eth1
NMI: 0
LOC: 31178646
ERR: 0
MIS: 0


# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

 

[luckman212]

Hmm all looks OK to me -- your USB controller is on IRQ 7 and not sharing that IRQ with any other devs. No errs on the interrupts either. Maybe you want to try a different hw clocksource?

You can see what options are available by typing:
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

if you have acpi_pm or hpet, try switching to one of those via e.g.
echo hpet > /sys/devices/system/clocksource/clocksource0/current_clocksource

The change should take effect immediately, and then re-run your dahdi_test to see if the results are any better.

 

[nievz]

A step in the right direction!

I only had acpi_pm, jiffies, tsc, and pit available. pit provides the worst result, while with jiffies i gained .006 better performance. So i'm sticking with this source for now:

# dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.990% 99.990% 99.990% 99.912% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990%
99.990% 99.912% 99.990% 99.990% 99.990% 99.892% 99.892% 99.892%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.912%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.912% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.912% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.912% 99.990% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990%
99.893% 99.893% 99.912% 99.893% 99.990% 99.990% 99.990% 99.990%
99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.990% 99.912%
99.990% 99.990% 99.990% 99.991% 99.990% 99.990% 99.990% 99.991%
99.990% 99.990% 99.991% 99.990% 99.991% 99.912% 99.991% 99.991%
99.991% 99.990% 99.991% 99.991% 99.991% 99.991% 99.991% 99.991%
99.991% 99.991% 99.991% 99.912% 99.991% 99.991% 99.991% 99.991%
--- Results after 440 passes ---
Best: 99.992 -- Worst: 99.797 -- Average: 99.981922, Difference: 99.993971

 

[luckman212]

That looks pretty good to me!
To make that change permanent, you can edit /etc/grub.conf

find the line that begins with "kernel" e.g.:
kernel /vmlinuz-2.6.18-194.17.4.el5 ro root=LABEL=/
and add:
clocksource=acpi_pm

save & reboot and test again.

 

[nievz]

Done. I'm now going to check how stable it is with this as the clock. Thanks for all your help. Eventually, I'll get an analog card for timing, any recommendations? We run pure voip so no need for PSTN terminations, just for timing.

Thanks for all your help!

 

[luckman212]

If you're pure voip why do you want to buy an analog card? just curious-- the UT50 should be an adequate timing source once you work out the kinks. If you don't want a "dongle" sticking out of your PBX then you could buy the internal version (UT51) and stick it on the motherboard header.

 

[nievz]

I'm still not that confident with the 99.98-99.992 accuracy i'm getting from Voicetime, not to mention the occational dips to 99.7xx. I'm very convinced that this has something to do with the architecture of the HP Proliant server I'm using, the circuitry doesn't favor USB. If i'm going to migrate 300 conference bridge users to this server, i've got to be absolutely sure that it can handle the load.

 

[gregc]

Would you be willing to rent/sale the UT50 you have? I've been itching to try it out on an Atom setup to see if it helps improve our timing.

With the info here from you and luckman, I too have updated our source to jiffies which provided a .01 gain on average and an overall more stable timing then tsc but still below optimal.

-Greg

 

[nievz]

Jiffies unreliable

So this is the first day I used jiffies as the system clocksource and around the middle of the day, while there were 20 users on the bridge, their calls suddenly started dropping like flies. I'm back to tsc now.

@greg
unfortunately our UT50's are company tracked and owned, so i'm sorry it's not possible to loan them to you. But you should see good performance from them. It's just that i'm using a specialized server, that's not optimized for USB, that's why i'm having these issues. I actually installed asterisk on an Avaya 8720 server, which is manufactured by HP.

 

[nievz]

Built another server using PIAF 2.0.6.0

I'm using a Digium TDM410P for timing on this new server. But, I'am getting WARNING[2426] app_meetme.c: Unable to write frame to channel SIP/trunk-00000017 error occasionally. The last time this happened was on a call that has been active for about 11 hours. The SIP call just hang up. I am in an all SIP setup except for the Digium card which i only use for timing.

PIAF Installed Version = 2.0.6.0 Running on *HARDWARE* │
│ FreePBX Version = 2.9.0.7 │
│ Running Asterisk Version = 1.8.8.0 │
│ Asterisk Source Version = 1.8.8.0 │
│ Dahdi Source Version = 2.6.0+2.6.0 │
│ Libpri Source Version = 1.4.12 │
│ Operating System = CentOS release 6.2 (Final) │
│ Kernel Version = 2.6.32-71.29.1.el6.i686 – 32 Bit

I'm wondering if this has something to do with timing. Anyone had this error before? Here's my result (it really fluctuates:

dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.988% 99.919% 99.986% 99.984% 99.985% 99.986% 99.890% 99.917%
99.986% 99.888% 99.987% 99.916% 99.986% 99.986% 99.889% 99.916%
99.889% 99.916% 99.986% 99.987% 99.889% 99.916% 99.987% 99.985%
99.889% 99.917% 99.986% 99.890% 99.917% 99.889% 99.987% 99.917%
99.986% 99.987% 99.889% 99.916% 99.890% 99.986% 99.917% 99.889%
99.916% 99.889% 99.986% 99.913% 99.983% 99.987% 99.889% 99.987%
99.916% ^C
--- Results after 49 passes ---
Best: 99.988% — Worst: 99.888% — Average: 99.940410%
Cummulative Accuracy (not per pass): 99.988

By the way, the server where i am running Trixbox and was using a Sangoma UT-50, i swapped it to a Digium TDM410P and i'm now getting stable 99.99x% accuracy. All good on that server. This new server on PIAF has the same hardware but not getting good result...

 

[nievz]

help

guys any suggestion? i'm running the exact same hardware on the Trixbox and PIAF but getting terrible dahdi_result in PIAF. I've disabled all unneeded PCI devices, and have the same BIOS settings as the other box.

I do see the PIAF server is missing tsc for its available clocksource, why is that? i only have acpi_pm. The Trixbox is using tsc.


[UPDATE]
I figured it out guys! i added nohz=off highres=off clocksource=tsc in grub.conf. I'm now getting the following result (although Trixbox still shows better timing):


$ dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.984% 99.988% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.986% 99.985% 99.985% 99.985% 99.985% 99.985% 99.984%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985% 99.985%
99.985% 99.985% 99.985% ^C
--- Results after 75 passes ---
Best: 99.988% -- Worst: 99.984% -- Average: 99.984975%
Cummulative Accuracy (not per pass): 99.985

 

[helvetico]

most accurate timing on system with internal clock source

I want to share my experience which is the result of quite some painful empirical trial and error.

Summary: The Best result I achieved by digging out and compiling the old dahdi_dummy module. This is done by uncommenting the second line referring to dahdi_dummy.o in the Kbuild file typically located in /usr/src/dahdi-linux-2.5.0.2/drivers/dahdi and compiling and installing the dahdi driver as usual.

Background: The new standard non kernel module based DAHDI clocksource showed me results with dahdi_test -v in the area of 99.6XX%, while I could measure straight 99.999% by bringing back dahdi_dummy on the same hardware, being a blade system where there is no option to add a PCI or USB external clock source. I have compared various internal timing options and sticked to the hpet clock. However that isn't available on older machines but in my opinion not the main factor.

Restrictions: This method does not work anymore on DAHDI 2.6 as something broke the compilation when having that line uncommented.

Doubts: I would be more than curious to hear from the DAHDI architects or some knowledgable source why the dahdi_dummy was abandoned for an apparently worse alternative and what outlook there is for systems without clocking device.

 

[nievz]

app_meetme.c: Unable to write frame to channel SIP/trunk-00000017

I'm still frequently getting this error

app_meetme.c: Unable to write frame to channel SIP/trunk-00000017

I found out my PiAF is using res_timing_timerfd.so for internal timing so i added the following line in modules.conf to force it to use res_timing_dahdi.so:

noload => res_timing_timerfd.so

I will update this thread if this resolves the problem.

By the way, my dahdi_test results have been wonderful:
$ dahdi_test
Opened pseudo dahdi interface, measuring accuracy...
99.999% 99.995% 99.997% 99.997% 99.997% 99.997% 99.997% 99.997%
99.997% 99.996% 99.997% 99.997% 99.997% 99.997% 99.997% 99.997%
99.997% 99.997% 99.997% 99.998% 99.997% 99.997% 99.997% 99.997%
99.997% 99.997% 99.997% 99.996% 99.990% 99.997% 99.997% 99.997%
99.997% 99.997% 99.997% 99.998% 99.997% 99.997% 99.997% 99.997% ^C
--- Results after 40 passes ---
Best: 99.999% -- Worst: 99.990% -- Average: 99.996973%
Cummulative Accuracy (not per pass): 99.997

I love Asterisk!!!