Created attachment 157677 [details]
strace of the traceroute command

Since we send now `-N 15" simultaneous probes at once, it might confuse ppp links.
What happens if you try `-I' (icmp) or `-T' (tcp) tracerouting?

Could you try:
  traceroute -N 1 www.yahoo.com
on your ppp link?

If it helps, try various `-N' values up to 15.

ping?...

Well I am running kubuntu but this looks to be the same problem so hopefully
this is useful.  Your supposition looks correct.  On limited sampling -N 7 works
while -N 9 doesn't.  -N 8 works sometimes.

tcl@gilbereth:~$ ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.35.79.128  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:3431 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3809 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:2561146 (2.4 MB)  TX bytes:701211 (684.7 KB)

tcl@gilbereth:~$ traceroute -V
Modern traceroute for Linux, version 2.0.9, Sep 28 2007
Copyright (c) 2006  Dmitry Butskoy,   License: GPL
tcl@gilbereth:~$  dpkg -l traceroute
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                       Version                    Description
+++-==========================-==========================-====================================================================
ii  traceroute                 2.0.9~rc1-1                Traces the route taken
by packets over an IPv4/IPv6 network
tcl@gilbereth:~$ uname -a
Linux gilbereth 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux
tcl@gilbereth:~$

root@gilbereth:~# traceroute -N 7 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98 (172.18.51.98)  2881.429 ms  2883.331 ms  2883.308 ms

root@gilbereth:~# traceroute -N 8 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98 (172.18.51.98)  114.550 ms  116.491 ms  110.569 ms

root@gilbereth:~# traceroute -N 8 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -I 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -T 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -T -N 7 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets

Well,

Could you check also with DNS disabled (traceroute -n) ? Does the level of 7
changed then?

What is your modem driver?

Mostly the software is vanilla kububtu 7.10 up to date as of a week ago.  I have
a USB Huawei E220 which is switched to modem mode by huaweiAktBbo (the only
added software) before using kppp front end to connect.  In the main it runs a
acceptable HSDPA (T-Mobile UK) connection.

traceroute -n makes no apparent difference.

tcl@gilbereth:~$ traceroute -N 9 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 7 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  98.758 ms  108.747 ms  104.480 ms
 2  10.64.74.141  114.419 ms  114.388 ms  194.320 ms
 3  172.18.14.61  214.223 ms  120.348 ms  179.743 ms
tcl@gilbereth:~$ traceroute -N 9 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * *
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  123.504 ms  117.673 ms  118.587 ms
 2  10.64.74.141  124.656 ms  125.548 ms  175.530 ms
 3  172.18.14.61  176.522 ms  177.484 ms  149.788 ms
tcl@gilbereth:~$

Thanks.

"send: No buffer space available" mean that send(2) syscall returns failure with
errno == ENOBUFS

It is not clear for me now, whether it is generated by the general socket stack,
or someway by pppd daemon. Is it possible for you to attach here a strace output
of your pppd daemon?

(After the ppp connection is established, determine the pid of pppd daemon
(pidof pppd), then run "strace -p pid_number -o out_file", and invoke
"traceroute" with -N > 8 on another terminal...)

Sure.

strace -p `pidof pppd` -o fred
Process 12070 attached - interrupt to quit
Process 12070 detached
root@gilbereth:/root# cat fred
select(13, [11 12], NULL, [11 12], {11, 812000} <unfinished ...>
root@gilbereth:/root#

and

tcl@gilbereth:~$ traceroute -N 10 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$

It seems the reason is the too small txqueuelen value (of 3) for ppp0 interface.

Could you increase this to, say 10 (by ifconfig or ip link) and report what
happens? (I'm not sure that the value of 10 is not too big for it though... :) )

I'll think these days about a fixing of the issue (and ship the next 2.0.10
version)...

That looks like it!

tcl@gilbereth:~$ traceroute -N 10 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.97  201.602 ms  220.378 ms  221.324 ms
 2  10.64.74.233  221.290 ms 10.64.74.229  230.269 ms 10.64.74.225  231.252 ms
 3  172.18.14.61  241.232 ms  240.303 ms  250.242 ms
 4  149.254.195.194  350.207 ms  178.736 ms  167.928 ms
 5  10.126.168.94  168.909 ms  168.885 ms  160.908 ms
 6  193.159.226.21  168.932 ms  160.826 ms  208.950 ms
 7  213.206.131.69  249.870 ms  168.729 ms  149.864 ms
 8  213.206.129.151  140.856 ms  148.862 ms  149.817 ms
 9  213.206.128.63  168.804 ms  160.903 ms  189.038 ms
10  82.195.189.186  209.883 ms  189.797 ms  268.118 ms
11  209.85.252.40  130.894 ms  130.927 ms 209.85.252.42  129.872 ms
12  64.233.175.213  289.043 ms 66.249.95.146  270.894 ms  308.984 ms
13  216.239.46.25  339.797 ms 72.14.233.113  340.936 ms  359.038 ms
14  66.249.94.92  278.035 ms  290.026 ms  309.922 ms
15  66.249.94.118  299.785 ms 72.14.236.130  260.780 ms 72.14.236.134  274.121 ms
16  72.14.207.99  296.041 ms  249.844 ms  259.799 ms
tcl@gilbereth:~$ ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.32.106.202  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:3737 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3749 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10
          RX bytes:3486420 (3.3 MB)  TX bytes:519159 (506.9 KB)
tcl@gilbereth:~$

Tom,

Could you compile and test the next version rc:
http://dmitry.butskoy.name/traceroute/traceroute-2.0.10rc1.tar.gz

(certainly, with tx_queue_len == 3, as usual).

If you cannot compile for some reason, I'll create an rpm for you.

ping?

Yeah, I didn't have the modem for a few days.

That seems to have fixed it.

tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$
traceroute -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$
./traceroute -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  91.580 ms  93.528 ms  100.601 ms
 2  10.64.74.141  102.516 ms  110.489 ms  112.455 ms
 3  172.18.14.60  121.435 ms  108.785 ms  109.731 ms
 4  * * *
 5  10.126.168.94  118.686 ms  128.659 ms  127.940 ms
 6  193.159.226.21  209.741 ms  210.686 ms  219.651 ms
 7  213.206.131.69  139.677 ms  137.939 ms  139.878 ms
 8  213.206.129.151  138.899 ms  129.898 ms  88.626 ms
 9  213.206.128.63  228.593 ms  220.839 ms  211.925 ms
10  82.195.189.186  99.961 ms  89.848 ms  90.806 ms
11  209.85.252.42  89.857 ms  89.706 ms 209.85.252.40  108.788 ms
12  64.233.175.213  160.007 ms  159.900 ms  159.935 ms
13  72.14.233.113  169.956 ms  169.936 ms  180.007 ms
14  66.249.94.90  179.019 ms  180.137 ms 72.14.236.183  179.133 ms
15  66.249.94.118  179.829 ms  179.839 ms  179.962 ms
16  72.14.207.99  187.936 ms  170.763 ms  180.952 ms
tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$ ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.35.99.222  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:577 errors:0 dropped:0 overruns:0 frame:0
          TX packets:892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:99786 (97.4 KB)  TX bytes:77588 (75.7 KB)

tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$

traceroute-2.0.10-1.fc8 has been submitted as an update for Fedora 8

traceroute-2.0.10-1.fc7 has been submitted as an update for Fedora 7

traceroute-2.0.10-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

traceroute-2.0.10-1.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.