Bug 245438 - traceroute fails with the error -- "send: No buffer space available"
traceroute fails with the error -- "send: No buffer space available"
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: traceroute (Show other bugs)
7
i386 Linux
low Severity low
: ---
: ---
Assigned To: Dmitry Butskoy
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-06-23 04:07 EDT by Steve
Modified: 2008-04-22 18:43 EDT (History)
3 users (show)

See Also:
Fixed In Version: 2.0.10-1.fc8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-04-22 18:42:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
strace of the traceroute command (24.62 KB, application/octet-stream)
2007-06-23 04:07 EDT, Steve
no flags Details

  None (edit)
Description Steve 2007-06-23 04:07:46 EDT
Description of problem:
traceroute fails with the error "send: No buffer space available", over a ppp
connection. Executing traceroute as both normal as well as root user seems to
generate the error.
----------------------------------------------
[sfernand@localhost ~]$ ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:560 (560.0 b)  TX bytes:560 (560.0 b)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:220.224.18.202  P-t-P:220.224.134.38  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:4165 errors:20 dropped:0 overruns:0 frame:0
          TX packets:4551 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:2939846 (2.8 MiB)  TX bytes:595249 (581.2 KiB)
[sfernand@localhost ~]$ rpm -q traceroute 
traceroute-2.0.3-1.1.fc7
[sfernand@localhost ~]$ uname -a
Linux localhost.localdomain 2.6.21-1.3194.fc7 #1 SMP Wed May 23 22:35:01 EDT
2007 i686 i686 i386 GNU/Linux
[sfernand@localhost ~]$ sudo traceroute www.yahoo.com
traceroute to www.yahoo.com (87.248.113.14), 30 hops max, 40 byte packets
send: No buffer space available
[sfernand@localhost ~]$ traceroute www.yahoo.com
traceroute to www.yahoo.com (87.248.113.14), 30 hops max, 40 byte packets
send: No buffer space available
[sfernand@localhost ~]$
----------------------------------------------

I am attaching an strace of the command.

I have not yet verified whether this happens over an ethernet (or any other
type of) connection. Will do so and update this report.

Please let me know if you need any other information.


Version-Release number of selected component (if applicable):
traceroute-2.0.3-1.1.fc7


How reproducible: Consistently reproducible.


Steps to Reproduce:
1. Install traceroute-2.0.3-1.1.fc7
2. Set up a ppp conection
3. Execute the command:
$ traceroute www.yahoo.com

Actual results:
[sfernand@localhost ~]$ sudo traceroute www.yahoo.com
traceroute to www.yahoo.com (87.248.113.14), 30 hops max, 40 byte packets
send: No buffer space available

Expected results:
$ traceroute www.yahoo.com
traceroute to www.yahoo.com (209.191.93.52), 30 hops max, 40 byte packets
 1 192.168.0.3 (192.168.0.3) 0.894 ms 1.059 ms 1.175 ms
 2 * * *
 3 ge-0-2-0-153.aggr01.crchtx.grandecom.net (216.82.202.190) 17.706 ms 21.113 ms
23.371 ms
 4 so-2-0-3-0.core01.smrctx.grandecom.net (24.155.121.13) 36.283 ms 40.006 ms
42.232 ms
 5 so-0-0-0-0.aggr01.austtx.grandecom.net (24.155.121.8) 45.834 ms 48.727 ms
52.601 ms
 ....
 ....


Additional info:
A google for this threw up a similar bug report for CentOS 5.0
http://bugs.centos.org/view.php?id=1955

I shall verify this on RHEL5 too and update this report.
Comment 1 Steve 2007-06-23 04:07:46 EDT
Created attachment 157677 [details]
strace of the traceroute command
Comment 2 Dmitry Butskoy 2007-07-30 10:05:19 EDT
Since we send now `-N 15" simultaneous probes at once, it might confuse ppp links.
What happens if you try `-I' (icmp) or `-T' (tcp) tracerouting?

Could you try:
  traceroute -N 1 www.yahoo.com
on your ppp link?

If it helps, try various `-N' values up to 15.

Comment 3 Dmitry Butskoy 2008-01-28 11:13:11 EST
ping?...
Comment 4 Tom 2008-03-29 09:42:34 EDT
Well I am running kubuntu but this looks to be the same problem so hopefully
this is useful.  Your supposition looks correct.  On limited sampling -N 7 works
while -N 9 doesn't.  -N 8 works sometimes.

tcl@gilbereth:~$ ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.35.79.128  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:3431 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3809 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:2561146 (2.4 MB)  TX bytes:701211 (684.7 KB)

tcl@gilbereth:~$ traceroute -V
Modern traceroute for Linux, version 2.0.9, Sep 28 2007
Copyright (c) 2006  Dmitry Butskoy,   License: GPL
tcl@gilbereth:~$  dpkg -l traceroute
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                       Version                    Description
+++-==========================-==========================-====================================================================
ii  traceroute                 2.0.9~rc1-1                Traces the route taken
by packets over an IPv4/IPv6 network
tcl@gilbereth:~$ uname -a
Linux gilbereth 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux
tcl@gilbereth:~$

root@gilbereth:~# traceroute -N 7 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98 (172.18.51.98)  2881.429 ms  2883.331 ms  2883.308 ms

root@gilbereth:~# traceroute -N 8 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98 (172.18.51.98)  114.550 ms  116.491 ms  110.569 ms

root@gilbereth:~# traceroute -N 8 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -I 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -T 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
root@gilbereth:~# traceroute -T -N 7 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets

Comment 5 Dmitry Butskoy 2008-03-31 10:15:24 EDT
Well,

Could you check also with DNS disabled (traceroute -n) ? Does the level of 7
changed then?

What is your modem driver?
Comment 6 Tom 2008-04-01 07:14:04 EDT
Mostly the software is vanilla kububtu 7.10 up to date as of a week ago.  I have
a USB Huawei E220 which is switched to modem mode by huaweiAktBbo (the only
added software) before using kppp front end to connect.  In the main it runs a
acceptable HSDPA (T-Mobile UK) connection.

traceroute -n makes no apparent difference.

tcl@gilbereth:~$ traceroute -N 9 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 7 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  98.758 ms  108.747 ms  104.480 ms
 2  10.64.74.141  114.419 ms  114.388 ms  194.320 ms
 3  172.18.14.61  214.223 ms  120.348 ms  179.743 ms
tcl@gilbereth:~$ traceroute -N 9 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * *
send: No buffer space available
tcl@gilbereth:~$ traceroute -N 8 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  123.504 ms  117.673 ms  118.587 ms
 2  10.64.74.141  124.656 ms  125.548 ms  175.530 ms
 3  172.18.14.61  176.522 ms  177.484 ms  149.788 ms
tcl@gilbereth:~$
Comment 7 Dmitry Butskoy 2008-04-01 07:49:27 EDT
Thanks.

"send: No buffer space available" mean that send(2) syscall returns failure with
errno == ENOBUFS

It is not clear for me now, whether it is generated by the general socket stack,
or someway by pppd daemon. Is it possible for you to attach here a strace output
of your pppd daemon?

(After the ppp connection is established, determine the pid of pppd daemon
(pidof pppd), then run "strace -p pid_number -o out_file", and invoke
"traceroute" with -N > 8 on another terminal...)
Comment 8 Tom 2008-04-01 08:13:37 EDT
Sure.

strace -p `pidof pppd` -o fred
Process 12070 attached - interrupt to quit
Process 12070 detached
root@gilbereth:/root# cat fred
select(13, [11 12], NULL, [11 12], {11, 812000} <unfinished ...>
root@gilbereth:/root#

and

tcl@gilbereth:~$ traceroute -N 10 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~$

Comment 9 Dmitry Butskoy 2008-04-01 12:48:23 EDT
It seems the reason is the too small txqueuelen value (of 3) for ppp0 interface.

Could you increase this to, say 10 (by ifconfig or ip link) and report what
happens? (I'm not sure that the value of 10 is not too big for it though... :) )

I'll think these days about a fixing of the issue (and ship the next 2.0.10
version)...
Comment 10 Tom 2008-04-01 13:04:26 EDT
That looks like it!

tcl@gilbereth:~$ traceroute -N 10 -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.97  201.602 ms  220.378 ms  221.324 ms
 2  10.64.74.233  221.290 ms 10.64.74.229  230.269 ms 10.64.74.225  231.252 ms
 3  172.18.14.61  241.232 ms  240.303 ms  250.242 ms
 4  149.254.195.194  350.207 ms  178.736 ms  167.928 ms
 5  10.126.168.94  168.909 ms  168.885 ms  160.908 ms
 6  193.159.226.21  168.932 ms  160.826 ms  208.950 ms
 7  213.206.131.69  249.870 ms  168.729 ms  149.864 ms
 8  213.206.129.151  140.856 ms  148.862 ms  149.817 ms
 9  213.206.128.63  168.804 ms  160.903 ms  189.038 ms
10  82.195.189.186  209.883 ms  189.797 ms  268.118 ms
11  209.85.252.40  130.894 ms  130.927 ms 209.85.252.42  129.872 ms
12  64.233.175.213  289.043 ms 66.249.95.146  270.894 ms  308.984 ms
13  216.239.46.25  339.797 ms 72.14.233.113  340.936 ms  359.038 ms
14  66.249.94.92  278.035 ms  290.026 ms  309.922 ms
15  66.249.94.118  299.785 ms 72.14.236.130  260.780 ms 72.14.236.134  274.121 ms
16  72.14.207.99  296.041 ms  249.844 ms  259.799 ms
tcl@gilbereth:~$ ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.32.106.202  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:3737 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3749 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10
          RX bytes:3486420 (3.3 MB)  TX bytes:519159 (506.9 KB)
tcl@gilbereth:~$
Comment 11 Dmitry Butskoy 2008-04-07 11:50:04 EDT
Tom,

Could you compile and test the next version rc:
http://dmitry.butskoy.name/traceroute/traceroute-2.0.10rc1.tar.gz

(certainly, with tx_queue_len == 3, as usual).

If you cannot compile for some reason, I'll create an rpm for you.
Comment 12 Dmitry Butskoy 2008-04-11 07:41:55 EDT
ping?
Comment 13 Tom 2008-04-12 05:10:20 EDT
Yeah, I didn't have the modem for a few days.

That seems to have fixed it.

tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$
traceroute -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
send: No buffer space available
tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$
./traceroute -n 72.14.207.99
traceroute to 72.14.207.99 (72.14.207.99), 30 hops max, 40 byte packets
 1  172.18.51.98  91.580 ms  93.528 ms  100.601 ms
 2  10.64.74.141  102.516 ms  110.489 ms  112.455 ms
 3  172.18.14.60  121.435 ms  108.785 ms  109.731 ms
 4  * * *
 5  10.126.168.94  118.686 ms  128.659 ms  127.940 ms
 6  193.159.226.21  209.741 ms  210.686 ms  219.651 ms
 7  213.206.131.69  139.677 ms  137.939 ms  139.878 ms
 8  213.206.129.151  138.899 ms  129.898 ms  88.626 ms
 9  213.206.128.63  228.593 ms  220.839 ms  211.925 ms
10  82.195.189.186  99.961 ms  89.848 ms  90.806 ms
11  209.85.252.42  89.857 ms  89.706 ms 209.85.252.40  108.788 ms
12  64.233.175.213  160.007 ms  159.900 ms  159.935 ms
13  72.14.233.113  169.956 ms  169.936 ms  180.007 ms
14  66.249.94.90  179.019 ms  180.137 ms 72.14.236.183  179.133 ms
15  66.249.94.118  179.829 ms  179.839 ms  179.962 ms
16  72.14.207.99  187.936 ms  170.763 ms  180.952 ms
tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$ ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol
          inet addr:10.35.99.222  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:577 errors:0 dropped:0 overruns:0 frame:0
          TX packets:892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:99786 (97.4 KB)  TX bytes:77588 (75.7 KB)

tcl@gilbereth:~/Downloads/software-build/traceroute-2.0.10/traceroute$
Comment 14 Fedora Update System 2008-04-17 09:02:58 EDT
traceroute-2.0.10-1.fc8 has been submitted as an update for Fedora 8
Comment 15 Fedora Update System 2008-04-17 09:05:59 EDT
traceroute-2.0.10-1.fc7 has been submitted as an update for Fedora 7
Comment 16 Fedora Update System 2008-04-22 18:42:24 EDT
traceroute-2.0.10-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2008-04-22 18:43:15 EDT
traceroute-2.0.10-1.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.