Description of problem:
When the net-snmp agent receives a SNMP packet including mulitple vars binding, it will generate multiple SMUX-PDUs for a SMUX peer.
The request-id field of the PDUs should remain the same for the several SMUX-PDUs.
Our customer's application relies on the request-id to remain the same.
With net-snmp 5.5-31, the request-id does change for each SMUX-PDUs, which is in contradiction with the paragraph 3.1.5 of the RFC 1227:
When the SNMP agent constructs an SNMP GetRequest-PDU,
GetNextRequest-PDU, or SetRequest-PDU, for a SMUX peer, the
request_id field of the SNMP takes a special meaning: if an SNMP
agent generates multiple PDUs for a SMUX peer, upon receipt of a
single PDU from the network management station, then the request_id
field of the PDUs sent to the SMUX peer must take the same value
(which need bear no relationship to the value of the request_id field
of the PDU originally received by the SNMP agent.)"
This did not occur with older ucd-snmp-4.x
Version-Release number of selected component (if applicable):
Always for our customer
Steps to Reproduce:
1. Send a request with mulitple vars binding
2. Capture the network traffic
3. Check the request-id field
The request-id is incremented
The request-id remains the same, as with ucd-snmp 4.x
The code path in agent/mibgroup/smux/smux.c is very similar between ucd-snmp-4.x and net-snmp-5.x.
As far as I can see, the request-id is a global static variable "smux_reqid" which is incremented only in smux_snmp_process().
By looking at the debug logs from ucd-snmp and net-snmp, smux_snmp_process() is being called as many times in both cases, so I would expect "smux_reqid" to be incremented the same.
Yet in the case of ucd-snmp the request-id remains the same, I must be missing something here but I can't find out what...
$ grep dumpx_smux_snmp_process net-snmp5.5.31.log
dumpx_smux_snmp_process:A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06
dumpx_smux_snmp_process:A2 27 02 01 02 02 01 00 02 01 00 30 1C 30 1A 06
dumpx_smux_snmp_process:A2 25 02 01 03 02 01 00 02 01 00 30 1A 30 18 06
dumpx_smux_snmp_process:A2 22 02 01 04 02 01 00 02 01 00 30 17 30 15 06
$ grep dumpx_smux_snmp_process ucd-snmp.log
dumpx_smux_snmp_process: A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06
dumpx_smux_snmp_process: A2 27 02 01 01 02 01 00 02 01 00 30 1C 30 1A 06
dumpx_smux_snmp_process: A2 25 02 01 01 02 01 00 02 01 00 30 1A 30 18 06
dumpx_smux_snmp_process: A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06
It's increasing in net-snmp5.5.31.log but not in ucd-snmp.log
There is one similar report upstream here:
But I haven't found any follow up upstream.
There's one change in the reqid that landed in net-snmp around the 5.x time:
but I am not sure how this could relate to the observed behavior. Also, our customer is using 32bits in both cases (ucd-snmp and net-snmp), so that would rule out a 64bit issue.
Created attachment 502299 [details]
simple (and dummy) reproducer
see the description inside the script
Created attachment 502564 [details]
simple reproducer of set-request with 3 variables
Here is better reproducer, now with set-requests, taken directly from customer's wireshark traces.
Created attachment 512456 [details]
I've enjoyed some quiet days in our office when everyone sane is on vacation and I think I have a patch. But it touches *really* ancient code and I don't even have a SMUX agent with SNMP-SET support to test with. Do we have any internally? I tried Dell OpenManage server, but I failed to get SNMP-SET working.
Created attachment 512642 [details]
experimental patch v2
I am sorry, last patch did only part of the job. This second version should be much better when processing multiple variables in one request.
Test build: http://download.devel.redhat.com/brewroot///scratch/jsafrane/task_3483848/
I've rebuilt the package with latest RHEL 6.2 patches and stored on my page where it will stay longer than in brew.
The reproducer from bug 394591 makes snmpd crash. I think it might be related to this bugfix.
Below is a backtrace from ppc64 but snmpd crashes on all platforms (ppc64 was available ATM)
Program received signal SIGABRT, Aborted.
0x00000fff8ec752c0 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
#0 0x00000fff8ec752c0 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00000fff8ec77274 in abort () at abort.c:92
#2 0x00000fff8ecb823c in __libc_message (do_abort=<value optimized out>,
fmt=0xfff8eda5b20 "*** glibc detected *** %s: %s: 0x%s ***\n")
#3 0x00000fff8ecc01f4 in malloc_printerr (action=<value optimized out>,
str=0xfff8eda6030 "double free or corruption (out)", ptr=<value optimized out>) at malloc.c:6283
#4 0x00000fff8f58cf34 in netsnmp_handler_free (handler=0x10008eed7d0) at agent_handler.c:591
#5 0x00000fff8f58d000 in netsnmp_handler_registration_free (reginfo=0x10008d79910) at agent_handler.c:633
#6 0x00000fff8f59520c in smux_peer_cleanup (sd=<value optimized out>) at mibgroup/smux/smux.c:1898
#7 0x00000fff8f5969b8 in smux_snmp_process (exact=<value optimized out>, objid=0x10008f61020,
len=0x10008f61000, return_len=0xfffc9a4fda0, return_type=0xfffc9a4fdaf <incomplete sequence \350>,
sd=<value optimized out>) at mibgroup/smux/smux.c:1561
#8 0x00000fff8f59723c in var_smux_get_new (root=0x10008ed3c10, root_len=7, name=0x10008f61020,
length=0x10008f61000, exact=<value optimized out>, var_len=0xfffc9a4fda0,
var_type=0xfffc9a4fdaf <incomplete sequence \350>) at mibgroup/smux/smux.c:379
#9 0x00000fff8f59807c in smux_handler (handler=<value optimized out>, reginfo=0x10008d79910,
reqinfo=0x10008f05630, requests=0x10008ef9590) at mibgroup/smux/smux.c:327
#10 0x00000fff8f58f448 in netsnmp_call_handler (reginfo=0x10008d79910, reqinfo=0x10008f05630,
requests=0x10008ef9590) at agent_handler.c:440
#11 netsnmp_call_handlers (reginfo=0x10008d79910, reqinfo=0x10008f05630, requests=0x10008ef9590)
#12 0x00000fff8f57ad24 in handle_var_requests (asp=0x10008f10b00) at snmp_agent.c:2611
#13 0x00000fff8f57d13c in handle_pdu (asp=0x10008f10b00) at snmp_agent.c:3407
#14 0x00000fff8f57ffa8 in netsnmp_handle_request (asp=0x10008f10b00, status=<value optimized out>)
#15 0x00000fff8f580dbc in handle_snmp_packet (op=<value optimized out>, session=<value optimized out>,
reqid=<value optimized out>, pdu=<value optimized out>, magic=<value optimized out>) at snmp_agent.c:1929
#16 0x00000fff8eb3517c in _sess_process_packet (sessp=0x10008f064a0, sp=0x10008f2bdc0, isp=0x10008f10080,
transport=<value optimized out>, opaque=<value optimized out>, olength=<value optimized out>,
packetptr=<value optimized out>, length=<value optimized out>) at snmp_api.c:5604
#17 0x00000fff8eb35c24 in _sess_read (sessp=Cannot access memory at address 0xfffc9a502a8
) at snmp_api.c:6043
Cannot access memory at address 0xfffc9a50400
I am also getting another snmpd crash when running the query on quagga/zebra
# snmpwalk -c public -v1 localhost .184.108.40.206.220.127.116.11.2
Timeout: No Response from localhost
Program received signal SIGSEGV, Segmentation fault.
0x00007f965867c82c in __libc_free (mem=0x7f965c848a50) at malloc.c:3724
3724 ar_ptr = arena_for_chunk(p);
#0 0x00007f965867c82c in __libc_free (mem=0x7f965c848a50) at malloc.c:3724
#1 0x00007f965a24d945 in netsnmp_handler_free (handler=0x7f965c8d89a0) at agent_handler.c:591
#2 0x00007f965a24d9c2 in netsnmp_handler_registration_free (reginfo=0x7f965c8f8340) at agent_handler.c:633
#3 0x00007f965a254520 in smux_peer_cleanup (sd=11) at mibgroup/smux/smux.c:1898
#4 0x00007f965a257c2a in smux_pdu_process (fd=11, data=0x7fffad842930 "", length=2)
#5 0x00007f965a2584ed in smux_process (fd=11) at mibgroup/smux/smux.c:815
#6 0x00007f965a6a2f9a in receive (argc=<value optimized out>, argv=<value optimized out>) at snmpd.c:1205
#7 main (argc=<value optimized out>, argv=<value optimized out>) at snmpd.c:1060
(In reply to comment #34)
> I am also getting another snmpd crash when running the query on quagga/zebra
This needs to be fixed.
I've fixed it upstream in git commit 94b710d27e2972831dbc17b27011f2d11da8afc1
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
In previous net-snmp versions, snmpd did not distinguish between outgoing SMUX messages and always incremented their Request-Id, even when multiple SMUX messages were sent as result of one incoming SNMP request with multiple variables. RFC 1227 requires, that these SMUX messages should have the same Request-Id. With this update, snmpd properly recognizes multiple SMUX outgoing messages due to one incoming SNMP request and sets the same Request-Id to them.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.