Bug 708370

Summary:

net-snmp increments request-id when generating multiple SMUX-PDUs for a SMUX peer

Product:

Red Hat Enterprise Linux 6

Reporter:

Olivier Fourdan <ofourdan>

Component:

net-snmp

Assignee:

Jan Safranek <jsafrane>

Status:

CLOSED ERRATA

QA Contact:

BaseOS QE Security Team <qe-baseos-security>

Severity:

high

Docs Contact:

Priority:

high

Version:

6.0

CC:

cward, kem, ksrot, rkhadgar, rvokal, smarkovi

Target Milestone:

Keywords:

OtherQA

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

In previous net-snmp versions, snmpd did not distinguish between outgoing SMUX messages and always incremented their Request-Id, even when multiple SMUX messages were sent as result of one incoming SNMP request with multiple variables. RFC 1227 requires, that these SMUX messages should have the same Request-Id. With this update, snmpd properly recognizes multiple SMUX outgoing messages due to one incoming SNMP request and sets the same Request-Id to them.

Story Points:

---

Clone Of:

Environment:

Last Closed:

2011-12-06 17:11:55 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
simple (and dummy) reproducer	none
simple reproducer of set-request with 3 variables	none
experimental patch	none
experimental patch v2	none

Description Olivier Fourdan 2011-05-27 13:22:24 UTC

Description of problem:

When the net-snmp agent receives a SNMP packet including mulitple vars binding, it will generate multiple SMUX-PDUs for a SMUX peer.

The request-id field of the PDUs should remain the same for the several SMUX-PDUs. 

Our customer's application relies on the request-id to remain the same.

With net-snmp 5.5-31, the request-id does change for each SMUX-PDUs, which is in contradiction with the paragraph 3.1.5 of the RFC 1227:

  "3.1.5. Request-ID
   When the SNMP agent constructs an SNMP GetRequest-PDU,
   GetNextRequest-PDU, or SetRequest-PDU, for a SMUX peer, the
   request_id field of the SNMP takes a special meaning: if an SNMP
   agent generates multiple PDUs for a SMUX peer, upon receipt of a
   single PDU from the network management station, then the request_id
   field of the PDUs sent to the SMUX peer must take the same value
   (which need bear no relationship to the value of the request_id field
   of the PDU originally received by the SNMP agent.)"

   See http://www.faqs.org/rfcs/rfc1227.html

This did not occur with older ucd-snmp-4.x

Version-Release number of selected component (if applicable):

net-snmp-5.5

How reproducible:

Always for our customer

Steps to Reproduce:
1. Send a request with mulitple vars binding
2. Capture the network traffic
3. Check the request-id field
  
Actual results:

The request-id is incremented

Expected results:

The request-id remains the same, as with ucd-snmp 4.x

Additional info:

The code path in agent/mibgroup/smux/smux.c is very similar between ucd-snmp-4.x and net-snmp-5.x.

As far as I can see, the request-id is a global static variable "smux_reqid" which is incremented only in smux_snmp_process().

By looking at the debug logs from ucd-snmp and net-snmp, smux_snmp_process() is being called as many times in both cases, so I would expect "smux_reqid" to be incremented the same.

Yet in the case of ucd-snmp the request-id remains the same, I must be missing something here but I can't find out what...

$ grep dumpx_smux_snmp_process net-snmp5.5.31.log 
dumpx_smux_snmp_process:A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06 
dumpx_smux_snmp_process:A2 27 02 01 02 02 01 00 02 01 00 30 1C 30 1A 06 
dumpx_smux_snmp_process:A2 25 02 01 03 02 01 00 02 01 00 30 1A 30 18 06 
dumpx_smux_snmp_process:A2 22 02 01 04 02 01 00 02 01 00 30 17 30 15 06 
                                    ^^
                                   reqid

$ grep dumpx_smux_snmp_process ucd-snmp.log
dumpx_smux_snmp_process:  A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06
dumpx_smux_snmp_process:  A2 27 02 01 01 02 01 00 02 01 00 30 1C 30 1A 06
dumpx_smux_snmp_process:  A2 25 02 01 01 02 01 00 02 01 00 30 1A 30 18 06
dumpx_smux_snmp_process:  A2 22 02 01 01 02 01 00 02 01 00 30 17 30 15 06
                                      ^^
                                     reqid

It's increasing in net-snmp5.5.31.log but not in ucd-snmp.log

There is one similar report upstream here:

  http://www.mail-archive.com/net-snmp-users@lists.sourceforge.net/msg09860.html

But I haven't found any follow up upstream.

There's one change in the reqid that landed in net-snmp around the 5.x time:

  http://sourceforge.net/tracker/?func=detail&atid=312694&aid=1225963&group_id=12694

but I am not sure how this could relate to the observed behavior. Also, our customer is using 32bits in both cases (ucd-snmp and net-snmp), so that would rule out a 64bit issue.

Comment 5 Jan Safranek 2011-06-01 15:04:10 UTC

Created attachment 502299 [details]
simple (and dummy) reproducer

see the description inside the script

Comment 6 Jan Safranek 2011-06-02 15:01:13 UTC

Created attachment 502564 [details]
simple reproducer of set-request with 3 variables

Here is better reproducer, now with set-requests, taken directly from customer's wireshark traces.

Comment 16 Jan Safranek 2011-07-12 15:39:13 UTC

Created attachment 512456 [details]
experimental patch

I've enjoyed some quiet days in our office when everyone sane is on vacation and I think I have a patch. But it touches *really* ancient code and I don't even have a SMUX agent with SNMP-SET support to test with. Do we have any internally? I tried Dell OpenManage server, but I failed to get SNMP-SET working.

Comment 18 Jan Safranek 2011-07-13 11:51:05 UTC

Created attachment 512642 [details]
experimental patch v2

I am sorry, last patch did only part of the job. This second version should be much better when processing multiple variables in one request.

Test build: http://download.devel.redhat.com/brewroot///scratch/jsafrane/task_3483848/

Comment 28 Jan Safranek 2011-08-12 10:17:08 UTC

I've rebuilt the package with latest RHEL 6.2 patches and stored on my page where it will stay longer than in brew.
http://people.redhat.com/jsafrane/bugs/708370/

Comment 32 Karel Srot 2011-08-22 14:19:20 UTC

The reproducer from bug 394591 makes snmpd crash. I think it might be related to this bugfix.

Below is a backtrace from ppc64 but snmpd crashes on all platforms (ppc64 was available ATM)

Program received signal SIGABRT, Aborted.
0x00000fff8ec752c0 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00000fff8ec752c0 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00000fff8ec77274 in abort () at abort.c:92
#2  0x00000fff8ecb823c in __libc_message (do_abort=<value optimized out>, 
    fmt=0xfff8eda5b20 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3  0x00000fff8ecc01f4 in malloc_printerr (action=<value optimized out>, 
    str=0xfff8eda6030 "double free or corruption (out)", ptr=<value optimized out>) at malloc.c:6283
#4  0x00000fff8f58cf34 in netsnmp_handler_free (handler=0x10008eed7d0) at agent_handler.c:591
#5  0x00000fff8f58d000 in netsnmp_handler_registration_free (reginfo=0x10008d79910) at agent_handler.c:633
#6  0x00000fff8f59520c in smux_peer_cleanup (sd=<value optimized out>) at mibgroup/smux/smux.c:1898
#7  0x00000fff8f5969b8 in smux_snmp_process (exact=<value optimized out>, objid=0x10008f61020, 
    len=0x10008f61000, return_len=0xfffc9a4fda0, return_type=0xfffc9a4fdaf  <incomplete sequence \350>, 
    sd=<value optimized out>) at mibgroup/smux/smux.c:1561
#8  0x00000fff8f59723c in var_smux_get_new (root=0x10008ed3c10, root_len=7, name=0x10008f61020, 
    length=0x10008f61000, exact=<value optimized out>, var_len=0xfffc9a4fda0, 
    var_type=0xfffc9a4fdaf  <incomplete sequence \350>) at mibgroup/smux/smux.c:379
#9  0x00000fff8f59807c in smux_handler (handler=<value optimized out>, reginfo=0x10008d79910, 
    reqinfo=0x10008f05630, requests=0x10008ef9590) at mibgroup/smux/smux.c:327
#10 0x00000fff8f58f448 in netsnmp_call_handler (reginfo=0x10008d79910, reqinfo=0x10008f05630, 
    requests=0x10008ef9590) at agent_handler.c:440
#11 netsnmp_call_handlers (reginfo=0x10008d79910, reqinfo=0x10008f05630, requests=0x10008ef9590)
    at agent_handler.c:521
#12 0x00000fff8f57ad24 in handle_var_requests (asp=0x10008f10b00) at snmp_agent.c:2611
#13 0x00000fff8f57d13c in handle_pdu (asp=0x10008f10b00) at snmp_agent.c:3407
#14 0x00000fff8f57ffa8 in netsnmp_handle_request (asp=0x10008f10b00, status=<value optimized out>)
    at snmp_agent.c:3203
#15 0x00000fff8f580dbc in handle_snmp_packet (op=<value optimized out>, session=<value optimized out>, 
    reqid=<value optimized out>, pdu=<value optimized out>, magic=<value optimized out>) at snmp_agent.c:1929
#16 0x00000fff8eb3517c in _sess_process_packet (sessp=0x10008f064a0, sp=0x10008f2bdc0, isp=0x10008f10080, 
    transport=<value optimized out>, opaque=<value optimized out>, olength=<value optimized out>, 
    packetptr=<value optimized out>, length=<value optimized out>) at snmp_api.c:5604
#17 0x00000fff8eb35c24 in _sess_read (sessp=Cannot access memory at address 0xfffc9a502a8
) at snmp_api.c:6043
Cannot access memory at address 0xfffc9a50400

Comment 34 Karel Srot 2011-08-23 12:43:36 UTC

I am also getting another snmpd crash when running the query on quagga/zebra

# snmpwalk -c public -v1 localhost .1.3.6.1.2.1.4.24.2
Timeout: No Response from localhost

Program received signal SIGSEGV, Segmentation fault.
0x00007f965867c82c in __libc_free (mem=0x7f965c848a50) at malloc.c:3724
3724	  ar_ptr = arena_for_chunk(p);
(gdb) bt
#0  0x00007f965867c82c in __libc_free (mem=0x7f965c848a50) at malloc.c:3724
#1  0x00007f965a24d945 in netsnmp_handler_free (handler=0x7f965c8d89a0) at agent_handler.c:591
#2  0x00007f965a24d9c2 in netsnmp_handler_registration_free (reginfo=0x7f965c8f8340) at agent_handler.c:633
#3  0x00007f965a254520 in smux_peer_cleanup (sd=11) at mibgroup/smux/smux.c:1898
#4  0x00007f965a257c2a in smux_pdu_process (fd=11, data=0x7fffad842930 "", length=2)
    at mibgroup/smux/smux.c:884
#5  0x00007f965a2584ed in smux_process (fd=11) at mibgroup/smux/smux.c:815
#6  0x00007f965a6a2f9a in receive (argc=<value optimized out>, argv=<value optimized out>) at snmpd.c:1205
#7  main (argc=<value optimized out>, argv=<value optimized out>) at snmpd.c:1060

Comment 35 Jan Safranek 2011-08-23 14:37:03 UTC

(In reply to comment #34)
> I am also getting another snmpd crash when running the query on quagga/zebra

This needs to be fixed.

Comment 36 Jan Safranek 2011-08-23 16:21:32 UTC

I've fixed it upstream in git commit 94b710d27e2972831dbc17b27011f2d11da8afc1

Comment 44 Jan Safranek 2011-11-23 16:35:26 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
In previous net-snmp versions, snmpd did not distinguish between outgoing SMUX messages and always incremented their Request-Id, even when multiple SMUX messages were sent as result of one incoming SNMP request with multiple variables. RFC 1227 requires, that these SMUX messages should have the same Request-Id. With this update, snmpd properly recognizes multiple SMUX outgoing messages due to one incoming SNMP request and sets the same Request-Id to them.

Comment 45 errata-xmlrpc 2011-12-06 17:11:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1524.html