Bug 1753506

Summary: net-snmpd invalid free
Product: [Fedora] Fedora Reporter: Josef Ridky <jridky>
Component: net-snmpAssignee: Josef Ridky <jridky>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 30CC: bryan, extras-qa, jliedy, jridky, jsafrane, pavel, ravpatil, wattersm, zdohnal
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: net-snmp-5.8-19.fc30 net-snmp-5.8-19.fc32 net-snmp-5.8-19.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1663027 Environment:
Last Closed: 2020-04-25 02:41:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1663027    
Bug Blocks:    

Description Josef Ridky 2019-09-19 06:38:47 UTC
--- Additional comment from  on 2019-09-17 22:14:43 CEST ---

fedora 30, snmp 5.8-10

==11498== Invalid free() / delete / delete[] / realloc()
==11498==    at 0x4839A0C: free (vg_replace_malloc.c:540)
==11498==    by 0x4C548CC: usm_rgenerate_out_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C55458: usm_secmod_rgenerate_out_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF46B8: snmpv3_packet_realloc_rbuild (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF86B4: snmp_build (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF8DC9: netsnmp_build_packet (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF9026: _build_initial_pdu_packet (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BFAE8A: snmp_sess_async_send (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x48A5817: netsnmp_wrap_up_request (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x48A82AD: netsnmp_check_delegated_requests (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x48A8DF0: netsnmp_check_outstanding_agent_requests (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x10D948: ??? (in /usr/sbin/snmpd)
==11498==  Address 0x14491000 is 0 bytes inside a block of size 104 free'd
==11498==    at 0x4839A0C: free (vg_replace_malloc.c:540)
==11498==    by 0x4C53A57: usm_rgenerate_out_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C55458: usm_secmod_rgenerate_out_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF46B8: snmpv3_packet_realloc_rbuild (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF86B4: snmp_build (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF8DC9: netsnmp_build_packet (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BF9026: _build_initial_pdu_packet (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x48A5747: netsnmp_wrap_up_request (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x48A82AD: netsnmp_check_delegated_requests (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x48A8DF0: netsnmp_check_outstanding_agent_requests (in /usr/lib64/libnetsnmpagent.so.35.0.0)
==11498==    by 0x10D948: ??? (in /usr/sbin/snmpd)
==11498==    by 0x10D1D4: ??? (in /usr/sbin/snmpd)
==11498==  Block was alloc'd at
==11498==    at 0x483AB1A: calloc (vg_replace_malloc.c:762)
==11498==    by 0x4C50B00: usm_process_in_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C51E8C: usm_secmod_process_in_msg (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BFEC70: snmpv3_parse (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4BFFF72: ??? (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C01846: ??? (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C02115: _sess_read (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C02C3C: snmp_sess_read2 (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x4C02C8A: snmp_read2 (in /usr/lib64/libnetsnmp.so.35.0.0)
==11498==    by 0x10DD81: ??? (in /usr/sbin/snmpd)
==11498==    by 0x10D1D4: ??? (in /usr/sbin/snmpd)
==11498==    by 0x4D9FF42: (below main) (in /usr/lib64/libc-2.29.so)

--- Additional comment from Josef Ridky on 2019-09-18 08:43:08 CEST ---

@Pavel do you have trap forwarding enabled in configuration?

--- Additional comment from  on 2019-09-18 11:47:06 CEST ---

No, it is disabled. In my config snmpd is master (agentx), agentx transport is unix sockets and subagent is agentx client based on agent++ library. security model is USM.

interesting it fails only on some tables (and no problem with the same table in <= fc28 os)

under valgrind it does not crash (think because of valgrind virtual machine) and shows some interesting messages like

send response: USM encryption error (build string: bad header, length too short: 2 < 11)

or

send response: USM encryption error

or 

send response: USM encryption error (Can't build OID for variable)

i tried to review subagent source code but didnt manage to find any difference between "good" and "bad" tables

may be PDU size limitation ? "bad" table contains 7 columns and one of columns has length ~ 200 characters, snmpd does not crash when subagent returns 1 table entry but immediately crashes when i add second record. anyway i don't see this crash in previous versions of snmpd

Comment 1 Josef Ridky 2019-09-19 07:06:40 UTC
I have prepared scratch build with possible solution.

@pavel Are you able to test it on your case?

Scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=37736050

Comment 2 pavel 2019-09-19 15:21:44 UTC
5.8-13, fc30: snmpd daemon do not crash, command line tools work as required but now i see problem with perl module SNMP::Session

i have test table with 2000 records, snmptable fetches all rows, SNMP::Session::get_table stops after ~127 rows with status:

(genError) A general failure occured

(tooBig) Response message would have been too large.

can successfully retrieve the same table under fc28

Comment 3 Michael Watters 2019-12-30 20:29:29 UTC
This issue also appears to affect RHEL/CentOS 8.  The snmpd service crashes every few hours with log entries similar to below.


Dec 30 01:25:39 host.example.com  systemd-coredump[788]: Process 30476 (snmpd) of user 0 dumped core.
                                                                         
                                                                         Stack trace of thread 30476:
                                                                         #0  0x00007f26c126e93f raise (libc.so.6)
                                                                         #1  0x00007f26c1258c95 abort (libc.so.6)
                                                                         #2  0x00007f26c12b1d57 __libc_message (libc.so.6)
                                                                         #3  0x00007f26c12b868c malloc_printerr (libc.so.6)
                                                                         #4  0x00007f26c12ba027 _int_free (libc.so.6)
                                                                         #5  0x00007f26c2bd1275 usm_generate_out_msg (libnetsnmp.so.35)
                                                                         #6  0x00007f26c2bd1ae9 usm_secmod_generate_out_msg (libnetsnmp.so.35)
                                                                         #7  0x00007f26c2b74ed2 snmpv3_packet_build (libnetsnmp.so.35)
                                                                         #8  0x00007f26c2b76f07 snmp_build (libnetsnmp.so.35)
                                                                         #9  0x00007f26c2b7742b netsnmp_build_packet (libnetsnmp.so.35)
                                                                         #10 0x00007f26c2b776d6 _build_initial_pdu_packet (libnetsnmp.so.35)
                                                                         #11 0x00007f26c33df5c9 netsnmp_wrap_up_request (libnetsnmpagent.so.35)
                                                                         #12 0x00007f26c33e277b netsnmp_handle_request (libnetsnmpagent.so.35)
                                                                         #13 0x00007f26c33e2a9a handle_snmp_packet (libnetsnmpagent.so.35)
                                                                         #14 0x00007f26c2b7f712 _sess_process_packet (libnetsnmp.so.35)
                                                                         #15 0x00007f26c2b80776 _sess_read (libnetsnmp.so.35)
                                                                         #16 0x00007f26c2b812dd snmp_sess_read2 (libnetsnmp.so.35)
                                                                         #17 0x00007f26c2b8132b snmp_read2 (libnetsnmp.so.35)
                                                                         #18 0x000055dd2afb9377 receive (snmpd)
                                                                         #19 0x000055dd2afb866e main (snmpd)
                                                                         #20 0x00007f26c125a813 __libc_start_main (libc.so.6)
                                                                         #21 0x000055dd2afb8abe _start (snmpd)

Package info is as follows.

[root@host ~]# rpm -qi net-snmp
Name        : net-snmp
Epoch       : 1
Version     : 5.8
Release     : 7.el8_0.2
Architecture: x86_64

Comment 4 Fedora Update System 2020-04-09 16:26:03 UTC
FEDORA-2020-c3ab547325 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3ab547325

Comment 5 Fedora Update System 2020-04-09 16:26:04 UTC
FEDORA-2020-44b769adaf has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-44b769adaf

Comment 6 Fedora Update System 2020-04-09 16:26:06 UTC
FEDORA-2020-7303a69282 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-7303a69282

Comment 7 Fedora Update System 2020-04-10 16:56:23 UTC
FEDORA-2020-c3ab547325 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-c3ab547325`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c3ab547325

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2020-04-10 21:36:55 UTC
FEDORA-2020-44b769adaf has been pushed to the Fedora 30 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-44b769adaf`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-44b769adaf

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2020-04-11 18:50:59 UTC
FEDORA-2020-7303a69282 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-7303a69282`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7303a69282

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2020-04-25 02:41:00 UTC
FEDORA-2020-44b769adaf has been pushed to the Fedora 30 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 Fedora Update System 2020-04-26 02:50:44 UTC
FEDORA-2020-7303a69282 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 12 Fedora Update System 2020-04-28 02:55:53 UTC
FEDORA-2020-c3ab547325 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.