Bug 1261424 - The function of snmp doesn't work after auto install with snmp_password parameter
Summary: The function of snmp doesn't work after auto install with snmp_password param...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node
Version: 3.5.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-3.6.2
: 3.6.0
Assignee: Ryan Barry
QA Contact: cshao
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-09 10:01 UTC by cshao
Modified: 2016-03-09 14:37 UTC (History)
13 users (show)

(edit)
Cause: Auto-installing RHEV-H and setting an SNMP password using the snmp_password kernel argument to enable SNMP

Consequence: SNMP did not work

Fix: RHEV-H is now updting the password correctly

Result: SNMP is enabled after auto-installation
Clone Of:
(edit)
Last Closed: 2016-03-09 14:37:14 UTC


Attachments (Terms of Use)
/var/log/*.* (164.87 KB, application/x-gzip)
2015-09-09 10:01 UTC, cshao
no flags Details
/var/log/*.* (102.02 KB, application/x-gzip)
2015-12-16 02:46 UTC, cshao
no flags Details
snmp-fail1.png (270.65 KB, image/png)
2015-12-16 02:48 UTC, cshao
no flags Details
snmp-fail2.png (237.30 KB, image/png)
2015-12-16 02:49 UTC, cshao
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0378 normal SHIPPED_LIVE ovirt-node bug fix and enhancement update for RHEV 3.6 2016-03-09 19:06:36 UTC
oVirt gerrit 46533 master MERGED Persist all of /var/lib/net-snmp instead of just snmpd.conf Never
oVirt gerrit 49033 ovirt-3.6 MERGED Persist all of /var/lib/net-snmp instead of just snmpd.conf Never
oVirt gerrit 50634 master MERGED We should cat the old snmp_conf, not the snmp_conf dir 2015-12-22 12:01 UTC
oVirt gerrit 50914 ovirt-3.6 MERGED We should cat the old snmp_conf, not the snmp_conf dir 2015-12-22 15:00 UTC
Red Hat Bugzilla 1232412 None None None Never

Internal Trackers: 1232412

Description cshao 2015-09-09 10:01:07 UTC
Created attachment 1071665 [details]
/var/log/*.*

Description of problem:
The function of snmp doesn't work after auto install with snmp_password parameter.


# snmpwalk -v3 -u root -l authPriv -a SHA -A redhat123 -x AES -X redhat123 $ip sysDescr
snmpwalk: Unknown engine ID (Sub-id not found: (top) -> sysDescr)


# service snmpd status
Redirecting to /bin/systemctl status  snmpd.service
snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
   Loaded: loaded (/usr/lib/systemd/system/snmpd.service; enabled)
   Active: active (running) since Tue 2015-09-08 07:27:33 UTC; 2h 43min ago
 Main PID: 1233 (snmpd)
   CGroup: /system.slice/snmpd.service
           └─1233 /usr/sbin/snmpd -LS0-6d -f

Sep 08 07:27:33 localhost snmpd[1233]: buffer too small to read octet string (17 < 17)
Sep 08 07:27:33 localhost snmpd[1233]: Turning on AgentX master support.
Sep 08 07:27:33 localhost snmpd[1233]: Cannot rename /var/lib/net-snmp/snmpd.conf to /var/lib/net-snmp/snmpd.0.conf
Sep 08 07:27:33 localhost snmpd[1233]: Cannot unlink /var/lib/net-snmp/snmpd.conf
Sep 08 07:27:33 localhost snmpd[1233]: NET-SNMP version 5.7.2
Sep 08 07:27:33 localhost systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..



Version-Release number of selected component (if applicable):
rhev-hypervisor7-7.1-20150827.1.el7ev
ovirt-node-3.2.3-20.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Auto install RHEV-H with below parameters
BOOTIF=eth0 storage_init=/dev/sda adminpw=xxx snmp_password=redhat123 firstboot
2. Reboot host
3. Login RHEV-H and check the status of snmp.
4. Check snmp function with below cmd.
#snmpwalk -v3 -u root -l authPriv -a SHA -A redhat123 -x AES -X redhat123 $ip sysDescr.
5. Set new snmp password on TUI.

Test result:
1. Auto install with "snmp" parameter can enable snmp.
2. In SNMP menu, SNMP show as enable status.
3. After Step4: SNMP function doesn't work.
# snmpwalk -v3 -u root -l authPriv -a SHA -A redhat123 -x AES -X redhat123 $ip sysDescr
snmpwalk: Unknown engine ID (Sub-id not found: (top) -> sysDescr)
4. After Step5: An error occurred while applying the changes:    

│(1/1) Enabling SNMP and setting the password                                                                        
│An error occurred while applying the changes:                  
│Command 'snmpusm -v 3 -u root -n "" -l authNoPriv -a SHA -A redhat123 localhost passwd redhat123 11111111 -x AES' returned non-zero exit status 1  


Expected results:
The function of snmp can work after auto install with snmp_password parameter

Additional info:
No such issue with TUI install.

Comment 1 Fabian Deutsch 2015-09-16 10:33:19 UTC
Does this bug require a different fix than bug 1232412?

Comment 2 Ryan Barry 2015-09-16 14:14:41 UTC
(In reply to Fabian Deutsch from comment #1)
> Does this bug require a different fix than bug 1232412?

I have no idea. Was that fix included in ovirt-node-3.2.3-20?

Comment 3 Ryan Barry 2015-09-16 19:43:12 UTC
Jan -

Is there a way to disable this behavior?

RHEV-H is a read-only root image which keeps persistent files on another partition and bind mounts them. This behavior appears to have changed in 7.1

Even in instances where snmp is properly configured (and auth works) prior to a reboot, restarting snmp appears to attempt to rename (doesn't work -- bind mount) and unlink every time, while also erasing or ignoring previous authorization.

This behavior can be observed by grabbing the named image (rhev-hypervisor7-7.1-20150827.1 from brew) and booting it in KVM.

Add "BOOTIF=ens3 storage_init adminpw=42OLJtXwdXtEY snmp_password=redhat123 node.break=post-installer" to the kernel args

Autoinstall will complete. SNMP works. Reboot, and snmp does not work, with errors about renaming in the snmp service logs.

This is a change from RHEL 7.0

Is there a workaround?

Comment 4 Ryan Barry 2015-09-16 19:43:51 UTC
(In reply to Ryan Barry from comment #3)
> Jan -
> 
> Is there a way to disable this behavior?
> 
> RHEV-H is a read-only root image which keeps persistent files on another
> partition and bind mounts them. This behavior appears to have changed in 7.1
> 
> Even in instances where snmp is properly configured (and auth works) prior
> to a reboot, restarting snmp appears to attempt to rename (doesn't work --
> bind mount) and unlink every time, while also erasing or ignoring previous
> authorization.
> 
> This behavior can be observed by grabbing the named image
> (rhev-hypervisor7-7.1-20150827.1 from brew) and booting it in KVM.
> 
> Add "BOOTIF=ens3 storage_init adminpw=42OLJtXwdXtEY snmp_password=redhat123
> node.break=post-installer" to the kernel args
> 
> Autoinstall will complete. SNMP works. Reboot, and snmp does not work, with
> errors about renaming in the snmp service logs.
> 
> This is a change from RHEL 7.0
> 
> Is there a workaround?

Note: that hashed password is "ovirt", and you log in as the "admin" user

Comment 5 Jan Safranek 2015-09-22 08:34:37 UTC
There were no interesting changes in writing to /var/lib/net-snmp/snmpd.conf in RHEL 7.1 or 7.2.

snmpd expects it can either rename or remove its own persistent state file in /var:

    if (rename(file, fileold)) {
        snmp_log(LOG_ERR, "Cannot rename %s to %s\n", file, fileold);
        /* moving it failed, try nuking it, as leaving
         * it around is very bad. */
        if (unlink(file) == -1)
            snmp_log(LOG_ERR, "Cannot unlink %s\n", file);
    }

And then snmpd *appends* its persistent state to already existing file -> problems on the next startup.

If it was working in RHEL 7.0 or anytime earlier, something must have changed on RHEV side.

Comment 7 Jan Safranek 2015-09-22 08:38:59 UTC
Out of curiosity, can you bind-mount whole /var/lib/net-snmp directory instead of just snmpd.conf file? snmpd would be much happier then.

Comment 8 Ryan Barry 2015-09-22 15:57:15 UTC
(In reply to Jan Safranek from comment #7)
> Out of curiosity, can you bind-mount whole /var/lib/net-snmp directory
> instead of just snmpd.conf file? snmpd would be much happier then.

We can do this. I just tested this, and it works as expected. Thanks, Jan.

Comment 12 cshao 2015-12-16 02:45:41 UTC
Met new error during auto install with snmp parameters.

Test version:
rhev-hypervisor7-7.2-20151210.1
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

Test steps:
1. Auto install RHEV-H with below parameters
BOOTIF=eth0 storage_init=/dev/sda adminpw=xxx snmp_password=redhat123 firstboot

Test result:
Auto install failed with snmp parameters.

2015-12-16 02:05:59,443      ERROR '<class 'subprocess.CalledProcessError'>' on transaction '<Transaction elements='[<ConfigureSNMP 'Enabling SNMP and setting the password'>]' title='Configuring SNMP' at 0x71cf810>': Command 'snmpusm -v 3 -u root -n "" -l authNoPriv -a SHA -A cat: /var/lib/net-snmp/: Is a directory localhost passwd cat: /var/lib/net-snmp/: Is a directory redhat123 -x AES' returned non-zero exit status 1 - 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 162, in run_transaction
    e.commit()
  File "/usr/lib/python2.7/site-packages/ovirt/node/setup/snmp/snmp_model.py", line 109, in commit
    enable_snmpd(snmp_password)
  File "/usr/lib/python2.7/site-packages/ovirt/node/setup/snmp/snmp_model.py", line 51, in enable_snmpd
    change_password(oldpwd)
  File "/usr/lib/python2.7/site-packages/ovirt/node/setup/snmp/snmp_model.py", line 37, in change_password
    process.check_call(pwd_change_cmd, shell=True)
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/process.py", line 139, in check_call
    return int(subprocess.check_call(*args, **kwargs))
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'snmpusm -v 3 -u root -n "" -l authNoPriv -a SHA -A cat: /var/lib/net-snmp/: Is a directory localhost passwd cat: /var/lib/net-snmp/: Is a directory redhat123 -x AES' returned non-zero exit status 1
2015-12-16 02:05:59,485       INFO Captured nothing
2015-12-16 02:05:59,488      ERROR '<class 'subprocess.CalledProcessError'>' on transaction '<Transaction elements='[<PrepareInstallation 'Prepare installation'>, <AutomaticDiskPartitioning 'Performing automatic disk partitioning'>, <ConfigureNetworking 'Configuring network'>, <ConfigureLogging 'Configuring Logging'>, <PerformInstallation 'Transferring image'>, <ConfigureKdump 'Configuring KDump'>, <InstallBootloader 'Installing Bootloader'>, <RunHooks 'Running Hooks'>]' title='Automatic Installation' at 0x4b69e50>': Command '['python', '/etc/ovirt-config-boot.d/snmp_autoinstall.py']' returned non-zero exit status 1 - 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/console.py", line 162, in run_transaction
    e.commit()
  File "/usr/libexec/ovirt-auto-install", line 223, in commit
    hooks.Hooks.post_auto_install()
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/hooks.py", line 47, in post_auto_install
    Hooks.__run(Hooks.legacy_hooks_directory)
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/hooks.py", line 70, in __run
    output = process.check_output(["python", script])
  File "/usr/lib/python2.7/site-packages/ovirt/node/utils/process.py", line 159, in check_output
    stdout = subprocess.check_output(*args, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)


I have to assigned this bug.

Comment 13 cshao 2015-12-16 02:46 UTC
Created attachment 1106268 [details]
/var/log/*.*

Comment 14 cshao 2015-12-16 02:48 UTC
Created attachment 1106269 [details]
snmp-fail1.png

Comment 15 cshao 2015-12-16 02:49 UTC
Created attachment 1106270 [details]
snmp-fail2.png

Comment 16 Ryan Barry 2015-12-17 04:54:05 UTC
Typo that got missed during code review...

Comment 17 Eyal Edri 2015-12-17 13:57:35 UTC
RHEV 3.6.0 BETA2 is out, any open bugs are moved to the BETA3 milestone.

Comment 22 cshao 2016-01-13 09:21:56 UTC
Test version:
rhev-hypervisor7-7.2-20160107.0
ovirt-node-3.6.1-2.0.el7ev.noarch

Test steps:
1. Auto install RHEV-H with below parameters
BOOTIF=eth0 storage_init=/dev/sda adminpw=xxx snmp_password=redhat123 firstboot
2. Reboot host
3. Login RHEV-H and check the status of snmp.
4. Check snmp function with below cmd.
#snmpwalk -v3 -u root -l authPriv -a SHA -A redhat123 -x AES -X redhat123 $ip sysDescr.
5. Set new snmp password on TUI.

Test result:
1. Auto install with "snmp" parameter can enable snmp.
2. In SNMP menu, SNMP show as enable status.
3. After Step4: SNMP function work well.
4. After Step5: set new password - pass

So the bug is fixed, change bug status to VERIFIED.

Comment 24 errata-xmlrpc 2016-03-09 14:37:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html


Note You need to log in before you can comment on or make changes to this bug.