Bug 1248371

Summary: An exception SystemError Error replacing `/etc/shadow' when setting password in RHEV-H
Product: Red Hat Enterprise Virtualization Manager Reporter: Ying Cui <ycui>
Component: ovirt-nodeAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED ERRATA QA Contact: Ying Cui <ycui>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.5.4CC: agkesos, bmcclain, cshao, cwu, gklein, huiwa, lsurette, ycui, ykaul
Target Milestone: ovirt-3.6.0-rcKeywords: Regression, ZStream
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-node-3.3.0-0.4.20150906git14a6024.el7ev Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1250399 (view as bug list) Environment:
Last Closed: 2016-03-09 14:33:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1250399    
Attachments:
Description Flags
6.7-20150828 failed installation none

Description Ying Cui 2015-07-30 07:54:16 UTC 
split new bug from https://bugzilla.redhat.com/show_bug.cgi?id=1247576#c4

SystemError error when setting password in RHEV-H, and password can not set successful.
SystemError: Error replacing `/etc/shadow': Device or resource busy


Test version:
rhev-hypervisor-6-6.7-20150717.0
ovirt-node-3.2.3-14.el6.noarch
rhev-hypervisor-7-7.1-20150717.0
ovirt-node-3.2.3-11.el7.noarch


Test Steps:
1. TUI installation RHEV-H 6.7 successful.
2. Login RHEV-H
3. Set password in Security TUI, the first time set password, there is no error display, the password can be updated and set successful.
4. Reboot RHEV-H
5. Set password in Security TUI, the second time to set password here, then you may 50% can reproduce this SystemError.
6. if don't encounter this error, then Reboot RHEV-H
7. Set password in Security TUI, the third time to set password here, then you probably 90% can reproduce this SystemError.
8. if don't encounter this error, then Reboot RHEV-H
8. Set password in Security TUI, the fourth time to set password, then 99.99% can reproduce this SystemError.


2015-07-29 03:50:58,414       INFO An exception during the transaction: Error replacing `/etc/shadow': Device or resource busy
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt/node/ui/__init__.py", line 949, in __run_transaction
  File "/usr/lib/python2.6/site-packages/ovirt/node/plugins.py", line 188, in dry_or
  File "/usr/lib/python2.6/site-packages/ovirt/node/ui/__init__.py", line 949, in <lambda>
  File "/usr/lib/python2.6/site-packages/ovirt/node/setup/core/security_page.py", line 124, in commit
  File "/usr/lib/python2.6/site-packages/ovirt/node/utils/security.py", line 99, in set_password
  File "/usr/lib/python2.6/site-packages/ovirtnode/password.py", line 44, in set_password
SystemError: Error replacing `/etc/shadow': Device or resource busy
2015-07-29 03:50:58,425       INFO Changes were merged successfully


#./ovirt-config-password
...
Changing password for user 'admin'.
New password: 
Retype new Password: 
[WARNING] You have provided a weak password! Strong passwords contain a mix of uppercase, lowercase, numeric and punctuation characters. They are six or more characters long and do not contain dictionary words. Reason: it is based on a dictionary word
Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
  File "/usr/lib/python2.6/site-packages/ovirt/node/tools/password.py", line 179, in <module>
  File "/usr/lib64/python2.6/cmd.py", line 142, in cmdloop
  File "/usr/lib64/python2.6/cmd.py", line 219, in onecmd
  File "/usr/lib/python2.6/site-packages/ovirt/node/tools/password.py", line 92, in do_admin
  File "/usr/lib/python2.6/site-packages/ovirt/node/tools/password.py", line 106, in do_set_admin_password
  File "/usr/lib/python2.6/site-packages/ovirt/node/tools/password.py", line 157, in __ask_and_set_user_pasword
  File "/usr/lib/python2.6/site-packages/ovirt/node/utils/security.py", line 99, in set_password
  File "/usr/lib/python2.6/site-packages/ovirtnode/password.py", line 44, in set_password
SystemError: Error replacing `/etc/shadow': Device or resource busy


Additional info
1. In RHEVM TUI to set password, we also can reproduce this issue.
2. In CIM TUI to set password, we also can reproduce this issue.
3. And when upgrading via TUI, then set new password when TUI upgrade, then still encounter this error and password will not set new after upgrade.
Comment 1 Ying Cui 2015-07-30 08:04:14 UTC 
correct myself here for bug description version info:
rhev-hypervisor7-7.1-20150728.1 
ovirt-node-3.2.3-14.el7
rhev-hypervisor-6-6.7-20150728.0
ovirt-node-3.2.3-14.el6
Comment 2 Ying Cui 2015-07-30 08:16:53 UTC 
Workaround:
 1. If you encounter the error
 2. Drop to shell
 3. unmount /etc/shadow
 4. cp /config/etc/shadow /etc/shadow
 5. /usr/libexec/ovirt-config-password
 6. Set the password 

step 6 will set new password successful.
Comment 5 Fabian Deutsch 2015-07-31 13:13:37 UTC 
Patch has been tested (on el6 and el7) as follows:

1. Clean install
2. Login to TUI
3. Set a new password on the security page
4. Reboot
5. Login to TUI
6. set password again
7. Reboot
8. Login to TUI

Results:
After 3: Setting password worked
After 5: Login works
After 6: Setting password worked
After 8: Login works
Comment 8 Alexandros Gkesos 2015-09-23 09:51:58 UTC 
Created attachment 1076139 [details]
6.7-20150828 failed installation

Customer has the same issue during installation of RHEV-H 6.7-20150828 (attached photo)
Comment 9 Ying Cui 2015-11-12 06:58:57 UTC 
(In reply to Alexandros Gkesos from comment #8)
> Created attachment 1076139 [details]
> 6.7-20150828 failed installation
> 
> Customer has the same issue during installation of RHEV-H 6.7-20150828
> (attached photo)

Alexandros, the customer ticket 01507493 is closed yet, does customer still reproduce this issue on last rhevh 6.7 build: rhev-hypervisor6-6.7-20151015.1.el6ev?

I can not reproduce this during RHEV-H 6.7-20150828 installation, but for password issue, we have one bug on upgrade flow: Bug 1263648 - [7.2_3.5][node]"Error replacing '/etc/passwd': Device or resource busy" occurs when upgrade from 7.1 to RHEV-H 7.2.
Comment 10 Ying Cui 2015-11-12 07:15:08 UTC 
Test Steps:
1. TUI installation RHEV-H 7.2 for 3.6.0 successful.
2. Login RHEV-H
3. Set admin password in Security TUI, successful.
4. Set SNMP password in TUI, PASS.
5. Set CIM password in TUI, PASS.
6. Set root password in TUI,PASS.
7. Reboot RHEV-H
8. repeat step 2 - 7 four times.

All password set successful. no "An exception SystemError Error replacing `/etc/shadow'" error anymore.

# rpm -qa ovirt-node 
ovirt-node-3.3.0-0.18.20151022git82dc52c.el7ev.noarch

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20151025.0.el7ev)

For upgrade flow, we trace the issue by Bug 1263648 - [7.2_3.5][node]"Error replacing '/etc/passwd': Device or resource busy" occurs when upgrade from 7.1 to RHEV-H 7.2.
Comment 11 Ying Cui 2015-11-12 07:16:34 UTC 
To verify this bug, I am pending on comment 9 needinfo feedback.
Comment 12 Alexandros Gkesos 2015-11-12 07:25:03 UTC 
Hello Ying,

Customer hadn't responded for one month and the ticket was auto-archived.
I don't have any further data on that.
Comment 13 Ying Cui 2015-11-12 07:31:01 UTC 
(In reply to Alexandros Gkesos from comment #12)
> Hello Ying,
> 
> Customer hadn't responded for one month and the ticket was auto-archived.
> I don't have any further data on that.

Thanks Alexandros.
I will verify this bug according to comment 10, and rhevh 3.5.z stream bug #1250399 also verified 3 months ago, If customer still encounter the issue, free free to reopen the bug or new one, and all fresh logs as a must to provide. Thanks.
Comment 15 errata-xmlrpc 2016-03-09 14:33:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html