Bug 1263648 - [7.2_3.5][node]"Error replacing '/etc/passwd': Device or resource busy" occurs when upgrade from 7.1 to RHEV-H 7.2
[7.2_3.5][node]"Error replacing '/etc/passwd': Device or resource busy" occur...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node (Show other bugs)
3.5.6
Unspecified Unspecified
high Severity high
: ovirt-3.6.2
: 3.6.0
Assigned To: Ryan Barry
Huijuan Zhao
: Reopened, ZStream
Depends On: 1286242 1269840
Blocks: 1283643 1288535
  Show dependency treegraph
 
Reported: 2015-09-16 06:56 EDT by Huijuan Zhao
Modified: 2016-03-09 09:38 EST (History)
14 users (show)

See Also:
Fixed In Version: ovirt-node-3.6.1-1.0.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1288535 (view as bug list)
Environment:
Last Closed: 2016-03-09 09:38:14 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
upgrade error screenshot (31.63 KB, image/png)
2015-09-16 06:56 EDT, Huijuan Zhao
no flags Details
upgrade error log (7.80 MB, application/x-gzip)
2015-09-16 06:59 EDT, Huijuan Zhao
no flags Details
upgrade failed log (5.84 MB, application/x-gzip)
2015-11-02 06:59 EST, Huijuan Zhao
no flags Details
upgrade failed screenshot (590.19 KB, image/png)
2015-11-02 07:00 EST, Huijuan Zhao
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48216 master ABANDONED password: Replace libuser and update the persist/unpersist calls Never
oVirt gerrit 49325 master MERGED recipe: Let udevd use a recursive slave mnt namespace Never
oVirt gerrit 49526 ovirt-3.6 MERGED recipe: Let udevd use a recursive slave mnt namespace Never
oVirt gerrit 50162 ovirt-3.5 MERGED recipe: Let udevd use a recursive slave mnt namespace Never
oVirt gerrit 50481 master MERGED installer: Use shared subtree for initramfs regeneration 2015-12-22 07:02 EST
oVirt gerrit 50508 master MERGED init: Ensure that /etc will be a rshared subtree 2015-12-22 07:02 EST
oVirt gerrit 50915 ovirt-3.6 MERGED init: Ensure that /etc will be a rshared subtree 2015-12-22 10:00 EST
oVirt gerrit 50916 ovirt-3.6 MERGED installer: Use shared subtree for initramfs regeneration 2015-12-22 10:00 EST
oVirt gerrit 51292 ovirt-3.5 MERGED init: Ensure that /etc will be a rshared subtree 2016-01-06 11:57 EST

  None (edit)
Description Huijuan Zhao 2015-09-16 06:56:52 EDT
Created attachment 1073968 [details]
upgrade error screenshot

Description of problem:
Exception occurs when upgrade from 7.1 GA to 7.2

Version-Release number of selected component (if applicable):
rhev-hypervisor-7-7.2-20150913.0
ovirt-node-3.2.3-20.el7.noarch

How reproducible:
100%
QA Whiteboard: node
Regression bug

Steps to Reproduce:
1. Clean install rhev-hypervisor-7-7.1-20150911.0
2. TUI Upgrade to rhev-hypervisor-7-7.2-20150913.0 step by step

Actual results:
after step2, there is error report:
------------------
Exception: System error("Error replacing '/etc/passwd': Device or resource busy",)
------------------

Expected results:
after step2, upgrade should successful

Additional info:
Comment 1 Huijuan Zhao 2015-09-16 06:59:07 EDT
Created attachment 1073970 [details]
upgrade error log
Comment 2 cshao 2015-09-17 05:31:20 EDT
After setting new password during upgrade, reboot rhevh, the password did not change to new one.
Comment 3 Fabian Deutsch 2015-09-17 09:14:59 EDT

*** This bug has been marked as a duplicate of bug 1246117 ***
Comment 4 cshao 2015-09-22 03:05:26 EDT
Update test steps for more clearly.

Test steps:
1. Clean install rhev-hypervisor-7-7.1-20150911.0
2. TUI Upgrade to rhev-hypervisor-7-7.2-20150913.0
3. Setting new password during upgrade, reboot.
4. Login RHEV-H with new password.

Actual results:
1. after step2, there is error report:
Exception: System error("Error replacing '/etc/passwd': Device or resource busy",)

2. After step 4, failed to login with new password. But old password still can work fine.
Comment 6 Fabian Deutsch 2015-10-19 15:52:47 EDT
Likely a dupe of bug 1269840
Comment 7 Huijuan Zhao 2015-11-02 06:58:01 EST
Still encounter this issue on rhev-hypervisor-7-7.2-20151029.0.

Test steps:
1. Clean install rhev-hypervisor-7-7.1-20151015.0
2. Deploy HE on rhevh
3. TUI Upgrade to rhev-hypervisor-7-7.2-20151029.0
4. Setting new password during upgrade.

Actual results:
1. after step4, there is error report:
Exception: System error("Error replacing '/etc/passwd': Device or resource busy",)

Please refer to attachment: "upgrade failed log", "upgrade failed screenshot".

So I will change the status to assigned.
Comment 8 Huijuan Zhao 2015-11-02 06:59 EST
Created attachment 1088542 [details]
upgrade failed log
Comment 9 Huijuan Zhao 2015-11-02 07:00 EST
Created attachment 1088543 [details]
upgrade failed screenshot
Comment 11 Doron Fediuck 2015-11-03 04:49:52 EST
Taken from the ovirt-node.log file:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 126, in __run
  File "/usr/lib/python2.7/site-packages/ovirt/node/plugins.py", line 188, in dry_or
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 120, in do_commit
  File "/usr/lib/python2.7/site-packages/ovirt/node/installer/core/progress_page.py", line 266, in commit
  File "/usr/lib/python2.7/site-packages/ovirtnode/password.py", line 45, in set_password
SystemError: Error replacing `/etc/passwd': Device or resource busy
Comment 12 Fabian Deutsch 2015-11-03 05:59:08 EST
Okay, I see that ovirt-node-3.2.3-20.el7.noarch is used in the tested build. This build is known to be broken wrt to persistence.

Please try a 3.5 build with at least ovirt-node-3.2.3-24 (we need the "ntp" patch a380c6be248c88e68c8b278bcbe278c5bceb7abc), the latest 3.5.6 build should have that ovirt-node build.
Comment 13 Huijuan Zhao 2015-11-03 06:08:24 EST
Tested build rhev-hypervisor-7-7.2-20151029.0, have the ovirt-node-3.2.3-26.el7.noarch
Comment 14 Douglas Schilling Landgraf 2015-11-03 09:20:32 EST
(In reply to Huijuan Zhao from comment #13)
> Tested build rhev-hypervisor-7-7.2-20151029.0, have the
> ovirt-node-3.2.3-26.el7.noarch

Hi Huijuan,

Could you please clarify the last comment? Still happening to you?
Comment 15 Fabian Deutsch 2015-11-03 09:57:25 EST
Yes, it was still happening with the new build as well, see comment 7.
Comment 16 Douglas Schilling Landgraf 2015-11-03 12:24:49 EST
I have reproduced the report:

Installed rhev-hypervisor-7-7.1-20151015.0 and upgraded to rhev-hypervisor-7-7.2-20151029.0 via Text User Interface.

However, I would like to share the workaround for now, upgrade via RHEV-M or ovirt-node-upgrade tool. 

Upgraded from rhev-hypervisor7-7.1-20150911.0.iso to
rhev-hypervisor-7-7.2-20150913.0.iso worked without any issue.

or

Upgraded from rhev-hypervisor-7-7.1-20151015.0.iso to
rhev-hypervisor-7-7.2-20151029.0.iso worked without any issue.

Huijuan, Could you please confirm the workaround also worked for you?

Thanks!
Comment 17 Huijuan Zhao 2015-11-04 00:15:43 EST
Hi, Douglas

The workaround you shared: upgrade via RHEV-M or ovirt-node-upgrade tool also worked without any issue in my test.

Just upgrade via TUI failed with error report:

Upgraded from rhev-hypervisor7-7.1-20150911.0.iso to
rhev-hypervisor-7-7.2-20150913.0.iso  via TUI failed.

or

Upgraded from rhev-hypervisor-7-7.1-20151015.0.iso to
rhev-hypervisor-7-7.2-20151029.0.iso via TUI failed.

The error report "System error("Error replacing '/etc/passwd': Device or resource busy",)" likely caused by setup adminpw during TUI upgrade.

Upgrade via RHEV-M or ovirt-node-upgrade tool will skip setup adminpw, so that worked well.
Comment 18 Douglas Schilling Landgraf 2015-11-04 16:19:01 EST
Reducing the priority for now as we have workaround. The only affects cases where users want to upgrade *and* also change the admin password.
Comment 21 Huijuan Zhao 2015-11-06 02:44:15 EST
Ying,
  I tested the following cases each for 5 times, detailed info please refer to following:

1. TUI Upgrade AND change password during upgrade
Failed with error report.
How reproducible: 100%

2. Automatic Upgrade AND change password during upgrade
Without any issue, upgrade successful.

3. TUI Upgrade AND _AFTERWARDS_ change password in TUI
Failed with error report during TUI upgrade.
After TUI upgrade, reboot, change password in TUI successful without any issue.

4. Automatic Upgrade AND _AFTERWARDS_ change password in TUI
Without any issue, upgrade successful.
Change password in TUI successful.
Comment 25 Fabian Deutsch 2015-11-27 09:53:13 EST
The problem seems to be that udevd on RHEL 7.2 is using a slave subtree. That,s fine.
But all mounts within that subtree should become slaves - this is especially important for our config bind-mounts in /etc.

This patch should solve it:
https://github.com/systemd/systemd/pull/2050
Comment 29 Huijuan Zhao 2015-12-11 02:11:59 EST
Still encounter this issue in rhev-hypervisor-7-7.2-20151210.1

Test version:
rhev-hypervisor-7-7.2-20151210.1.e17ev
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

Test steps:
1. TUI install rhev-hypervisor-7-7.1-20151015.0
2. TUI Upgrade to rhev-hypervisor-7-7.2-20151210.1
3. Setting new password during upgrade, reboot.
4. Login RHEV-H with new password.

Actual results:
1. after step2, there is error report:
Exception: System error("Error replacing '/etc/passwd': Device or resource busy",)

2. After step 4, failed to login with new password. But old password still can work fine.

So this bug is not fix in rhev-hypervisor-7-7.2-20151210.1.e17ev, change the status to assigned.
Comment 30 Ying Cui 2015-12-14 07:14:41 EST
(In reply to Huijuan Zhao from comment #29)
> 
> 2. After step 4, failed to login with new password. But old password still
> can work fine.

This reminds me of one Bug 1251867 - Failed to reset the initial password after RHEV-H automatic installation.

The bug 1251867 was closed dup of bug 1246117, but the bug 1246117 was verified.

Huijuan, please check the bug 1251867, if the bug is you mentioned in comment 29, we have to reopen the bug 1251867, too.
Comment 31 Huijuan Zhao 2015-12-14 21:24:34 EST
Ying, I checked the bug 1251867, no issue in rhev-hypervisor-7-7.2-20151210.1.e17ev.

Test Steps:
1. PXE install rhev-hypervisor-7-7.2-20151210.1.e17ev with arguments:
management_server=10.66.108.1 BOOTIF=em1 adminpw=OKr05SbCu3D3g storage_init=/dev/sda reinstall ssh_pwauth=1
2. After the system install successful, approve the system on RHEV-M and the system status is up.
3. Input the initial UNIX password and set the new password successful, and login the TUI successful, reboot the system.
4. In the login session, input the new password, successful to login the TUI.

Test results:
In step4, can login with the new password

So bug 1251867 is fixed in rhev-hypervisor-7-7.2-20151210.1.e17ev.
Comment 32 Fabian Deutsch 2015-12-15 05:40:16 EST
The remaining problem seemed to be that the mount propagation of /etc got broken.

Huijuan, can you please try if the following steps are fixing the issues for you:

1. Boot ISO
2. Do the first install
3. Reboot
4. Boot from ISO again
5. In installer, hit F2, dro pto shell
6. Run: mount --make-rshared /etc
7. Run: exit - to get back into installer
8. Run update/installation/reinstallation and set a new password

After step 8: No error is shown
Comment 33 Huijuan Zhao 2015-12-16 04:48:28 EST
Fabian, I tried the steps as Comment 32, after step 8: No error is shown.

Test version:
rhev-hypervisor-7-7.2-20151210.1.e17ev
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

Test steps:
1. Boot ISO (rhev-hypervisor-7-7.2-20151129.1)
2. Do the first install
3. Reboot
4. Boot from ISO again (rhev-hypervisor-7-7.2-20151210.1)
5. In installer, hit F2, drop to shell
6. Run: mount --make-rshared /etc
7. Run: exit - to get back into installer
8. Run update and set a new password

Test results:
After step 8: No error is shown
Comment 34 Fabian Deutsch 2015-12-17 02:16:02 EST
Great, thanks for testing, we'll merge the patch shortly.
Comment 35 Huijuan Zhao 2015-12-31 04:11:41 EST
Test Version:
rhev-hypervisor-7-7.2-20151229.0
ovirt-node-3.6.1-1.0.el7.noarch

Steps to Reproduce:
1. Clean install rhev-hypervisor-7-7.2-20151129.1
2. TUI Upgrade to rhev-hypervisor-7-7.2-20151229.0
3. Setting new password during upgrade, reboot.
4. Login RHEV-H with new password.

Test results:
Upgrade successful, login RHEV-H with new password successful.

So this bug is fixed on rhev-hypervisor-7-7.2-20151229.0, change the status to VERIFIED.
Comment 37 errata-xmlrpc 2016-03-09 09:38:14 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html

Note You need to log in before you can comment on or make changes to this bug.