Bug 1251151
Summary: | Can not login RHEV-H after automatic installation on physical machine: Authentication token manipulation error | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Ying Cui <ycui> | ||||||||||||
Component: | ovirt-node | Assignee: | Douglas Schilling Landgraf <dougsland> | ||||||||||||
Status: | CLOSED ERRATA | QA Contact: | Ying Cui <ycui> | ||||||||||||
Severity: | urgent | Docs Contact: | |||||||||||||
Priority: | urgent | ||||||||||||||
Version: | 3.5.4 | CC: | cshao, cwu, fdeutsch, gklein, huiwa, leiwang, lsurette, mburman, mgoldboi, rbarry, tlitovsk, ycui, ykaul | ||||||||||||
Target Milestone: | ovirt-3.6.1 | Keywords: | AutomationBlocker, TestBlocker, ZStream | ||||||||||||
Target Release: | 3.6.0 | ||||||||||||||
Hardware: | Unspecified | ||||||||||||||
OS: | Unspecified | ||||||||||||||
Whiteboard: | |||||||||||||||
Fixed In Version: | ovirt-node-3.3.0-0.14.20151013git5f84da0.el7 | Doc Type: | Bug Fix | ||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||
Clone Of: | |||||||||||||||
: | 1269840 (view as bug list) | Environment: | |||||||||||||
Last Closed: | 2016-03-09 14:34:33 UTC | Type: | Bug | ||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||
Documentation: | --- | CRM: | |||||||||||||
Verified Versions: | Category: | --- | |||||||||||||
oVirt Team: | Node | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
Embargoed: | |||||||||||||||
Bug Depends On: | |||||||||||||||
Bug Blocks: | 1199035, 1254510, 1269840 | ||||||||||||||
Attachments: |
|
Description
Ying Cui
2015-08-06 14:54:51 UTC
Created attachment 1060006 [details]
sosreport
Created attachment 1060008 [details]
varlog
I'm not able to reproduce this on a local system (tested CDROM installs on a R300 and a 9020), and I was able to immediately log into QE's system and change the password as normal. Testing USB now, but this looks ok to me so far. I'll keep trying to reproduce. Ying - If you get a chance, can you log in as root and provide the last couple logs from dmesg and the journal when this happens? Also, output from attaching strace to login during a failed attempt may be very long, but potentially helpful. I'm also not able to reproduce this with USB installs Ryan, for dmesg log and secure log, you can check above the attachment 1060008 [details].
<snip>
[ 0.000000] Command line: initrd=initrd0.img root=live:CDLABEL=rhev-hypervisor7-7.1-20150805.0 rootfstype=auto ro rd.live.image rd.live.check crashkernel=256M rd_NO_MULTIPATH rootflags=ro elevator=deadline install quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 storage_init=/dev/sda storage_vol=:::::: adminpw=OKr05SbCu3D3g firstboot BOOTIF=em1 BOOT_IMAGE=vmlinuz0
</snip>
# cat /var/log/secure
Aug 7 03:51:35 dhcp-10-118 usermod[1810]: change user 'admin' password
Aug 7 03:51:35 dhcp-10-118 chage[1816]: changed password expiry for admin
Aug 7 03:51:47 dhcp-10-118 sshd[1935]: Server listening on 0.0.0.0 port 22.
Aug 7 03:51:47 dhcp-10-118 sshd[1935]: Server listening on :: port 22.
Aug 7 03:51:35 dhcp-10-118 usermod[1810]: change user 'admin' password
Aug 7 03:51:35 dhcp-10-118 chage[1816]: changed password expiry for admin
Aug 7 03:51:47 dhcp-10-118 sshd[1935]: Server listening on 0.0.0.0 port 22.
Aug 7 03:51:47 dhcp-10-118 sshd[1935]: Server listening on :: port 22.
Aug 7 03:57:23 localhost sshd[2419]: Server listening on 0.0.0.0 port 22.
Aug 7 03:57:23 localhost sshd[2419]: Server listening on :: port 22.
Aug 7 03:57:59 localhost login: pam_unix(login:account): expired password for user admin (root enforced)
Aug 7 03:58:01 localhost unix_chkpwd[16708]: password check failed for user (admin)
Aug 7 03:58:01 localhost login: pam_unix(login:chauthtok): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=admin
Aug 7 03:58:04 localhost login: Authentication token manipulation error
Aug 7 04:06:07 localhost sshd[2444]: Server listening on 0.0.0.0 port 22.
Aug 7 04:06:07 localhost sshd[2444]: Server listening on :: port 22.
Aug 7 04:06:34 localhost login: pam_unix(login:account): expired password for user admin (root enforced)
Aug 7 04:06:37 localhost unix_chkpwd[16595]: password check failed for user (admin)
Aug 7 04:06:37 localhost login: pam_unix(login:chauthtok): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=admin
Aug 7 04:06:39 localhost login: Authentication token manipulation error
Aug 7 06:14:37 localhost sshd[2446]: Server listening on 0.0.0.0 port 22.
Aug 7 06:14:37 localhost sshd[2446]: Server listening on :: port 22.
Aug 7 06:15:22 localhost login: pam_unix(login:account): expired password for user admin (root enforced)
Aug 7 06:15:26 localhost unix_chkpwd[16605]: password check failed for user (admin)
Aug 7 06:15:26 localhost login: pam_unix(login:chauthtok): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=admin
Aug 7 06:15:27 localhost login: Authentication token manipulation error
# cat /etc/pam.d/password-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass nullok
auth required pam_deny.so
account required pam_unix.so
password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
# cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass nullok
auth required pam_deny.so
account required pam_unix.so
password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
# getent passwd admin
admin:x:1001:1001::/home/admin:/usr/libexec/ovirt-admin-shell
# getent shadow admin
admin:OKr05SbCu3D3g:0:0:99999:7:::
# ls -Z /etc/passwd
-rw-r--r--. root root system_u:object_r:passwd_file_t:s0 /etc/passwd
# ls -Z /etc/shadow
----------. root root system_u:object_r:shadow_t:s0 /etc/shadow
Created attachment 1060252 [details] all_info including fresh logs: journalctl sosreport-localhost-20150807070827.tar.xz varlog.tar.bz2 I lower the severity to medium. Because we can reproduce this issue on _previous_ RHEV-H as well rhev-hypervisor7-7.1-20150603.0.iso and rhevh-6.6-20150512.0.el6ev.iso. And devel side can not reproduce it. Moved this severity to Urgent again, because QE encountered this issue lots of times during automatic installation on physical machine. It is affecting our test now. Can you please check with enforcing=0 during installation? Hi Ying, Is it possible to autoinstall adding rootpw as well in the kernel args params and provide the machine via ssh for us to debug? Thanks! *** Bug 1246833 has been marked as a duplicate of this bug. *** Hi I can see this too with the auto install scripts. Ping me on IRC and I can provide machines with the error . Neither enforcing=0 or rootpw helps Can you attach logs from the failed run? A summary on the observations: ntpd.service is using the PrivateTmp service directive, which creates a slave subtree of / with private /tmp and /var/tmp. Somehow, the exact reason is still unknown, the umounts of the bind mounts in the root ns are not correctly propagated to the ntpd mount ns - thus the bind mounts are kept there. Because the bind mounts still exist in th entpd mount ns, the files can not be removed (because they still serve as targets for the bind mounts in the ntpd mounts ns). A workaround is to stop ntpd or (this is the fix for now) launch ntpd without PrivateTmp. *** Bug 1242366 has been marked as a duplicate of this bug. *** *** Bug 1270203 has been marked as a duplicate of this bug. *** There two new patches 47167 and 47168 are review in Progress, not merged into Fixed In Version: ovirt-node-3.3.0-0.11.20151008git33a0533.el7ev. And tested on build rhev-hypervisor7-7.2-20151009.0(ovirt-node-3.3.0-0.13.20151008git03eefb5.el7ev.noarch) Still can reproduce this error the same as original bug description. So it is still affecting our automated installation testing, about 60+ test cases. After confirmed with Fabian, patches 47167 and 47168 are for 3.5.z. So all patches are already included into rhev-hypervisor7-7.2-20151009.0(ovirt-node-3.3.0-0.13.20151008git03eefb5.el7ev.noarch). Then I have to assign this bug, new fresh log will be provided later. Created attachment 1081907 [details]
varlog_ovirt-node-3.3.0-0.13.20151008git03eefb5
Created attachment 1081908 [details]
sosreport_ovirt-node-3.3.0-0.13.20151008git03eefb5
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0378.html |