Bug 973331 - IPA Server will not reboot after install
IPA Server will not reboot after install
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: freeipa (Show other bugs)
19
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Rob Crittenden
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-11 13:11 EDT by Dean Hunter
Modified: 2013-07-25 20:33 EDT (History)
6 users (show)

See Also:
Fixed In Version: freeipa-3.2.2-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-25 20:33:38 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg.txt (355.11 KB, text/plain)
2013-06-12 10:52 EDT, Dean Hunter
no flags Details

  None (edit)
Description Dean Hunter 2013-06-11 13:11:19 EDT
Description of problem:

I have built a new Fedora 19 Beta VM, applied all available updates, rebooted, and installed freeipa-server.  The scripts all completed without error, but the VM will not reboot after installing freeipa-server.  When I try to connect to the VM console the last line of text on the display says "Starting Network Manager Wait Online".  I am unable to connect with SSH: "ssh: connect to host ipa port 22: Connection refused".

Where do I start?


Version-Release number of selected component (if applicable):

freeipa-server


How reproducible: Consistent
Comment 1 Alexander Bokovoy 2013-06-11 14:57:59 EDT
Hi Dean, please follow http://freedesktop.org/wiki/Software/systemd/Debugging/ to diagnose which service actually blocks.
Comment 2 Dean Hunter 2013-06-11 15:55:51 EDT
Following the instructions I have waited 10 minutes and I can not get a command prompt.  From the Virtual Machine Manager window for the IPA server, if I select View, Text Consoles, Serial 1 then I get a blinking block cursor in the upper left hand corner.  If, instead, I select Send Key, Ctrl+Alt+F2 then I get a blinking underscore cursor. Neither display will prompt nor respond to the Enter key.  The display also will not respond to a Crtl+Alt+Delete.

I am sorry, but I do not know how to edit the kernel command line as the next step requests.
Comment 3 Dean Hunter 2013-06-11 16:35:30 EDT
OK! I think I got to the kernel command line by typing e while the grub2 menu was displayed.  I added "emergency" and selected f10.  It booted to a prompt and I used "journalctl -bx" to review the systemd journal.  I saw no obvious errors aside from 10 AVCs.  From the prompt I was able to reboot successfully.  Now the system will boot, but it will not shutdown!
Comment 4 Dean Hunter 2013-06-11 16:49:05 EDT
Selecting the previous version of the kernel from the GRUB menu allows start up and shut down to behave as expected.

Current version:  3.9.4-301.fc19.x86_64
Previous version: 3.9.2-301.fc19.x86_64
Comment 5 Dean Hunter 2013-06-11 18:08:49 EDT
Using the instructions for debugging systemd problems, I executed this sequence:

  systemctl enable debug-shell.service
  reboot
  ...
  yum install --assumeyes bind bind-dyndb-ldap freeipa-server

  cat >/etc/hosts <<EOD
127.0.0.1        localhost.localdomain  localhost
192.168.1.11     ipa.hunter.org         ipa
EOD

  ipa-server-install \\
    --admin-password adminpassword \\
    --domain hunter.org \\
    --ds-password dspassword \\
    --hostname ipa.hunter.org \\
    --idstart 128000000 \\
    --no-forwarders \\
    --realm HUNTER.ORG \\
    --setup-dns \\
    --ssh-trust-dns \\
    --unattended

  firewall-cmd --permanent --zone public --add-service dns
  firewall-cmd --permanent --zone public --add-service http
  firewall-cmd --permanent --zone public --add-service https
  firewall-cmd --permanent --zone public --add-service kerberos
  firewall-cmd --permanent --zone public --add-service kpasswd
  firewall-cmd --permanent --zone public --add-service ldap
  firewall-cmd --permanent --zone public --add-service ldaps
  firewall-cmd --reload
  firewall-cmd --list-all

  echo adminpassword | kinit admin

  ipa config-mod --defaultshell=/bin/bash 

  ipa host-add host.hunter.org \\
    --force \\
    --ip-address=192.168.1.10

  echo first | ipa user-add dean \\
    --first=Dean \\
    --last=Hunter \\
    --password \\
    --uid=128000001

  ipa automountkey-add default auto.direct \\
    --key /mnt/Shared \\
    --info '-fstype=nfs4,sec=krb5p host.hunter.org:/srv/nfs/Shared'

  ipa automountlocation-add VM

  ipa automountkey-add VM auto.direct \\
    --key /mnt/Shared \\
    --info '-fstype=nfs4,sec=krb5p host.hunter.org:/srv/nfs/Shared'

  ipa automountmap-add-indirect VM auto.home \\
    --mount /home
 
  ipa automountkey-add VM auto.home \\
    --key '*' \\
    --info '-fstype=nfs4,sec=krb5p host.hunter.org:/srv/nfs/home/&'

  kdestroy

  reboot

Using:

  systemctl list-jobs

at the debug shell prompt I can see that the ipa.service stop job is running, never finishes, and 41 other jobs are waiting for it.

[root@ipa /]# ps -ef | grep ipa
root      4953     1  0 16:48 ?        00:00:01 /usr/bin/python /usr/sbin/ipactl stop
root      5089  4953  0 16:48 ?        00:00:00 /bin/systemctl stop ipa-otpd.socket

[root@ipa /]#
Comment 6 Martin Kosek 2013-06-12 06:11:04 EDT
Lukas or Michal, can you please advise how should we continue with debugging? It seems like "/bin/systemctl stop ipa-otpd.socket" is stuck.

This is how the ipa-otpd.socket is configured:

# cat /usr/lib/systemd/system/ipa-otpd.socket
[Unit]
Description=ipa-otpd socket

[Socket]
ListenStream=/var/kerberos/krb5kdc/DEFAULT.socket
ExecStopPre=/usr/bin/unlink /var/kerberos/krb5kdc/DEFAULT.socket
SocketMode=0600
Accept=true

[Install]
WantedBy=krb5kdc.service
Comment 7 Michal Sekletar 2013-06-12 09:56:26 EDT
Well, we will need more information to figure out what is wrong. Please provide some logs, preferably with systemd_log.level=debug. Please use how to article on systemd debugging [1]. It would be great to see the difference between 3.9.4 and 3.9.2 kernels. 


[1] http://freedesktop.org/wiki/Software/systemd/Debugging/
Comment 8 Dean Hunter 2013-06-12 10:51:35 EDT
I hope I did this correctly:

1) Requested a reboot from the graphical console

2) Ctrl+Alt+F9 and killed the "/bin/systemctl stop ipa-otpd.socket" process

3) Used the grub menu to edit the line that starts with:

   linux   /vmlinuz-....

   and appended:

   systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M

4) Waited for the boot to complete and the CPU activity to reach its idle state

5) Requested a reboot from the graphical console

6) Ctrl+Alt+F9, "dmesg >dmesg.txt", and killed the "/bin/systemctl stop ipa-otpd.socket" process

7) Waited for the boot to complete

8) scp'ed /dmesg.txt

I am sorry, but since the rebuild of comment 5 the problem occurs on both the current and previous versions of the kernel.
Comment 9 Dean Hunter 2013-06-12 10:52:09 EDT
Created attachment 760200 [details]
dmesg.txt
Comment 10 Martin Kosek 2013-06-14 03:30:39 EDT
Michal, did the information that Dean provided help?
Comment 11 Michal Sekletar 2013-06-17 08:12:06 EDT
ipa.service should include ordering dependency After= on all of its dependencies. When stopping services, ipactl should call systemctl using --no-block option in order to remedy deadlock during shutdown transaction.
Comment 12 Martin Kosek 2013-06-17 09:28:23 EDT
Thanks for cooperation Michal, I will create an upstream ticket to fix it.
Comment 13 Martin Kosek 2013-06-17 09:29:03 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3729
Comment 14 Dean Hunter 2013-07-14 12:14:14 EDT
To work around the problem, stop IPA before rebooting:

  systemctl stop ipa.service
Comment 15 Martin Kosek 2013-07-15 06:23:21 EDT
Thanks for the workaround.

Note that there is already a patch submitted upstream and ideally we would to have this issue fixed with next bugfixing update for Fedora 19.
Comment 16 Martin Kosek 2013-07-16 07:01:54 EDT
Fixed upstream:

master: fb166e8f5c0a959f0f1dd9ceb5d9cc8ddd3e7ef6
ipa-3-2: f99bbb97e71cec50754c756d06a905cb879fdf4d
Comment 17 Dean Hunter 2013-07-16 10:43:46 EDT
Yay!
Comment 18 Fedora Update System 2013-07-17 12:49:17 EDT
freeipa-3.2.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/freeipa-3.2.2-1.fc19
Comment 19 Fedora Update System 2013-07-18 01:52:12 EDT
Package freeipa-3.2.2-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing freeipa-3.2.2-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-13224/freeipa-3.2.2-1.fc19
then log in and leave karma (feedback).
Comment 20 Dean Hunter 2013-07-18 13:45:41 EDT
Correction verified and karma updated.

However, now none of the NFS services are starting:

[root@ipa19 ~]# systemctl list-units | grep nfs
[root@ipa19 ~]# systemctl start nfs-server.service
[root@ipa19 ~]# systemctl start nfs-secure-server.service
[root@ipa19 ~]# systemctl start nfs-secure.service
[root@ipa19 ~]# systemctl list-units | grep nfs
proc-fs-nfsd.mount          loaded active mounted   RPC Pipe File System
nfs-idmap.service           loaded active running   NFSv4 ID-name mapping daemon
nfs-mountd.service          loaded active running   NFS Mount Daemon
nfs-rquotad.service         loaded active running   NFS Remote Quota Server
nfs-secure-server.service   loaded active running   Secure NFS Server
nfs-secure.service          loaded active running   Secure NFS
nfs-server.service          loaded active exited    NFS Server
[root@ipa19 ~]# 

And there has never been any response to 972363.
Comment 21 Martin Kosek 2013-07-19 03:25:23 EDT
Thanks for verification! I am really not sure why there is no response in Bug 972363 - I tried to bring this Bug to attention via other channels.
Comment 22 Dean Hunter 2013-07-19 10:57:41 EDT
Reading the unit files shows that the NFS server should be enabled via nfs.target rather than via nfs-server.service and nfs-lock.service as I had previously learned.  However, I believe there is still an error with the NFS secure client as described in Bug 972363.

I know NFS is off the subject of this bug report, but I wanted to let y'all know because I only use NFS through IPA auto mount.

Thank you to the team for the fixes.  Now I can begin upgrading my network to Fedora 19.
Comment 23 Fedora Update System 2013-07-25 20:33:38 EDT
freeipa-3.2.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.