Description of problem: Upgrading from Fedora 18 to Fedora 19 fails with errors in ipa-upgrade.log Version-Release number of selected component (if applicable): freeipa-server-3.1.5-1.fc18.x86_64 freeipa-server-3.2.1-1.fc19.x86_64 How reproducible: Consistent Steps to Reproduce: 1. yum --assumeyes update 2. yum --assumeyes install fedup 3. fedup --network 19 4. reboot Actual results: Errors in ipaupgrade.log and IPA fails to start after the reboot Expected results: No errors in upaupgrade.log and IPA starts after the reboot Additional info:
Created attachment 776193 [details] ipaupgrade.log
This looks relevant: File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 115, in __upgrade ld = ldapupdate.LDAPUpdate(dm_password='', ldapi=True, live_run=self.live_run, plugins=True) File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 174, in __init__ conn.do_external_bind(self.pw_name) File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1734, in do_external_bind self.conn.sasl_interactive_bind_s, timeout, None, auth_tokens) File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1720, in __bind_with_wait self.__wait_for_connection(timeout) File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1706, in __wait_for_connection wait_for_open_socket(lurl.hostport, timeout) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1109, in wait_for_open_socket raise e error: [Errno 2] No such file or directory This is most probably a problem in SELinux policy. I investigated this error with Mirek Grepl when developing FreeIPA 3.2.2. Can you check /var/log/audit/audit.log? There is probably an AVC relevant to DS not being able to create it's socket. This should be fixed in selinux-policy-3.12.1-65.fc19 + FreeIPA 3.2.2. Please report if that helps you.
How do I upgrade to selinux-policy-3.12.1-65.fc19 and FreeIPA 3.2.2 when they are still in updates-testing? I am using: fedup --network 19 \ --instrepo http://host.hunter.org/repos/fedora19/iso with local repositories: local-fedora http://host.hunter.org/repos/fedora19/fedora local-updates http://host.hunter.org/repos/fedora19/updates local-testing http://host.hunter.org/repos/fedora19/testing I do NOT want to apply all packages from local-testing.
I tried moving SELinux 3.12.1-65 and FreeIPA 3.2.2-1 packages from local-testing to local-updates, updating the metadata, and performing the upgrade again. There are still errors in the ipaupgrade.log.
These are the installed packages after the upgrade: [root@host ~]# ssh root@ipa2 Last login: Mon Jul 22 15:04:47 2013 [root@ipa2 ~]# yum list selinux* Loaded plugins: langpacks, refresh-packagekit Installed Packages selinux-policy.noarch 3.12.1-65.fc19 installed selinux-policy-devel.noarch 3.12.1-65.fc19 installed selinux-policy-doc.noarch 3.12.1-65.fc19 installed selinux-policy-targeted.noarch 3.12.1-65.fc19 installed Available Packages selinux-policy-minimum.noarch 3.12.1-65.fc19 local-updates selinux-policy-mls.noarch 3.12.1-65.fc19 local-updates [root@ipa2 ~]# yum list freeipa* Loaded plugins: langpacks, refresh-packagekit Installed Packages freeipa-admintools.x86_64 3.2.2-1.fc19 installed freeipa-client.x86_64 3.2.2-1.fc19 installed freeipa-python.x86_64 3.2.2-1.fc19 installed freeipa-server.x86_64 3.2.2-1.fc19 installed Available Packages freeipa-server-selinux.x86_64 3.2.1-1.fc19 local-fedora freeipa-server-strict.x86_64 3.2.2-1.fc19 local-updates freeipa-server-trust-ad.x86_64 3.2.2-1.fc19 local-updates [root@ipa2 ~]#
Created attachment 777053 [details] ipaupgrade.log from freeipa 3.2.2
This means that a DS socket was not created. Dean, is there any relevant AVC in /var/log/audit/audit.log? Dean, can you please also get the label for the DS socket? For example with this command: # ls -laZ /var/run/slapd-*.socket Mirek (CC-ed), we may need to check this behavior, it seems similar to the behavior you fixed in selinux-policy-3.12.1-65.fc19.
[root@ipa2 ~]# ausearch --message avc ---- time->Mon Jul 22 09:48:33 2013 type=AVC msg=audit(1374504513.040:17): avc: denied { read } for pid=476 comm="abrtd" name="abrt" dev="dm-1" ino=654174 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:initrc_tmp_t:s0 tclass=dir ---- time->Mon Jul 22 09:48:33 2013 type=AVC msg=audit(1374504513.040:18): avc: denied { read } for pid=476 comm="abrtd" name="abrt" dev="dm-1" ino=654174 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:initrc_tmp_t:s0 tclass=dir [root@ipa2 ~]# ls -laZ /var/run/slapd-*.socket srw-rw-rw-. root root system_u:object_r:dirsrv_var_run_t:s0 /var/run/slapd-HUNTER-ORG.socket [root@ipa2 ~]#
restorecon -R -v /var/tmp/abrt The labeling is OK.
[root@ipa2 ~]# restorecon -R -v /var/tmp/abrt [root@ipa2 ~]# ls -dlZ /var/tmp/abrt drwxr-xr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 /var/tmp/abrt [root@ipa2 ~]# ls -alZ /var/tmp/abrt drwxr-xr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 . drwxrwxrwt. root root system_u:object_r:tmp_t:s0 .. [root@ipa2 ~]#
Just to answer your question I overlooked: (In reply to Dean Hunter from comment #3) > How do I upgrade to selinux-policy-3.12.1-65.fc19 and FreeIPA 3.2.2 when > they are still in updates-testing? # yum update freeipa-server selinux-policy --enablerepo=updates-testing As for the issue itself, I suspect this may be a crash in the 389 Directory Server we already found ourselves (Bug 987767). Dean, did abrt catch any coredump we can use to confirm that this is the same issue? Maybe it would catch the coredump if you put SELinux to permissive in case you still hit this issue.
I am sorry, but I do not understand. Where is this sequence: yum --assumeyes update yum --assumeyes install fedup fedup --network 19 reboot would I insert: yum update freeipa-server selinux-policy --enablerepo=updates-testing - There are no problems detected by the Automatic Bug Reporting Tool. - The two AVC messages occurred five and one half hours before the Fedora 18 to Fedora 19 upgrade was started. - I am unable to view any details of bug 987767. It is as if someone entered the heading but no comments.
(In reply to Dean Hunter from comment #12) > I am sorry, but I do not understand. Where is this sequence: > > yum --assumeyes update > yum --assumeyes install fedup > fedup --network 19 > reboot > > would I insert: > > yum update freeipa-server selinux-policy --enablerepo=updates-testing Ah - in the upgrade scenario this won't work. But IIUC, FreeIPA 3.2.2 should be pulled by fedup when it hits stable repo. > - There are no problems detected by the Automatic Bug Reporting Tool. > - The two AVC messages occurred five and one half hours before the Fedora 18 > to Fedora 19 upgrade was started. Ok, I thought this happened during or after the upgrade. Your issues should not be caused by this issue then. > - I am unable to view any details of bug 987767. It is as if someone > entered the heading but no comments. This is be cause there are some sensitive data in Bug 987767 description which prevents it from showing to all users. It's clone Bug 987705 has more visible data in.
Anyway, I am going to append this bug to 3.2.2 release as it should be fixed by it.
freeipa-3.2.2-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/FEDORA-2013-13224/freeipa-3.2.2-1.fc19
To elaborate on comment 4: mv /srv/http/repos/fedora19/updates-testing/freeipa* \ /srv/http/repos/fedora19/updates mv /srv/http/repos/fedora19/updates-testing/selinux-policy* \ /srv/http/repos/fedora19/updates createrepo --update /srv/http/repos/fedora19/updates createrepo --update /srv/http/repos/fedora19/updates-testing Then: yum --assumeyes update yum --assumeyes install fedup fedup --network 19 \ --instrepo http://host.hunter.org/repos/fedora19/iso reboot This sequence upgraded FreeIPA to 3.2.2-1, but there were still errors, see the second attachment.
Ok, (In reply to Dean Hunter from comment #16) ... > This sequence upgraded FreeIPA to 3.2.2-1, but there were still errors, see > the second attachment. I would need more information to evaluate this. This error means that ipa-upgradeconfig could not find DS socket After you reboot and see this error message, is IPA properly started? # ipactl status Is DS socket in place? # ls -laZ /var/run/slapd-*.socket Are there any errors in DS log? (/var/log/dirsrv/slapd-*/errors) I will assume there was no AVC logged during the upgrade process as you confirmed in Comment 12. Does the upgrade finish properly if you run it after Fedora 18 -> Fedora 19 upgrade? You can re-run it with these commands: # /usr/sbin/ipa-ldap-updater --upgrade # /usr/sbin/ipa-upgradeconfig
[root@host ~]# ssh root@ipa2 Last login: Thu Jul 25 09:30:36 2013 from host.hunter.org [root@ipa2 ~]# uname -a Linux ipa2.hunter.org 3.9.11-200.fc18.x86_64 #1 SMP Mon Jul 22 21:04:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@ipa2 ~]# rpm -q freeipa-server freeipa-server-3.2.2-1.fc19.x86_64 [root@ipa2 ~]# ipactl status Directory Service: STOPPED Directory Service must be running in order to obtain status of other services ipa: INFO: The ipactl command was successful [root@ipa2 ~]# ls -laZ /var/run/slapd-*.socket srw-rw-rw-. root root system_u:object_r:dirsrv_var_run_t:s0 /var/run/slapd-HUNTER-ORG.socket [root@ipa2 ~]# ls -l /var/log/dirsrv/slapd-*/errors -rw-------. 1 dirsrv dirsrv 23286 Jul 25 09:51 /var/log/dirsrv/slapd-HUNTER-ORG/errors [root@ipa2 ~]# /usr/sbin/ipa-ldap-updater --upgrade Upgrading IPA: [1/8]: stopping directory server [2/8]: saving configuration [3/8]: disabling listeners [4/8]: starting directory server [5/8]: upgrading server PRE_UPDATE Parsing update file '/usr/share/ipa/updates/10-60basev2.update' Parsing update file '/usr/share/ipa/updates/10-60basev3.update' Parsing update file '/usr/share/ipa/updates/10-70ipaotp.update' ... Done Updating existing entry: cn=CAcert,cn=ipa,cn=etc,dc=hunter,dc=org Done [6/8]: stopping directory server [7/8]: restoring configuration [8/8]: starting directory server Done. The ipa-ldap-updater command was successful [root@ipa2 ~]# /usr/sbin/ipa-upgradeconfig [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] [Verifying that KDC configuration is using ipa-kdb backend] [Fix DS schema file syntax] [Removing self-signed CA] Configuring ipa-otpd [1/2]: starting ipa-otpd [2/2]: configuring ipa-otpd to start on boot Done configuring ipa-otpd. [Checking for deprecated KDC configuration files] [Setting up Firefox extension] /usr/share/ipa/html/krb.js exists, skipping install of Firefox extension [Add missing CA DNS records] [Enabling persistent search in DNS] [Enabling serial autoincrement in DNS] [Updating GSSAPI configuration in DNS] [Updating pid-file configuration in DNS] Changes to named.conf have been made, restart named [Enable certificate renewal] [Verifying that CA service certificate profile is updated] [Certificate renewal should stop the CA] Already configured to stop CA The ipa-upgradeconfig command was successful [root@ipa2 ~]# ipactl stop Stopping Directory Service ipa: INFO: The ipactl command was successful [root@ipa2 ~]# ipactl start Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Starting ipa_memcached Service Starting httpd Service Starting pki-tomcatd Service Failed to start pki-tomcatd Service Shutting down Aborting ipactl [root@ipa2 ~]#
Created attachment 778294 [details] /var/log/dirsrv/slapd-HUNTER-ORG/errors The IPA server was rebuilt on Fedora 18 on 24/Jul/2013 about 15:20. The IPA server was upgraded to Fedora 19 on 25/Jul/2013 about 9:30.
I am concerned that the kernel does not appear to have been updated even though freeipa was updated: [root@ipa2 ~]# uname -a Linux ipa2.hunter.org 3.9.11-200.fc18.x86_64 #1 SMP Mon Jul 22 21:04:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@ipa2 ~]# rpm -q freeipa-server freeipa-server-3.2.2-1.fc19.x86_64
freeipa-3.2.2-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
Judging on Comment 18, it seems to me that the upgrade in the new environment worked fine. Not sure about your Comment 20 though, after fedup process, VM/machine should boot in F19 kernel - worked for me when I was doing F18->F19 update lately.