Bug 986559 - Upgrade to 3.2.1-1 from 3.1.5-1 fails
Upgrade to 3.2.1-1 from 3.1.5-1 fails
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: freeipa (Show other bugs)
19
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Rob Crittenden
Fedora Extras Quality Assurance
:
Depends On: 987767
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-20 11:46 EDT by Dean Hunter
Modified: 2013-07-26 02:16 EDT (History)
6 users (show)

See Also:
Fixed In Version: freeipa-3.2.2-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-25 20:33:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ipaupgrade.log (23.85 KB, text/x-log)
2013-07-20 11:57 EDT, Dean Hunter
no flags Details
ipaupgrade.log from freeipa 3.2.2 (23.27 KB, text/plain)
2013-07-22 16:18 EDT, Dean Hunter
no flags Details
/var/log/dirsrv/slapd-HUNTER-ORG/errors (22.74 KB, text/plain)
2013-07-25 11:17 EDT, Dean Hunter
no flags Details

  None (edit)
Description Dean Hunter 2013-07-20 11:46:00 EDT
Description of problem:

Upgrading from Fedora 18 to Fedora 19 fails with errors in ipa-upgrade.log


Version-Release number of selected component (if applicable):

freeipa-server-3.1.5-1.fc18.x86_64
freeipa-server-3.2.1-1.fc19.x86_64


How reproducible: Consistent


Steps to Reproduce:

1. yum --assumeyes update
2. yum --assumeyes install fedup
3. fedup --network 19
4. reboot


Actual results:

Errors in ipaupgrade.log and IPA fails to start after the reboot


Expected results:

No errors in upaupgrade.log and IPA starts after the reboot


Additional info:
Comment 1 Dean Hunter 2013-07-20 11:57:56 EDT
Created attachment 776193 [details]
ipaupgrade.log
Comment 2 Martin Kosek 2013-07-22 02:57:42 EDT
This looks relevant:

  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 115, in __upgrade
    ld = ldapupdate.LDAPUpdate(dm_password='', ldapi=True, live_run=self.live_run, plugins=True)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 174, in __init__
    conn.do_external_bind(self.pw_name)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1734, in do_external_bind
    self.conn.sasl_interactive_bind_s, timeout, None, auth_tokens)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1720, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1706, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1109, in wait_for_open_socket
    raise e
error: [Errno 2] No such file or directory

This is most probably a problem in SELinux policy. I investigated this error with Mirek Grepl when developing FreeIPA 3.2.2.

Can you check /var/log/audit/audit.log? There is probably an AVC relevant to DS not being able to create it's socket.

This should be fixed in selinux-policy-3.12.1-65.fc19 + FreeIPA 3.2.2. Please report if that helps you.
Comment 3 Dean Hunter 2013-07-22 14:10:12 EDT
How do I upgrade to selinux-policy-3.12.1-65.fc19 and FreeIPA 3.2.2 when they are still in updates-testing?

I am using:

  fedup --network 19 \
    --instrepo http://host.hunter.org/repos/fedora19/iso

with local repositories:

  local-fedora    http://host.hunter.org/repos/fedora19/fedora
  local-updates   http://host.hunter.org/repos/fedora19/updates
  local-testing   http://host.hunter.org/repos/fedora19/testing

I do NOT want to apply all packages from local-testing.
Comment 4 Dean Hunter 2013-07-22 16:12:57 EDT
I tried moving SELinux 3.12.1-65 and FreeIPA 3.2.2-1 packages from local-testing to local-updates, updating the metadata, and performing the upgrade again.  There are still errors in the ipaupgrade.log.
Comment 5 Dean Hunter 2013-07-22 16:15:38 EDT
These are the installed packages after the upgrade:

[root@host ~]# ssh root@ipa2
Last login: Mon Jul 22 15:04:47 2013

[root@ipa2 ~]# yum list selinux*
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
selinux-policy.noarch                    3.12.1-65.fc19            installed    
selinux-policy-devel.noarch              3.12.1-65.fc19            installed    
selinux-policy-doc.noarch                3.12.1-65.fc19            installed    
selinux-policy-targeted.noarch           3.12.1-65.fc19            installed    
Available Packages
selinux-policy-minimum.noarch            3.12.1-65.fc19            local-updates
selinux-policy-mls.noarch                3.12.1-65.fc19            local-updates

[root@ipa2 ~]# yum list freeipa*
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
freeipa-admintools.x86_64                 3.2.2-1.fc19             installed    
freeipa-client.x86_64                     3.2.2-1.fc19             installed    
freeipa-python.x86_64                     3.2.2-1.fc19             installed    
freeipa-server.x86_64                     3.2.2-1.fc19             installed    
Available Packages
freeipa-server-selinux.x86_64             3.2.1-1.fc19             local-fedora 
freeipa-server-strict.x86_64              3.2.2-1.fc19             local-updates
freeipa-server-trust-ad.x86_64            3.2.2-1.fc19             local-updates

[root@ipa2 ~]#
Comment 6 Dean Hunter 2013-07-22 16:18:12 EDT
Created attachment 777053 [details]
ipaupgrade.log from freeipa 3.2.2
Comment 7 Martin Kosek 2013-07-23 02:41:31 EDT
This means that a DS socket was not created.

Dean, is there any relevant AVC in /var/log/audit/audit.log?
Dean, can you please also get the label for the DS socket? For example with this command:

# ls -laZ /var/run/slapd-*.socket

Mirek (CC-ed), we may need to check this behavior, it seems similar to the behavior you fixed in selinux-policy-3.12.1-65.fc19.
Comment 8 Dean Hunter 2013-07-23 08:17:25 EDT
[root@ipa2 ~]# ausearch --message avc
----
time->Mon Jul 22 09:48:33 2013
type=AVC msg=audit(1374504513.040:17): avc:  denied  { read } for  pid=476 comm="abrtd" name="abrt" dev="dm-1" ino=654174 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:initrc_tmp_t:s0 tclass=dir
----
time->Mon Jul 22 09:48:33 2013
type=AVC msg=audit(1374504513.040:18): avc:  denied  { read } for  pid=476 comm="abrtd" name="abrt" dev="dm-1" ino=654174 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:initrc_tmp_t:s0 tclass=dir

[root@ipa2 ~]# ls -laZ /var/run/slapd-*.socket
srw-rw-rw-. root root system_u:object_r:dirsrv_var_run_t:s0 /var/run/slapd-HUNTER-ORG.socket

[root@ipa2 ~]#
Comment 9 Miroslav Grepl 2013-07-23 10:10:09 EDT
restorecon -R -v /var/tmp/abrt


The labeling is OK.
Comment 10 Dean Hunter 2013-07-23 12:22:14 EDT
[root@ipa2 ~]# restorecon -R -v /var/tmp/abrt
[root@ipa2 ~]# ls -dlZ /var/tmp/abrt
drwxr-xr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 /var/tmp/abrt
[root@ipa2 ~]# ls -alZ /var/tmp/abrt
drwxr-xr-x. abrt abrt system_u:object_r:abrt_var_cache_t:s0 .
drwxrwxrwt. root root system_u:object_r:tmp_t:s0       ..
[root@ipa2 ~]#
Comment 11 Martin Kosek 2013-07-24 02:53:53 EDT
Just to answer your question I overlooked:

(In reply to Dean Hunter from comment #3)
> How do I upgrade to selinux-policy-3.12.1-65.fc19 and FreeIPA 3.2.2 when
> they are still in updates-testing?

# yum update freeipa-server selinux-policy --enablerepo=updates-testing

As for the issue itself, I suspect this may be a crash in the 389 Directory Server we already found ourselves (Bug 987767).

Dean, did abrt catch any coredump we can use to confirm that this is the same issue? Maybe it would catch the coredump if you put SELinux to permissive in case you still hit this issue.
Comment 12 Dean Hunter 2013-07-24 09:54:24 EDT
I am sorry, but I do not understand.  Where is this sequence:

  yum --assumeyes update
  yum --assumeyes install fedup
  fedup --network 19
  reboot

would I insert:

  yum update freeipa-server selinux-policy --enablerepo=updates-testing


- There are no problems detected by the Automatic Bug Reporting Tool.
- The two AVC messages occurred five and one half hours before the Fedora 18 to Fedora 19 upgrade was started.
- I am unable to view any details of bug 987767.  It is as if someone entered the heading but no comments.
Comment 13 Martin Kosek 2013-07-24 10:15:43 EDT
(In reply to Dean Hunter from comment #12)
> I am sorry, but I do not understand.  Where is this sequence:
> 
>   yum --assumeyes update
>   yum --assumeyes install fedup
>   fedup --network 19
>   reboot
> 
> would I insert:
> 
>   yum update freeipa-server selinux-policy --enablerepo=updates-testing

Ah - in the upgrade scenario this won't work. But IIUC, FreeIPA 3.2.2 should be pulled by fedup when it hits stable repo.

> - There are no problems detected by the Automatic Bug Reporting Tool.
> - The two AVC messages occurred five and one half hours before the Fedora 18
> to Fedora 19 upgrade was started.

Ok, I thought this happened during or after the upgrade. Your issues should not  be caused by this issue then.

> - I am unable to view any details of bug 987767.  It is as if someone
> entered the heading but no comments.

This is be cause there are some sensitive data in Bug 987767 description which prevents it from showing to all users. It's clone Bug 987705 has more visible data in.
Comment 14 Martin Kosek 2013-07-24 10:17:18 EDT
Anyway, I am going to append this bug to 3.2.2 release as it should be fixed by it.
Comment 15 Fedora Update System 2013-07-24 10:20:02 EDT
freeipa-3.2.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/FEDORA-2013-13224/freeipa-3.2.2-1.fc19
Comment 16 Dean Hunter 2013-07-24 12:02:45 EDT
To elaborate on comment 4:

  mv /srv/http/repos/fedora19/updates-testing/freeipa* \
     /srv/http/repos/fedora19/updates

  mv /srv/http/repos/fedora19/updates-testing/selinux-policy* \
     /srv/http/repos/fedora19/updates

  createrepo --update /srv/http/repos/fedora19/updates
  createrepo --update /srv/http/repos/fedora19/updates-testing

Then:

  yum --assumeyes update
  yum --assumeyes install fedup

  fedup --network 19 \
    --instrepo http://host.hunter.org/repos/fedora19/iso

  reboot

This sequence upgraded FreeIPA to 3.2.2-1, but there were still errors, see the second attachment.
Comment 17 Martin Kosek 2013-07-25 02:53:44 EDT
Ok, (In reply to Dean Hunter from comment #16)
...
> This sequence upgraded FreeIPA to 3.2.2-1, but there were still errors, see
> the second attachment.

I would need more information to evaluate this. This error means that ipa-upgradeconfig could not find DS socket


After you reboot and see this error message, is IPA properly started?

# ipactl status

Is DS socket in place?

# ls -laZ /var/run/slapd-*.socket

Are there any errors in DS log? (/var/log/dirsrv/slapd-*/errors)

I will assume there was no AVC logged during the upgrade process as you confirmed in Comment 12.

Does the upgrade finish properly if you run it after Fedora 18 -> Fedora 19 upgrade? You can re-run it with these commands:

# /usr/sbin/ipa-ldap-updater --upgrade
# /usr/sbin/ipa-upgradeconfig
Comment 18 Dean Hunter 2013-07-25 11:12:11 EDT
[root@host ~]# ssh root@ipa2
Last login: Thu Jul 25 09:30:36 2013 from host.hunter.org

[root@ipa2 ~]# uname -a
Linux ipa2.hunter.org 3.9.11-200.fc18.x86_64 #1 SMP Mon Jul 22 21:04:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@ipa2 ~]# rpm -q freeipa-server
freeipa-server-3.2.2-1.fc19.x86_64

[root@ipa2 ~]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ls -laZ /var/run/slapd-*.socket
srw-rw-rw-. root root system_u:object_r:dirsrv_var_run_t:s0 /var/run/slapd-HUNTER-ORG.socket

[root@ipa2 ~]# ls -l /var/log/dirsrv/slapd-*/errors
-rw-------. 1 dirsrv dirsrv 23286 Jul 25 09:51 /var/log/dirsrv/slapd-HUNTER-ORG/errors

[root@ipa2 ~]# /usr/sbin/ipa-ldap-updater --upgrade
Upgrading IPA:
  [1/8]: stopping directory server
  [2/8]: saving configuration
  [3/8]: disabling listeners
  [4/8]: starting directory server
  [5/8]: upgrading server
PRE_UPDATE
Parsing update file '/usr/share/ipa/updates/10-60basev2.update'
Parsing update file '/usr/share/ipa/updates/10-60basev3.update'
Parsing update file '/usr/share/ipa/updates/10-70ipaotp.update'
...
Done
Updating existing entry: cn=CAcert,cn=ipa,cn=etc,dc=hunter,dc=org
Done
  [6/8]: stopping directory server
  [7/8]: restoring configuration
  [8/8]: starting directory server
Done.
The ipa-ldap-updater command was successful

[root@ipa2 ~]# /usr/sbin/ipa-upgradeconfig
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
[Removing self-signed CA]
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
[Checking for deprecated KDC configuration files]
[Setting up Firefox extension]
/usr/share/ipa/html/krb.js exists, skipping install of Firefox extension
[Add missing CA DNS records]
[Enabling persistent search in DNS]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Enable certificate renewal]
[Verifying that CA service certificate profile is updated]
[Certificate renewal should stop the CA]
Already configured to stop CA
The ipa-upgradeconfig command was successful

[root@ipa2 ~]# ipactl stop
Stopping Directory Service
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

[root@ipa2 ~]#
Comment 19 Dean Hunter 2013-07-25 11:17:35 EDT
Created attachment 778294 [details]
/var/log/dirsrv/slapd-HUNTER-ORG/errors

The IPA server was rebuilt on Fedora 18 on 24/Jul/2013 about 15:20.

The IPA server was upgraded to Fedora 19 on 25/Jul/2013 about 9:30.
Comment 20 Dean Hunter 2013-07-25 11:27:12 EDT
I am concerned that the kernel does not appear to have been updated even though freeipa was updated:

[root@ipa2 ~]# uname -a
Linux ipa2.hunter.org 3.9.11-200.fc18.x86_64 #1 SMP Mon Jul 22 21:04:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@ipa2 ~]# rpm -q freeipa-server
freeipa-server-3.2.2-1.fc19.x86_64
Comment 21 Fedora Update System 2013-07-25 20:33:45 EDT
freeipa-3.2.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 22 Martin Kosek 2013-07-26 02:16:47 EDT
Judging on Comment 18, it seems to me that the upgrade in the new environment worked fine. Not sure about your Comment 20 though, after fedup process, VM/machine should boot in F19 kernel - worked for me when I was doing F18->F19 update lately.

Note You need to log in before you can comment on or make changes to this bug.