Bug 1130304 - initial mount of nfs filesystem by pacemaker fails, succeeds later
Summary: initial mount of nfs filesystem by pacemaker fails, succeeds later
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ga
: Installer
Assignee: Crag Wolfe
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-14 19:44 UTC by Lars Kellogg-Stedman
Modified: 2014-08-21 18:08 UTC (History)
10 users (show)

Fixed In Version: openstack-foreman-installer-2.0.21-1.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, the nfs-utils package was not installed before a Image sevice or Block Storage service NFS export was attempted to be mounted by a HA controller node and so the NFS mount operation failed. This update ensures that the nfs-utils package is installed before any NFS mounts are made. As a result, NFS mount now succeed during the initial puppet run on the HA controller nodes, whether or not the nfs-utils package has already been installed before puppet is executed.
Clone Of:
Environment:
Last Closed: 2014-08-21 18:08:59 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1090 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory 2014-08-22 15:28:08 UTC

Description Lars Kellogg-Stedman 2014-08-14 19:44:43 UTC
My HA controller deployment got stuck at 60% (and I did *not* make that the subject of this ticket!).  It looks like this was because pacemaker showed the fs-varlibglanceimages in a failed state.

If I look in /var/log/messages, I see:

Aug 14 19:12:27 mac525400c1a0af Filesystem(fs-varlibglanceimages)[23867]: ERROR: Couldn't mount filesystem 172.16.0.2:/srv/glance on /var/lib/glance/images
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [ mount: wrong fs type, bad option, bad superblock on 172.16.0.2:/srv/glance, ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [        missing codepage or helper program, or other error ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [        (for several filesystems (e.g. nfs, cifs) you might ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [        need a /sbin/mount.<type> helper program) ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [  ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [        In some cases useful info is found in syslog - try ]
Aug 14 19:12:27 mac525400c1a0af lrmd[13683]: notice: operation_finished: fs-varlibglanceimages_start_0:23867:stderr [        dmesg | tail or so. ]
Aug 14 19:12:27 mac525400c1a0af crmd[13686]: notice: process_lrm_event: LRM operation fs-varlibglanceimages_start_0 (call=64, rc=1, cib-update=83, confirmed=true) unknown error
Aug 14 19:12:27 mac525400c1a0af crmd[13686]: notice: process_lrm_event: mac525400c1a0af.localdomain-fs-varlibglanceimages_start_0:64 [ mount: wrong fs type, bad option, bad superblock on 172.16.0.2:/srv/glance,\n       missing codepage or helper program, or other error\n       (for several filesystems (e.g. nfs, cifs) you might\n       need a /sbin/mount.<type> helper program)\n\n       In some cases useful info is found in syslog - try\n       dmesg | tail or so.\n ]

Looking in /var/log/yum.log, I see:

Aug 14 19:15:27 Installed: 1:nfs-utils-1.3.0-0.el7.x86_64

Given the timestamps, I wonder if there is a race condition here and we need to ensure that the nfs-utils package is installed prior to pacemaker starting to manage resources.

Comment 1 Lars Kellogg-Stedman 2014-08-14 19:48:16 UTC
Clearing the failures with:

    pcs resource cleanup fs-varlibglanceimages

...cleared the failures, and pacemaker was able to start the resource (and mount the filesystem) without error.

Comment 3 Crag Wolfe 2014-08-14 22:31:04 UTC
Patch posted: https://github.com/redhat-openstack/astapor/pull/354

Comment 4 Crag Wolfe 2014-08-14 22:32:09 UTC
BTW, nice debugging Lars.  Thanks.

Comment 5 Jason Guiditta 2014-08-15 20:28:08 UTC
Merged

Comment 8 Leonid Natapov 2014-08-20 12:32:13 UTC
openstack-foreman-installer-2.0.21-1.el6ost

NFS mount succeeds.

Comment 9 errata-xmlrpc 2014-08-21 18:08:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html


Note You need to log in before you can comment on or make changes to this bug.