Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1623433 - Brick fails to come online after shutting down and restarting a node [NEEDINFO]
Brick fails to come online after shutting down and restarting a node
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhgs-server-container (Show other bugs)
cns-3.10
Unspecified Unspecified
unspecified Severity high
: ---
: OCS 3.11
Assigned To: Saravanakumar
Rachael
https://github.com/gluster/gluster-co...
:
Depends On:
Blocks: 1610903 1568868 1589277 1629575
  Show dependency treegraph
 
Reported: 2018-08-29 06:55 EDT by Rachael
Modified: 2018-10-24 01:59 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, bricks were not properly mounted in the gluster pod and lvmetad processes (one in the container, one on the host) were competing for resources. Hence, this prevented certain logical volumes from being detected and/or available which resulted in failed mounted bricks. Now with this fix, do not rely on the lvmetad service, and ensure it is not started in the container. Hence, resulting in a single metadata daemon process running and managing the devices and logical volumes.
Story Points: ---
Clone Of:
: 1623465 (view as bug list)
Environment:
Last Closed: 2018-10-24 01:57:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
madam: needinfo? (nberry)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github gluster/gluster-containers/pull/104 None None None 2018-10-02 05:42 EDT
Red Hat Product Errata RHBA-2018:2990 None None None 2018-10-24 01:59 EDT

  None (edit)
Comment 6 Raghavendra Talur 2018-08-29 09:41:02 EDT
I think I found one problem with the mount script. Please look at the output of 

sh-4.2# cat /var/log/glusterfs/container/mountfstab 
mount: special device /dev/mapper/vg_8377c3ad7380bc5110664247eff77dc6-brick_2de9b07bd586eeae37a378737b137c96 does not exist
mount command exited with code 32


Now, I am able to mount the same device using the command:

mount -a --fstab /var/lib/heketi/fstab


We need to debug this further. At the minimum we need to ensure that the pod does not start at all if /etc/systemd/system/gluster-setup.service fails.
Comment 14 Humble Chirammal 2018-08-30 10:26:20 EDT
The upstream fix PR # https://github.com/gluster/gluster-containers/pull/103
Comment 16 Humble Chirammal 2018-08-31 07:51:15 EDT
This is interesting that the issue is always seen for bricks which are part of BHV.
Comment 31 Humble Chirammal 2018-09-05 04:15:22 EDT
saravana, can you please confirm or update the workaround mentioned in c#29 ?
Comment 36 Humble Chirammal 2018-09-07 04:18:34 EDT
Workaround: ( Thanks to Atin++ and Saravana++)


Mount  the brick(s) :
# mount -a --fstab /var/lib/heketi/fstab

Start the corresponding volume
# gluster volume start <volume name> force
Comment 43 Michael Adam 2018-09-20 16:48:29 EDT
The explanation is very plausible, and let's treat the image of
https://bugzilla.redhat.com/show_bug.cgi?id=1536511#c21
as a proposed patch for this problem.
Comment 44 Michael Adam 2018-09-20 16:54:01 EDT
proposing for 3.11.0
Comment 49 Anjana 2018-10-12 05:46:54 EDT
Updated doc text kindly review for technical accuracy.
Comment 50 Niels de Vos 2018-10-12 07:05:09 EDT
Looks good to me, thanks! I just changed the last words to "logical volumes", as there are multiple of them, not only one.
Comment 51 Anjana 2018-10-12 09:13:09 EDT
Neils, thank you for the update.
Comment 53 errata-xmlrpc 2018-10-24 01:57:39 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2990

Note You need to log in before you can comment on or make changes to this bug.