Bug 1676591

Summary: Unable to start gluster-block-target.service after upgrading and rebooting to glusterfs-server-3.12.2-40
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Christian Koep <ckoep>
Component: gluster-blockAssignee: Prasanna Kumar Kalever <prasanna.kalever>
Status: CLOSED ERRATA QA Contact: Nitin Goyal <nigoyal>
Severity: high Docs Contact:
Priority: unspecified    
Version: ocs-3.11CC: abhishku, ckoep, hchiramm, kramdoss, nigoyal, pkarampu, pprakash, prasanna.kalever, puebele, rcyriac, rgeorge, rhs-bugs, sankarshan, vbellur, xiubli
Target Milestone: ---Keywords: Regression, ZStream
Target Release: OCS 3.11.z Batch Update 2   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: gluster-block-0.2.1-31.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-27 04:55:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christian Koep 2019-02-12 15:59:41 UTC
Description of problem:

After upgrading Red Hat Gluster Storage to the latest version (via yum update), the `gluster-blockd.service` fails to start with the following error(s).

# systemctl status gluster-blockd
...
iglusterfs-p03.infra.vs.ch systemd[1]: Dependency failed for Gluster block storage utility.
iglusterfs-p03.infra.vs.ch systemd[1]: Job gluster-blockd.service/start failed with result 'dependency'.

# systemctl status gluster-block-target.service
● gluster-block-target.service - Restore LIO kernel target configuration
   Loaded: loaded (/usr/lib/systemd/system/gluster-block-target.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-02-12 11:56:30 CET; 3h 0min ago
  Process: 58618 ExecStop=/usr/bin/ps cax | /usr/bin/grep -wq [t]cmu-runner && /usr/bin/targetctl clear (code=exited, status=1/FAILURE)
  Process: 58943 ExecStart=/usr/bin/targetctl restore (code=exited, status=1/FAILURE)
 Main PID: 58943 (code=exited, status=1/FAILURE)

Feb 12 11:56:30 <snip> target[58943]: return _default_decoder.decode(s)
Feb 12 11:56:30 <snip> target[58943]: File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
Feb 12 11:56:30 <snip> target[58943]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Feb 12 11:56:30 <snip> target[58943]: File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
Feb 12 11:56:30 <snip> target[58943]: raise ValueError("No JSON object could be decoded")
Feb 12 11:56:30 <snip> target[58943]: ValueError: No JSON object could be decoded
Feb 12 11:56:30 <snip> systemd[1]: gluster-block-target.service: main process exited, code=exited, status=1/FAILURE
Feb 12 11:56:30 <snip> systemd[1]: Failed to start Restore LIO kernel target configuration.
Feb 12 11:56:30 <snip> systemd[1]: Unit gluster-block-target.service entered failed state.
Feb 12 11:56:30 <snip> systemd[1]: gluster-block-target.service failed.

Version-Release number of selected component (if applicable):

$ grep gluster installed-rpms 
glusterfs-3.12.2-40.el7rhgs.x86_64                          Tue Feb 12 08:51:12 2019
glusterfs-api-3.12.2-40.el7rhgs.x86_64                      Tue Feb 12 08:51:12 2019
glusterfs-client-xlators-3.12.2-40.el7rhgs.x86_64           Tue Feb 12 08:51:12 2019
glusterfs-cli-3.12.2-40.el7rhgs.x86_64                      Tue Feb 12 08:51:12 2019
glusterfs-fuse-3.12.2-40.el7rhgs.x86_64                     Tue Feb 12 08:51:12 2019
glusterfs-geo-replication-3.12.2-40.el7rhgs.x86_64          Tue Feb 12 08:51:14 2019
glusterfs-libs-3.12.2-40.el7rhgs.x86_64                     Tue Feb 12 08:51:12 2019
glusterfs-rdma-3.12.2-40.el7rhgs.x86_64                     Tue Feb 12 08:51:15 2019
glusterfs-server-3.12.2-40.el7rhgs.x86_64                   Tue Feb 12 08:51:12 2019
gluster-block-0.2.1-30.el7rhgs.x86_64                       Tue Feb 12 08:48:01 2019
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64               Mon May 28 14:22:18 2018
gluster-nagios-common-0.2.4-1.el7rhgs.noarch                Thu May 24 11:49:14 2018

How reproducible:
Unclear at this point.

Steps to Reproduce:
Unclear at this point.

Actual results:
Pods are unable to mount gluster block volumes / gluster file volumes are unaffected.

Expected results:
Pods are able to mount gluster block volumes.

Additional info: 
Posting additional info into the private comment section.

Comment 28 errata-xmlrpc 2019-03-27 04:55:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0665