Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1811398

Summary: Newly-installed RHV hosts with Gluster storage domains go non-operational [RHV clone - 4.3.8]
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: vdsmAssignee: Gobinda Das <godas>
Status: CLOSED ERRATA QA Contact: SATHEESARAN <sasundar>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.3.7CC: aoconnor, eshenitz, gveitmic, jortialc, lsurette, mavital, mkalinin, nsoffer, pelauter, sabose, sbonazzo, srevivo, tnisan, ycui
Target Milestone: ovirt-4.3.8Keywords: ZStream
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1800803 Environment:
Last Closed: 2020-03-08 11:07:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1800803    
Bug Blocks:    

Description RHV bug bot 2020-03-08 11:05:12 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1800803 +++
======================================================================

Description of problem:

After installation of new RHEL 7.7 host in a DC with Gluster storage domains, it goes non-operational. 

The vdsm logs are full of;

2020-02-07 16:26:37,577-0500 WARN  (jsonrpc/5) [storage.PersistentDict] Could not parse line `^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@


In debug mode, it's reading beyond the end of the SD metadata;

2020-02-06 05:35:14,744-0500 DEBUG (jsonrpc/2) [storage.PersistentDict] read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdata21', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=00000001-0001-0001-0001-000000000311', 'REMOTE_PATH=mmr14:/vmdata21', 'ROLE=Master', 'SDUUID=22b2b7a2-65f3-47a9-b6f8-168a50451b2d', 'TYPE=GLUSTERFS', 'VERSION=4', '_SHA_CKSUM=842d15901567b9c08654a838b33bea46d5cce9a6', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\....
2020-02-06 05:35:14,744-0500 WARN  (jsonrpc/2) [storage.PersistentDict] Could not parse line `^@^@^@^@^@^@^@^@^@^@^@....


So, it appears to be reading and parsing the entire 512-byte block.

Other hosts in the same DC have 'vdsm-4.30.33.1' and are working fine. I downgraded the system in question to 'vdsm-4.30.33.1' and it still went non-operational.


Version-Release number of selected component (if applicable):

RHV 4.3.7
RHEL 7.7 host;
    vdsm-4.30.38-1.el7ev.x86_64
    glusterfs-6.0-29.el7rhgs.x86_64


How reproducible:

Appears to be reproducible on newly-installed/upgraded RHV hosts with Gluster storage.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

(Originally by Gordon Watson)

Comment 12 RHV bug bot 2020-03-08 11:05:45 UTC
Sandro,

Can you please have a look at it.
It seems like there is some versions problem here.

(Originally by Eyal Shenitzky)

Comment 22 RHV bug bot 2020-03-08 11:06:11 UTC
Regarding the warnings, if we cannot get the gluster fix for bug 1737141,
we can backport this change in vdsm to 4.3 to avoid the unwanted warnings.
It will be easy to backport since it depends on oop.readFile which does
not exist in 4.3.

commit 0c459b90007947ed303db660b288565969367b59
Author: Nir Soffer <nsoffer>
Date:   Fri Aug 2 23:17:28 2019 +0300

    fileVolume: Strip padding when reading volume metadata
    
    VolumeMetadata.from_lines() does not warn about extra zero padding, but
    there is no reason to return the padding when reading volume metadata
    from storage.
    
    Replacing oop.directReadLines() with oop.readFile(direct=True) simplify
    the code, reading volume metadata the same way we do for domain
    metadata, and allows removing oop.directReadLines().
    
    Change-Id: I6d264e49d53329b2b8eef5a5af4e8fd38d272681
    Signed-off-by: Nir Soffer <nsoffer>

(Originally by Nir Soffer)

Comment 25 RHV bug bot 2020-03-08 11:06:18 UTC
(In reply to Nir Soffer from comment #22)
> Regarding the warnings, if we cannot get the gluster fix for bug 1737141,
> we can backport this change in vdsm to 4.3 to avoid the unwanted warnings.
> It will be easy to backport since it depends on oop.readFile which does
> not exist in 4.3.
> 
> commit 0c459b90007947ed303db660b288565969367b59
> Author: Nir Soffer <nsoffer>
> Date:   Fri Aug 2 23:17:28 2019 +0300
> 
>     fileVolume: Strip padding when reading volume metadata
>     
>     VolumeMetadata.from_lines() does not warn about extra zero padding, but
>     there is no reason to return the padding when reading volume metadata
>     from storage.
>     
>     Replacing oop.directReadLines() with oop.readFile(direct=True) simplify
>     the code, reading volume metadata the same way we do for domain
>     metadata, and allows removing oop.directReadLines().
>     
>     Change-Id: I6d264e49d53329b2b8eef5a5af4e8fd38d272681
>     Signed-off-by: Nir Soffer <nsoffer>

Gordon, can you please open a new bug for that?

(Originally by Marina Kalinin)

Comment 26 RHV bug bot 2020-03-08 11:06:21 UTC
> 
> Gordon, can you please open a new bug for that?

Marina, 

I just opened BZ 1801892.

Regards, GFW.

(Originally by Gordon Watson)

Comment 29 RHV bug bot 2020-03-08 11:06:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0534

(Originally by errata-xmlrpc)

Comment 31 RHV bug bot 2020-03-08 11:06:33 UTC
(In reply to Eyal Shenitzky from comment #12)
> Sandro,
> 
> Can you please have a look at it.
> It seems like there is some versions problem here.

Looks like this has been already clarified.

(Originally by Sandro Bonazzola)