Description of problem: After installation of new RHEL 7.7 host in a DC with Gluster storage domains, it goes non-operational. The vdsm logs are full of; 2020-02-07 16:26:37,577-0500 WARN (jsonrpc/5) [storage.PersistentDict] Could not parse line `^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ In debug mode, it's reading beyond the end of the SD metadata; 2020-02-06 05:35:14,744-0500 DEBUG (jsonrpc/2) [storage.PersistentDict] read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdata21', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=00000001-0001-0001-0001-000000000311', 'REMOTE_PATH=mmr14:/vmdata21', 'ROLE=Master', 'SDUUID=22b2b7a2-65f3-47a9-b6f8-168a50451b2d', 'TYPE=GLUSTERFS', 'VERSION=4', '_SHA_CKSUM=842d15901567b9c08654a838b33bea46d5cce9a6', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\.... 2020-02-06 05:35:14,744-0500 WARN (jsonrpc/2) [storage.PersistentDict] Could not parse line `^@^@^@^@^@^@^@^@^@^@^@.... So, it appears to be reading and parsing the entire 512-byte block. Other hosts in the same DC have 'vdsm-4.30.33.1' and are working fine. I downgraded the system in question to 'vdsm-4.30.33.1' and it still went non-operational. Version-Release number of selected component (if applicable): RHV 4.3.7 RHEL 7.7 host; vdsm-4.30.38-1.el7ev.x86_64 glusterfs-6.0-29.el7rhgs.x86_64 How reproducible: Appears to be reproducible on newly-installed/upgraded RHV hosts with Gluster storage. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Sandro, Can you please have a look at it. It seems like there is some versions problem here.
Regarding the warnings, if we cannot get the gluster fix for bug 1737141, we can backport this change in vdsm to 4.3 to avoid the unwanted warnings. It will be easy to backport since it depends on oop.readFile which does not exist in 4.3. commit 0c459b90007947ed303db660b288565969367b59 Author: Nir Soffer <nsoffer> Date: Fri Aug 2 23:17:28 2019 +0300 fileVolume: Strip padding when reading volume metadata VolumeMetadata.from_lines() does not warn about extra zero padding, but there is no reason to return the padding when reading volume metadata from storage. Replacing oop.directReadLines() with oop.readFile(direct=True) simplify the code, reading volume metadata the same way we do for domain metadata, and allows removing oop.directReadLines(). Change-Id: I6d264e49d53329b2b8eef5a5af4e8fd38d272681 Signed-off-by: Nir Soffer <nsoffer>
(In reply to Nir Soffer from comment #22) > Regarding the warnings, if we cannot get the gluster fix for bug 1737141, > we can backport this change in vdsm to 4.3 to avoid the unwanted warnings. > It will be easy to backport since it depends on oop.readFile which does > not exist in 4.3. > > commit 0c459b90007947ed303db660b288565969367b59 > Author: Nir Soffer <nsoffer> > Date: Fri Aug 2 23:17:28 2019 +0300 > > fileVolume: Strip padding when reading volume metadata > > VolumeMetadata.from_lines() does not warn about extra zero padding, but > there is no reason to return the padding when reading volume metadata > from storage. > > Replacing oop.directReadLines() with oop.readFile(direct=True) simplify > the code, reading volume metadata the same way we do for domain > metadata, and allows removing oop.directReadLines(). > > Change-Id: I6d264e49d53329b2b8eef5a5af4e8fd38d272681 > Signed-off-by: Nir Soffer <nsoffer> Gordon, can you please open a new bug for that?
> > Gordon, can you please open a new bug for that? Marina, I just opened BZ 1801892. Regards, GFW.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0534
(In reply to Eyal Shenitzky from comment #12) > Sandro, > > Can you please have a look at it. > It seems like there is some versions problem here. Looks like this has been already clarified.