Bug 1612152
| Summary: | [GSS] Crashes in glusterVdoVolumeList seen in messages file. | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Ben Turner <bturner> | |
| Component: | vdsm | Assignee: | Kaustav Majumder <kmajumde> | |
| Status: | CLOSED ERRATA | QA Contact: | SATHEESARAN <sasundar> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.2.2 | CC: | bturner, danken, dblack, dfediuck, godas, guillaume.pavese, kmajumde, lsurette, mkalinin, rdlugyhe, sabose, sbose, seamurph, sgoodman, srevivo, tnisan, ycui | |
| Target Milestone: | ovirt-4.4.1 | Flags: | lsvaty:
testing_plan_complete-
|
|
| Target Release: | 4.4.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | vdsm-4.40.20 | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, Virtual Data Optimizer (VDO) statistics were not available for VDO volumes with an error, so VDO monitoring from VDSM caused a traceback. This update fixes the issue by correctly handling the different outputs from the VDO statistics tool.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1721097 (view as bug list) | Environment: | ||
| Last Closed: | 2020-08-04 13:26:06 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | Gluster | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1721097 | |||
|
Description
Ben Turner
2018-08-03 15:30:38 UTC
This is not a vdsm crash, but an error querying vdo stats, as the vdo module is not present. The error is safe to be ignored as per Denis. Denis, do you want to comment? Is this on a RHEL 7.5 system? Ben, can you help to answer Sahina's question? Right, it means vdo module is not installed/working correctly. It can be safely ignored. (In reply to Denis Chaplygin from comment #7) > Right, it means vdo module is not installed/working correctly. It can be > safely ignored. So, is there another bug confirming this behavior or that it is fixed in next version or something so that we can close that bug as duplicate of it? Hi Denis,
I see this when vdo is installed and running. We need to handle cases where - VDO statistics: not available
Error in supervdsm.log
MainProcess|jsonrpc/5::DEBUG::2018-09-21 14:31:47,941::supervdsm_server::96::SuperVdsm.ServerCallback::(wrapper) call vdoVolumeList with () {}
MainProcess|jsonrpc/5::DEBUG::2018-09-21 14:31:47,942::commands::65::root::(execCmd) /usr/bin/taskset --cpu-list 0-31 /bin/vdo status (cwd None)
MainProcess|jsonrpc/5::DEBUG::2018-09-21 14:31:48,413::commands::86::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/5::ERROR::2018-09-21 14:31:48,490::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in vdoVolumeList
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper
res = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/gluster/thinstorage.py", line 95, in vdoVolumeList
for mapper, stats in iteritems(data["VDO statistics"]):
File "/usr/lib/python2.7/site-packages/six.py", line 576, in iteritems
return iter(d.iteritems(**kw))
AttributeError: 'str' object has no attribute 'iteritems'
vdo status output:
VDO status:
Date: '2018-09-21 14:36:42+05:30'
Node: rhsdev-grafton4-nic2.lab.eng.blr.redhat.com
Kernel module:
Loaded: true
Name: kvdo
Version information:
kvdo version: 6.1.0.181
Configuration:
File: /etc/vdoconf.yml
Last modified: '2018-07-06 18:27:02'
VDOs:
vdo_sdb:
Acknowledgement threads: 1
Activate: enabled
Bio rotation interval: 64
Bio submission threads: 4
Block map cache size: 128M
Block map period: 16380
Block size: 4096
CPU-work threads: 2
Compression: enabled
Configured write policy: sync
Deduplication: enabled
Device mapper status: 0 2936012800 vdo /dev/sdb albserver online cpu=2,bio=4,ack=1,bioRotationInterval=64
Emulate 512 byte: enabled
Hash zone threads: 1
Index checkpoint frequency: 0
Index memory setting: 0.25
Index parallel factor: 0
Index sparse: disabled
Index status: online
Logical size: 1400G
Logical threads: 1
Physical size: 190240M
Physical threads: 1
Read cache: enabled
Read cache size: 20M
Slab size: 32G
Storage device: /dev/sdb
VDO statistics:
/dev/mapper/vdo_sdb:
1K-blocks: 194805760
1K-blocks available: 158031652
1K-blocks used: 36774108
512 byte emulation: true
KVDO module bios used: 63766
KVDO module bytes used: 586508896
KVDO module peak bio count: 160524
KVDO module peak bytes used: 1143910024
bios acknowledged discard: 0
bios acknowledged flush: 39010980
bios acknowledged fua: 34634281
bios acknowledged partial discard: 0
bios acknowledged partial flush: 0
bios acknowledged partial fua: 0
bios acknowledged partial read: 29881488
bios acknowledged partial write: 915785
bios acknowledged read: 65605641
bios acknowledged write: 106957648
bios in discard: 0
bios in flush: 39010980
bios in fua: 34634281
bios in partial discard: 0
bios in partial flush: 0
bios in partial fua: 0
bios in partial read: 101917
bios in partial write: 915785
bios in progress discard: 0
bios in progress flush: 0
bios in progress fua: 0
bios in progress read: 0
bios in progress write: 0
bios in read: 65605641
bios in write: 106957648
bios journal completed discard: 0
bios journal completed flush: 0
bios journal completed fua: 0
bios journal completed read: 32768
bios journal completed write: 176081539
bios journal discard: 0
bios journal flush: 176081539
bios journal fua: 176081539
bios journal read: 32768
bios journal write: 176081539
bios meta completed discard: 0
bios meta completed flush: 0
bios meta completed fua: 0
bios meta completed read: 38712
bios meta completed write: 200858893
bios meta discard: 0
bios meta flush: 177013669
bios meta fua: 176081541
bios meta read: 38712
bios meta write: 200858893
bios out completed discard: 0
bios out completed flush: 0
bios out completed fua: 0
bios out completed read: 32552330
bios out completed write: 66473542
bios out discard: 0
bios out flush: 0
bios out fua: 0
bios out read: 32552330
bios out write: 66473542
bios page cache completed discard: 0
bios page cache completed flush: 0
bios page cache completed fua: 0
bios page cache completed read: 3334
bios page cache completed write: 28898
bios page cache discard: 0
bios page cache flush: 3
bios page cache fua: 0
bios page cache read: 3334
bios page cache write: 28898
block map cache pressure: 0
block map cache size: 134217728
block map clean pages: 2709
block map dirty pages: 625
block map discard required: 0
block map failed pages: 0
block map failed reads: 0
block map failed writes: 0
block map fetch required: 3334
block map flush count: 23887
block map found in cache: 264958494
block map free pages: 29434
block map incoming pages: 0
block map outgoing pages: 0
block map pages loaded: 3334
block map pages saved: 28895
block map read count: 134317562
block map read outgoing: 44
block map reclaimed: 79
block map wait for page: 787162
block map write count: 131431428
block size: 4096
completed recovery count: 1
compressed blocks written: 23624169
compressed fragments in packer: 31
compressed fragments written: 53649807
current VDO IO requests in progress: 31
current dedupe queries: 0
data blocks used: 2414920
dedupe advice stale: 7455
dedupe advice timeouts: 0
dedupe advice valid: 9756065
entries indexed: 63432491
flush out: 39010980
instance: 1
invalid advice PBN count: 0
journal blocks batching: 0
journal blocks committed: 176081539
journal blocks started: 176081539
journal blocks writing: 0
journal blocks written: 176081539
journal commits requested count: 0
journal disk full count: 0
journal entries batching: 0
journal entries committed: 262861901
journal entries started: 262861901
journal entries writing: 0
journal entries written: 262861901
logical blocks: 367001600
logical blocks used: 6580537
maximum VDO IO requests in progress: 2000
maximum dedupe queries: 1984
no space error count: 0
operating mode: normal
overhead blocks used: 6778607
physical blocks: 48701440
posts found: 9780277
posts not found: 55994283
queries found: 0
queries not found: 0
read cache accesses: 13447159
read cache data hits: 10471636
read cache hits: 10887649
read only error count: 0
read-only recovery count: 0
recovery progress (%): N/A
reference blocks written: 740026
release version: 131337
saving percent: 63
slab count: 5
slab journal blocked count: 0
slab journal blocks written: 192096
slab journal disk full count: 0
slab journal flush count: 43296
slab journal tail busy count: 0
slab summary blocks written: 192096
slabs opened: 1
slabs reopened: 15
updates found: 53197347
updates not found: 19
used percent: 18
version: 26
write amplification ratio: 2.5
write policy: sync
vdo_sdd1:
Acknowledgement threads: 1
Activate: enabled
Bio rotation interval: 64
Bio submission threads: 4
Block map cache size: 128M
Block map period: 16380
Block size: 4096
CPU-work threads: 2
Compression: enabled
Configured write policy: auto
Deduplication: enabled
Device mapper status: not available
Emulate 512 byte: enabled
Hash zone threads: 1
Index checkpoint frequency: 0
Index memory setting: 0.25
Index parallel factor: 0
Index sparse: disabled
Index status: not available
Logical size: 5000G
Logical threads: 1
Physical size: 50G
Physical threads: 1
Read cache: enabled
Read cache size: 20M
Slab size: 32G
Storage device: /dev/sdd1
VDO statistics: not available
This bug has not been marked as blocker for oVirt 4.3.0. Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1. Ovirt-4.3.1 already released, so moving to ovirt-4.3.2 Denis, any update on this patch? PInged reviewers, will try to get it merged asap Any update? Tal, need some help with patch reviews. Can you help move this forward? (In reply to Sahina Bose from comment #17) > Tal, need some help with patch reviews. Can you help move this forward? Discussed offline. Can you provide pm_ack? sync2jira sync2jira Kaustav, can you rebase the patch and own this? Moved to 4.4.1 not being marked as blocker for 4.4.0 and we are preparing to GA. Tested with RHV Manager Version 4.4.1.8-0.7.el8ev and vdsm-4.40.22-1.el8ev.x86_64 Ran the soak test for about 4 weeks, with the setup up and running with VDO volumes, no such errors popped up in the vdsm.log Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:3246 |