Bug 1739106

Summary:	[RFE] Compare storage with database for discrepancies
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Marina Kalinin <mkalinin>
Component:	rhv-log-collector-analyzer	Assignee:	Benny Zlotnik <bzlotnik>
Status:	CLOSED ERRATA	QA Contact:	Petr Kubica <pkubica>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.3.5	CC:	aoconnor, bzlotnik, dougsland, gscott, gveitmic, lleistne, michal.skrivanek, pelauter, pkubica, rdlugyhe, sbonazzo, tcarlin
Target Milestone:	ovirt-4.3.8	Keywords:	FieldEngineering, FutureFeature, ZStream
Target Release:	---	Flags:	lsvaty: testing_plan_complete-
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rhv-log-collector-analyzer-0.2.15-0.el7ev	Doc Type:	Enhancement
Doc Text:	This release adds a new feature, rhv-image-discrepancies, which reports discrepancies between images in the engine database and storage. The report lists images that are present in one location but missing from the other. Also, for images that are present in both locations, the report lists discrepancies in the values of attributes such as status, parent_id, and type.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-13 15:24:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	902971, 1489145, 1547336

Description Marina Kalinin 2019-08-08 13:44:34 UTC

We need a tool to detect discrepancy between the database and the actual storage. A mismatch between the two can happen due to various reasons, and even if a bug that caused it is fixed, the discrepancy is still there and we need a way to find those discrepancies to avoid future problems, like broken snapshots (most common scenario, but not only).

How this can be implemented:
- Today, vdsm-tool already provides storage contents from SPM in an sqlite format[1]. 
- We have the database on the other hand available. 
- Now we just need to compare the two databases.

Outcome: 
In log collector analyzer we will have this output and next step would be to have this also be an insight rule.

Existing RFEs:
bz#1580130 - [RFE] Need an FSCK-like tool to find and clean up inconsistencies between the database and storage domains
This RFE here comes with a more narrow specific request to implement the missing step.


[1] vdsm-tool dump-volume-chains <sd_uuid> -o sqlite -f out.db
https://gerrit.ovirt.org/#/c/89150/

Comment 5 Michal Skrivanek 2019-11-19 14:59:50 UTC

rhv-log-collector-analyzer-0.2.14-0.el7ev

Comment 6 Petr Kubica 2019-11-28 12:50:46 UTC

Hi,
I would like to ask how to use this new feature?

I used "rhv-log-collector-analyzer --live" which generates a html report but there was nothing about comparing storage with database for discrepancies.

It seems like the patch is not included in build (according to comment #5)
tested with rhv-log-collector-analyzer-0.2.14-0.el7ev.noarch

# rpm -ql rhv-log-collector-analyzer | grep rhv-image-discrepancies.py
or
# find / -name "rhv-image-discrepancies.py"
didn't find anything

Comment 7 Benny Zlotnik 2019-11-28 13:29:09 UTC

If I am not mistaken, it should be in under /bin so ./rhv-image-discrepancies should work

Comment 8 Petr Kubica 2019-11-28 13:41:15 UTC

Ah, yes, you are right.

And I tried to run the utility, got:
# rhv-image-discrepancies 
Traceback (most recent call last):
  File "/usr/bin/rhv-image-discrepancies", line 22, in <module>
    import engine_db_query as db
ModuleNotFoundError: No module named 'engine_db_query'

Comment 9 Benny Zlotnik 2019-12-01 10:14:02 UTC

Douglas, shouldn't engine_db_query be available? It's a dependency of rhv-log-collector-analyzer

Comment 10 Sandro Bonazzola 2019-12-02 08:29:49 UTC

Not blocking 4.3.7 on this, re-targeting to 4.3.8.

Comment 11 Douglas Schilling Landgraf 2019-12-03 14:39:49 UTC

(In reply to Petr Kubica from comment #8)
> Ah, yes, you are right.
> 
> And I tried to run the utility, got:
> # rhv-image-discrepancies 
> Traceback (most recent call last):
>   File "/usr/bin/rhv-image-discrepancies", line 22, in <module>
>     import engine_db_query as db
> ModuleNotFoundError: No module named 'engine_db_query'

Hi Petr, do you have the machine available so I can take a look?

Thanks!

Comment 12 Douglas Schilling Landgraf 2019-12-05 18:36:01 UTC

(In reply to Douglas Schilling Landgraf from comment #11)
> (In reply to Petr Kubica from comment #8)
> > Ah, yes, you are right.
> > 
> > And I tried to run the utility, got:
> > # rhv-image-discrepancies 
> > Traceback (most recent call last):
> >   File "/usr/bin/rhv-image-discrepancies", line 22, in <module>
> >     import engine_db_query as db
> > ModuleNotFoundError: No module named 'engine_db_query'
> 
> Hi Petr, do you have the machine available so I can take a look?
> 
> Thanks!

Just reproduced the report. The rhv-image-discrepancies script is python3 hard-coded.

# cat /usr/bin/rhv-image-discrepancies | grep python3
#!/usr/bin/python3

However, engine-db-query is python2 only at this moment.

Solution:

1. For now, I would replace in rhv-image-discrepancies #!/usr/bin/python3 to #!/usr/bin/env python
2. Open a bug to rhv-log-collector-analyzer and engine-db-query to port missing bits to python3 and build a new rpm.

Just a note, the default python in RHEL 7.7 is still python2. I would target such builds in python3 to RHEL8 based distro.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

# ls -la /usr/bin/python
lrwxrwxrwx. 1 root root 7 Dec  5 10:56 /usr/bin/python -> python2

Comment 20 Petr Kubica 2020-01-23 15:40:31 UTC

Verified in rhv-log-collector-analyzer-0.2.15-0.el7ev.noarch

Comment 24 errata-xmlrpc 2020-02-13 15:24:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:0498

Comment 28 Marina Kalinin 2020-02-20 20:23:04 UTC

*** Bug 1489145 has been marked as a duplicate of this bug. ***