Bug 890983
| Summary: | vdsm: dumpStorageTable.py exits on KeyError for buildVolumesChain | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Dafna Ron <dron> | ||||
| Component: | vdsm | Assignee: | Mark Huth <mhuth> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Elad <ebenahar> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.1.0 | CC: | abaron, amureini, bazulay, cpelland, hateya, iheim, lpeer, lyarwood, mhuth, mkalinin, scohen, sgrinber, ykaul, zdover | ||||
| Target Milestone: | --- | Keywords: | Patch, Regression | ||||
| Target Release: | 3.2.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | storage | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
does not require docs text
requires_doc_text- flag set
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2013-06-10 20:38:51 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
To elaborate on the 2nd expected results: Since we assume this will be executed when storage issues are suspected, whatever it finds it should print right away, not go over all the storage domain / pool / data / whatever, so we can get at least a partial view of the storage. This issue can be reproduced by running dumpStorageTable when previewing a snapshot. A particular parent volume in an image will have 2 child volumes now and dumpStorageTable doesn't account for that. I remember seeing this problem with dumpStorageTable a long while ago and being the bad support person I am - didn't report it or fix it :( Here volumes e41db184-6809-4589-80ee-bc0a06c1e291 and 02e80561-3420-4711-b72d-d633c1f59e66 have a single parent volume PU_93721462-a4fe-4b09-b73d-16d1e04cebb9 [root@rhevh02 vdsm]# lvs -o +tags | grep IU_400ddc 02e80561-3420-4711-b72d-d633c1f59e66 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi----- 1.00g PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_12 8f23a06e-a69f-4cd7-a50c-8b86edff9b20 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a--- 1.00g IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,PU_e41db184-6809-4589-80ee-bc0a06c1e291,MD_11 93721462-a4fe-4b09-b73d-16d1e04cebb9 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi----- 3.88g PU_00000000-0000-0000-0000-000000000000,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_5 e41db184-6809-4589-80ee-bc0a06c1e291 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a--- 1.00g PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,MD_10,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21 And when that is the case, dumpStorageTable barfs: [root@rhevh02 vdsm]# python dumpStorageTable.pyc Traceback (most recent call last): File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module> File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__ File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain KeyError: '00000000-0000-0000-0000-000000000000' Also, this problem (well, at least when using snapshot previews) isn't a regression as I remember seeing this problem over 12 months ago with RHEV2.2. Dafna, are you able to confirm if you are using snapshot previews when you generated the error in comment #0? If not, then I stand corrected. this was on our production environment and there are a lot of users there so I have no way of knowing of we had a vm with a preview.
however, I ran DumpStorageTable with and without a previewed snapshot on my own setup I did get the below when I had a previewed snapshot:
[root@gold-vdsd vdsm]# python dumpStorageTable.py
Traceback (most recent call last):
File "dumpStorageTable.py", line 299, in <module>
rc, msg = StorageTable(vdscli.connect()).show()
File "dumpStorageTable.py", line 34, in __init__
self._buildStorageTable()
File "dumpStorageTable.py", line 174, in _buildStorageTable
self._getSds(self.pool)
File "dumpStorageTable.py", line 85, in _getSds
self._getImages(disks)
File "dumpStorageTable.py", line 98, in _getImages
self._getVols(self.pool, sd, image)
File "dumpStorageTable.py", line 122, in _getVols
self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
File "dumpStorageTable.py", line 132, in _buildVolumesChain
next = volParents[prev]
KeyError: '00000000-0000-0000-0000-000000000000'
Created attachment 693064 [details] Allow dumpStorageTable.py to work with volumes with multiple leaf volumes, eg snapshot previews Step 1: Before patch: [root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py Traceback (most recent call last): File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module> rc, msg = StorageTable(vdscli.connect()).show() File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__ self._buildStorageTable() File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable self._getVms(self.pool, files) File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms self._getImages(disks) File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages self._getVols(self.pool, sd, image) File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image) File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain next = volParents[prev] KeyError: '1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8' Step 2: Applying patch: [root@rhevh-8 ~]# patch -b -i 0001-BZ890983-allow-dumpStorageTable-to-handle-images-wit.patch /usr/share/vdsm/dumpStorageTable.py patching file /usr/share/vdsm/dumpStorageTable.py Step 3: After patch: [root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | Vms | Domains | Images | Volumes | Template | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | | pbandark_test_pool-1 | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | | pbandark_test_pool-2 | Master | | 35cb0bf3-6eac-49f1-8203-937b7c75e030 | | | | Data200G | | | | | | | | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | | | | | c72d4cc0-e4de-4c00-b167-f74b0c2072ef | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template | | test | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent | | akovari_00753631_soa | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ Image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 was in a snapshot preview and thus had 2 volume chains and leaf volumes. The patch now handles that. Step 4: After commiting (or undoing) the snapshot preview for image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301: [root@rhevh-8 vdsm]# python /usr/share/vdsm/dumpStorageTable.py +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | Vms | Domains | Images | Volumes | Template | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | | pbandark_test_pool-1 | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | | pbandark_test_pool-2 | Master | | 35cb0bf3-6eac-49f1-8203-937b7c75e030 | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template | | test | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent | | akovari_00753631_soa | Master | | | | | | Data200G | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ Mark, did you submit the patch upstream to gerrit? (gerrit.ovirt.org) Hi Ayal, I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/ Never submitted to gerrit before and I hope I did it correctly - I followed the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit I'm getting emails that the build failed though: <email> ... Patch Set 1: Fails Build Failed http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1443/ : SUCCESS ... </email> I have to admit, the patch attached to this BZ (https://bugzilla.redhat.com/attachment.cgi?id=693064) didn't apply cleanly to the upstream code and I had to apply it manually, but I have tested it again and the re-patched dumpStorageTable does work: [root@rhevh02 ~]# python ./dumpStorageTable.py ... +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 2239c875-03cc-4b0d-9756-6ea73ea0a0e5 | 7a818b38-07f6-49a5-a8e3-cdb2153295e8 | 400ddc46-9ec6-4786-b43b-80c4bf379c21 | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent | | monitoredGuest | Master | | e41db184-6809-4589-80ee-bc0a06c1e291 | | | | data1 | | 80182192-1fc7-4094-b066-777b03d690d0 | | | | | | | | | | | | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent | | | | | e41db184-6809-4589-80ee-bc0a06c1e291 | | | | | | 8f23a06e-a69f-4cd7-a50c-8b86edff9b20 | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ (In reply to comment #9) > Hi Ayal, > > I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/ Thanks! > > Never submitted to gerrit before and I hope I did it correctly - I followed > the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit > > I'm getting emails that the build failed though: > > <email> > ... > Patch Set 1: Fails > > Build Failed > > http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE You can go to: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/console and see the errors (e.g. vdsm/dumpStorageTable.py:112:80: E501 line too long (96 characters) Additional comments on the patch though as there is no reason to inflate the bugzilla with the review process. Done... oVirt Jenkins CI Server 8:11 AM Patch Set 2: Verified Build Successful http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1498/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1461/ : SUCCESS This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag. Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information: * Cause: What actions or circumstances cause this bug to present. * Consequence: What happens when the bug presents. * Fix: What was done to fix the bug. * Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug. For further details on the Cause, Consequence, Fix, Result format please refer to: https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes Thanks in advance. Setting requires_doc_text to minus because I don't think this BZ fix warrants a release note. Checked on SF12 vdsm-4.10.2-13.0.el6ev.x86_64 [root@nott-vds1 ~]# python /usr/share/vdsm/dumpStorageTable.py +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | Vms | Domains | Images | Volumes | Template | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | ec738a7f-5672-418e-b1a7-021ba33153b3 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | eaceceee-8083-42b6-a43f-c461015522f7 | 4b816021-ce7a-4e5e-86ce-69109936e1fa | Template independent | | vm-test2 | Regular | | | | | | SD-iSCSI2 | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | ac10e4c1-ab7f-43ed-a563-3f6fb723f909 | 9feea3f7-b571-4e10-a5e3-dd38504f6f9e | | | | | vm-test4 | Master | | | | | | SD-iSCSI | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | 52192ea4-851b-43f1-8e75-6925b70c78c9 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | c95e0c92-1cd7-41c4-b6fd-e00b8047259e | 193315ef-8458-4c1d-af9f-1b996118c268 | Template independent | | vm-iscsi-4-stateless | Regular | | cabc8d83-83fb-4dee-bc46-0e3cfc7e2125 | | | | SD-iSCSI2 | | | | | | | | | | +--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html |
Description of problem: I tried running dumpStorageTable.py on vdsm to try and trouble shoot some issues on rhevm-3. the script exists with error: root@master-vds9 vdsm]# python /usr/share/vdsm/dumpStorageTable.py Traceback (most recent call last): File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module> rc, msg = StorageTable(vdscli.connect()).show() File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__ self._buildStorageTable() File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable self._getVms(self.pool, files) File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms self._getImages(disks) File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages self._getVols(self.pool, sd, image) File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image) File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain next = volParents[prev] KeyError: '00000000-0000-0000-0000-000000000000' Version-Release number of selected component (if applicable): vdsm-4.9.6-44.1.el6_3.x86_64 How reproducible: Steps to Reproduce: 1. run dumpStorageTable.py 2. 3. Actual results: KeyError: '00000000-0000-0000-0000-000000000000' Expected results: 1. we need to fix this particular issue 2. we need to fix the script to exit in specific places so it will help us fix the issues as we go. Additional info: