This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 890983 - vdsm: dumpStorageTable.py exits on KeyError for buildVolumesChain
vdsm: dumpStorageTable.py exits on KeyError for buildVolumesChain
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.1.0
x86_64 Linux
unspecified Severity high
: ---
: 3.2.0
Assigned To: Mark Huth
Elad
storage
: Patch, Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-31 09:59 EST by Dafna Ron
Modified: 2016-02-10 14:15 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
does not require docs text requires_doc_text- flag set
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-10 16:38:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Allow dumpStorageTable.py to work with volumes with multiple leaf volumes, eg snapshot previews (3.95 KB, patch)
2013-02-04 19:08 EST, Mark Huth
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 12703 None None None Never

  None (edit)
Description Dafna Ron 2012-12-31 09:59:37 EST
Description of problem:

I tried running dumpStorageTable.py on vdsm to try and trouble shoot some issues on rhevm-3. 
the script exists with error: 

root@master-vds9 vdsm]# python /usr/share/vdsm/dumpStorageTable.py
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
    self._getVms(self.pool, files)
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
    self._getImages(disks)
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '00000000-0000-0000-0000-000000000000'

Version-Release number of selected component (if applicable):

vdsm-4.9.6-44.1.el6_3.x86_64

How reproducible:


Steps to Reproduce:
1. run dumpStorageTable.py
2.
3.
  
Actual results:

KeyError: '00000000-0000-0000-0000-000000000000'

Expected results:

1. we need to fix this particular issue
2. we need to fix the script to exit in specific places so it will help us fix the issues as we go. 

Additional info:
Comment 1 Yaniv Kaul 2012-12-31 10:07:29 EST
To elaborate on the 2nd expected results:
Since we assume this will be executed when storage issues are suspected, whatever it finds it should print right away, not go over all the storage domain / pool / data / whatever, so we can get at least a partial view of the storage.
Comment 3 Mark Huth 2013-01-08 22:20:19 EST
This issue can be reproduced by running dumpStorageTable when previewing a snapshot.  A particular parent volume in an image will have 2 child volumes now and dumpStorageTable doesn't account for that.  I remember seeing this problem with dumpStorageTable a long while ago and being the bad support person I am - didn't report it or fix it :(

Here volumes e41db184-6809-4589-80ee-bc0a06c1e291 and 02e80561-3420-4711-b72d-d633c1f59e66 have a single parent volume PU_93721462-a4fe-4b09-b73d-16d1e04cebb9

[root@rhevh02 vdsm]# lvs -o +tags | grep IU_400ddc
  02e80561-3420-4711-b72d-d633c1f59e66 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-----   1.00g                                            PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_12
  8f23a06e-a69f-4cd7-a50c-8b86edff9b20 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a---   1.00g                                            IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,PU_e41db184-6809-4589-80ee-bc0a06c1e291,MD_11
  93721462-a4fe-4b09-b73d-16d1e04cebb9 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-----   3.88g                                            PU_00000000-0000-0000-0000-000000000000,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_5 
  e41db184-6809-4589-80ee-bc0a06c1e291 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a---   1.00g                                            PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,MD_10,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21

And when that is the case, dumpStorageTable barfs:

[root@rhevh02 vdsm]# python dumpStorageTable.pyc
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
KeyError: '00000000-0000-0000-0000-000000000000'
Comment 4 Mark Huth 2013-01-08 22:42:19 EST
Also, this problem (well, at least when using snapshot previews) isn't a regression as I remember seeing this problem over 12 months ago with RHEV2.2.  Dafna, are you able to confirm if you are using snapshot previews when you generated the error in comment #0?  If not, then I stand corrected.
Comment 5 Dafna Ron 2013-02-03 10:17:20 EST
this was on our production environment and there are a lot of users there so I have no way of knowing of we had a vm with a preview. 
however, I ran DumpStorageTable with and without a previewed snapshot on my own setup I did get the below when I had a previewed snapshot: 

[root@gold-vdsd vdsm]# python dumpStorageTable.py
Traceback (most recent call last):
  File "dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "dumpStorageTable.py", line 174, in _buildStorageTable
    self._getSds(self.pool)
  File "dumpStorageTable.py", line 85, in _getSds
    self._getImages(disks)
  File "dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '00000000-0000-0000-0000-000000000000'
Comment 7 Mark Huth 2013-02-04 19:08:41 EST
Created attachment 693064 [details]
Allow dumpStorageTable.py to work with volumes with multiple leaf volumes, eg snapshot previews

Step 1: Before patch:

[root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
    self._getVms(self.pool, files)
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
    self._getImages(disks)
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8'

Step 2: Applying patch:

[root@rhevh-8 ~]# patch -b -i 0001-BZ890983-allow-dumpStorageTable-to-handle-images-wit.patch /usr/share/vdsm/dumpStorageTable.py
patching file /usr/share/vdsm/dumpStorageTable.py

Step 3: After patch:

[root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-1                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-2                 | Master                               |                                      | 35cb0bf3-6eac-49f1-8203-937b7c75e030 |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
|                                      |                                      |                                      | c72d4cc0-e4de-4c00-b167-f74b0c2072ef |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template                             |
| test                                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent                 |
| akovari_00753631_soa                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+

Image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 was in a snapshot preview and thus had 2 volume chains and leaf volumes.  The patch now handles that.


Step 4: After commiting (or undoing) the snapshot preview for image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301:

[root@rhevh-8 vdsm]# python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-1                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-2                 | Master                               |                                      | 35cb0bf3-6eac-49f1-8203-937b7c75e030 |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template                             |
| test                                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent                 |
| akovari_00753631_soa                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
Comment 8 Ayal Baron 2013-03-04 07:59:07 EST
Mark, did you submit the patch upstream to gerrit? (gerrit.ovirt.org)
Comment 9 Mark Huth 2013-03-04 21:24:06 EST
Hi Ayal,

I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/

Never submitted to gerrit before and I hope I did it correctly - I followed the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit

I'm getting emails that the build failed though:

<email>
...
Patch Set 1: Fails

Build Failed 

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1443/ : SUCCESS
...
</email>

I have to admit, the patch attached to this BZ (https://bugzilla.redhat.com/attachment.cgi?id=693064) didn't apply cleanly to the upstream code and I had to apply it manually, but I have tested it again and the re-patched dumpStorageTable does work:

[root@rhevh02 ~]# python ./dumpStorageTable.py
...
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 2239c875-03cc-4b0d-9756-6ea73ea0a0e5 | 7a818b38-07f6-49a5-a8e3-cdb2153295e8 | 400ddc46-9ec6-4786-b43b-80c4bf379c21 | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent                 |
| monitoredGuest                       | Master                               |                                      | e41db184-6809-4589-80ee-bc0a06c1e291 |                                      |
|                                      | data1                                |                                      | 80182192-1fc7-4094-b066-777b03d690d0 |                                      |
|                                      |                                      |                                      |                                      |                                      |
|                                      |                                      |                                      | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent                 |
|                                      |                                      |                                      | e41db184-6809-4589-80ee-bc0a06c1e291 |                                      |
|                                      |                                      |                                      | 8f23a06e-a69f-4cd7-a50c-8b86edff9b20 |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
Comment 10 Ayal Baron 2013-03-05 02:58:56 EST
(In reply to comment #9)
> Hi Ayal,
> 
> I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/

Thanks!

> 
> Never submitted to gerrit before and I hope I did it correctly - I followed
> the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit
> 
> I'm getting emails that the build failed though:
> 
> <email>
> ...
> Patch Set 1: Fails
> 
> Build Failed 
> 
> http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE

You can go to: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/console
and see the errors (e.g. vdsm/dumpStorageTable.py:112:80: E501 line too long (96 characters)

Additional comments on the patch though as there is no reason to inflate the bugzilla with the review process.
Comment 12 Mark Huth 2013-03-06 17:29:45 EST
Done...

oVirt Jenkins CI Server		8:11 AM

Patch Set 2: Verified
Build Successful
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1498/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1461/ : SUCCESS
Comment 14 Cheryn Tan 2013-04-03 03:01:38 EDT
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.
Comment 15 Mark Huth 2013-04-03 17:43:26 EDT
Setting requires_doc_text to minus because I don't think this BZ fix warrants a release note.
Comment 16 Elad 2013-04-04 05:30:21 EDT
Checked on SF12
vdsm-4.10.2-13.0.el6ev.x86_64



[root@nott-vds1 ~]#  python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| ec738a7f-5672-418e-b1a7-021ba33153b3 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | eaceceee-8083-42b6-a43f-c461015522f7 | 4b816021-ce7a-4e5e-86ce-69109936e1fa | Template independent                 |
| vm-test2                             | Regular                              |                                      |                                      |                                      |
|                                      | SD-iSCSI2                            |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| ac10e4c1-ab7f-43ed-a563-3f6fb723f909 | 9feea3f7-b571-4e10-a5e3-dd38504f6f9e |                                      |                                      |                                      |
| vm-test4                             | Master                               |                                      |                                      |                                      |
|                                      | SD-iSCSI                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 52192ea4-851b-43f1-8e75-6925b70c78c9 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | c95e0c92-1cd7-41c4-b6fd-e00b8047259e | 193315ef-8458-4c1d-af9f-1b996118c268 | Template independent                 |
| vm-iscsi-4-stateless                 | Regular                              |                                      | cabc8d83-83fb-4dee-bc46-0e3cfc7e2125 |                                      |
|                                      | SD-iSCSI2                            |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
Comment 18 errata-xmlrpc 2013-06-10 16:38:51 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0886.html

Note You need to log in before you can comment on or make changes to this bug.