Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 890983 - vdsm: dumpStorageTable.py exits on KeyError for buildVolumesChain
Summary: vdsm: dumpStorageTable.py exits on KeyError for buildVolumesChain
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.1.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.2.0
Assignee: Mark Huth
QA Contact: Elad
URL:
Whiteboard: storage
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-31 14:59 UTC by Dafna Ron
Modified: 2016-02-10 19:15 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
does not require docs text requires_doc_text- flag set
Clone Of:
Environment:
Last Closed: 2013-06-10 20:38:51 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)
Allow dumpStorageTable.py to work with volumes with multiple leaf volumes, eg snapshot previews (3.95 KB, patch)
2013-02-05 00:08 UTC, Mark Huth
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0886 0 normal SHIPPED_LIVE Moderate: rhev 3.2 - vdsm security and bug fix update 2013-06-11 00:25:02 UTC
oVirt gerrit 12703 0 None None None Never

Description Dafna Ron 2012-12-31 14:59:37 UTC
Description of problem:

I tried running dumpStorageTable.py on vdsm to try and trouble shoot some issues on rhevm-3. 
the script exists with error: 

root@master-vds9 vdsm]# python /usr/share/vdsm/dumpStorageTable.py
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
    self._getVms(self.pool, files)
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
    self._getImages(disks)
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '00000000-0000-0000-0000-000000000000'

Version-Release number of selected component (if applicable):

vdsm-4.9.6-44.1.el6_3.x86_64

How reproducible:


Steps to Reproduce:
1. run dumpStorageTable.py
2.
3.
  
Actual results:

KeyError: '00000000-0000-0000-0000-000000000000'

Expected results:

1. we need to fix this particular issue
2. we need to fix the script to exit in specific places so it will help us fix the issues as we go. 

Additional info:

Comment 1 Yaniv Kaul 2012-12-31 15:07:29 UTC
To elaborate on the 2nd expected results:
Since we assume this will be executed when storage issues are suspected, whatever it finds it should print right away, not go over all the storage domain / pool / data / whatever, so we can get at least a partial view of the storage.

Comment 3 Mark Huth 2013-01-09 03:20:19 UTC
This issue can be reproduced by running dumpStorageTable when previewing a snapshot.  A particular parent volume in an image will have 2 child volumes now and dumpStorageTable doesn't account for that.  I remember seeing this problem with dumpStorageTable a long while ago and being the bad support person I am - didn't report it or fix it :(

Here volumes e41db184-6809-4589-80ee-bc0a06c1e291 and 02e80561-3420-4711-b72d-d633c1f59e66 have a single parent volume PU_93721462-a4fe-4b09-b73d-16d1e04cebb9

[root@rhevh02 vdsm]# lvs -o +tags | grep IU_400ddc
  02e80561-3420-4711-b72d-d633c1f59e66 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-----   1.00g                                            PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_12
  8f23a06e-a69f-4cd7-a50c-8b86edff9b20 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a---   1.00g                                            IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,PU_e41db184-6809-4589-80ee-bc0a06c1e291,MD_11
  93721462-a4fe-4b09-b73d-16d1e04cebb9 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-----   3.88g                                            PU_00000000-0000-0000-0000-000000000000,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21,MD_5 
  e41db184-6809-4589-80ee-bc0a06c1e291 7a818b38-07f6-49a5-a8e3-cdb2153295e8 -wi-a---   1.00g                                            PU_93721462-a4fe-4b09-b73d-16d1e04cebb9,MD_10,IU_400ddc46-9ec6-4786-b43b-80c4bf379c21

And when that is the case, dumpStorageTable barfs:

[root@rhevh02 vdsm]# python dumpStorageTable.pyc
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
KeyError: '00000000-0000-0000-0000-000000000000'

Comment 4 Mark Huth 2013-01-09 03:42:19 UTC
Also, this problem (well, at least when using snapshot previews) isn't a regression as I remember seeing this problem over 12 months ago with RHEV2.2.  Dafna, are you able to confirm if you are using snapshot previews when you generated the error in comment #0?  If not, then I stand corrected.

Comment 5 Dafna Ron 2013-02-03 15:17:20 UTC
this was on our production environment and there are a lot of users there so I have no way of knowing of we had a vm with a preview. 
however, I ran DumpStorageTable with and without a previewed snapshot on my own setup I did get the below when I had a previewed snapshot: 

[root@gold-vdsd vdsm]# python dumpStorageTable.py
Traceback (most recent call last):
  File "dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "dumpStorageTable.py", line 174, in _buildStorageTable
    self._getSds(self.pool)
  File "dumpStorageTable.py", line 85, in _getSds
    self._getImages(disks)
  File "dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '00000000-0000-0000-0000-000000000000'

Comment 7 Mark Huth 2013-02-05 00:08:41 UTC
Created attachment 693064 [details]
Allow dumpStorageTable.py to work with volumes with multiple leaf volumes, eg snapshot previews

Step 1: Before patch:

[root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 299, in <module>
    rc, msg = StorageTable(vdscli.connect()).show()
  File "/usr/share/vdsm/dumpStorageTable.py", line 34, in __init__
    self._buildStorageTable()
  File "/usr/share/vdsm/dumpStorageTable.py", line 172, in _buildStorageTable
    self._getVms(self.pool, files)
  File "/usr/share/vdsm/dumpStorageTable.py", line 63, in _getVms
    self._getImages(disks)
  File "/usr/share/vdsm/dumpStorageTable.py", line 98, in _getImages
    self._getVols(self.pool, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 122, in _getVols
    self.imagesList[imageDom] = self._buildVolumesChain(leafvol,vollen,volParents, sd, image)
  File "/usr/share/vdsm/dumpStorageTable.py", line 132, in _buildVolumesChain
    next = volParents[prev]
KeyError: '1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8'

Step 2: Applying patch:

[root@rhevh-8 ~]# patch -b -i 0001-BZ890983-allow-dumpStorageTable-to-handle-images-wit.patch /usr/share/vdsm/dumpStorageTable.py
patching file /usr/share/vdsm/dumpStorageTable.py

Step 3: After patch:

[root@rhevh-8 ~]# python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-1                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-2                 | Master                               |                                      | 35cb0bf3-6eac-49f1-8203-937b7c75e030 |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
|                                      |                                      |                                      | c72d4cc0-e4de-4c00-b167-f74b0c2072ef |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template                             |
| test                                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent                 |
| akovari_00753631_soa                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+

Image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 was in a snapshot preview and thus had 2 volume chains and leaf volumes.  The patch now handles that.


Step 4: After commiting (or undoing) the snapshot preview for image 57f2cced-4d68-4c9c-afa4-4b2a3faa8301:

[root@rhevh-8 vdsm]# python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 4466324c-d589-4416-8b62-93315163aa48 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | f8346847-d5a4-41f6-b082-cc99e303fefd | abe31b5c-7d22-4058-be7c-d9efc0658abb | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-1                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| bb765ba6-336f-48ea-b92f-0b8a7d9a46b9 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 57f2cced-4d68-4c9c-afa4-4b2a3faa8301 | eeace406-f0ab-4e9f-add0-b29a1e8dbef6 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 |
| pbandark_test_pool-2                 | Master                               |                                      | 35cb0bf3-6eac-49f1-8203-937b7c75e030 |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 0f9c92be-049b-4cf1-ae74-6fa28f283794 | b5aec7e9-33c5-4fc7-878a-f7382e64299e | 94b5ebca-c2b1-4436-bfa1-b87fc4bcafa3 | 1116e3ca-ed71-44e6-b5fd-b0ecee70fcb8 | Template                             |
| test                                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 64796488-2092-4739-b4b8-acf87431c87e | b5aec7e9-33c5-4fc7-878a-f7382e64299e | b6a2ad65-ff69-497a-8608-90c23e9dedf7 | f4ad98c9-6385-485d-b572-efcb0893b275 | Template independent                 |
| akovari_00753631_soa                 | Master                               |                                      |                                      |                                      |
|                                      | Data200G                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+

Comment 8 Ayal Baron 2013-03-04 12:59:07 UTC
Mark, did you submit the patch upstream to gerrit? (gerrit.ovirt.org)

Comment 9 Mark Huth 2013-03-05 02:24:06 UTC
Hi Ayal,

I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/

Never submitted to gerrit before and I hope I did it correctly - I followed the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit

I'm getting emails that the build failed though:

<email>
...
Patch Set 1: Fails

Build Failed 

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1443/ : SUCCESS
...
</email>

I have to admit, the patch attached to this BZ (https://bugzilla.redhat.com/attachment.cgi?id=693064) didn't apply cleanly to the upstream code and I had to apply it manually, but I have tested it again and the re-patched dumpStorageTable does work:

[root@rhevh02 ~]# python ./dumpStorageTable.py
...
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 2239c875-03cc-4b0d-9756-6ea73ea0a0e5 | 7a818b38-07f6-49a5-a8e3-cdb2153295e8 | 400ddc46-9ec6-4786-b43b-80c4bf379c21 | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent                 |
| monitoredGuest                       | Master                               |                                      | e41db184-6809-4589-80ee-bc0a06c1e291 |                                      |
|                                      | data1                                |                                      | 80182192-1fc7-4094-b066-777b03d690d0 |                                      |
|                                      |                                      |                                      |                                      |                                      |
|                                      |                                      |                                      | 93721462-a4fe-4b09-b73d-16d1e04cebb9 | Template independent                 |
|                                      |                                      |                                      | e41db184-6809-4589-80ee-bc0a06c1e291 |                                      |
|                                      |                                      |                                      | 8f23a06e-a69f-4cd7-a50c-8b86edff9b20 |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+

Comment 10 Ayal Baron 2013-03-05 07:58:56 UTC
(In reply to comment #9)
> Hi Ayal,
> 
> I have submitted it just now - http://gerrit.ovirt.org/#/c/12703/

Thanks!

> 
> Never submitted to gerrit before and I hope I did it correctly - I followed
> the steps in http://wiki.ovirt.org/Working_with_oVirt_Gerrit
> 
> I'm getting emails that the build failed though:
> 
> <email>
> ...
> Patch Set 1: Fails
> 
> Build Failed 
> 
> http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/ : FAILURE

You can go to: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1480/console
and see the errors (e.g. vdsm/dumpStorageTable.py:112:80: E501 line too long (96 characters)

Additional comments on the patch though as there is no reason to inflate the bugzilla with the review process.

Comment 12 Mark Huth 2013-03-06 22:29:45 UTC
Done...

oVirt Jenkins CI Server		8:11 AM

Patch Set 2: Verified
Build Successful
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1498/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1461/ : SUCCESS

Comment 14 Cheryn Tan 2013-04-03 07:01:38 UTC
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.

Comment 15 Mark Huth 2013-04-03 21:43:26 UTC
Setting requires_doc_text to minus because I don't think this BZ fix warrants a release note.

Comment 16 Elad 2013-04-04 09:30:21 UTC
Checked on SF12
vdsm-4.10.2-13.0.el6ev.x86_64



[root@nott-vds1 ~]#  python /usr/share/vdsm/dumpStorageTable.py
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
|                 Vms                  |               Domains                |                Images                |               Volumes                |               Template               |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| ec738a7f-5672-418e-b1a7-021ba33153b3 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | eaceceee-8083-42b6-a43f-c461015522f7 | 4b816021-ce7a-4e5e-86ce-69109936e1fa | Template independent                 |
| vm-test2                             | Regular                              |                                      |                                      |                                      |
|                                      | SD-iSCSI2                            |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| ac10e4c1-ab7f-43ed-a563-3f6fb723f909 | 9feea3f7-b571-4e10-a5e3-dd38504f6f9e |                                      |                                      |                                      |
| vm-test4                             | Master                               |                                      |                                      |                                      |
|                                      | SD-iSCSI                             |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| 52192ea4-851b-43f1-8e75-6925b70c78c9 | 75c83dc1-24e2-4f05-bcb1-30638060f5cb | c95e0c92-1cd7-41c4-b6fd-e00b8047259e | 193315ef-8458-4c1d-af9f-1b996118c268 | Template independent                 |
| vm-iscsi-4-stateless                 | Regular                              |                                      | cabc8d83-83fb-4dee-bc46-0e3cfc7e2125 |                                      |
|                                      | SD-iSCSI2                            |                                      |                                      |                                      |
|                                      |                                      |                                      |                                      |                                      |
+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+

Comment 18 errata-xmlrpc 2013-06-10 20:38:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0886.html


Note You need to log in before you can comment on or make changes to this bug.