Created attachment 817451 [details] engine and vdsm logs Description of problem: prepareImage for vm fails with unboundLocalError on nfs after live snapshot creation: Thread-452::DEBUG::2013-10-30 02:32:22,464::BindingXMLRPC::984::vds::(wrapper) client [10.35.161.81]::call vmSnapshot with ('acb85533-3254-4292-a5a4-3dbd51a2e5d9', [{'baseVolumeID': 'af20b747-3a78-492f-8efe-eb82d288c865', 'domainID': 'd6eecfb6-d147-4604-92db-cd2e931456a7', 'volumeID': '8098ae12-4da2-4052-afc5-540efb0d6a8d', 'imageID': '190e3bfd-9df9-4fb5-964e-ddb553cf15ca'}], 'd6eecfb6-d147-4604-92db-cd2e931456a7,ca5152ee-739b-4f51-be5e-7576e39513d9,faf6bd71-6d94-4545-9917-b40467907b1a,788ce11e-5736-47d7-b08b-e575d14ef67f,a3a6a9d7-94af-4f90-81ff-b990c38eb6c3,59c94c97-385b-4581-9f93-928cc4f8b98a') {} flowID [5fe9cf5b] Thread-452::DEBUG::2013-10-30 02:32:22,464::task::579::TaskManager.Task::(_updateState) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::moving from state init -> state preparing Thread-452::INFO::2013-10-30 02:32:22,465::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='d6eecfb6-d147-4604-92db-cd2e931456a7', spUUID='ca5152ee-739b-4f51-be5e-7576e39513d9', imgUUID='190e3bfd-9df9-4fb5-964e-ddb553cf15ca', leafUUID='8098ae12-4da2-4052-afc5-540efb0d6a8d') Thread-452::DEBUG::2013-10-30 02:32:22,465::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.d6eecfb6-d147-4604-92db-cd2e931456a7`ReqID=`d91343c3-9697-4d65-a82a-5a04da1f79db`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3250' at 'prepareImage' Thread-452::DEBUG::2013-10-30 02:32:22,465::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.d6eecfb6-d147-4604-92db-cd2e931456a7' for lock type 'shared' Thread-452::DEBUG::2013-10-30 02:32:22,465::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.d6eecfb6-d147-4604-92db-cd2e931456a7' is free. Now locking as 'shared' (1 active user) Thread-452::DEBUG::2013-10-30 02:32:22,466::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.d6eecfb6-d147-4604-92db-cd2e931456a7`ReqID=`d91343c3-9697-4d65-a82a-5a04da1f79db`::Granted request Thread-452::DEBUG::2013-10-30 02:32:22,467::task::811::TaskManager.Task::(resourceAcquired) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::_resourcesAcquired: Storage.d6eecfb6-d147-4604-92db-cd2e931456a7 (shared) Thread-452::DEBUG::2013-10-30 02:32:22,467::task::974::TaskManager.Task::(_decref) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::ref 1 aborting False Thread-452::WARNING::2013-10-30 02:32:22,472::fileUtils::167::Storage.fileUtils::(createdir) Dir /var/run/vdsm/storage/d6eecfb6-d147-4604-92db-cd2e931456a7 already exists Thread-452::DEBUG::2013-10-30 02:32:22,472::fileSD::443::Storage.StorageDomain::(createImageLinks) img run dir already exists: /var/run/vdsm/storage/d6eecfb6-d147-4604-92db-cd2e931456a7/190e3bfd-9df9-4fb5-964e-ddb553cf15ca Thread-452::DEBUG::2013-10-30 02:32:22,473::fileVolume::528::Storage.Volume::(validateVolumePath) validate path for af20b747-3a78-492f-8efe-eb82d288c865 Thread-452::DEBUG::2013-10-30 02:32:22,475::fileVolume::528::Storage.Volume::(validateVolumePath) validate path for 3d3b2184-0132-44af-8fc5-fa2371f11e56 Thread-452::ERROR::2013-10-30 02:32:22,476::task::850::TaskManager.Task::(_setError) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 857, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3283, in prepareImage return {'path': leafPath, 'info': leafInfo, UnboundLocalError: local variable 'leafInfo' referenced before assignment Thread-452::DEBUG::2013-10-30 02:32:22,478::task::869::TaskManager.Task::(_run) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::Task._run: 32f7feca-9226-487a-929c-04b0af37f2cd ('d6eecfb6-d147-4604-92db-cd2e931456a7', 'ca5152ee-739b-4f51-be5e-7576e39513d9', '190e3bfd-9df9-4fb5-964e-ddb553cf15ca', '8098ae12-4da2-4052-afc5-540efb0d6a8d') {} failed - stopping task Thread-452::DEBUG::2013-10-30 02:32:22,478::task::1194::TaskManager.Task::(stop) Task=`32f7feca-9226-487a-929c-04b0af37f2cd`::stopping in state preparing (force False) Version-Release number of selected component (if applicable): vdsm-4.13.0-0.5.beta1.el6ev.x86_64 How reproducible: ? reproduced only through automation at the moment Steps to Reproduce: 2 hosts, 2 data domains 1. Create vm with a single disk and install OS on it (run on HSM) 2. Create live snapshot of vm - this snapshot fails Actual results: live snapshot creation fails Expected results: snapshot creation should succeed Additional info:
The new volume 8098ae12-4da2-4052-afc5-540efb0d6a8d is not visible for the HSM when the prepare image is called. This is unrelated to vdsm. Reproduced with or without vdsm from the cmd line.
Reproduction: Starting state: HSM: [root@aqua-vds4 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ ├── dom_md │ │ ├── ids │ │ ├── inbox │ │ ├── leases │ │ ├── metadata │ │ └── outbox │ ├── images │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ └── master │ ├── tasks │ └── vms └── __DIRECT_IO_TEST__ SPM: root@aqua-vds5 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt ├── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ │ ├── dom_md │ │ │ ├── ids │ │ │ ├── inbox │ │ │ ├── leases │ │ │ ├── metadata │ │ │ └── outbox │ │ ├── images │ │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ │ └── master │ │ ├── tasks │ │ └── vms │ └── __DIRECT_IO_TEST__ └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2 ├── 6207c259-bed2-4cc8-8cdf-836da5892c98 │ ├── dom_md │ │ ├── ids │ │ ├── inbox │ │ ├── leases │ │ ├── metadata │ │ └── outbox │ └── images └── __DIRECT_IO_TEST__ 2) A new snapshot is created SPM: [root@aqua-vds5 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt ├── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ │ ├── dom_md │ │ │ ├── ids │ │ │ ├── inbox │ │ │ ├── leases │ │ │ ├── metadata │ │ │ └── outbox │ │ ├── images │ │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.lease │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.meta │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ │ └── master │ │ ├── tasks │ │ │ └── 53adcf8b-3106-4ab7-bea7-453c8e52ae71 │ │ │ ├── 53adcf8b-3106-4ab7-bea7-453c8e52ae71.job.0 │ │ │ ├── 53adcf8b-3106-4ab7-bea7-453c8e52ae71.recover.0 │ │ │ ├── 53adcf8b-3106-4ab7-bea7-453c8e52ae71.result │ │ │ └── 53adcf8b-3106-4ab7-bea7-453c8e52ae71.task │ │ └── vms │ └── __DIRECT_IO_TEST__ └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2 ├── 6207c259-bed2-4cc8-8cdf-836da5892c98 │ ├── dom_md │ │ ├── ids │ │ ├── inbox │ │ ├── leases │ │ ├── metadata │ │ └── outbox │ └── images └── __DIRECT_IO_TEST__ But HSM remains the same as before (steady state): [root@aqua-vds4 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ ├── dom_md │ │ ├── ids │ │ ├── inbox │ │ ├── leases │ │ ├── metadata │ │ └── outbox │ ├── images │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ └── master │ ├── tasks │ └── vms └── __DIRECT_IO_TEST__ 3) Create a new file in the storage server: [root@tiger ~]# touch /fastpass/gadi/nfs1/added_file.void HSM and SPM hosts can see the added file. 4) Seting the HSM on maintenance and activating it (i.e. mount and umount): HSM: [root@aqua-vds4 ~]# ls -l /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/ total 4 -rw-r--r--. 1 root root 0 Nov 4 2013 added_file.void drwxr-xr-x. 5 vdsm kvm 4096 Nov 4 2013 c66bd14e-7652-4efb-b064-31bb65717a35 -rwxr-xr-x. 1 vdsm kvm 0 Sep 24 14:47 __DIRECT_IO_TEST__ SPM: (Can't view the file): [root@aqua-vds5 ~]# ls -al /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 total 12 drwxr-xr-x. 3 vdsm kvm 4096 Nov 4 2013 . drwxr-xr-x. 4 vdsm kvm 4096 Nov 3 17:13 .. drwxr-xr-x. 5 vdsm kvm 4096 Nov 4 2013 c66bd14e-7652-4efb-b064-31bb65717a35 -rwxr-xr-x. 1 vdsm kvm 0 Sep 24 14:47 __DIRECT_IO_TEST__
5) Creating a new directory in the storage server causes the clients to refresh the view of the mount. Files are still stale. [root@tiger ~]# mkdir -pv /fastpass/gadi/nfs1/another_dir.void mkdir: created directory `/fastpass/gadi/nfs1/another_dir.void' [root@aqua-vds5 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ ├── ef33f066-b713-445b-bd05-f9cd586b58ee -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2/ef33f066-b713-445b-bd05-f9cd586b58ee │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt ├── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 │ ├── added_file.void │ ├── another_dir.void │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ │ ├── dom_md │ │ │ ├── ids │ │ │ ├── inbox │ │ │ ├── leases │ │ │ ├── metadata │ │ │ └── outbox │ │ ├── images │ │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.lease │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.meta │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ │ └── master │ │ ├── tasks │ │ └── vms │ │ └── 833fa4bb-77e4-4e2d-aadc-3f77e16636fc │ │ └── 833fa4bb-77e4-4e2d-aadc-3f77e16636fc.ovf │ └── __DIRECT_IO_TEST__ └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2 ├── __DIRECT_IO_TEST__ └── ef33f066-b713-445b-bd05-f9cd586b58ee ├── dom_md │ ├── ids │ ├── inbox │ ├── leases │ ├── metadata │ └── outbox └── images └── 458b2e09-6af7-4e2e-9368-5c2b179b9dc2 ├── ac8a7de5-d04a-456a-b7e0-365f20a07d4f ├── ac8a7de5-d04a-456a-b7e0-365f20a07d4f.lease └── ac8a7de5-d04a-456a-b7e0-365f20a07d4f.meta 6) Creating another file produces the same results: [root@tiger ~]# touch /fastpass/gadi/nfs1/reconfirmation.void (Same on aqua's) 7) Create another dir causes refresh [root@tiger ~]# mkdir -pv /fastpass/gadi/nfs1/reconfirmation_dir.void mkdir: created directory `/fastpass/gadi/nfs1/reconfirmation_dir.void' [root@aqua-vds5 ~]# tree /rhev/data-center/ /rhev/data-center/ ├── 5849b030-626e-47cb-ad90-3ce782d831b3 │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 │ ├── ef33f066-b713-445b-bd05-f9cd586b58ee -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2/ef33f066-b713-445b-bd05-f9cd586b58ee │ └── mastersd -> /rhev/data-center/mnt/tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1/c66bd14e-7652-4efb-b064-31bb65717a35 ├── hsm-tasks └── mnt ├── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs1 │ ├── added_file.void │ ├── another_dir.void │ ├── c66bd14e-7652-4efb-b064-31bb65717a35 │ │ ├── dom_md │ │ │ ├── ids │ │ │ ├── inbox │ │ │ ├── leases │ │ │ ├── metadata │ │ │ └── outbox │ │ ├── images │ │ │ └── 43a88945-62f8-4dd3-88dd-d55b71244723 │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.lease │ │ │ ├── 32eb4805-c03b-4727-9e88-9d559359057a.meta │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511 │ │ │ ├── 8bac43ab-fb3d-41a2-843a-45e1af21b511.lease │ │ │ └── 8bac43ab-fb3d-41a2-843a-45e1af21b511.meta │ │ └── master │ │ ├── tasks │ │ └── vms │ │ └── 833fa4bb-77e4-4e2d-aadc-3f77e16636fc │ │ └── 833fa4bb-77e4-4e2d-aadc-3f77e16636fc.ovf │ ├── __DIRECT_IO_TEST__ │ ├── reconfirmation_dir.void │ └── reconfirmation.void └── tiger.qa.lab.tlv.redhat.com:_fastpass_gadi_nfs2 ├── __DIRECT_IO_TEST__ └── ef33f066-b713-445b-bd05-f9cd586b58ee ├── dom_md │ ├── ids │ ├── inbox │ ├── leases │ ├── metadata │ └── outbox └── images └── 458b2e09-6af7-4e2e-9368-5c2b179b9dc2 ├── ac8a7de5-d04a-456a-b7e0-365f20a07d4f ├── ac8a7de5-d04a-456a-b7e0-365f20a07d4f.lease └── ac8a7de5-d04a-456a-b7e0-365f20a07d4f.meta
Gadi, Can you, please, try to reproduce that bug with kernel newer than kernel-2.6.32-426.el. Lately we had a lot of NFS issues that should be fixed in that build of kernel.
> 3) Create a new file in the storage server: > [root@tiger ~]# touch /fastpass/gadi/nfs1/added_file.void > > HSM and SPM hosts can see the added file. > > HSM and SPM hosts CAN'T see the added file.
Rerunning this scenario (live snapshot of vm on HSM) as well as scenario tested by Eduardo in comment 4 succeeds with kernel 2.6.32-428.el6.x86_64
I am moving it to ON_QA, based on comment #7.
(In reply to Sergey Gotliv from comment #8) > I am moving it to ON_QA, based on comment #7. As per comment 7, Live Snapshot of vm that is running on HSM succeeds. Moving to Verified. kernel 2.6.32-428.el6.x86_64 vdsm-4.13.0-0.5.beta1.el6ev.x86_64
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag. Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information: * Cause: What actions or circumstances cause this bug to present. * Consequence: What happens when the bug presents. * Fix: What was done to fix the bug. * Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug. For further details on the Cause, Consequence, Fix, Result format please refer to: https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes Thanks in advance.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0040.html
Note that BZ#1024784 should have been close "Not a bug" and not to be included in the errata.
*** Bug 1080106 has been marked as a duplicate of this bug. ***