Bug 2068104
| Summary: | hybrid backup VM after a snapshot fails - start_nbd_server error=Bitmap does not exist | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Evelina Shames <eshames> | ||||
| Component: | BLL.Storage | Assignee: | Benny Zlotnik <bzlotnik> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Evelina Shames <eshames> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.5.0 | CC: | ahadas, bugs, bzlotnik | ||||
| Target Milestone: | ovirt-4.5.0 | Keywords: | Automation | ||||
| Target Release: | 4.5.0.1 | Flags: | pm-rhel:
ovirt-4.5?
ahadas: blocker- |
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | ovirt-engine-4.5.0.1 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-04-28 09:26:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP. The issue is specific to performing incremental backup on RAW disks with COW overlay, so doing incremental backup in which a RAW disk with a snapshot participates is enough to reproduce this with non hybrid backup. Non hybrid backup can be enabled with: $ engine-config -s UseHybridBackup=false Tested on ovirt-engine-4.5.0.4-0.1.el8ev.noarch Incremental backup after creating a snapshot works as expected. Moving to 'Verified'. This bugzilla is included in oVirt 4.5.0 release, published on April 20th 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Created attachment 1868126 [details] logs Description of problem: Incremental backup (using SDK script backup_vm.py) after taking a snapshot fails: VDSM log: 2022-03-24 14:42:43,073+0200 ERROR (jsonrpc/5) [storage.taskmanager.task] (Task='7bb81375-a869-4bc3-a4b3-f6062cded97a') U nexpected error (task:877) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in _run return fn(*args, **kargs) File "<decorator-gen-231>", line 2, in start_nbd_server File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 3522, in start_nbd_server url = nbd.start_server(server_id, config) File "/usr/lib/python3.6/site-packages/vdsm/storage/nbd.py", line 133, in start_server bitmap=cfg.bitmap) vdsm.storage.exception.BitmapDoesNotExist: Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center /mnt/blockSD/dc0b82d1-f5ce-48d3-9bf3-301abbf2b3f1/images/44d693f0-b000-44e0-9c5e-125551234981/412c8f19-425c-40de-b927-3dc f246ab917', 'bitmap': '1dc099d8-99e3-48aa-8026-3979b5461ba4'}" 2022-03-24 14:42:43,074+0200 ERROR (jsonrpc/5) [storage.dispatcher] FINISH start_nbd_server error=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/blockSD/dc0b82d1-f5ce-48d3-9bf3-301abbf2b3f1/images/44d693f0-b000-44e0-9c5e-125551234981/412c8f19-425c-40de-b927-3dcf246ab917', 'bitmap': '1dc099d8-99e3-48aa-8026-3979b5461ba4'}" (dispatcher:83) Engine log: 2022-03-24 14:42:43,077+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-29) [c2db9189-e65b-4ffe-890e-a4520f9c4dc5] Failed in 'StartNbdServerVDS' method 2022-03-24 14:42:43,078+02 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.StartNbdServerVDSCommand] (default task-29) [c2db9189-e65b-4ffe-890e-a4520f9c4dc5] Unexpected return value: Status [code=947, message=Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/blockSD/dc0b82d1-f5ce-48d3-9bf3-301abbf2b3f1/images/44d693f0-b000-44e0-9c5e-125551234981/412c8f19-425c-40de-b927-3dcf246ab917', 'bitmap': '1dc099d8-99e3-48aa-8026-3979b5461ba4'}"] 2022-03-24 14:42:43,084+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [c2db9189-e65b-4ffe-890e-a4520f9c4dc5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_3 command StartNbdServerVDS failed: Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/blockSD/dc0b82d1-f5ce-48d3-9bf3-301abbf2b3f1/images/44d693f0-b000-44e0-9c5e-125551234981/412c8f19-425c-40de-b927-3dcf246ab917', 'bitmap': '1dc099d8-99e3-48aa-8026-3979b5461ba4'}" Version-Release number of selected component (if applicable): vdsm-4.50.0.10-1.el8ev.x86_64 engine-4.5.0-0.237.el8ev Same test on 4.4 works (engine-4.4.10.7-0.4.el8ev, vdsm-4.40.100.2-1.el8ev.x86_64) How reproducible: Always with 2 automation tests Steps to Reproduce: - Create a VM with disks: virtio_scsicow virtio_scsiraw virtiocow virtioraw - Run the VM - Create a full backup for the QCOW disks - Download the backed up disks - Create an incremental backups for the QCOW disks of the VM - Power off the VM - Create a snapshot for the VM - Run the VM - Add a content to backed up disks on the VM - Create another incremental backup for the VM, on the same disks - fails Actual results: Fails - Bitmap does not exist Expected results: The backup should succeed. Additional info: Engine and VDSM Logs are attached