Bug 893184 - 3.2 - DC upgrade failure with error "NoSpaceLeftOnDomain: No space left on domain <id>"
Summary: 3.2 - DC upgrade failure with error "NoSpaceLeftOnDomain: No space left on do...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.2.0
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ---
: 3.2.0
Assignee: Federico Simoncelli
QA Contact: Elad
URL:
Whiteboard: storage
Depends On:
Blocks: 894034 894038 915537
TreeView+ depends on / blocked
 
Reported: 2013-01-08 20:05 UTC by Sadique Puthen
Modified: 2018-12-01 16:10 UTC (History)
17 users (show)

Fixed In Version: vdsm-4.10.2-4.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 894038 (view as bug list)
Environment:
Last Closed:
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sadique Puthen 2013-01-08 20:05:05 UTC
Description of problem:

We upgraded Pune RHEVM and clusters to 3.1 successfully. Upgrading dc to 3.1 compatibility mode fails with below error. This upgrade will change the sds from v1 to v3 directly.

802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::316::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 274, in startSpm
    self._upgradePool(expectedDomVersion, __securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 414, in _upgradePool
    self._convertDomain(self.masterDomain, str(targetDomVersion))
  File "/usr/share/vdsm/storage/sp.py", line 1033, in _convertDomain
    domain.getRealDomain(), isMsd, targetFormat)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 342, in convert
    converter(repoPath, hostId, imageRepo, isMsd)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 192, in v3DomainConverter
    v3ReallocateMetadataSlot(domain, allVolumes)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 168, in v3ReallocateMetadataSlot
    raise se.NoSpaceLeftOnDomain(domain.sdUUID)
NoSpaceLeftOnDomain: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::317::Storage.StoragePool::(startSpm) failed: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::DEBUG::2013-01-08 06:45:50,968::sp::323::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process 

SPM is currently Contending between available hypervisors with above error without anyone taking the role. This brought down the environment without being able to start any vms. Still no luck

We increase the size of /dev/2f8b2751-7704-49a5-83ea-1e822b259c64/metadata lv from 512m to 1.5g, but issue still continues.

We need a fix for this before the release, otherwise this can cause longer outage to customers than expected or scheduled. No customer will agree to a full shutdown of env due to upgrade.

Comment 12 Federico Simoncelli 2013-01-10 14:59:15 UTC
Author: Federico Simoncelli <fsimonce>
Date:   Wed Jan 9 09:57:52 2013 +0200

    upgrade: force upgrade to v2 before upgrading to v3
    
    During the upgrade of a domain to version 3 vdsm reallocates the
    metadata slots that are higher than 1947 (given a leases LV of 2Gb)
    in order to use the same offsets for the volume leases (BZ#882276
    and git commit hash 2ba76e3).
    This has no effect when the domain is version 0 since the metadata
    slots offsets are fixed (the first physical extent of the LV) and
    they can't be reallocated. In such case the domain must be upgraded
    to version 2 first.

http://gerrit.ovirt.org/#/c/10792/

Comment 14 Elad 2013-02-26 15:30:36 UTC
tested on SF8. I was manage to upgrade from 3.0 to 3.1 and to 3.2 and from SD V1 format to V3 format.

Comment 18 Itamar Heim 2013-06-11 09:48:47 UTC
3.2 has been released

Comment 19 Itamar Heim 2013-06-11 09:48:56 UTC
3.2 has been released

Comment 20 Itamar Heim 2013-06-11 09:57:26 UTC
3.2 has been released


Note You need to log in before you can comment on or make changes to this bug.