Bug 893184 - 3.2 - DC upgrade failure with error "NoSpaceLeftOnDomain: No space left on domain <id>"
3.2 - DC upgrade failure with error "NoSpaceLeftOnDomain: No space left on do...
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
x86_64 Linux
high Severity urgent
: ---
: 3.2.0
Assigned To: Federico Simoncelli
: ZStream
Depends On:
Blocks: 894034 894038 915537
  Show dependency treegraph
Reported: 2013-01-08 15:05 EST by Sadique Puthen
Modified: 2016-02-10 14:57 EST (History)
17 users (show)

See Also:
Fixed In Version: vdsm-4.10.2-4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 894038 (view as bug list)
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Sadique Puthen 2013-01-08 15:05:05 EST
Description of problem:

We upgraded Pune RHEVM and clusters to 3.1 successfully. Upgrading dc to 3.1 compatibility mode fails with below error. This upgrade will change the sds from v1 to v3 directly.

802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::316::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 274, in startSpm
    self._upgradePool(expectedDomVersion, __securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 414, in _upgradePool
    self._convertDomain(self.masterDomain, str(targetDomVersion))
  File "/usr/share/vdsm/storage/sp.py", line 1033, in _convertDomain
    domain.getRealDomain(), isMsd, targetFormat)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 342, in convert
    converter(repoPath, hostId, imageRepo, isMsd)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 192, in v3DomainConverter
    v3ReallocateMetadataSlot(domain, allVolumes)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 168, in v3ReallocateMetadataSlot
    raise se.NoSpaceLeftOnDomain(domain.sdUUID)
NoSpaceLeftOnDomain: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::317::Storage.StoragePool::(startSpm) failed: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::DEBUG::2013-01-08 06:45:50,968::sp::323::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process 

SPM is currently Contending between available hypervisors with above error without anyone taking the role. This brought down the environment without being able to start any vms. Still no luck

We increase the size of /dev/2f8b2751-7704-49a5-83ea-1e822b259c64/metadata lv from 512m to 1.5g, but issue still continues.

We need a fix for this before the release, otherwise this can cause longer outage to customers than expected or scheduled. No customer will agree to a full shutdown of env due to upgrade.
Comment 12 Federico Simoncelli 2013-01-10 09:59:15 EST
Author: Federico Simoncelli <fsimonce@redhat.com>
Date:   Wed Jan 9 09:57:52 2013 +0200

    upgrade: force upgrade to v2 before upgrading to v3
    During the upgrade of a domain to version 3 vdsm reallocates the
    metadata slots that are higher than 1947 (given a leases LV of 2Gb)
    in order to use the same offsets for the volume leases (BZ#882276
    and git commit hash 2ba76e3).
    This has no effect when the domain is version 0 since the metadata
    slots offsets are fixed (the first physical extent of the LV) and
    they can't be reallocated. In such case the domain must be upgraded
    to version 2 first.

Comment 14 Elad 2013-02-26 10:30:36 EST
tested on SF8. I was manage to upgrade from 3.0 to 3.1 and to 3.2 and from SD V1 format to V3 format.
Comment 18 Itamar Heim 2013-06-11 05:48:47 EDT
3.2 has been released
Comment 19 Itamar Heim 2013-06-11 05:48:56 EDT
3.2 has been released
Comment 20 Itamar Heim 2013-06-11 05:57:26 EDT
3.2 has been released

Note You need to log in before you can comment on or make changes to this bug.