Description of problem:
We upgraded Pune RHEVM and clusters to 3.1 successfully. Upgrading dc to 3.1 compatibility mode fails with below error. This upgrade will change the sds from v1 to v3 directly.
802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::316::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 274, in startSpm
self._upgradePool(expectedDomVersion, __securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper
return f(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 414, in _upgradePool
self._convertDomain(self.masterDomain, str(targetDomVersion))
File "/usr/share/vdsm/storage/sp.py", line 1033, in _convertDomain
domain.getRealDomain(), isMsd, targetFormat)
File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 342, in convert
converter(repoPath, hostId, imageRepo, isMsd)
File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 192, in v3DomainConverter
v3ReallocateMetadataSlot(domain, allVolumes)
File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 168, in v3ReallocateMetadataSlot
raise se.NoSpaceLeftOnDomain(domain.sdUUID)
NoSpaceLeftOnDomain: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::317::Storage.StoragePool::(startSpm) failed: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64'
802c79cd-5c97-4673-b16b-2efba481c88f::DEBUG::2013-01-08 06:45:50,968::sp::323::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
SPM is currently Contending between available hypervisors with above error without anyone taking the role. This brought down the environment without being able to start any vms. Still no luck
We increase the size of /dev/2f8b2751-7704-49a5-83ea-1e822b259c64/metadata lv from 512m to 1.5g, but issue still continues.
We need a fix for this before the release, otherwise this can cause longer outage to customers than expected or scheduled. No customer will agree to a full shutdown of env due to upgrade.
Comment 12Federico Simoncelli
2013-01-10 14:59:15 UTC
Author: Federico Simoncelli <fsimonce>
Date: Wed Jan 9 09:57:52 2013 +0200
upgrade: force upgrade to v2 before upgrading to v3
During the upgrade of a domain to version 3 vdsm reallocates the
metadata slots that are higher than 1947 (given a leases LV of 2Gb)
in order to use the same offsets for the volume leases (BZ#882276
and git commit hash 2ba76e3).
This has no effect when the domain is version 0 since the metadata
slots offsets are fixed (the first physical extent of the LV) and
they can't be reallocated. In such case the domain must be upgraded
to version 2 first.
http://gerrit.ovirt.org/#/c/10792/
Description of problem: We upgraded Pune RHEVM and clusters to 3.1 successfully. Upgrading dc to 3.1 compatibility mode fails with below error. This upgrade will change the sds from v1 to v3 directly. 802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::316::Storage.StoragePool::(startSpm) Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 274, in startSpm self._upgradePool(expectedDomVersion, __securityOverride=True) File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper return f(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 414, in _upgradePool self._convertDomain(self.masterDomain, str(targetDomVersion)) File "/usr/share/vdsm/storage/sp.py", line 1033, in _convertDomain domain.getRealDomain(), isMsd, targetFormat) File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 342, in convert converter(repoPath, hostId, imageRepo, isMsd) File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 192, in v3DomainConverter v3ReallocateMetadataSlot(domain, allVolumes) File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 168, in v3ReallocateMetadataSlot raise se.NoSpaceLeftOnDomain(domain.sdUUID) NoSpaceLeftOnDomain: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64' 802c79cd-5c97-4673-b16b-2efba481c88f::ERROR::2013-01-08 06:45:50,967::sp::317::Storage.StoragePool::(startSpm) failed: No space left on domain 2f8b2751-7704-49a5-83ea-1e822b259c64: '2f8b2751-7704-49a5-83ea-1e822b259c64' 802c79cd-5c97-4673-b16b-2efba481c88f::DEBUG::2013-01-08 06:45:50,968::sp::323::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process SPM is currently Contending between available hypervisors with above error without anyone taking the role. This brought down the environment without being able to start any vms. Still no luck We increase the size of /dev/2f8b2751-7704-49a5-83ea-1e822b259c64/metadata lv from 512m to 1.5g, but issue still continues. We need a fix for this before the release, otherwise this can cause longer outage to customers than expected or scheduled. No customer will agree to a full shutdown of env due to upgrade.