Bug 902838

Description of problem:
spmStart throws a MetaDataKeyNotFoundError exception while upgrading a domain from v2 to v3 due to a partially removed volume (lv present, metadata blanked).

This is due to the following try catch block missing the MetaDataKeyNotFoundError exception from img.prepare on line 251 :

vdsm/vdsm/storage/imageRepository/formatConverter.py

~~~
250             try:
251                 for vol in img.prepare(domain.sdUUID, imgUUID):
252                     try:
253                         v3ResetMetaVolSize(vol)  # BZ#811880
254                     except qemuImg.QImgError:
255                         log.error("It is not possible to read the volume %s "
256                                   "using qemu-img, the content looks damaged",
257                                   vol.volUUID, exc_info=True)
258 
259             except se.VolumeDoesNotExist:
260                 log.error("It is not possible to prepare the image %s, the "
261                           "volume chain looks damaged", imgUUID, exc_info=True)
262 
263             finally:
264                 try:
265                     img.teardown(domain.sdUUID, imgUUID)
266                 except:
267                     log.debug("Unable to teardown the image %s, this error is "
268                               "not critical since the volume might be in use",
269                               imgUUID, exc_info=True)
~~~

Suggested fix would be to simply catch the MetaDataKeyNotFoundError and continue as we do in finally.

Example vdsm.log attached with ~12 examples of spmStart failing due to this.

I've included an example snippet below :
~~~

39985 Thread-13913::DEBUG::2013-01-22 12:53:58,448::BindingXMLRPC::171::vds::(wrapper) [10.141.149.254]
39986 Thread-13913::DEBUG::2013-01-22 12:53:58,448::task::588::TaskManager.Task::(_updateState) Task=`13697099-e662-4d6a-bd4b-0268d46c100a`::moving from state init -> state preparing
39987 Thread-13913::INFO::2013-01-22 12:53:58,449::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='7b2ded83-73c1-412a-b7d9-765e21fb1e42', prevID=-1, prevLVER='114', recoveryMode=None, scsiFencing='false', maxHostID=      250, domVersion='3', options=None)
[..]
42691 13697099-e662-4d6a-bd4b-0268d46c100a::DEBUG::2013-01-22 12:55:36,688::formatConverter::269::Storage.v3DomainConverter::(v3DomainConverter) Unable to teardown the image 69b5106e-5d4a-4642-8be7-d5d39cfbaede, this error is not critica      l since the volume might be in use
42692 Traceback (most recent call last):
42693   File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 265, in v3DomainConverter
42694   File "/usr/share/vdsm/storage/image.py", line 379, in teardown
42695   File "/usr/share/vdsm/storage/image.py", line 319, in getChain
42696   File "/usr/share/vdsm/storage/volume.py", line 660, in isShared
42697   File "/usr/share/vdsm/storage/volume.py", line 621, in getVolType
42698   File "/usr/share/vdsm/storage/volume.py", line 865, in getMetaParam
42699 MetaDataKeyNotFoundError: Meta Data key not found error: ("{'NONE': '##################################################################################################################################################################      #######################################################################################################################################################################################################################################      #############################################################################################################'}:VOLTYPE",)
[..]
42776 13697099-e662-4d6a-bd4b-0268d46c100a::ERROR::2013-01-22 12:55:38,929::sp::316::Storage.StoragePool::(startSpm) Unexpected error
42777 Traceback (most recent call last):
42778   File "/usr/share/vdsm/storage/sp.py", line 274, in startSpm
42779   File "/usr/share/vdsm/storage/securable.py", line 63, in wrapper
42780   File "/usr/share/vdsm/storage/sp.py", line 414, in _upgradePool
42781   File "/usr/share/vdsm/storage/sp.py", line 1033, in _convertDomain
42782   File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 342, in convert
42783   File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 251, in v3DomainConverter
42784   File "/usr/share/vdsm/storage/image.py", line 364, in prepare
42785   File "/usr/share/vdsm/storage/image.py", line 319, in getChain
42786   File "/usr/share/vdsm/storage/volume.py", line 660, in isShared
42787   File "/usr/share/vdsm/storage/volume.py", line 621, in getVolType
42788   File "/usr/share/vdsm/storage/volume.py", line 865, in getMetaParam
42789 MetaDataKeyNotFoundError: Meta Data key not found error: ("{'NONE': '##################################################################################################################################################################      #######################################################################################################################################################################################################################################      #############################################################################################################'}:VOLTYPE",)
42790 13697099-e662-4d6a-bd4b-0268d46c100a::ERROR::2013-01-22 12:55:38,929::sp::317::Storage.StoragePool::(startSpm) failed: Meta Data key not found error: ("{'NONE': '#####################################################################      #######################################################################################################################################################################################################################################      ##########################################################################################################################################################################################################'}:VOLTYPE",)
42791 13697099-e662-4d6a-bd4b-0268d46c100a::DEBUG::2013-01-22 12:55:38,929::sp::323::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
~~~

A simple workaround to this situation is to lvrename the volume LVs that form the image and also lvchange --deltag any LV tags that they might have.

Version-Release number of selected component (if applicable):
4.9.6-44.2

How reproducible:
Always where a volume's LV is still present but it's metadata is blanked out.

Steps to Reproduce:
1. Remove a volume from engine.
2. Blank out the volumes MD block.
3. Attempt to upgrade the domain from V2 to V3.
  
Actual results:
Failure to upgrade due to the uncaught MetaDataKeyNotFoundError exception.


Expected results:
Upgrade catches the MetaDataKeyNotFoundError exception and completes successfully.

Additional info:
For the original customer this all started with a  failure in 3.0 to cleanly remove this image/volume, something I will look into shortly with data from older cases.