Created attachment 1022154 [details] openstack logs - /var/log/cinder/* Description of problem: Installed openstack on a VM all in one , cinder backend is local LVM . repo was installed from : https://repos.fedorapeople.org/repos/openstack/openstack-kilo/testing/el7/ Started to run tempest code , there are few failures . From /var/log/ : u'compute', u'name': u'nova'}], u'request_id': u'req-b162155b-cc4a-4d6c-9c81-2843ff17adfb', u'user_domain': None, u'read_deleted': u'no', u'tenant': u'2e69e7ebba7f460e91369b9cfb132769'} unpack_context /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203 2015-05-05 05:23:40.298 6084 DEBUG cinder.volume.manager [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Volume 86651feb-a9ed-4ae0-ba55-e971d6b15ef2: creating export initialize_connection /usr/lib/python2.7/site-packages/cinder/volume/manager.py:1082 2015-05-05 05:23:40.299 6084 DEBUG cinder.volume.targets.tgt [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Failed to open config for Volume volume-86651feb-a9ed-4ae0-ba55-e971d6b15ef2: [Errno 2] No such file or directory: u'/var/lib/cinder/volumes/volume-86651feb-a9ed-4ae0-ba55-e971d6b15ef2' _get_target_chap_auth /usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py:134 2015-05-05 05:23:40.320 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199 2015-05-05 05:23:40.470 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target" returned: 96 in 0.150s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225 2015-05-05 05:23:40.472 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] u'sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258 2015-05-05 05:23:40.474 6084 ERROR oslo_messaging.rpc.dispatcher [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Exception during message handling: Unexpected error while running command. Command: None Exit code: - Stdout: u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target\nExit code: 96\nStdout: u''\nStderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\\n'" Stderr: None 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1084, in initialize_connection 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher volume) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 606, in create_export 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher volume_path) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 208, in create_export 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher chap_auth) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 174, in create_iscsi_target 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher run_as_root=True) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 143, in execute 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 233, in execute 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher cmd=sanitized_cmd) 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command. 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Exit code: 96 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Stdout: u'' 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Stderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\n' 2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher 2015-05-05 05:23:40.572 6084 ERROR oslo_messaging._drivers.common [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Returning exception Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target Exit code: 96 Stdout: u'' Stderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\n' to caller 2015-05-05 05:23:40.573 6084 ERROR oslo_messaging._drivers.common [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1084, in initialize_connection\n volume)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 606, in create_export\n volume_path)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 208, in create_export\n chap_auth)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 174, in create_iscsi_target\n run_as_root=True)\n', ' File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 143, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 233, in execute\n cmd=sanitized_cmd)\n', "ProcessExecutionError: Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target\nExit code: 96\nStdout: u''\nStderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\\n'\n"] 2015-05-05 05:23:40.574 6084 DEBUG oslo_messaging._drivers.amqp [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] UNIQUE_ID is 3ca248fc9b144c7a81a41ef74c89fa40. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258 2015-05-05 05:23:40.579 6084 DEBUG oslo_messaging._drivers.amqp [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] UNIQUE_ID is cb44031fdbab4f72b42f1847b01efdb5. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258 Version-Release number of selected component (if applicable): How reproducible: Run tempest api tests , errors are in : tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments[id-7fa563fe-f0f7-43eb-9e22-a1ece036b513] tempest.api.volume.test_volumes_snapshots.VolumesV1SnapshotTestJSON.test_snapshot_create_with_volume_in_use[compute,id-b467b54c-07a4-446d-a1cf-651dedcc3ff1] tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use[compute,id-b467b54c-07a4-446d-a1cf-651dedcc3ff1] tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume] tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume] tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4717-910f-956cce965722,image,volume] tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks[compute,id-79165fb4-5534-4b9d-8429-97ccffb8f86e,image,volume] tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern[compute,id-608e604b-1d63-4a82-8e3e-91bc665c90b4,image,network] Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
/usr/share/cinder/cinder-dist.conf should be setting iscsi_helper=lioadm for el7. tgtadm is not supported on el7. Can you post your cinder-dist.conf and cinder.conf? (Also this should probably be posted under RDO instead of RHOS.)
Created attachment 1022938 [details] cinder-dist.conf
Created attachment 1022947 [details] cinder.conf
This is happening because this section in cinder.conf doesn't have iscsi_helper set to "lioadm": [rhos-6-lvm] volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver lvm_type = default This is not a valid configuration on el7 since it defaults to iscsi_helper = tgtadm. From the cinder volume log: 2015-05-05 03:43:38.236 5856 DEBUG cinder.openstack.common.service [req-ff8ab064-27b7-4099-af3a-32f853f00940 - - - - -] rhos-6-lvm.iscsi_helper = tgtadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2199
Ok , thanks . fixed my automation , is it a valid case ? if not we can close it. I dont get into this issue with rhos6 . Benny
No, tgtadm is not supported on el7.