RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 1218598 - RDO - LVM backend - Executable not found: tgtadm (filter match = tgtadm)\n'
Summary: RDO - LVM backend - Executable not found: tgtadm (filter match = tgtadm)\n'
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: RDO
Classification: Community
Component: openstack-cinder
Version: Kilo
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Eric Harney
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-05 11:26 UTC by bkopilov
Modified: 2016-04-26 14:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-05-07 14:55:23 UTC
Embargoed:


Attachments (Terms of Use)
openstack logs - /var/log/cinder/* (6.50 MB, application/x-gzip)
2015-05-05 11:26 UTC, bkopilov
no flags Details
cinder-dist.conf (474 bytes, text/plain)
2015-05-07 06:26 UTC, bkopilov
no flags Details
cinder.conf (79.52 KB, text/plain)
2015-05-07 06:27 UTC, bkopilov
no flags Details

Description bkopilov 2015-05-05 11:26:35 UTC
Created attachment 1022154 [details]
openstack logs -  /var/log/cinder/*

Description of problem:
Installed openstack on a VM all in one , cinder backend is local LVM .
repo was installed from : 
https://repos.fedorapeople.org/repos/openstack/openstack-kilo/testing/el7/


Started to run tempest code , there are few failures .
From /var/log/ :
u'compute', u'name': u'nova'}], u'request_id': u'req-b162155b-cc4a-4d6c-9c81-2843ff17adfb', u'user_domain': None, u'read_deleted': u'no', u'tenant': u'2e69e7ebba7f460e91369b9cfb132769'} unpack_context /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:203
2015-05-05 05:23:40.298 6084 DEBUG cinder.volume.manager [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Volume 86651feb-a9ed-4ae0-ba55-e971d6b15ef2: creating export initialize_connection /usr/lib/python2.7/site-packages/cinder/volume/manager.py:1082
2015-05-05 05:23:40.299 6084 DEBUG cinder.volume.targets.tgt [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Failed to open config for Volume volume-86651feb-a9ed-4ae0-ba55-e971d6b15ef2: [Errno 2] No such file or directory: u'/var/lib/cinder/volumes/volume-86651feb-a9ed-4ae0-ba55-e971d6b15ef2' _get_target_chap_auth /usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py:134
2015-05-05 05:23:40.320 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
2015-05-05 05:23:40.470 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target" returned: 96 in 0.150s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
2015-05-05 05:23:40.472 6084 DEBUG oslo_concurrency.processutils [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] u'sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
2015-05-05 05:23:40.474 6084 ERROR oslo_messaging.rpc.dispatcher [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Exception during message handling: Unexpected error while running command.
Command: None
Exit code: -
Stdout: u"Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target\nExit code: 96\nStdout: u''\nStderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\\n'"
Stderr: None
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1084, in initialize_connection
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     volume)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 606, in create_export
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     volume_path)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 208, in create_export
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     chap_auth)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 174, in create_iscsi_target
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     run_as_root=True)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 143, in execute
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 233, in execute
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher     cmd=sanitized_cmd)
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Exit code: 96
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher Stderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\n'
2015-05-05 05:23:40.474 6084 TRACE oslo_messaging.rpc.dispatcher 
2015-05-05 05:23:40.572 6084 ERROR oslo_messaging._drivers.common [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] Returning exception Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target
Exit code: 96
Stdout: u''
Stderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\n' to caller
2015-05-05 05:23:40.573 6084 ERROR oslo_messaging._drivers.common [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1084, in initialize_connection\n    volume)\n', '  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 606, in create_export\n    volume_path)\n', '  File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 208, in create_export\n    chap_auth)\n', '  File "/usr/lib/python2.7/site-packages/cinder/volume/targets/tgt.py", line 174, in create_iscsi_target\n    run_as_root=True)\n', '  File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 143, in execute\n    return processutils.execute(*cmd, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 233, in execute\n    cmd=sanitized_cmd)\n', "ProcessExecutionError: Unexpected error while running command.\nCommand: sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgtadm --lld iscsi --op show --mode target\nExit code: 96\nStdout: u''\nStderr: u'/usr/bin/cinder-rootwrap: Executable not found: tgtadm (filter match = tgtadm)\\n'\n"]
2015-05-05 05:23:40.574 6084 DEBUG oslo_messaging._drivers.amqp [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] UNIQUE_ID is 3ca248fc9b144c7a81a41ef74c89fa40. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2015-05-05 05:23:40.579 6084 DEBUG oslo_messaging._drivers.amqp [req-b162155b-cc4a-4d6c-9c81-2843ff17adfb 8b4142494d7643edaebd283452540d15 2e69e7ebba7f460e91369b9cfb132769 - - -] UNIQUE_ID is cb44031fdbab4f72b42f1847b01efdb5. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258


Version-Release number of selected component (if applicable):


How reproducible:
Run tempest api tests ,  errors are in :
 tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments[id-7fa563fe-f0f7-43eb-9e22-a1ece036b513]
 tempest.api.volume.test_volumes_snapshots.VolumesV1SnapshotTestJSON.test_snapshot_create_with_volume_in_use[compute,id-b467b54c-07a4-446d-a1cf-651dedcc3ff1]
 tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use[compute,id-b467b54c-07a4-446d-a1cf-651dedcc3ff1]
 tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
 tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
 tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup[compute,id-cbc752ed-b716-4717-910f-956cce965722,image,volume]
 tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks[compute,id-79165fb4-5534-4b9d-8429-97ccffb8f86e,image,volume]
 tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern[compute,id-608e604b-1d63-4a82-8e3e-91bc665c90b4,image,network]


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Eric Harney 2015-05-05 13:30:34 UTC
/usr/share/cinder/cinder-dist.conf should be setting iscsi_helper=lioadm for el7.  tgtadm is not supported on el7.

Can you post your cinder-dist.conf and cinder.conf?

(Also this should probably be posted under RDO instead of RHOS.)

Comment 5 bkopilov 2015-05-07 06:26:31 UTC
Created attachment 1022938 [details]
cinder-dist.conf

Comment 6 bkopilov 2015-05-07 06:27:41 UTC
Created attachment 1022947 [details]
cinder.conf

Comment 7 Eric Harney 2015-05-07 14:21:12 UTC
This is happening because this section in cinder.conf doesn't have iscsi_helper set to "lioadm":

[rhos-6-lvm]
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
lvm_type = default

This is not a valid configuration on el7 since it defaults to iscsi_helper = tgtadm.


From the cinder volume log:

2015-05-05 03:43:38.236 5856 DEBUG cinder.openstack.common.service [req-ff8ab064-27b7-4099-af3a-32f853f00940 - - - - -] rhos-6-lvm.iscsi_helper        = tgtadm log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2199

Comment 8 bkopilov 2015-05-07 14:54:10 UTC
Ok , thanks .
fixed my automation , is it a valid case ? if not we can close it.
I dont get into this issue with rhos6 .
Benny

Comment 9 Eric Harney 2015-05-07 14:55:23 UTC
No, tgtadm is not supported on el7.


Note You need to log in before you can comment on or make changes to this bug.