Description of problem: RHOS14 all-in-one(standalone) installation http://etherpad.corp.redhat.com/cUfOMARL2r After launching of instance it in ERROR state: | fault | {u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance ccb61483-91ce-4a14-8eb4-6c29057ecacc.', u'code': 500, u'details': u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 578, in build_instances\n raise exception.MaxRetriesExceeded(reason=msg)\n', u'created': u'2018-07-05T12:03:20Z'} | | flavor | m1.nano (0) /var/log/containers/nova/nova-compute.log:2018-07-05 12:03:18.966 1 DEBUG nova.compute.utils [req-68481d44-9d65-470d-b5f2-59bb905c8ab4 b9918b5bf57845ddab131766c44236dd 90419cda5d8c47bea32dfe9c157659c2 - default default] [instance: ccb61483-91ce-4a14-8eb4-6c29057ecacc] internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:358 /var/log/containers/nova/nova-conductor.log:2018-07-05 12:03:20.080 24 ERROR nova.scheduler.utils [req-2eee609b-3ce2-4c37-bbac-320dc9badbed 8b4d0b6b0e0543d280b96e34649500b8 7c06624ac5e84f129b8f3a52e700d43a - default default] [instance: ccb61483-91ce-4a14-8eb4-6c29057ecacc] Error from last host: standalone-0.localdomain (node standalone-0.localdomain): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1819, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance ccb61483-91ce-4a14-8eb4-6c29057ecacc was re-scheduled: internal error: process exited while connecting to monitor: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied\n'] Version-Release number of selected component (if applicable): [stack@standalone-0 ~]$ sudo rpm -qa "*openstack*" openstack-heat-engine-12.0.0-0.20180604085325.7d878a8.el7ost.noarch openstack-tripleo-common-9.1.1-0.20180623003933.5191b65.el7ost.noarch openstack-heat-agents-1.6.1-0.20180605100743.235e1ae.el7ost.noarch openstack-heat-api-12.0.0-0.20180604085325.7d878a8.el7ost.noarch openstack-tripleo-common-containers-9.1.1-0.20180623003933.5191b65.el7ost.noarch openstack-tripleo-puppet-elements-9.0.0-0.20180602004307.939b586.el7ost.noarch openstack-heat-common-12.0.0-0.20180604085325.7d878a8.el7ost.noarch python-openstackclient-lang-3.15.0-0.20180402052711.67edb39.el7ost.noarch openstack-selinux-0.8.15-0.20180524134826.b63283a.el7ost.noarch python2-openstacksdk-0.14.0-0.20180619125431.07d3828.el7ost.noarch openstack-heat-monolith-12.0.0-0.20180604085325.7d878a8.el7ost.noarch python2-openstackclient-3.15.0-0.20180402052711.67edb39.el7ost.noarch openstack-tripleo-heat-templates-9.0.0-0.20180625181147.ed26bd7.el7ost.noarch puppet-openstack_extras-13.1.1-0.20180612201733.a13141f.el7ost.noarch puppet-openstacklib-13.1.1-0.20180623141417.b48ac57.el7ost.noarch openstack-tripleo-image-elements-9.0.0-0.20180601015717.2ac38dd.el7ost.noarch [stack@standalone-0 ~]$ sudo cat /home/cloud-user/core_puddle_version 2018-07-03.1 2018-07-03.1[stack@standsudo rpm -qa "*selinux*" libselinux-2.5-12.el7.x86_64 libselinux-utils-2.5-12.el7.x86_64 libselinux-ruby-2.5-12.el7.x86_64 selinux-policy-targeted-3.13.1-192.el7_5.4.noarch container-selinux-2.66-1.el7.noarch openstack-selinux-0.8.15-0.20180524134826.b63283a.el7ost.noarch libselinux-python-2.5-12.el7.x86_64 selinux-policy-3.13.1-192.el7_5.4.noarch [stack@standalone-0 ~]$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-server 14.0-21 3797947368d3 29 hours ago 819 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-api 14.0-22 29ab96f36cdc 29 hours ago 981 MB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-central 14.0-22 9af1fbcde0d5 29 hours ago 699 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-l3-agent 14.0-22 5cc900971dd0 29 hours ago 888 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-api 14.0-22 fd20812be5c9 29 hours ago 893 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-placement-api 14.0-22 7b6d2087c08f 29 hours ago 893 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-compute 14.0-22 99766fe95f62 29 hours ago 1.34 GB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-openvswitch-agent 14.0-22 1a621da14966 29 hours ago 781 MB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-object 14.0-22 078d46b2beb0 29 hours ago 643 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-consoleauth 14.0-22 4d89a7f6a225 29 hours ago 855 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-metadata-agent 14.0-22 1b001e3d92c6 29 hours ago 752 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-scheduler 14.0-22 07db8600a589 29 hours ago 892 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-volume 14.0-22 9048a65593ad 29 hours ago 1.01 GB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-volume pcmklatest 9048a65593ad 29 hours ago 1.01 GB docker-registry.engineering.redhat.com/rhosp14/openstack-keystone 14.0-22 a8d2e41bad76 29 hours ago 691 MB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-proxy-server 14.0-22 986b582f1d99 29 hours ago 694 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-novncproxy 14.0-22 a01336f1f565 29 hours ago 856 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-evaluator 14.0-22 e860a54ebc23 29 hours ago 678 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-dhcp-agent 14.0-22 b8a3134fbc20 29 hours ago 888 MB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-notification 14.0-22 6ff9c003f541 29 hours ago 651 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-listener 14.0-22 36f1fe43d157 29 hours ago 678 MB docker-registry.engineering.redhat.com/rhosp14/openstack-glance-api 14.0-22 41f6093c3c03 29 hours ago 873 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-api 14.0-22 d4a64c3c868d 29 hours ago 678 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-engine 14.0-22 d3f64f017e48 29 hours ago 684 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-scheduler 14.0-22 f7d35ae8e5c3 29 hours ago 895 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-metricd 14.0-21 dff6497cb654 29 hours ago 1.02 GB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-account 14.0-22 7b42858554b7 29 hours ago 643 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-statsd 14.0-22 be7a46238f7a 29 hours ago 1.02 GB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-container 14.0-22 b6f85a6f3311 29 hours ago 643 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-api 14.0-22 0715d514b49a 29 hours ago 684 MB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-compute 14.0-22 83d2c3a0bc1d 29 hours ago 699 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-api-cfn 14.0-22 47c70328ec04 29 hours ago 684 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-api 14.0-21 1e23b1b2a8fc 29 hours ago 1.02 GB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-conductor 14.0-22 5e00e754f7ef 29 hours ago 855 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-notifier 14.0-22 ca2da5c76255 29 hours ago 678 MB docker-registry.engineering.redhat.com/rhosp14/openstack-panko-api 14.0-22 0f630e03c550 29 hours ago 691 MB docker-registry.engineering.redhat.com/rhosp14/openstack-redis 14.0-20 56a09aa8a43f 30 hours ago 622 MB docker-registry.engineering.redhat.com/rhosp14/openstack-redis pcmklatest 56a09aa8a43f 30 hours ago 622 MB docker-registry.engineering.redhat.com/rhosp14/openstack-horizon 14.0-24 f1cb0ae08970 30 hours ago 795 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-libvirt 14.0-31 e41735f505c7 30 hours ago 1.26 GB docker-registry.engineering.redhat.com/rhosp14/openstack-cron 14.0-30 90b3a0186267 30 hours ago 492 MB docker-registry.engineering.redhat.com/rhosp14/openstack-rabbitmq 14.0-30 c211ee672a10 30 hours ago 657 MB docker-registry.engineering.redhat.com/rhosp14/openstack-rabbitmq pcmklatest c211ee672a10 30 hours ago 657 MB docker-registry.engineering.redhat.com/rhosp14/openstack-iscsid 14.0-29 dff6ae55dd6c 30 hours ago 497 MB docker-registry.engineering.redhat.com/rhosp14/openstack-memcached 14.0-30 d8f389e82570 30 hours ago 530 MB docker-registry.engineering.redhat.com/rhosp14/openstack-mariadb 14.0-30 3206de0e48aa 30 hours ago 778 MB docker-registry.engineering.redhat.com/rhosp14/openstack-mariadb pcmklatest 3206de0e48aa 30 hours ago 778 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-server 14.0-20 f1e0d0cd60a0 2 days ago 815 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-api 14.0-21 c4ffec157472 2 days ago 977 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-l3-agent 14.0-21 807a6e22c699 2 days ago 884 MB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-central 14.0-21 a73b2dc8808c 2 days ago 733 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-placement-api 14.0-21 d78f4ce710e7 2 days ago 886 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-compute 14.0-21 02ff47867f0b 2 days ago 1.33 GB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-openvswitch-agent 14.0-21 e01f10735fe1 2 days ago 777 MB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-object 14.0-21 bdd96a5d82ff 2 days ago 676 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-consoleauth 14.0-21 d19fb84f4694 2 days ago 849 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-metadata-agent 14.0-21 e4d195c2f159 2 days ago 747 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-scheduler 14.0-21 60ad00ec9f1c 2 days ago 869 MB docker-registry.engineering.redhat.com/rhosp14/openstack-keystone 14.0-21 e70a78452fc1 2 days ago 723 MB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-proxy-server 14.0-21 714337ee5b77 2 days ago 728 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-api 14.0-21 82bdf0de8233 2 days ago 886 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-evaluator 14.0-21 dbf443926720 2 days ago 712 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-volume 14.0-21 cd9d1d2a01f0 2 days ago 1 GB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-novncproxy 14.0-21 178fd10775f0 2 days ago 849 MB docker-registry.engineering.redhat.com/rhosp14/openstack-neutron-dhcp-agent 14.0-21 4571cecb400e 2 days ago 884 MB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-notification 14.0-21 ff9e3d75ef14 2 days ago 685 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-listener 14.0-21 557c8140e009 2 days ago 712 MB docker-registry.engineering.redhat.com/rhosp14/openstack-glance-api 14.0-21 ae184809330d 2 days ago 868 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-api 14.0-21 233678074bc9 2 days ago 712 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-engine 14.0-21 23fb6085641a 2 days ago 717 MB docker-registry.engineering.redhat.com/rhosp14/openstack-cinder-scheduler 14.0-21 64d81c6d3ad4 2 days ago 890 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-metricd 14.0-20 0f4d392dab91 2 days ago 1 GB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-account 14.0-21 fe37f14ed82c 2 days ago 676 MB docker-registry.engineering.redhat.com/rhosp14/openstack-swift-container 14.0-21 00340f0ced61 2 days ago 676 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-statsd 14.0-21 868ef65ccf6d 2 days ago 1 GB docker-registry.engineering.redhat.com/rhosp14/openstack-ceilometer-compute 14.0-21 ceefe005560d 2 days ago 733 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-api 14.0-21 196d386bb694 2 days ago 717 MB docker-registry.engineering.redhat.com/rhosp14/openstack-heat-api-cfn 14.0-21 c41ad3d21371 2 days ago 717 MB docker-registry.engineering.redhat.com/rhosp14/openstack-gnocchi-api 14.0-20 44dbd76b7b56 2 days ago 1 GB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-conductor 14.0-21 f8654a45a016 2 days ago 848 MB docker-registry.engineering.redhat.com/rhosp14/openstack-aodh-notifier 14.0-21 c18ff26737d8 2 days ago 712 MB docker-registry.engineering.redhat.com/rhosp14/openstack-panko-api 14.0-21 f191d9fb59b6 2 days ago 725 MB docker-registry.engineering.redhat.com/rhosp14/openstack-horizon 14.0-23 6a6d0150192d 2 days ago 827 MB docker-registry.engineering.redhat.com/rhosp14/openstack-redis 14.0-19 6c8a2fbddaf5 2 days ago 597 MB docker-registry.engineering.redhat.com/rhosp14/openstack-nova-libvirt 14.0-30 9daabd913e2c 2 days ago 1.23 GB docker-registry.engineering.redhat.com/rhosp14/openstack-cron 14.0-29 195f7d4facbc 2 days ago 467 MB docker-registry.engineering.redhat.com/rhosp14/openstack-iscsid 14.0-28 32706d51ce58 2 days ago 471 MB docker-registry.engineering.redhat.com/rhosp14/openstack-rabbitmq 14.0-29 ab5c93d22065 2 days ago 632 MB docker-registry.engineering.redhat.com/rhosp14/openstack-memcached 14.0-29 bdb44d8b1eda 2 days ago 505 MB docker-registry.engineering.redhat.com/rhosp14/openstack-mariadb 14.0-29 650a49bfb6da 2 days ago 753 MB How reproducible: always Steps to Reproduce: 1. http://etherpad.corp.redhat.com/cUfOMARL2r Actual results: instance state = Error Expected results: instance state == Active Additional info:
Created attachment 1456761 [details] /var/log/containers
Please could you provide the audit logs containing the AVC denial. Also, is this reproducible without patches?
The patches are necessary to run this setup (cloud in a single node). This is the expected configuration for OSP14 for this feature. Will try and grab the avc logs
type=AVC msg=audit(1531151277.699:5956): avc: denied { entrypoint } for pid=166360 comm="libvirtd" path="/usr/libexec/qemu-kvm" dev="vda1" ino=80176023 scontext=system_u:system_r:svirt_tcg_t:s0:c115,c193 tcontext=system_u:object_r:container_share_t:s0 tclass=file
Created attachment 1457519 [details] audit.log
(In reply to Alex Schultz from comment #5) > type=AVC msg=audit(1531151277.699:5956): avc: denied { entrypoint } for > pid=166360 comm="libvirtd" path="/usr/libexec/qemu-kvm" dev="vda1" > ino=80176023 scontext=system_u:system_r:svirt_tcg_t:s0:c115,c193 > tcontext=system_u:object_r:container_share_t:s0 tclass=file Unless it's being mislabeled, looks like you might be running TCG. Is this nested virt by any chance? You might want to check that it's correctly enabled. https://docs.fedoraproject.org/quick-docs/en-US/using-nested-virtualization-in-kvm.html This looks like an private matter between libvirt, qemu, and their SELinux policy. I suspect you've stumbled down an unsupported (and little tested) code path, and fixing nested virt will make it go away. Looking at openstack-selinux, I don't think this can be an issue in our policy customisations because we don't define either the type or the transition. I'm going to punt this to libvirt to either fix or signpost appropriately.
The system_u:object_r:container_share_t context of /usr/libexec/qemu-kvm is incorrect, normally it is system_u:object_r:qemu_exec_t:s0. Moving back to OpenStack (openstack-containers looks like it could be the right component) since this has nothing to do with libvirt.
(In reply to Matthew Booth from comment #7) > (In reply to Alex Schultz from comment #5) > > type=AVC msg=audit(1531151277.699:5956): avc: denied { entrypoint } for > > pid=166360 comm="libvirtd" path="/usr/libexec/qemu-kvm" dev="vda1" > > ino=80176023 scontext=system_u:system_r:svirt_tcg_t:s0:c115,c193 > > tcontext=system_u:object_r:container_share_t:s0 tclass=file > > Unless it's being mislabeled, looks like you might be running TCG. Is this > nested virt by any chance? You might want to check that it's correctly > enabled. Yes, they are using TCG, from that SELinux context. And this bug is eerily similar to this other one (which is now moved to 'selinux-policy' component: https://bugzilla.redhat.com/show_bug.cgi?id=1538651 ("When using --libvirttype qemu to deploy an overcloud, VMs can't be launched because SELinux blocks it") And the fix (not yet merged in 'selinux-policy' package) is outlined here: https://bugzilla.redhat.com/show_bug.cgi?id=1538651#c12
Agree with Jiri Denemark, the correct label for /usr/libexec/qemu-kvm is: # matchpathcon /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm system_u:object_r:qemu_exec_t:s0 # And SELinux policy defines following entrypoint rule: # sesearch -s svirt_tcg_t -p entrypoint -A Found 1 semantic av rules: allow svirt_tcg_t qemu_exec_t : file { ioctl read getattr lock map execute entrypoint open } ; # Please run "restorecon -v /usr/libexec/qemu-kvm" to correct the label.
Kashyap seems to have found the older issue. Looking in openstack-selinux we do indeed already have: allow svirt_t container_share_t:file { entrypoint execute }; We probably want to replicate a bunch of svirt_t rules for svirt_tcg_t. Although as mentioned earlier I don't believe we actually support TCG, but we prefer that it works. *** This bug has been marked as a duplicate of bug 1538651 ***