Description of problem: Deployment with enabled NFS share for nova ephemeral storage fails. Podman fails to relable with mounted nfs in /var/lib/nova/instances: Jul 4 17:08:24 compute-0 podman[28047]: unable to start container "nova_virtlogd": relabel failed "/var/lib/nova": operation not supported Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Control process exited, code=exited status=125 Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Failed with result 'exit-code'. Jul 4 17:08:24 compute-0 systemd[1]: Failed to start nova_virtlogd container. Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Service RestartSec=100ms expired, scheduling restart. Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Scheduled restart job, restart counter is at 5. Jul 4 17:08:24 compute-0 systemd[1]: Stopped nova_virtlogd container. Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Start request repeated too quickly. Jul 4 17:08:24 compute-0 systemd[1]: tripleo_nova_virtlogd.service: Failed with result 'exit-code'. Jul 4 17:08:24 compute-0 systemd[1]: Failed to start nova_virtlogd container. Version-Release number of selected component (if applicable): RHOS_TRUNK-15.0-RHEL-8-20190701.n.0 How reproducible: always Steps to Reproduce: 1. deploy with parameter_defaults: NovaNfsEnabled: True NovaNfsShare: '192.168.24.1:/var/nfs' Actual results: "Completed $ podman create --name nova_virtlogd --label config_id=tripleo_step3 --label container_name=nova_virtlogd --label managed_by=paunch --label config_data={\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=d4af547ef9c398ce67c3abb8c19a394 7\"], \"image\": \"192.168.24.1:8787/rhosp15/openstack-nova-libvirt:20190625.1\", \"net\": \"host\", \"pid\": \"host\", \"privileged\": true, \"restart\": \"always\", \"security_opt\": \"label=disable\", \"start_order\": 0, \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localt$ me:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc$ pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/nova_virtlogd.json:/var/lib/kolla/config_files$ config.json:ro\", \"/var/lib/config-data/puppet-generated/nova_libvirt/:/var/lib/kolla/config_files/src:ro\", \"/lib/modules:/lib/modules:ro\", \"/dev:/dev\", \"/run:/run\", \"/sys/fs/cgroup:/sys/fs/cgroup\", \"/var/lib/nova:/var/lib/nova:shared,z\", \"/var/run/libvirt:/var/run/li$ virt:shared,z\", \"/var/lib/libvirt:/var/lib/libvirt\", \"/etc/libvirt/qemu:/etc/libvirt/qemu:ro\", \"/var/log/libvirt/qemu:/var/log/libvirt/qemu\"]} --conmon-pidfile=/var/run/nova_virtlogd.pid --detach=true --log-driver json-file --log-opt path=/var/log/containers/stdouts/nova_vi$ tlogd.log --env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env=TRIPLEO_CONFIG_HASH=d4af547ef9c398ce67c3abb8c19a3947 --net=host --pid=host --privileged=true --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/e$ tracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/ce$ t.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/kolla/config_files/nova_virtlogd.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-d$ ta/puppet-generated/nova_libvirt/:/var/lib/kolla/config_files/src:ro --volume=/lib/modules:/lib/modules:ro --volume=/dev:/dev --volume=/run:/run --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/var/lib/nova:/var/lib/nova:shared,z --volume=/var/run/libvirt:/var/run/libvirt:shared,z --volume=/var/lib/libvirt:/var/lib/libvirt --volume=/etc/libvirt/qemu:/etc/libvirt/qemu:ro --volume=/var/log/libvirt/qemu:/var/log/libvirt/qemu --security-opt=label=disable 192.168.24.1:8787/rhosp15/openstack-nova-libvirt:20190625.1", "stdout: 7686ab7e70c57886340d4273e4ee949339bcdaf33c211890fd2259bf7bf449fd", "stderr: ", "Creating systemd unit file: /etc/systemd/system/tripleo_nova_virtlogd.service", "Executing: systemctl daemon-reload", "Executing: systemctl enable --now tripleo_nova_virtlogd", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_nova_virtlogd.service → /etc/systemd/system/tripleo_nova_virtlogd.service.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "systemctl failed", "Traceback (most recent call last):", " File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 31, in systemctl", " subprocess.check_call(cmd)", " File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call", " raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "During handling of the above exception, another exception occurred:", " File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 98, in service_create", " systemctl.enable(service, now=True)", " File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 53, in enable", " systemctl(cmd, log)", " File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 33, in systemctl", " raise SystemctlException(str(err))", "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", " File \"/usr/lib/python3.6/site-packages/cliff/app.py\", line 401, in run_subcommand", " result = cmd.run(parsed_args)", " File \"/usr/lib/python3.6/site-packages/cliff/command.py\", line 184, in run", " return_code = self.take_action(parsed_args) or 0", " File \"/usr/lib/python3.6/site-packages/paunch/cmd.py\", line 104, in take_action", " healthcheck_disabled=parsed_args.healthcheck_disabled", " File \"/usr/lib/python3.6/site-packages/paunch/__init__.py\", line 80, in apply", " return builder.apply()", " File \"/usr/lib/python3.6/site-packages/paunch/builder/base.py\", line 111, in apply", " log=self.log)", " File \"/bin/paunch\", line 10, in <module>", " sys.exit(main())", " File \"/usr/lib/python3.6/site-packages/paunch/__main__.py\", line 39, in main", " return myapp.run(argv)", " File \"/usr/lib/python3.6/site-packages/cliff/app.py\", line 281, in run", " result = self.run_subcommand(remainder)" ] } NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* compute-0 : ok=182 changed=85 unreachable=0 failed=1 skipped=559 rescued=0 ignored=1 compute-1 : ok=182 changed=85 unreachable=0 failed=1 skipped=559 rescued=0 ignored=1 controller-0 : ok=241 changed=144 unreachable=0 failed=0 skipped=501 rescued=0 ignored=1 undercloud : ok=11 changed=7 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 Expected results: overcloud deploy successful Additional info: z flag for /var/lib/nova bind mount: ~~~ docker_config: step_3: nova_virtlogd: start_order: 0 image: {get_param: ContainerNovaLibvirtImage} ulimit: {get_param: ContainerNovaLibvirtUlimit} net: host pid: host security_opt: label=disable privileged: true restart: always volumes: list_concat: - {get_attr: [ContainersCommon, volumes]} - - /var/lib/kolla/config_files/nova_virtlogd.json:/var/lib/kolla/config_files/config.json:ro - /var/lib/config-data/puppet-generated/nova_libvirt/:/var/lib/kolla/config_files/src:ro - /lib/modules:/lib/modules:ro - /dev:/dev - /run:/run - /sys/fs/cgroup:/sys/fs/cgroup - /var/lib/nova:/var/lib/nova:shared,z <-------------- - /var/run/libvirt:/var/run/libvirt:shared,z - /var/lib/libvirt:/var/lib/libvirt - /etc/libvirt/qemu:/etc/libvirt/qemu:ro - /var/log/libvirt/qemu:/var/log/libvirt/qemu environment: - KOLLA_CONFIG_STRATEGY=COPY_ALWAYS ~~~ [1] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-libvirt-container-puppet.yaml#L623
Created attachment 1587638 [details] compute sosreport
This is a regression. Probably needs blocker flag.
Note: The fix in [1] will not set the z flag for the /var/lib/nova bind mount in case a nfs share is used for /var/lib/nova/instances. There is no workaround for the scheduled beta release for this configuration where nfs is used for nova ephemeral storage. [1] https://review.opendev.org/#/c/669317/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811