Bug 1719667 - [OSP14] minor update fails because : /var/log/containers/nova/nova-manage.log is owned by root:root on compute nodes
Summary: [OSP14] minor update fails because : /var/log/containers/nova/nova-manage.log...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z3
: 14.0 (Rocky)
Assignee: Martin Schuppert
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks: 1719673 1719691
TreeView+ depends on / blocked
 
Reported: 2019-06-12 10:23 UTC by Martin Schuppert
Modified: 2019-07-02 20:09 UTC (History)
4 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171733.9995be9.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1719673 1719691 (view as bug list)
Environment:
Last Closed: 2019-07-02 20:09:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1832482 0 None None None 2019-06-12 10:23:25 UTC
OpenStack gerrit 664854 0 'None' MERGED Fix log owner on computes during overcloud deploy runs 2020-05-08 05:21:12 UTC
Red Hat Product Errata RHBA-2019:1672 0 None None None 2019-07-02 20:09:52 UTC

Description Martin Schuppert 2019-06-12 10:23:26 UTC
Description of problem:

[1] switched to run nova-manage discovery as non root user.

                command: "/usr/bin/bootstrap_host_exec nova_compute su nova -s /bin/bash -c '/container-config-scripts/pyshim.sh /container-config-scripts/nova_cell_v2_discover_hosts.py'"

In case of updates where we already can have a nova-manage log owned by root make the overcloud deploy to fail with:

2019-06-12 05:30:08,789 p=13529 u=mistral | fatal: [compute-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "Error running ['docker', 'run', '--name', 'nova_cell_v2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_cell_v2_discover_hosts', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 0, \"image\": \"192.168.24.1:87
87/rhosp14/openstack-nova-compute:2019-04-23.1\", \"environment\": [\"TRIPLEO_DEPLOY_IDENTIFIER=1560329712\"], \"command\": \"su nova -s /bin/bash -c \\'/docker-config-scripts/nova_cell_v2_discover_hosts.py\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/et
c/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.
crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro\
", \"/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/docker-config-scripts/:/docker-config-scripts/\"], \"net\": \"host\", \"detach\": false}', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1560329712', '--net=host', '--user=ro
ot', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:
/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro
', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova_libvirt/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/docker-config-scripts/
:/docker-config-scripts/', '192.168.24.1:8787/rhosp14/openstack-nova-compute:2019-04-23.1', 'su', 'nova', '-s', '/bin/bash', '-c', \"'/docker-config-scripts/nova_cell_v2_discover_hosts.py'\"]. [1]",
        "",
        "stdout: ERROR:nova_cell_v2_discover_hosts:Cell v2 discovery failed with exit code 1, retrying",
        "ERROR:nova_cell_v2_discover_hosts:Cell v2 discovery failed with exit code 1, retrying",
        "stderr: Traceback (most recent call last):",
        " File \"/usr/bin/nova-manage\", line 10, in <module>",
        " sys.exit(main())",
        " File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 2325, in main",
        " logging.setup(CONF, \"nova\")",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 264, in setup",
        " _setup_logging_from_conf(conf, product_name, version)",
        " File \"/usr/lib/python2.7/site-packages/oslo_log/log.py\", line 353, in _setup_logging_from_conf",
        " filelog = file_handler(logpath)",
        " File \"/usr/lib64/python2.7/logging/handlers.py\", line 392, in __init__",
        " logging.FileHandler.__init__(self, filename, mode, encoding, delay)",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 902, in __init__",
        " StreamHandler.__init__(self, self._open())",
        " File \"/usr/lib64/python2.7/logging/__init__.py\", line 925, in _open",
        " stream = open(self.baseFilename, self.mode)",
        "IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'",
        "Traceback (most recent call last):"
    ]
}

We need to fix log owner ship on every deploy run on the computes, like we have on the
controller.

[1] https://review.opendev.org/#/c/652039/13/deployment/nova/nova-compute-container-puppet.yaml

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. deploy environment with openstack-tripleo-heat-templates-9.3.1-0.20190314162760.d0a6cb1.el7ost
2. update to openstack-tripleo-heat-templates-9.3.1-0.20190314162762.d0a6cb1
3. run overcloud deploy again

Actual results:
deploy fails on compute with  IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'

Expected results:
deploy is successful

Additional info:

Comment 1 Martin Schuppert 2019-06-12 10:27:57 UTC
This could be worked around deleting /var/log/containers/nova/nova-manage.log or chown it to nova:nova.

Comment 19 errata-xmlrpc 2019-07-02 20:09:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1672


Note You need to log in before you can comment on or make changes to this bug.