Description of problem: While deploying an undercloud, we hit the following issue: "Error running ['podman', 'run', '--name', 'glance_api_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=glance_api_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec glance_api su glance -s /bin/bash -c \\'/usr/local/bin/kolla_start\\'\", \"detach\": f alse, \"environment\": [\"KOLLA_BOOTSTRAP=True\", \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=3b62fadd631676c0920bffd3fda037b6\"], \"image\": \"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest\", \"net\": \"host\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc /hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\ ", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/glance:/var/log/glance:z\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated /glance_api/:/var/lib/kolla/config_files/src:ro\", \"/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\", \"/var/lib/glance:/var/lib/glance:slave,z\"]}', '--conmon-pidfile=/var/run/glance_api_db_sync.pid', '--log-driver', 'json-file', '--log-opt', 'path=/var/log/containers/stdouts/glance_api_db_sync.log', '--env=KOLLA_BOOTSTRAP=True', '--env=KO LLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=3b62fadd631676c0920bffd3fda037b6', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/et c/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_ known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/glance:/var/log/glance:z', '--volume=/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json', '--volume=/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro', '--volume=/etc/ceph:/var/lib/kolla/confi g_files/src-ceph:ro', '--volume=/var/lib/glance:/var/lib/glance:slave,z', 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest', '/usr/bin/bootstrap_host_exec', 'glance_api', 'su', 'glance', '-s', '/bin/bash', '-c', \"'/usr/local/bin/kolla_start'\"]. [1]", "stdout: ", "stderr: + sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Deleting /etc/glance/glance-api.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-api.conf", "INFO:__main__:Deleting /etc/glance/glance-cache.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glance-cache.conf", "INFO:__main__:Deleting /etc/glance/glance-image-import.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-image-import.conf to /etc/glance/glance-image-import.conf", "INFO:__main__:Deleting /etc/glance/glance-swift.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-swift.conf to /etc/glance/glance-swift.conf", "INFO:__main__:Creating directory /etc/my.cnf.d", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf", "INFO:__main__:Writing out command to execute", "INFO:__main__:Setting permission for /var/lib/glance", "++ cat /run_command", "+ CMD='/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'", "+ ARGS=", "+ [[ ! -n '' ]]", "+ . kolla_extend_start", "++ [[ ! -d /var/log/kolla/glance ]]", "++ mkdir -p /var/log/kolla/glance", "+++ stat -c %a /var/log/kolla/glance", "++ [[ 2700 != \\7\\5\\5 ]]", "++ chmod 755 /var/log/kolla/glance", "++ . /usr/local/bin/kolla_glance_extend_start", "+++ [[ -n 0 ]]", "+++ glance-manage db_sync", "Traceback (most recent call last):", " File \"/usr/bin/glance-manage\", line 6, in <module>", " from glance.cmd.manage import main", " File \"/usr/lib/python3.6/site-packages/glance/cmd/manage.py\", line 53, in <module>", " from glance.db import migration as db_migration", " File \"/usr/lib/python3.6/site-packages/glance/db/__init__.py\", line 22, in <module>", " from wsme.rest import json", "ImportError: cannot import name 'json'", "stderr: ", "stdout: 8b59e00b993202bc2b183bd41b0417d7dee219f20a3f28dc61b29a69f9edfe7d", "INFO:__main__:Creating directory /etc/keystone/fernet-keys", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/keystone/fernet-keys/0 to /etc/keystone/fernet-keys/0", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/keystone/fernet-keys/1 to /etc/keystone/fernet-keys/1", "INFO:__main__:Deleting /etc/httpd/conf.d", "INFO:__main__:Creating directory /etc/httpd/conf.d", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/10-keystone_wsgi_admin.conf to /etc/httpd/conf.d/10-keystone_wsgi_admin.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/10-keystone_wsgi_main.conf to /etc/httpd/conf.d/10-keystone_wsgi_main.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/ssl.conf to /etc/httpd/conf.d/ssl.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/10-keystone_wsgi.conf to /etc/httpd/conf.d/10-keystone_wsgi.conf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/access_compat.load to /etc/httpd/conf.d/access_compat.load", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/actions.load to /etc/httpd/conf.d/actions.load", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/alias.conf to /etc/httpd/conf.d/alias.conf", The important part is: "Traceback (most recent call last):", " File \"/usr/bin/glance-manage\", line 6, in <module>", " from glance.cmd.manage import main", " File \"/usr/lib/python3.6/site-packages/glance/cmd/manage.py\", line 53, in <module>", " from glance.db import migration as db_migration", " File \"/usr/lib/python3.6/site-packages/glance/db/__init__.py\", line 22, in <module>", " from wsme.rest import json", "ImportError: cannot import name 'json'", Version-Release number of selected component (if applicable): imagename: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-glance-api:latest How reproducible: Always Steps to Reproduce: 1. deploy an undercloud against 15-trunk (using rhos-release) 2. wait a bit 3. Tadaaaa Actual results: Failing due to missing dependency in glance-api container Expected results: Should go forward and succeed the deploy. Additional info:
This was likely caused due to a change in wsme https://opendev.org/x/wsme/commit/6f6dbfadefa68d44e3acb042d8da5411575f6925. There was a revert of the naming change https://opendev.org/x/wsme/commit/993d8a06d5c9f6fed13e03c1043c1a9594a88193 which may fix this now
@Alex: Good idea, but the stein upper-constraints has wsme at 0.9.3 [0] and that is also the version in rocky [1], and I'm assuming that we didn't have this problem in OSP14? [0] https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-constraints.txt#L410 [1] https://opendev.org/openstack/requirements/src/branch/stable/rocky/upper-constraints.txt#L387 The glance stein requirements.txt has WSME>=0.8.0. I think Sergii is correct, that wsme is missing from the container image.
It may be 0.9.3, but we aren't properly capping it in RDO and likely just pulled in that latest changes which is why it suddenly broke. wsme is a requirement for the python-glance package so it's likely not missing
@Alex: makes sense. What do we need to do to get the correct version into the build?
We should be capping wsme in rdo if there is an upper constraint: https://github.com/redhat-openstack/rdoinfo/blob/2104aa66243f31f7edf735be973df36934fed9a6/tags/rocky.yml#L1160 https://github.com/redhat-openstack/rdoinfo/blob/2104aa66243f31f7edf735be973df36934fed9a6/tags/stein.yml#L1211 Should probably ping the folks in #rdo on why this wasn't automagically handled at the end of the cycle. And then you'll likely want to sync with the release delivery folks on how to achieve this downstream correctly
*** This bug has been marked as a duplicate of bug 1710966 ***
For the record, I'm pining wsme in RDO to the fixed wsme commit https://review.rdoproject.org/r/#/c/20748 . Automatic pining from u-c has not worked fine for wsme because of a tooling issue (uppercase vs lowercase). We also look at how to fix that.