Hi David, I have tried to reproduce the issue, as mentioned by you in previous comment, but looks like there is no issue and the packages are removed during minor update. I have deployed OSP13 z4 and updated it to latest by job [1] and I have manually created vms on overcloud during tempest run step (after overcloud deploy and before undercloud update). [1] https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-13-from-z4-HA-ipv4/48/ You could see that the packages are there on compute node before update and removed after doing update and also there is not change in permission even after update. [root@compute-0 ~]# rpm -qa | grep nova puppet-nova-12.4.0-14.el7ost.noarch openstack-nova-conductor-17.0.7-5.el7ost.noarch python-nova-17.0.7-5.el7ost.noarch openstack-nova-compute-17.0.7-5.el7ost.noarch openstack-nova-scheduler-17.0.7-5.el7ost.noarch python2-novaclient-10.1.0-1.el7ost.noarch openstack-nova-migration-17.0.7-5.el7ost.noarch openstack-nova-api-17.0.7-5.el7ost.noarch openstack-nova-common-17.0.7-5.el7ost.noarch openstack-nova-console-17.0.7-5.el7ost.noarch lopenstack-nova-placement-api-17.0.7-5.el7ost.noarch openstack-nova-novncproxy-17.0.7-5.el7ost.noarch [root@compute-0 ~]# ls -lnrt /var/lib/nova/ total 0 drwxr-xr-x. 2 42436 42436 6 Dec 21 2018 tmp drwxr-xr-x. 2 42436 42436 6 Dec 21 2018 networks drwxr-xr-x. 2 42436 42436 6 Dec 21 2018 keys drwxr-xr-x. 2 42436 42436 6 Dec 21 2018 buckets drwxr-xr-x. 6 42436 42436 120 May 29 09:23 instances [root@compute-0 ~]# virsh list Id Name State ---------------------------------------------------- 6 instance-00000029 running [root@compute-0 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 53d4b47bb39e 192.168.24.1:8787/rhosp13/openstack-neutron-openvswitch-agent:2019-01-10.1 "kolla_start" 27 minutes ago Up 27 minutes (healthy) neutron_ovs_agent 61089b3a9402 192.168.24.1:8787/rhosp13/openstack-cron:2019-01-10.1 "kolla_start" 27 minutes ago Up 27 minutes logrotate_crond 01c16cd431bd 192.168.24.1:8787/rhosp13/openstack-ceilometer-compute:2019-01-10.1 "kolla_start" 27 minutes ago Up 27 minutes ceilometer_agent_compute e7231db89f0d 192.168.24.1:8787/rhosp13/openstack-nova-compute:2019-01-10.1 "kolla_start" 27 minutes ago Up 27 minutes (healthy) nova_compute 0e3a4611fbe1 192.168.24.1:8787/rhosp13/openstack-nova-compute:2019-01-10.1 "kolla_start" 27 minutes ago Up 27 minutes (healthy) nova_migration_target c21538cef450 192.168.24.1:8787/rhosp13/openstack-iscsid:2019-01-10.1 "kolla_start" 32 minutes ago Up 32 minutes (healthy) iscsid f601ec1f8ff6 192.168.24.1:8787/rhosp13/openstack-nova-libvirt:2019-01-10.1 "kolla_start" 32 minutes ago Up 32 minutes nova_libvirt 8b0931267b05 192.168.24.1:8787/rhosp13/openstack-nova-libvirt:2019-01-10.1 "kolla_start" 32 minutes ago Up 32 minutes nova_virtlogd ==================================== after minor update to latest version ==================================== [root@compute-0 ~]# rpm -qa | grep nova python2-novaclient-10.1.1-1.el7ost.noarch puppet-nova-12.5.0-8.el7ost.noarch [root@compute-0 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 209be78679b1 192.168.24.1:8787/rh-osbs/rhosp13-openstack-neutron-openvswitch-agent:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour (healthy) neutron_ovs_agent 476b6b2d1f64 192.168.24.1:8787/rh-osbs/rhosp13-openstack-nova-compute:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour (healthy) nova_compute 2c189d010835 192.168.24.1:8787/rh-osbs/rhosp13-openstack-cron:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour logrotate_crond aedb507a6f5e 192.168.24.1:8787/rh-osbs/rhosp13-openstack-nova-compute:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour (healthy) nova_migration_target ab6768805f43 192.168.24.1:8787/rh-osbs/rhosp13-openstack-ceilometer-compute:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour ceilometer_agent_compute def4a47c2b26 192.168.24.1:8787/rh-osbs/rhosp13-openstack-iscsid:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour (healthy) iscsid 5971b3755fd5 192.168.24.1:8787/rh-osbs/rhosp13-openstack-nova-libvirt:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour nova_libvirt 541366bd99e3 192.168.24.1:8787/rh-osbs/rhosp13-openstack-nova-libvirt:20200511.1 "dumb-init --singl..." About an hour ago Up About an hour nova_virtlogd [root@compute-0 ~]# virsh list Id Name State ---------------------------------------------------- 6 instance-00000029 running [root@compute-0 ~]# ls -larn /var/lib/nova/ total 4 drwx------. 2 42436 42436 20 May 29 13:41 .ssh drwxr-xr-x. 5 42436 42436 97 May 29 09:43 instances drwxr-xr-x. 3 42436 42436 46 May 29 09:20 .cinderclient drwxr-xr-x. 97 0 0 4096 May 29 13:37 .. drwxr-xr-x. 5 42436 42436 56 May 11 15:35 .
Hello Rajesh, Thanks for trying this. I believe I understand what happened. When I tried the update, I wanted to first try a simple yum update on one of the compute. No openstack packages were updated and that's because the rhel-7-server-openstack-13-rpms repo is disabled. This is a satellite deployed lab, and in my templates, I did have rhel-7-server-openstack-13-rpms in the list of enabled repos, so I suspect that tripleo disabled this repo itself during the initial deployment, or it's from infrared. For fun, I enabled the repo and reran the yum update. Since I had the packages on the node, and the repo was enabled, the packages were updated to the latest version, and indeed, the permissions on /var/lib/nova were reset to nova:nova instead of the nova container's nova user UID. I only have another compute on this environment, so instead of updating it with a plain yum update, I ran stack update on it. During the "update run", I see that the permission changed from 42436 to nova on the controllers, just like the customer had on his compute, but it doesn't matter because only nova-compute uses these folders. On the computes though, the nova packages are removed [1] before we proceed with the yum update, and the permissions [2] aren't altered. This behavior was introduced by this commit [a] to address this bug [b]. That's a good example why we need to run the updates through director. Closing this bug now. Thank you very much, DVD [a] https://review.opendev.org/#/c/628101/2 [b] https://bugzilla.redhat.com/show_bug.cgi?id=1656617 [1] ~~~ Jun 12 13:12:02 Updated: 2:docker-rhel-push-plugin-1.13.1-109.gitcccb291.el7_7.x86_64 Jun 12 13:12:02 Updated: 2:docker-common-1.13.1-109.gitcccb291.el7_7.x86_64 Jun 12 13:12:03 Updated: 2:docker-client-1.13.1-109.gitcccb291.el7_7.x86_64 Jun 12 13:12:05 Updated: 2:docker-1.13.1-109.gitcccb291.el7_7.x86_64 Jun 12 13:12:27 Erased: 1:openstack-nova-migration-17.0.13-2.el7ost.noarch Jun 12 13:12:27 Erased: 1:openstack-nova-compute-17.0.13-2.el7ost.noarch Jun 12 13:12:39 Erased: 1:openstack-nova-novncproxy-17.0.13-2.el7ost.noarch Jun 12 13:12:39 Erased: 1:openstack-nova-scheduler-17.0.13-2.el7ost.noarch Jun 12 13:12:39 Erased: 1:openstack-nova-placement-api-17.0.13-2.el7ost.noarch Jun 12 13:12:40 Erased: 1:openstack-nova-console-17.0.13-2.el7ost.noarch Jun 12 13:12:40 Erased: 1:openstack-nova-conductor-17.0.13-2.el7ost.noarch Jun 12 13:12:40 Erased: 1:openstack-nova-api-17.0.13-2.el7ost.noarch Jun 12 13:12:40 Erased: 1:openstack-nova-common-17.0.13-2.el7ost.noarch Jun 12 13:12:40 Erased: 1:python-nova-17.0.13-2.el7ost.noarch Jun 12 13:13:31 Updated: corosync-2.4.3-6.el7_7.1.x86_64 Jun 12 13:13:31 Updated: corosynclib-2.4.3-6.el7_7.1.x86_64 Jun 12 13:13:31 Updated: pacemaker-libs-1.1.20-5.el7_7.2.x86_64 Jun 12 13:13:32 Updated: pacemaker-cli-1.1.20-5.el7_7.2.x86_64 Jun 12 13:13:32 Updated: resource-agents-4.1.1-30.el7_7.4.x86_64 Jun 12 13:13:32 Updated: pacemaker-cluster-libs-1.1.20-5.el7_7.2.x86_64 Jun 12 13:13:32 Updated: pacemaker-1.1.20-5.el7_7.2.x86_64 Jun 12 13:13:32 Updated: clufter-common-0.77.1-1.el7.noarch Jun 12 13:13:32 Updated: clufter-bin-0.77.1-1.el7.x86_64 Jun 12 13:13:32 Updated: python-clufter-0.77.1-1.el7.noarch Jun 12 13:13:34 Updated: pcs-0.9.167-3.el7_7.1.x86_64 Jun 12 13:13:34 Updated: pacemaker-remote-1.1.20-5.el7_7.2.x86_64 Jun 12 13:13:34 Installed: 1:containers-common-0.1.37-3.el7.x86_64 Jun 12 13:13:34 Installed: python2-jmespath-0.9.0-6.el7_7.noarch Jun 12 13:13:34 Updated: python-semantic_version-2.4.2-2.el7.noarch Jun 12 13:13:34 Updated: 1:oci-systemd-hook-0.2.0-1.git05e6923.el7_6.x86_64 Jun 12 13:13:34 Updated: driverctl-0.108-1.el7_6.noarch Jun 12 13:13:34 Updated: 2:oci-umount-2.5-3.el7.x86_64 Jun 12 13:13:36 Updated: etcd-3.2.26-1.el7.x86_64 Jun 12 13:13:36 Updated: dpdk-18.11.2-1.el7.x86_64 Jun 12 13:13:36 Updated: 1:atomic-registries-1.22.1-29.gitb507039.el7.x86_64 Jun 12 13:13:36 Updated: python-websocket-client-0.56.0-3.git3c25814.el7.noarch Jun 12 13:13:48 Updated: 2:container-selinux-2.107-3.el7.noarch Jun 12 13:13:48 Updated: 1:python-docker-pycreds-0.3.0-11.el7.noarch Jun 12 13:13:48 Erased: 1:skopeo-containers-0.1.31-1.dev.gitae64ff7.el7.x86_64 Jun 12 13:13:48 Erased: python-jmespath-0.9.0-5.el7ost.noarch ~~~ [2] ~~~ [root@ess13z2-scpu-0 log]# ls -tlra /var/lib/nova/instances/ total 4 drwxr-xr-x. 5 nova nova 97 Feb 3 12:05 . drwxr-xr-x. 2 42436 42436 93 Jun 9 06:04 locks drwxr-xr-x. 2 42436 42436 54 Jun 9 06:04 _base drwxr-xr-x. 2 42436 42436 54 Jun 9 06:04 ffdb07b2-552d-4696-afb7-1f228e306b81 -rw-r--r--. 1 42436 42436 60 Jun 12 12:43 compute_nodes drwxr-xr-x. 3 42436 42436 23 Jun 12 13:12 .. ~~~