Description of problem: - Undercloud update fails with: puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/mysql -- defaults-extra-file=/root/.my.cnf -NBe show variables like '%_database' #mysql50#.config' returned 1: ERROR 1102 (42000): Incorrect database name '#mysql50#.config'", - Full error attached bellow: http://pastebin.test.redhat.com/777847 How reproducible: Steps to Reproduce: 1. Deploy an HA OPS15 enviroment. 2. Perform a minor update Actual results: - Jenkins job output: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/updates/job/DFG-pidone-updates-15_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-sanity/23/console
- Jenkins artifacts: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/updates/job/DFG-pidone-updates-15_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-sanity/23/artifact/
I am not sure if it is relevant, but after removing a hide directory called ".config" from "/var/lib/mysql" in the mysql container, the undercloud update finished success. The thing is that I don't know what was created that hide directory.
So the reason this failed in the job is that there is this spurious folder ".config/procps" inside a bunch of containers. E.g.: [root@undercloud-0 ~]# find /var/lib -iname procps | head -n20 /var/lib/glance/.config/procps /var/lib/ironic/.config/procps /var/lib/containers/storage/overlay/6b15aeff4374b2a01c01a15d0834b6cbc5425c3793fb5706ea70eba0acea70b4/diff/run/memcache/.config/procps /var/lib/containers/storage/overlay/6b15aeff4374b2a01c01a15d0834b6cbc5425c3793fb5706ea70eba0acea70b4/merged/run/memcache/.config/procps /var/lib/containers/storage/overlay/7a54086db668849301821c35636247ebe22b352258b327180c66b1f6b5a362b7/diff/root/.config/procps /var/lib/containers/storage/overlay/7a54086db668849301821c35636247ebe22b352258b327180c66b1f6b5a362b7/merged/root/.config/procps /var/lib/containers/storage/overlay/3f5e7b4f4910ffcb2e9a0dcc1d9f5708a3fea8221c8386d8937c63df37adb1cb/diff/root/.config/procps /var/lib/containers/storage/overlay/3f5e7b4f4910ffcb2e9a0dcc1d9f5708a3fea8221c8386d8937c63df37adb1cb/merged/root/.config/procps /var/lib/containers/storage/overlay/f70eafd773d41511ea2943d134d0069bb9aa96f19ed0cfd9a0c52bd72398c028/diff/var/tmp/.config/procps /var/lib/containers/storage/overlay/f70eafd773d41511ea2943d134d0069bb9aa96f19ed0cfd9a0c52bd72398c028/merged/var/tmp/.config/procps /var/lib/containers/storage/overlay/7700e4532f7fb7af093e2100001fdfeb484024d26a09b1ff03b161a60a842f8a/diff/root/.config/procps /var/lib/containers/storage/overlay/7700e4532f7fb7af093e2100001fdfeb484024d26a09b1ff03b161a60a842f8a/merged/root/.config/procps /var/lib/containers/storage/overlay/e970ef9fe6e8ca7d2490607c05df0fb89d8d330665facdc3656ff6f830da730c/diff/root/.config/procps /var/lib/containers/storage/overlay/e970ef9fe6e8ca7d2490607c05df0fb89d8d330665facdc3656ff6f830da730c/merged/root/.config/procps /var/lib/containers/storage/overlay/e5ef9f561ee3dd99c2dbae522de37765544b6a886107ae3c5385ab7ec05d0827/diff/root/.config/procps /var/lib/containers/storage/overlay/e5ef9f561ee3dd99c2dbae522de37765544b6a886107ae3c5385ab7ec05d0827/merged/root/.config/procps /var/lib/containers/storage/overlay/ce69b4310b8b1cabc67634959b339dc679f6bc1df8247cfcc852fdb3ea98c81c/diff/root/.config/procps /var/lib/containers/storage/overlay/ce69b4310b8b1cabc67634959b339dc679f6bc1df8247cfcc852fdb3ea98c81c/merged/root/.config/procps /var/lib/containers/storage/overlay/b622de8f366aec259ced04e5a5c99eec47f7a9d084bf7782cfef2f636932f6f3/diff/root/.config/procps /var/lib/containers/storage/overlay/b622de8f366aec259ced04e5a5c99eec47f7a9d084bf7782cfef2f636932f6f3/merged/root/.config/procps .... This trips up mysql because it things the '.config' folder is a DB folder. Now the important question is "who on earth is creating this folder" Now an interesting find is that if you call 'top' it actually (wtf) creates an empty ~/.config/procps folder: 857548 mkdir("/root/.config", 0700) = 0 857548 mkdir("/root/.config/procps", 0700) = 0 857548 openat(AT_FDCWD, "/root/.config/procps/toprc", O_RDONLY) = -1 ENOENT (No such file or directory) So my guess here is that something has ran top inside the containers? Maybe infrared, maybe something else? Note, I tried to run openstack undercloud upgrade just now on my env it worked okay: Install artifact is located at /home/stack/undercloud-install-20190709142929.tar.bzip2 ######################################################## Deployment successful!
Ok so this is actually infrared when collecting log plugins https://github.com/redhat-openstack/infrared/blob/master/plugins/collect-logs/tasks/collect_host_logs.yml#L174 : for cont in $($cont_cmd ps -a | awk {'print $NF'} | grep -v NAMES); do INFO_DIR="$BASE_CONT_EXTRA/containers/${cont}"; mkdir -p "$INFO_DIR"; INFO_FILE="$INFO_DIR/container_info.log"; if [[ "$cont_cmd" == "docker" ]]; then echo "+ $cont_cmd top $cont auxw" > "$INFO_FILE"; $cont_cmd top "$cont" auxw &>> "$INFO_FILE"; else echo "+ $cont_cmd top $cont user pid ppid pcpu vsz tty state time etime args" &>> "$INFO_FILE"; $cont_cmd top "$cont" user pid ppid pcpu vsz tty state time etime args &>> "$INFO_FILE"; fi echo "+ $cont_cmd exec $cont top -bwn1" >> "$INFO_FILE"; $cont_cmd exec "$cont" top -bwn1 &>> "$INFO_FILE"; echo "+ $cont_cmd inspect $cont" >> "$INFO_FILE"; $cont_cmd inspect "$cont" &>> "$INFO_FILE"; $cont_cmd logs "$cont" &> "$INFO_DIR/stdout.log"; echo "+ $cont_cmd exec -it --user root $cont /usr/bin/rpm -qa" >> "$INFO_FILE"; $cont_cmd exec -it --user root ${cont} /usr/bin/rpm -qa &>> "$INFO_FILE"; if [[ "$cont_cmd" == "docker" ]]; then $cont_cmd cp "$cont:/var/lib/kolla/config_files/config.json" "$INFO_DIR/config.json"; else mnt=$($cont_cmd mount "$cont"); cp "$mnt:/var/lib/kolla/config_files/config.json" "$INFO_DIR/config.json"; $cont_cmd umount $cont; fi done; So this should be filed as an infrared bug as it is quite harmful
Filed https://projects.engineering.redhat.com/browse/RHOSINFRA-2628 and am closing this one as not a bug (for once it was not our fault ;)
*** Bug 1730764 has been marked as a duplicate of this bug. ***
*** Bug 1733684 has been marked as a duplicate of this bug. ***