Red Hat Bugzilla – Bug 1463573
galera changes IP address after minor OSP8 upgrade
Last modified: 2018-01-07 05:34:43 EST
Description of problem:
When doing a minor OSP8 upgrade, with 3 controllers/1 compute, it fails because it loses mysql connectivity.
The issue is that galera is changing the IP that is listening to...
In my case : before the upgrade, it is configured to listen to 172.17.1.16. You can see it written on nova.conf, and i could test it was the case.
After doing the minor upgrade, it failed with error:
Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, \"Can't connect to MySQL server on '172.17.1.16' ([Errno 113] No route to host)
That happens in all db sync commands, causing minor upgrade to fail. When looking at galera, it is not listening now to 172.17.1.16, but to another ip:
telnet 172.17.1.16 3306
telnet: connect to address 172.17.1.16: No route to host
ps -ef | grep galera
mysql 4510 3554 0 08:26 ? 00:00:17 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep-provider=/usr/lib64/galera/libgalera_smm.so --wsrep-cluster-address=gcomm://overcloud-controller-0,overcloud-controller-1,overcloud-controller-2 --log-error=/var/log/mysqld.log --open-files-limit=16384 --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=c6e7fe90-55f1-11e7-8762-1af616643337:87403
You can see it listens to 172.17.1.14 instead:
lsof -n -p 4510 | grep LISTEN
mysqld 4510 mysql 11u IPv4 9842675 0t0 TCP 172.17.1.14:tram (LISTEN)
mysqld 4510 mysql 26u IPv4 9823774 0t0 TCP 172.17.1.14:mysql (LISTEN)
That was caused by a bug in documentation that was solved recently. Minor upgrade documentation, was telling to include the upgrade-from-vip.yaml environment.
And this was the one causing the IP change. I retried without including it, and the minor upgrade was successful.