Bug 1463573 - galera changes IP address after minor OSP8 upgrade
galera changes IP address after minor OSP8 upgrade
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: James Slagle
Arik Chernetsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 05:07 EDT by Yolanda Robla
Modified: 2018-01-07 05:34 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-21 15:42:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yolanda Robla 2017-06-21 05:07:25 EDT
Description of problem:


When doing a minor OSP8 upgrade, with 3 controllers/1 compute, it fails because it loses mysql connectivity.
The issue is that galera is changing the IP that is listening to...

In my case : before the upgrade, it is configured to listen to 172.17.1.16. You can see it written on nova.conf, and i could test it was the case.

After doing the minor upgrade, it failed with error:

Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, \"Can't connect to MySQL server on '172.17.1.16' ([Errno 113] No route to host)

That happens in all db sync commands, causing minor upgrade to fail. When looking at galera, it is not listening now to 172.17.1.16, but to another ip:

telnet 172.17.1.16 3306
Trying 172.17.1.16...

telnet: connect to address 172.17.1.16: No route to host


ps -ef | grep galera
mysql     4510  3554  0 08:26 ?        00:00:17 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep-provider=/usr/lib64/galera/libgalera_smm.so --wsrep-cluster-address=gcomm://overcloud-controller-0,overcloud-controller-1,overcloud-controller-2 --log-error=/var/log/mysqld.log --open-files-limit=16384 --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=c6e7fe90-55f1-11e7-8762-1af616643337:87403


You can see it listens to 172.17.1.14 instead:

 lsof -n -p 4510 | grep LISTEN 
mysqld  4510 mysql   11u     IPv4            9842675       0t0        TCP 172.17.1.14:tram (LISTEN)
mysqld  4510 mysql   26u     IPv4            9823774       0t0        TCP 172.17.1.14:mysql (LISTEN)
Comment 1 Yolanda Robla 2017-06-21 15:42:27 EDT
That was caused by a bug in documentation that was solved recently. Minor upgrade documentation, was telling to include the upgrade-from-vip.yaml environment.
And this was the one causing the IP change. I retried without including it, and the minor upgrade was successful.

Note You need to log in before you can comment on or make changes to this bug.