Description of problem: I tried to deploy minimal(1 controller + 1 compute) overcloud OSP12 with containerized openstack services and received failure. http://pastebin.test.redhat.com/483599 Version-Release number of selected component (if applicable): RHOSP 12 How reproducible: Always Steps to Reproduce: 1.Execute steps from http://etherpad.corp.redhat.com/testing-osp12-containers 2.Before Overcloud deployment - apply workaround for bug - https://bugzilla.redhat.com/show_bug.cgi?id=1448482http://paste.openstack.org/show/609370/ 3. Actual results: Heat Stack create failed. Expected results: Heat Stack created Additional info:
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Created attachment 1278144 [details] /var/log/messages from controller node /var/log/messages from controller node
The mysql_bootstrap returned 1 but mysql container happily runs: http://pastebin.test.redhat.com/483771
I dug up the log file from the stopped mysql_bootstrap container. The root cause is: 170512 10:39:47 [ERROR] /usr/libexec/mysqld: unknown option '--wsrep-new-cluster' Looks like the downstream binary we're using doesn't support galera clustering. 170512 10:39:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 170512 10:39:45 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170512 10:39:45 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170512 10:39:45 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 566 ... 170512 10:39:45 InnoDB: The InnoDB memory heap is disabled 170512 10:39:45 InnoDB: Mutexes and rw_locks use GCC atomic builtins 170512 10:39:45 InnoDB: Compressed tables use zlib 1.2.7 170512 10:39:45 InnoDB: Using Linux native AIO 170512 10:39:45 InnoDB: Initializing buffer pool, size = 128.0M 170512 10:39:45 InnoDB: Completed initialization of buffer pool InnoDB: The first specified data file ./ibdata1 did not exist: InnoDB: a new database to be created! 170512 10:39:45 InnoDB: Setting file ./ibdata1 size to 10 MB InnoDB: Database physically writes the file full: wait... 170512 10:39:45 InnoDB: Log file ./ib_logfile0 did not exist: new to be created InnoDB: Setting log file ./ib_logfile0 size to 5 MB InnoDB: Database physically writes the file full: wait... 170512 10:39:45 InnoDB: Log file ./ib_logfile1 did not exist: new to be created InnoDB: Setting log file ./ib_logfile1 size to 5 MB InnoDB: Database physically writes the file full: wait... InnoDB: Doublewrite buffer not found: creating new InnoDB: Doublewrite buffer created InnoDB: 127 rollback segment(s) active. InnoDB: Creating foreign key constraint system tables InnoDB: Foreign key constraint system tables created 170512 10:39:46 InnoDB: Waiting for the background threads to start 170512 10:39:47 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 0 170512 10:39:47 [Note] Plugin 'FEEDBACK' is disabled. 170512 10:39:47 [ERROR] /usr/libexec/mysqld: unknown option '--wsrep-new-cluster' 170512 10:39:47 [ERROR] Aborting 170512 10:39:47 InnoDB: Starting shutdown... 170512 10:39:47 InnoDB: Shutdown completed; log sequence number 1597945 170512 10:39:47 [Note] /usr/libexec/mysqld: Shutdown complete 170512 10:39:47 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended ~
Yea `/usr/libexec/mysqld --verbose --help | grep wsrep` doesn't print anything, while on upstream environment it prints the replication-related options.
()[mysql@overcloud-controller-0 /var/lib/mysql]$ rpm -qa | grep maria mariadb-libs-5.5.52-1.el7.x86_64 mariadb-server-5.5.52-1.el7.x86_64 mariadb-5.5.52-1.el7.x86_64 ()[mysql@overcloud-controller-0 /var/lib/mysql]$ rpm -qa | grep galera galera-25.3.5-7.el7ost.x86_64 Damien/Michele, can either of you tell what we're doing wrong that we don't have clustering support in our MariaDB server?
For upstream containers we're using mariadb 10 coming from RDO, if I'm not mistaken. (undercloud) [stack@undercloud ~]$ docker images | grep maria 192.168.24.1:8787/tripleoupstream/centos-binary-mariadb latest 71036f555ff5 10 hours ago 598.6 MB docker.io/tripleoupstream/centos-binary-mariadb latest 1f6497469546 9 days ago 599.1 MB 192.168.24.1:8787/tripleoupstream/centos-binary-mariadb <none> 1f6497469546 9 days ago 599.1 MB (undercloud) [stack@undercloud ~]$ docker run -ti --rm 71036f555ff5 bash ()[mysql@e6c68ba13a2c /]$ rpm -qa *mariadb* mariadb-config-10.1.20-1.el7.x86_64 mariadb-libs-10.1.20-1.el7.x86_64 mariadb-server-10.1.20-1.el7.x86_64 mariadb-common-10.1.20-1.el7.x86_64 mariadb-errmsg-10.1.20-1.el7.x86_64 mariadb-10.1.20-1.el7.x86_64 ()[mysql@e6c68ba13a2c /]$ rpm -qa *galera* galera-25.3.16-1.el7.x86_64 ()[mysql@e6c68ba13a2c /]$ yum info mariadb-server Loaded plugins: fastestmirror, ovl, priorities ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' Loading mirror speeds from cached hostfile * base: centos.mirrors.ovh.net * epel: mirror.uni-trier.de * extras: centos.mirror.fr.planethoster.net * updates: centos.mirror.fr.planethoster.net 27 packages excluded due to repository priority protections Installed Packages Name : mariadb-server Arch : x86_64 Epoch : 3 Version : 10.1.20 Release : 1.el7 Size : 92 M Repo : installed From repo : delorean-pike-testing Summary : The MariaDB server and related files URL : http://mariadb.org License : GPLv2 with exceptions and LGPLv2 and BSD Description : MariaDB is a multi-user, multi-threaded SQL database server. It is a : client/server implementation consisting of a server daemon (mysqld) : and many different client programs and libraries. This package contains : the MariaDB server and some accompanying files and directories. : MariaDB is a community developed branch of MySQL.
The ongoing reviews we have for overcloud HA don't use mariadb-server in the mysql kolla image, they use mariadb-server-galera [1] It looks like this is necessary as well for default non-HA overcloud deployments? Could you perhaps update the mysql image you're using on your local registry to use mariadb-server-galera and see if deploys passes? [1] https://review.openstack.org/#/c/456174/5/container-images/tripleo_kolla_template_overrides.j2
Reproduced. Checking the overcloud stack for failures the following is shown under stderr: deploy_stderr: | INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/lib/mysql 170517 19:46:35 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170517 19:46:35 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 51 ... 170517 19:46:35 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170517 19:46:35 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 59 ... The error in comment #4 is found in /var/log/containers/mysql/mariadb.log: 170517 19:46:37 [ERROR] /usr/libexec/mysqld: unknown option '--wsrep-new-cluster'
Thanks Damien! You were right, and we produced a workaround with Artem today (the only difference is that the downstream package is called mariadb-galera-server). After downloading images from registry, right before deploying the overcloud, build an image which has the correct package installed inside, and push it to the undercloud registry: https://paste.fedoraproject.org/paste/aMscm5CV1VPhz-4AYVyoXl5M1UNdIGYhyRLivL9gydE=/raw After that, edit docker-osp12.yaml file to point to the correct image tag: DockerMysqlImage: openstack-mariadb-docker:workaround And then deploy the overcloud. All mariadb containers look fine: [root@overcloud-controller-0 ~]# docker ps -a | grep mariadb 808760d1b2fa 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround "kolla_start" 11 minutes ago Up 11 minutes mysql 2e7898d79a55 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround "bash -c 'test -e /va" 11 minutes ago Exited (0) 11 minutes ago mysql_bootstrap 6c8321d59ccc 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround "/bin/bash -c 'chown " 11 minutes ago Exited (0) 11 minutes ago mysql_init_logs
I had to restart mariadb on the undercloud before re-running the overcloud deployment, following the workaround. Otherwise the OC deployment fails right away with: Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 6607c30a-5dc3-43b0-864d-1bd4c2e72f1f {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'1257fc583b28440aa6db50b81278ded7\', \'name\': u\'45e36286-7cc6-4290-9058-b48912c0e525\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'45e36286-7cc6-4290-9058-b48912c0e525'}, u'api': u'v2', u'headers': {u'Client-ID': u'c2cca7a4-f844-44c9-b382-ff91f097b240', u'X-Project-ID': u'1257fc583b28440aa6db50b81278ded7'}}}
Yeah, I also had to restart mariadb, forgot to write this info. Thx!
About the mariadb hiccup..I see that one as well from time to time. I opened this upstream bug to track it: https://bugs.launchpad.net/tripleo/+bug/1691951
Apparently fpaste expires in a very short time nowadays, the workaround from #10 is gone. Pasting it here instead, it's not that long after all. mkdir -p mariadb-workaround echo > mariadb-workaround/Dockerfile \ 'FROM 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-05-02.2 USER root RUN curl -O http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm && rpm -ivh rhos-release-latest.noarch.rpm && rhos-release 12-director -d RUN yum -y remove mariadb-server && yum -y install mariadb-galera-server USER mysql ' docker build --rm -t 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround ~/mariadb-workaround docker push 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround
*** Bug 1456977 has been marked as a duplicate of this bug. ***
In addition to comment #14: After that, edit docker-osp12.yaml file to point to the correct image tag: DockerMysqlImage: openstack-mariadb-docker:workaround (this was already mentioned in comment #10).
Quick update here, the next openstack-mariadb-docker image will contain the fix discussed in comment #14.
Assigning to Chris as UA. Thanks for the updates Damien.
(In reply to Alexander Chuzhoy from comment #16) > In addition to comment #14: > After that, edit docker-osp12.yaml file to point to the correct image tag: > > DockerMysqlImage: openstack-mariadb-docker:workaround > > > (this was already mentioned in comment #10). I think it's already included, what takes this bug to ON_QA ?
Verified: openstack-mariadb-docker 2017-06-15.2 The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457