Bug 1576083 - [OSP13][Undercloud backup/restor] When attemting CLI driven backups, it missing the file /root/undercloud-all-databases.sql and restore fails.
Summary: [OSP13][Undercloud backup/restor] When attemting CLI driven backups, it missi...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 13.0 (Queens)
Assignee: Carlos Camacho
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-08 18:51 UTC by Omri Hochman
Modified: 2018-05-18 20:55 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-17 16:48:21 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 566897 0 None MERGED [UC backup] - Add restore information 2020-09-07 12:04:26 UTC
OpenStack gerrit 567834 0 None MERGED Remove undercloud backup extraction commands and add clarification details. 2020-09-07 12:04:26 UTC

Description Omri Hochman 2018-05-08 18:51:37 UTC
[OSP13][Undercloud backup/restor] When attemting instructions for CLI driven undercloud backup,  it misses the file /root/undercloud-all-databases.sql and restores fails. 

CLI driven backups
------------------

There is an automated way of creating an Undercloud backup,
this CLI option allows the operator to run a database and filesystem backup.
By default, all databases are included in the backup, also, the folder `/home/stack`.


[stack@undercloud75 ~]$ source stackrc
(undercloud) [stack@undercloud75 ~]$ openstack undercloud backup --add-path /etc/hosts \
>                               --add-path /var/log/ \
>                               --add-path /var/lib/glance/images/ \
>                               --add-path /srv/node/ \
>                               --add-path /etc/

Started Mistral Workflow tripleo.undercloud_backup.v1.backup. Execution ID: 78716244-f8f4-49c5-b1e7-872bb3b4cb11
Waiting for messages on queue 'tripleo' with no timeout.
Undercloud Backup succeed
(undercloud) [stack@undercloud75 ~]$


(undercloud) [stack@undercloud ~]$ mkdir kk
(undercloud) [stack@undercloud ~]$ cd kk
(undercloud) [stack@undercloud kk]$ openstack container save undercloud-backups
(undercloud) [stack@undercloud kk]$ ls
UC-backup-20180501163744.tar
(undercloud) [stack@undercloud kk]$ ls -ltah
total 509M
-rw-rw-r--.  1 stack stack 509M May  1 16:38 UC-backup-20180501163744.tar
drwxrwxr-x.  2 stack stack   42 May  1 16:38 .
drwx------. 18 stack stack 4.0K May  1 16:38 ..


Results :

Error when Attempting to restore :
----------------------------------
[root@undercloud75 ~]#   tar -xC / -f UC-backup-20180501163744.tar etc/my.cnf.d/mariadb-server.cnf /root/undercloud-all-databases.sql
tar: /root/undercloud-all-databases.sql: Not found in archive
tar: Exiting with failure status due to previous errors

Comment 1 Carlos Camacho 2018-05-11 14:52:45 UTC
Hello, 

I managed to deploy a Queens box and ran the backup, afterward I unzipped the backup and restored the DB.

I pushed 2 reviews upstream

https://review.openstack.org/#/c/567834/
https://review.openstack.org/#/c/566897/


Both merged, does this makes the process more clear?

Comment 3 Carlos Camacho 2018-05-14 10:01:17 UTC
After applying 
https://review.openstack.org/#/c/568245/
https://review.openstack.org/#/c/564784/

The following commands should reinstall the UC from the backup (tested just now):


source ~/stackrc
rm -rf /var/tmp/test_uc_backup
mkdir -p /var/tmp/test_uc_backup
cd /var/tmp/test_uc_backup
openstack container delete undercloud-backups --recursive
openstack undercloud backup --exclude-path /home/stack/
openstack container save undercloud-backups
tar -xvf *.tar
gunzip *.gz
cat all-databases-*.sql | sudo mysql
for i in ceilometer glance heat ironic keystone neutron nova;do sudo mysql -e "drop user $i" || true;done
sudo mysql -e 'flush privileges'
openstack undercloud install

Comment 5 Omri Hochman 2018-05-16 15:04:40 UTC
Adding the Info : We've attempted this on my env got the following permission error when redeploying the undercloud: 

2018-02-22 08:33:03,203 INFO: Error: Systemd start for mariadb failed!
2018-02-22 08:33:03,204 INFO: journalctl log for mariadb:
2018-02-22 08:33:03,205 INFO: -- Logs begin at Tue 2018-02-20 14:30:10 EST, end at Thu 2018-02-22 08:33:03 EST. --
2018-02-22 08:33:03,206 INFO: Feb 22 08:29:50 undercloud75 systemd[1]: Starting MariaDB 10.1 database server...
2018-02-22 08:33:03,207 INFO: Feb 22 08:29:50 undercloud75 mysql-check-socket[30092]: Socket file /var/lib/mysql/mysql.sock exists.
2018-02-22 08:33:03,208 INFO: Feb 22 08:29:50 undercloud75 mysql-check-socket[30092]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, so it will be
removed automatically.
2018-02-22 08:33:03,209 INFO: Feb 22 08:29:51 undercloud75 mysql-prepare-db-dir[30116]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
2018-02-22 08:33:03,210 INFO: Feb 22 08:29:51 undercloud75 mysql-prepare-db-dir[30116]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prep
are-db-dir.
2018-02-22 08:33:03,211 INFO: Feb 22 08:29:51 undercloud75 mysqld[30153]: 2018-02-22  8:29:51 140221668534464 [Warning] option 'open_files_limit': unsigned value 18446744073709
551615 adjusted to 4294967295
2018-02-22 08:33:03,212 INFO: Feb 22 08:29:51 undercloud75 mysqld[30153]: 2018-02-22  8:29:51 140221668534464 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as pr
ocess 30153 ...
2018-02-22 08:33:03,213 INFO: Feb 22 08:32:02 undercloud75 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
2018-02-22 08:33:03,214 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: mariadb.service: control process exited, code=exited status=3
2018-02-22 08:33:03,214 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: Failed to start MariaDB 10.1 database server.
2018-02-22 08:33:03,215 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: Unit mariadb.service entered failed state.
2018-02-22 08:33:03,216 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: mariadb.service failed.
2018-02-22 08:33:03,216 INFO:
2018-02-22 08:33:03,217 INFO: Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Systemd start for mariadb failed!
2018-02-22 08:33:03,217 INFO: journalctl log for mariadb:
2018-02-22 08:33:03,218 INFO: -- Logs begin at Tue 2018-02-20 14:30:10 EST, end at Thu 2018-02-22 08:33:03 EST. --
2018-02-22 08:33:03,219 INFO: Feb 22 08:29:50 undercloud75 systemd[1]: Starting MariaDB 10.1 database server...
2018-02-22 08:33:03,220 INFO: Feb 22 08:29:50 undercloud75 mysql-check-socket[30092]: Socket file /var/lib/mysql/mysql.sock exists.
2018-02-22 08:33:03,221 INFO: Feb 22 08:29:50 undercloud75 mysql-check-socket[30092]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, so it will be
removed automatically.
2018-02-22 08:33:03,222 INFO: Feb 22 08:29:51 undercloud75 mysql-prepare-db-dir[30116]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
2018-02-22 08:33:03,223 INFO: Feb 22 08:29:51 undercloud75 mysql-prepare-db-dir[30116]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prep
are-db-dir.
2018-02-22 08:33:03,224 INFO: Feb 22 08:29:51 undercloud75 mysqld[30153]: 2018-02-22  8:29:51 140221668534464 [Warning] option 'open_files_limit': unsigned value 18446744073709
551615 adjusted to 4294967295
2018-02-22 08:33:03,225 INFO: Feb 22 08:29:51 undercloud75 mysqld[30153]: 2018-02-22  8:29:51 140221668534464 [Note] /usr/libexec/mysqld (mysqld 10.1.20-MariaDB) starting as pr
ocess 30153 ...
2018-02-22 08:33:03,226 INFO: Feb 22 08:32:02 undercloud75 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
2018-02-22 08:33:03,227 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: mariadb.service: control process exited, code=exited status=3
2018-02-22 08:33:03,228 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: Failed to start MariaDB 10.1 database server.
2018-02-22 08:33:03,228 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: Unit mariadb.service entered failed state.
2018-02-22 08:33:03,229 INFO: Feb 22 08:33:03 undercloud75 systemd[1]: mariadb.service failed.
2018-02-22 08:33:03,229 INFO:
2018-02-22 08:33:03,230 INFO: Notice: /Stage[main]/Mysql::Server::Service/Exec[wait_for_mysql_socket_to_open]: Dependency Service[mysqld] has failures: true
2018-02-22 08:33:03,231 INFO: Warning: /Stage[main]/Mysql::Server::Service/Exec[wait_for_mysql_socket_to_open]: Skipping because of failed dependencies
2018-02-22 08:33:03,232 INFO: Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
2018-02-22 08:33:03,233 INFO: Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
2018-02-22 08:33:04,312 INFO: Error: Failed to apply catalog: Execution of '/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM
 mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
2018-02-22 08:33:04,313 INFO: Changes:
2018-02-22 08:33:04,313 INFO:             Total: 1
2018-02-22 08:33:04,313 INFO: Events:
2018-02-22 08:33:04,314 INFO:           Failure: 1
2018-02-22 08:33:04,314 INFO:           Success: 1
2018-02-22 08:33:04,314 INFO:             Total: 2
2018-02-22 08:33:04,315 INFO: Resources:
2018-02-22 08:33:04,315 INFO:            Failed: 1
2018-02-22 08:33:04,315 INFO:         Restarted: 1
2018-02-22 08:33:04,316 INFO:           Changed: 1
2018-02-22 08:33:04,316 INFO:    Corrective change: 2
2018-02-22 08:33:04,316 INFO:           Skipped: 2
2018-02-22 08:33:04,317 INFO:       Out of sync: 2
2018-02-22 08:33:04,317 INFO:             Total: 50
2018-02-22 08:33:04,317 INFO: Time:
2018-02-22 08:33:04,318 INFO:           Archive: 0.00
2018-02-22 08:33:04,318 INFO:     Mysql datadir: 0.00
2018-02-22 08:33:04,318 INFO:            Anchor: 0.00
2018-02-22 08:33:04,319 INFO:           Package: 0.20
2018-02-22 08:33:04,319 INFO:              File: 0.58

Comment 6 Omri Hochman 2018-05-16 15:11:28 UTC
what we've done : 
(1) deploy UC + OC of osp13 
(2) backup undercloud 
(3) copy the backup files to side folder  
(4) revert the undercloud machine to clean rhel7.5
(5) install clean undercloud osp13
(6) copy the backup files to back undercloud machine 
(7) attempt to restore undercloud from backup files --> fails

Comment 7 Carlos Camacho 2018-05-17 16:48:21 UTC
Hi Omri, we tested this correctly yesterday, closing as the DB backup is done correctly. Please, feel free to reopen if you think the DB is not dumped.

Comment 8 Carlos Camacho 2018-05-17 16:53:18 UTC
THe last issue posted by Omri was because the MySQL server was broken not because the DB dump is not created.

Comment 9 Carlos Camacho 2018-05-17 16:54:05 UTC
Let's track the restore here: https://bugzilla.redhat.com/show_bug.cgi?id=1557794


Note You need to log in before you can comment on or make changes to this bug.