Bug 1916162 - tripleo-transfer can't transfer large databases
Summary: tripleo-transfer can't transfer large databases
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 16.1 (Train)
Hardware: All
OS: All
urgent
high
Target Milestone: z6
: 16.1 (Train on RHEL 8.2)
Assignee: Jesse Pretorius
QA Contact: Jose Luis Franco
URL:
Whiteboard:
: 1904681 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-14 11:29 UTC by Sergii Golovatiuk
Modified: 2024-12-20 19:31 UTC (History)
8 users (show)

Fixed In Version: tripleo-ansible-0.5.1-1.20210310113105.902c3c8.el8ost openstack-tripleo-heat-templates-11.3.2-1.20210310113344.29a02c1.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-26 11:43:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1908425 0 None None None 2021-01-21 14:09:18 UTC
OpenStack gerrit 778650 0 None MERGED Remove tripleo_transfer cleanup.yml reference 2021-04-07 13:13:43 UTC
OpenStack gerrit 778665 0 None MERGED [tripleo_transfer] Do transfers directly from src to dst 2021-04-07 13:13:49 UTC
Red Hat Issue Tracker UPG-2490 0 None None None 2021-11-04 05:45:31 UTC
Red Hat Product Errata RHSA-2021:2119 0 None None None 2021-05-26 11:44:24 UTC

Description Sergii Golovatiuk 2021-01-14 11:29:42 UTC
Description of problem:

Working on bz#1904681 we found that 

1. We dump and compress database on controller-1
2. We transfer it to undercloud from controller-1,
3. We transfer it from undercloud to controller-0, unpack it and start database.

Usually our undercloud is pretty small 10-50GB. However, we have clients that have openstack DB larger than 1TB. It means that transfer will never happen.


How reproducible:
Always

Steps to Reproduce:
1. Create uc as 10Gb as OSP13
2. Create overcloud DB as 100Gb and try to run FFWD as OSP13
3. Run FFWD

Actual results:
Fail

Expected results:
Pass


Additional info:

Instead of doing transfer to undercloud we should initiate transfer from controller-1 to controller-0 directly which has enough space after leapp (consider it fresh system). In that case, transfer will be fast as we transfer 1 time instead of 2 times and transfer won't depend on diskspace of undercloud.

I did a quick test and was able to transfer file with

- hosts: controller-0
  tasks:    
   - name: Copy DB from Controller-1 to Controller-0
     synchronize: src=~/DB.sql dest=~/DB.sql
     delegate_to: controller-1

In this case, file is transferred without copying to/from undercloud.

Comment 2 Jose Luis Franco 2021-01-21 16:45:28 UTC
*** Bug 1904681 has been marked as a duplicate of this bug. ***

Comment 28 Jose Luis Franco 2021-05-05 14:22:41 UTC
Validated in: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/ffu/job/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-HA/140

http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-HA/140/undercloud-0/home/stack/overcloud_upgrade_run-ceph-0,controller-0.log.gz

2021-05-04 13:35:51 | TASK [Transfer mysql data] *****************************************************
2021-05-04 13:35:51 | Tuesday 04 May 2021  13:35:35 +0000 (0:00:00.062)       0:02:59.617 *********** 
2021-05-04 13:35:51 | 
2021-05-04 13:35:51 | TASK [tripleo-transfer : install requirements in src and dest hosts] ***********
2021-05-04 13:35:51 | Tuesday 04 May 2021  13:35:35 +0000 (0:00:00.173)       0:02:59.790 *********** 
2021-05-04 13:35:51 | ok: [undercloud -> 192.168.24.22] => (item=controller-1) => {"ansible_loop_var": "item", "changed": false, "item": "controller-1", "msg": "", "rc": 0, "results": ["rsync-3.1.2-10.el7.x86_64 providing rsync is already installed", "openssh-clients-7.4p1-21.el7.x86_64 providing openssh-clients is already installed"]}
2021-05-04 13:35:51 | ok: [undercloud -> 192.168.24.24] => (item=controller-0) => {"ansible_loop_var": "item", "changed": false, "item": "controller-0", "msg": "Nothing to do", "rc": 0, "results": []}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : generate ssh key-pair in source host] *****************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:51 +0000 (0:00:15.927)       0:03:15.717 *********** 
2021-05-04 13:36:08 | changed: [undercloud -> 192.168.24.22] => {"changed": true, "comment": "", "filename": "/root/transfer_key", "fingerprint": "SHA256:p3DPYEcb/c1z7Sh+S/F4dYRaU1P0ADTJfMrhMgwYBeg", "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQGgo3YKO/RIYAnSNjir1gxbwX+L5y+6VLu6t0b0LWTfG+PJMS/QeDG7ysLFBSly9sandJlkHi+vvCnauFBvirc35e+V+ShG+t5fEtcpSOaNxlZoilEvw8M/IyDQeI2PTeMg7+qc2sMZJI5EOOHF6VtUXnKcxoZgBE4USpWlv9Kujl46g4y63wicazIG22Lbaeku1jSzrmyXw+gX19rQd09p6q7MANjwfZ/1T9xn0mkwOxlkJVG9M801ZI5Atm45erPFdVX427h1FbUSigYFDsbcqOiBaUhhtuhbO3YqXJEh66dfacjEMpMoangdSwP0Ph1sQvXy/IT0T0vCM6noFp6dZMRfZ41YebNwo9pKb0ATvT80IsLBJ6acD5P9q2wfhYbDcVvL5ZLZ1sgdTzlBuwP4PKcWXgQEQx9SPBNO2U7VI0LIWBh0oOo0ZszlFTcnnOpHNLB/2e4upmXuPkXp9kSDA5S2hRuo/uPnohSwZWs94btSBRRXLbtNWN3MYqCHLx2pNySGub7GU3OQ2hSHl40JTsVDG3zGuqsrQawDRSbkrzf6rImVWhh+sm8lm5AoCtB5B518YmxxowwfKuRUEIOPhQFfFrT4nDCvDzsX5Bylg9/o6yb4HmAvhcKSYPfHOcQoODS/C8Zx54BtQ1q2ZwV56tYfYoUcyatzM5uweTbw==", "size": 4096, "type": "rsa"}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : set authorized-keys in destination host] **************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:52 +0000 (0:00:00.947)       0:03:16.665 *********** 
2021-05-04 13:36:08 | changed: [undercloud -> 192.168.24.24] => {"changed": true, "comment": "Added by tripleo-transfer", "exclusive": false, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQGgo3YKO/RIYAnSNjir1gxbwX+L5y+6VLu6t0b0LWTfG+PJMS/QeDG7ysLFBSly9sandJlkHi+vvCnauFBvirc35e+V+ShG+t5fEtcpSOaNxlZoilEvw8M/IyDQeI2PTeMg7+qc2sMZJI5EOOHF6VtUXnKcxoZgBE4USpWlv9Kujl46g4y63wicazIG22Lbaeku1jSzrmyXw+gX19rQd09p6q7MANjwfZ/1T9xn0mkwOxlkJVG9M801ZI5Atm45erPFdVX427h1FbUSigYFDsbcqOiBaUhhtuhbO3YqXJEh66dfacjEMpMoangdSwP0Ph1sQvXy/IT0T0vCM6noFp6dZMRfZ41YebNwo9pKb0ATvT80IsLBJ6acD5P9q2wfhYbDcVvL5ZLZ1sgdTzlBuwP4PKcWXgQEQx9SPBNO2U7VI0LIWBh0oOo0ZszlFTcnnOpHNLB/2e4upmXuPkXp9kSDA5S2hRuo/uPnohSwZWs94btSBRRXLbtNWN3MYqCHLx2pNySGub7GU3OQ2hSHl40JTsVDG3zGuqsrQawDRSbkrzf6rImVWhh+sm8lm5AoCtB5B518YmxxowwfKuRUEIOPhQFfFrT4nDCvDzsX5Bylg9/o6yb4HmAvhcKSYPfHOcQoODS/C8Zx54BtQ1q2ZwV56tYfYoUcyatzM5uweTbw==", "key_options": null, "keyfile": "/home/tripleo-admin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tripleo-admin", "validate_certs": true}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : check flag file existence in destination host] ********
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:53 +0000 (0:00:00.744)       0:03:17.409 *********** 
2021-05-04 13:36:08 | ok: [undercloud -> 192.168.24.24] => {"changed": false, "stat": {"exists": false}}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : fail if flag file exists] *****************************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:53 +0000 (0:00:00.594)       0:03:18.004 *********** 
2021-05-04 13:36:08 | skipping: [undercloud] => {"changed": false, "skip_reason": "Conditional result was False"}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : ensure directory for flag file exists] ****************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:53 +0000 (0:00:00.094)       0:03:18.099 *********** 
2021-05-04 13:36:08 | changed: [undercloud -> 192.168.24.24] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/tripleo/transfer-flags", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 6, "state": "directory", "uid": 0}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : create the flag file] *********************************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:54 +0000 (0:00:00.595)       0:03:18.695 *********** 
2021-05-04 13:36:08 | changed: [undercloud -> 192.168.24.24] => {"changed": true, "dest": "/var/lib/tripleo/transfer-flags/var-lib-mysql", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 0, "state": "file", "uid": 0}
2021-05-04 13:36:08 | 
2021-05-04 13:36:08 | TASK [tripleo-transfer : synchronize both directories] *************************
2021-05-04 13:36:08 | Tuesday 04 May 2021  13:35:54 +0000 (0:00:00.399)       0:03:19.094 *********** 
2021-05-04 13:36:08 | changed: [undercloud -> 192.168.24.22] => {"changed": true, "cmd": "/usr/bin/rsync -v --delay-updates -F --ignore-times --compress --archive --delete --rsync-path='sudo rsync' --rsh='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/transfer_key' /var/lib/mysql/ tripleo-admin.24.24:/var/lib/mysql", "delta": "0:00:13.465667", "end": "2021-05-04 13:36:08.538203", "rc": 0, "start": "2021-05-04 13:35:55.072536", "stderr": "Warning: Permanently added '192.168.24.24' (ECDSA) to the list of known hosts.", "stderr_lines": ["Warning: Permanently added '192.168.24.24' (ECDSA) to the list of known hosts."], "stdout": "building file list ... done\n./\naria_log.00000001\naria_log_control\ngalera.cache\ngrastate.dat\nib_logfile0\nib_logfile1\nibdata1\nmulti-master.info\naodh/alarm.frm\naodh
......................................................................................................
"panko/trait_int.frm", "panko/trait_int.ibd", "panko/trait_text.frm", "panko/trait_text.ibd", "performance_schema/", "performance_schema/db.opt", "", "sent 26,566,966 bytes  received 1,007,006 bytes  2,042,516.44 bytes/sec", "total size is 425,033,003  speedup is 15.41"]}
2021-05-04 13:36:11 | 
2021-05-04 13:36:11 | 
2021-05-04 13:36:11 | TASK [tripleo-transfer : clean up keys in source host] *************************
2021-05-04 13:36:11 | Tuesday 04 May 2021  13:36:08 +0000 (0:00:13.948)       0:03:33.044 *********** 
2021-05-04 13:36:11 | changed: [undercloud -> 192.168.24.22] => (item=~/transfer_key) => {"ansible_loop_var": "item", "changed": true, "item": "~/transfer_key", "path": "/root/transfer_key", "state": "absent"}
2021-05-04 13:36:11 | changed: [undercloud -> 192.168.24.22] => (item=~/transfer_key.pub) => {"ansible_loop_var": "item", "changed": true, "item": "~/transfer_key.pub", "path": "/root/transfer_key.pub", "state": "absent"}
2021-05-04 13:36:11 | 
2021-05-04 13:36:11 | TASK [tripleo-transfer : remove public key from authorized keys in destination host] ***
2021-05-04 13:36:11 | Tuesday 04 May 2021  13:36:09 +0000 (0:00:00.650)       0:03:33.694 *********** 
2021-05-04 13:36:11 | changed: [undercloud -> 192.168.24.24] => {"backup": "", "changed": true, "found": 1, "msg": "1 line(s) removed"}

PACKAGES:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-HA/140/undercloud-0/var/log/dnf.rpm.log.gz

2021-05-04T11:48:15Z SUBDEBUG Installed: tripleo-ansible-0.5.1-1.20210323173506.el8ost.noarch
2021-05-04T11:48:15Z SUBDEBUG Installed: openstack-tripleo-heat-templates-11.3.2-1.20210408163452.el8ost.noarch

Comment 34 errata-xmlrpc 2021-05-26 11:43:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenStack Platform 16.1.6 (tripleo-ansible) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2119


Note You need to log in before you can comment on or make changes to this bug.