Bug 1563783 - [OSP13] openstack undercloud backup fails over: /undercloud-backups/UC-backup-20180222101308.tar 413 Request Entity Too Large [first 60 chars of response] [NEEDINFO]
Summary: [OSP13] openstack undercloud backup fails over: /undercloud-backups/UC-backup...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Carlos Camacho
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-04 17:04 UTC by Omri Hochman
Modified: 2018-06-27 13:50 UTC (History)
8 users (show)

Fixed In Version: instack-undercloud-8.4.0-4.el7ost openstack-tripleo-common-8.6.1-6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:50:03 UTC
Target Upstream Version:
knylande: needinfo? (ccamacho)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1761412 None None None 2018-04-05 08:44:35 UTC
OpenStack gerrit 559024 None MERGED Allow uploading big files to swift (5GB) 2020-02-17 09:01:22 UTC
OpenStack gerrit 559349 None MERGED Enable missing support for large objects in Swift 2020-02-17 09:01:22 UTC
OpenStack gerrit 559927 None MERGED Enable missing support for large objects in Swift 2020-02-17 09:01:22 UTC
OpenStack gerrit 563977 None MERGED Allow uploading big files to swift (5GB) 2020-02-17 09:01:22 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:50:53 UTC

Description Omri Hochman 2018-04-04 17:04:10 UTC
[OSP13] openstack undercloud backup fails over: /undercloud-backups/UC-backup-20180222101308.tar 413 Request Entity Too Large  [first 60 chars of response] 


Environment: 
------------
instack-8.1.1-0.20180313084439.0d768a3.el7ost.noarch
instack-undercloud-8.4.0-2.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-0.20180327213843.f25e2d8.el7ost.noarch
python-swift-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-swift-container-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-swift-proxy-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-swift-object-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
python2-swiftclient-3.5.0-1.el7ost.noarch
openstack-swift-account-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
puppet-swift-12.3.1-0.20180221111701.5bdce0a.el7ost.noarch


Steps:
------

(1) Attempt to perform OSP13 Undercloud backup,
 according to instructions: 

https://review.openstack.org/#/c/558429/1/doc/source/install/controlplane_backup_restore/01_undercloud_backup.rst 

results command exist with the following failure:
--------------------------------------------------
(undercloud) [stack@undercloud75 ~]$
(undercloud) [stack@undercloud75 ~]$ openstack undercloud backup --add-path /etc/hosts \
>                               --add-path /var/log/ \
>                               --add-path /var/lib/glance/images/ \
>                               --add-path /srv/node/ \
>                               --add-path /etc/
Started Mistral Workflow tripleo.undercloud_backup.v1.backup. Execution ID: 3711cf24-716f-4686-ad80-93956852cc33
Waiting for messages on queue 'tripleo' with no timeout.




Undercloud backup finished with errors
Output: {u'execution': {u'created_at': u'2018-02-22 15:08:57',
                u'id': u'3711cf24-716f-4686-ad80-93956852cc33',
                u'input': {u'queue_name': u'tripleo',
                           u'sources_path': u'/etc/,/etc/hosts,/home/stack/,/srv/node/,/var/lib/glance/images/,/var/log/'},
                u'name': u'tripleo.undercloud_backup.v1.backup',
                u'params': {u'namespace': u''},
                u'spec': {u'description': u'This workflow will launch the Undercloud backup',
                          u'input': [{u'sources_path': u'/home/stack/'},
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'backup',
                          u'tags': [u'tripleo-common-managed'],
                          u'tasks': {u'cleanup_backup': {u'action': u'tripleo.undercloud.remove_temp_dir',
..
..
..
..
..
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'upload_backup': {u'action': u'tripleo.undercloud.upload_backup_to_swift',
                                                        u'input': {u'backup_path': u'<% $.backup_path.path %>'},
                                                        u'name': u'upload_backup',
                                                        u'on-error': u'send_message',
                                                        u'on-success': u'cleanup_backup',
                                                        u'publish': {u'message': u'<% task().result %>',
                                                                     u'status': u'SUCCESS'},
                                                        u'publish-on-error': {u'message': u'<% task().result %>',
                                                                              u'status': u'FAILED'},
                                                        u'type': u'direct',
                                                        u'version': u'2.0'}},
                          u'version': u'2.0'},
                u'updated_at': u'2018-02-22 15:08:57'},
 u'message': {u'msg': u'Object PUT failed: https://192.168.0.2:13808/v1/AUTH_5b0b3efc458a4fc4af8e21487556aeca/undercloud-backups/UC-backup-20180222101308.tar 413 Request Entity Too Large  [first 60 chars of response] <html><h1>Request Entity Too Large</h1><p>The body of your r'},
 u'status': u'FAILED'}
(undercloud) [stack@undercloud75 ~]$
(undercloud) [stack@undercloud75 ~]$
(undercloud) [stack@undercloud75 ~]$
(undercloud) [stack@undercloud75 ~]$


Checking the size of file and HardDisk ( seems there's enough space on HD):
---------------------------------------------------------------------------

[root@undercloud75 undercloud-backup-JjgPDr]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       104G   31G   74G  30% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G  4.0K  7.8G   1% /dev/shm
tmpfs           7.8G  788K  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/0
[root@undercloud75 undercloud-backup-JjgPDr]# ls -lah
total 8.2G
drwx------.  2 mistral mistral   86 Feb 22 10:09 .
drwxrwxrwt. 14 root    root    4.0K Feb 22 10:08 ..
-rw-r--r--.  1 mistral mistral  35M Feb 22 10:09 all-databases-20180222100900.sql.gz
-rw-r--r--.  1 mistral mistral 8.2G Feb 22 10:13 filesystem-20180222100929.tar

Comment 1 Carlos Camacho 2018-04-05 12:42:58 UTC
Both fixes are merged/backported

Comment 4 Omri Hochman 2018-04-20 18:54:54 UTC
Adding blocker flag -  As the backup/restore undercloud is a regression test for DFG:DF

Comment 10 Omri Hochman 2018-05-08 12:53:17 UTC
Verified : 
[stack@undercloud75 ~]$ source stackrc
(undercloud) [stack@undercloud75 ~]$ openstack undercloud backup --add-path /etc/hosts \
>                               --add-path /var/log/ \
>                               --add-path /var/lib/glance/images/ \
>                               --add-path /srv/node/ \
>                               --add-path /etc/

Started Mistral Workflow tripleo.undercloud_backup.v1.backup. Execution ID: 78716244-f8f4-49c5-b1e7-872bb3b4cb11
Waiting for messages on queue 'tripleo' with no timeout.
Undercloud Backup succeed
(undercloud) [stack@undercloud75 ~]$
(undercloud) [stack@undercloud75 ~]$ rpm -qa | grep instack
instack-undercloud-8.4.1-4.el7ost.noarch

Comment 13 errata-xmlrpc 2018-06-27 13:50:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.