Bug 1723747 - Swift container fails when the network file doesn't exist in openstack overcloud deploy command
Summary: Swift container fails when the network file doesn't exist in openstack overcl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 15.0 (Stein)
Assignee: Cédric Jeanneret
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-25 09:33 UTC by Sasha Smolyak
Modified: 2019-09-26 10:52 UTC (History)
6 users (show)

Fixed In Version: python-tripleoclient-11.4.1-0.20190702170413.d009dd7.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:34 UTC
Target Upstream Version:
cjeanner: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 667562 0 None MERGED Ensure files exist before building/uploading plan 2020-03-02 09:44:48 UTC
OpenStack gerrit 668596 0 None MERGED Ensure files exist before building/uploading plan 2020-03-02 09:44:48 UTC
OpenStack gerrit 668689 0 None MERGED Ensure files exist before building/uploading plan 2020-03-02 09:44:48 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:51 UTC

Internal Links: 1726321 1726322

Description Sasha Smolyak 2019-06-25 09:33:58 UTC
Description of problem:
When running openstack overcloud deploy -n /home/stack/virt/network/network_data_1.yaml and the file network_data_1.yaml doesn't exist, get a big error:

Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 400, in _deploy_tripleo_heat_templates
    validate_stack=False)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 206, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 293, in _load_passwords
    return plan_env['passwords']
KeyError: 'passwords'

because the command is trying to build a plan and fails, without validating first that the file exists.

Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190619.n.1

How reproducible:
100%

Steps to Reproduce:
1. deploy undercloud
2. Try to deploy overcloud with a non-existing network file

Actual results:
Deploy fails because of empty plan

Expected results:
Error explaining that the file doesn't exist

Additional info:

Comment 1 Cédric Jeanneret 2019-06-25 12:56:36 UTC
Heya,

so basically, we need a pre-flight check that will loop over environment files passed as argument and ensure they are all existing. It shouldn't be that hard to add, I'll have a look. this week.

Cheers,

C.

Comment 2 Sasha Smolyak 2019-06-25 14:53:54 UTC
So. It didn't happen on the latest puddle (RHOS_TRUNK-15.0-RHEL-8-20190621.n.1) at the first time, got an error that "The following files were not found: /home/stack/virt/network/network_data_1.yaml"
And then when I've tried once more, I got exactly the "passwords" error again.

Scenario:
1. Try to deploy with a non-existing network file:
(undercloud) [stack@undercloud-0 ~]$ ./overcloud_deploy.sh 
Creating Swift container to store the plan
Creating plan from template files in: /tmp/tripleoclient-nwyg8t6l/tripleo-heat-templates
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 407, in _deploy_tripleo_heat_templates
    validate_stack=False)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 169, in create_plan_from_templates
    plan_env_file, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 55, in _upload_templates
    constants.OVERCLOUD_NETWORKS_FILE, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 280, in _upload_file
    with open(local_filename, 'rb') as file_content:
FileNotFoundError: [Errno 2] No such file or directory: '/home/stack/virt/network_data_1.yaml'
[Errno 2] No such file or directory: '/home/stack/virt/network_data_1.yaml'

2. Without fixing the file, try to run the command again:
(undercloud) [stack@undercloud-0 ~]$ ./overcloud_deploy.sh 
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 400, in _deploy_tripleo_heat_templates
    validate_stack=False)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 206, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 293, in _load_passwords
    return plan_env['passwords']
KeyError: 'passwords'
'passwords'

Comment 3 Cédric Jeanneret 2019-06-26 06:49:28 UTC
Can't reproduce it:

[stack@undercloud ~]$ ./deploy-overcloud.sh
Error: The following files were not found: /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml
[stack@undercloud ~]$ ./deploy-overcloud.sh 
Output env file exists, moving it to backup.
Error: The following files were not found: /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml
[stack@undercloud ~]$ ./deploy-overcloud.sh 
Output env file exists, moving it to backup.
Error: The following files were not found: /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml

Script:
openstack overcloud deploy \
  --templates /usr/share/openstack-tripleo-heat-templates/ \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
  --environment-directory ~/overcloud-yml \
  --ntp-server pool.ntp.org --libvirt-type qemu 2>&1 | tee -a ~/install-overcloud.log


This is with latest-RHOS_TRUNK-15-RHEL-8 providing
python3-tripleo-common-10.7.1-0.20190625154433.5751fc7.el8ost.noarch                                                                                                                                                                          
python3-tripleoclient-11.4.1-0.20190611084233.f36d899.el8ost.noarch                                                                                                                                                                           
openstack-tripleo-common-10.7.1-0.20190625154433.5751fc7.el8ost.noarch

The type of the file doesn't count imho (being a network env file, a telemetry env file or whatever).

Care to recheck?

Cheers,

C.

Comment 4 Cédric Jeanneret 2019-06-26 08:54:49 UTC
Quick update:
network env is passed as "-n", not "-e".

In this case, indeed, the plan is created and uploaded without any check on the existence of the passed file for network.

Complete output in this case (first run):
Output env file exists, moving it to backup.
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 400, in _deploy_tripleo_heat_templates
    validate_stack=False)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 232, in update_plan_from_templates
    plan_env_file, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 55, in _upload_templates
    constants.OVERCLOUD_NETWORKS_FILE, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 280, in _upload_file
    with open(local_filename, 'rb') as file_content:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml'
[Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry-11.yaml'
Removing the current plan files
Uploading new plan files

And there, since the plan files are uploaded to swift, we hit the issue with the next run, with or without plan correction:
Output env file exists, moving it to backup.
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 400, in _deploy_tripleo_heat_templates
    validate_stack=False)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 206, in update_plan_from_templates
    passwords = _load_passwords(swift_client, name)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 293, in _load_passwords
    return plan_env['passwords']
KeyError: 'passwords'
'passwords'


Now I know what to look for :).

Comment 5 Cédric Jeanneret 2019-06-26 09:27:21 UTC
Correction pushed upstream.

Comment 7 Cédric Jeanneret 2019-07-02 05:38:51 UTC
Upstream backport to Stein added.

Comment 9 Cédric Jeanneret 2019-07-02 15:15:29 UTC
patch merged upstream stable/stein - it still needs to go down to queens upstream. Upstream Rocky backport added

Comment 10 Cédric Jeanneret 2019-07-03 05:36:33 UTC
Pushing right component..

Comment 16 Sasha Smolyak 2019-07-15 10:50:06 UTC
Checked with non-existing files of type: --plan-environment-file --answers-file --networks-file
Tried several times, then deployed a correct configuration. Deployment passed. Verified.

Comment 18 errata-xmlrpc 2019-09-21 11:23:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.