Bug 1380670

Summary: Deployment fails: Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/puppet/post.yaml
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: python-tripleoclientAssignee: mathieu bultel <mbultel>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: agurenko, dbecker, hbrock, ipetrova, jcoufal, jschluet, jslagle, mbultel, mburns, morazi, nsantos, ohochman, rhel-osp-director-maint, sathlang
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-5.2.0-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-21 10:11:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1337797    

Description Marius Cornea 2016-09-30 09:57:18 UTC
Description of problem:
Deployment fails with the following message:

Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/puppet/post.yaml

/usr/share/openstack-tripleo-heat-templates/puppet/post.yaml doesn't exist but /usr/share/openstack-tripleo-heat-templates/puppet/post.j2.yaml exists

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-0.20160922100830.75c20d5.1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates
openstack overcloud deploy --templates \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/storage-environment.yaml \
-e ~/templates/disk-layout.yaml \
--control-scale 1 \
--control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \
--compute-scale 1 \
--compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
--ntp-server clock.redhat.com  


Actual results:
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/puppet/post.yaml

Expected results:
Deployment succeds.

Additional info:

Comment 1 Sofer Athlan-Guyot 2016-09-30 10:07:24 UTC
Adding upstream bug.

Comment 2 Omri Hochman 2016-09-30 12:40:37 UTC
discussion & possible workaround (by: Bandini): 
"
Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml
https://bugs.launchpad.net/tripleo/+bug/1624727 

The most annoying one at the moment is this one. You basically need to
"cd <tht-folder>" and then run "swift download overcloud" otherwise the
deploy will fail. This one is also blocking the CI upgrade job
implementation.  Hopefully we can unblock this one soon."

Comment 3 Omri Hochman 2016-10-04 11:44:42 UTC
*** Bug 1381525 has been marked as a duplicate of this bug. ***

Comment 5 Sofer Athlan-Guyot 2016-10-05 13:00:37 UTC
I've tested the upstream review on OSP10 during upgrade, using: 


    curl https://review.openstack.org/changes/379547/revisions/current/patch?download | \
    base64 -d | \
    sudo patch  -d /usr/lib/python2.7/site-packages/ -p1

And the init stage passes fine afterwards.

Comment 6 Omri Hochman 2016-10-06 01:11:20 UTC
(In reply to Sofer Athlan-Guyot from comment #5)

^^ worked for me as well

Comment 7 Sofer Athlan-Guyot 2016-10-06 09:19:49 UTC
Just to add that yesterday the upstream patch was rebase against a code base that make the latest review patch fails downstream.

The current working patch is stuck at revision 17:



    curl https://review.openstack.org/changes/379547/revisions/17/patch?download | \
    base64 -d | \
    sudo patch  -d /usr/lib/python2.7/site-packages/ -p1

Will track it to check when code upstream make its way downstream so that we can have the latest again.

Comment 8 Sofer Athlan-Guyot 2016-10-06 10:06:03 UTC
Oki, I've identified the missing upstream commit that makes the patching process fails.  I'm adding it to the tracker as it's required to have this patch.

The current workaround is then:

    curl https://review.openstack.org/changes/378740/revisions/a6221608bd7641d7eeac6ad3733d37e51989d9f9/patch?download | \
        base64 -d | \
        sudo patch  -d /usr/lib/python2.7/site-packages/ -p1

    curl https://review.openstack.org/changes/379547/revisions/current/patch?download | \
        base64 -d | \
        sudo patch  -d /usr/lib/python2.7/site-packages/ -p1


The change 378740 made its way to stable/newton on the 6th of October.
So while it makes its way to the puddle, we can have access to the
latest upstream 379547 patch using both patches.

Comment 9 James Slagle 2016-10-10 15:22:40 UTC
patches merged to newton

Comment 13 errata-xmlrpc 2016-12-14 16:06:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Comment 14 Irina Petrova 2017-04-20 17:41:02 UTC
Hi...

It looks like we have a regression:
python-tripleoclient-5.4.1-1.el7ost.noarch                  Mon Apr  3 16:31:19 2017


// Basically:

/usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml     << Does NOT exist
/usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.j2.yaml  << DOES exist

- vs - [originally in this BZ]:

/usr/share/openstack-tripleo-heat-templates/puppet/post.yaml doesn't exist but 
/usr/share/openstack-tripleo-heat-templates/puppet/post.j2.yaml exists


// In details:

>> Step 3.4.4. Installing the Upgrade Scripts [1]
~~~
Processing environment files /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
Error Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml processing environment file /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
Rewriting /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml environment to /tmp/tripleoclient-Th1Z6p/tripleo-heat-templates/env-major-upgrade-pacemaker-init-tz8kja.yaml
Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1183, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 403, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 469, in _deploy_tripleo_heat_templates
    cleanup=not parsed_args.no_cleanup)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 203, in _process_multiple_environments
    env_path=f.name)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 308, in process_environment_and_files
    env_base_url)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 346, in resolve_environment_urls
    is_object=is_object, object_request=object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 159, in get_file_contents
    file_content = utils.read_url_content(str_url)
  File "/usr/lib/python2.7/site-packages/heatclient/common/utils.py", line 341, in read_url_content
    raise exc.CommandError(_('Could not fetch contents for %s') % url)
CommandError: Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml
clean_up DeployOvercloud: Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1183, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 403, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 469, in _deploy_tripleo_heat_templates
    cleanup=not parsed_args.no_cleanup)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 203, in _process_multiple_environments
    env_path=f.name)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 308, in process_environment_and_files
    env_base_url)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 346, in resolve_environment_urls
    is_object=is_object, object_request=object_request)
  File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 159, in get_file_contents
    file_content = utils.read_url_content(str_url)
  File "/usr/lib/python2.7/site-packages/heatclient/common/utils.py", line 341, in read_url_content
    raise exc.CommandError(_('Could not fetch contents for %s') % url)
CommandError: Could not fetch contents for file:///usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml

END return value: 1
~~~

~~~
[stack@... ~]$ ls -l /usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml
ls: cannot access /usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_init.yaml: No such file or directory

[stack@... ~]$ cd /usr/share/openstack-tripleo-heat-templates/extraconfig/tasks/
[stack@... tasks]$ ls
aodh_data_migration.sh                            major_upgrade_check.sh                   major_upgrade_controller_pacemaker_4.sh  major_upgrade_pacemaker_migrations.sh        pacemaker_maintenance_mode.sh       yum_update_noop.yaml
major_upgrade_block_storage.sh                    major_upgrade_compute.sh                 major_upgrade_controller_pacemaker_5.sh  major_upgrade_pacemaker.yaml                 pacemaker_resource_restart.sh       yum_update.sh
major_upgrade_ceilometer_wsgi_mitaka_newton.yaml  major_upgrade_controller_pacemaker_1.sh  major_upgrade_controller_pacemaker_6.sh  mitaka_to_newton_aodh_data_migration.yaml    post_puppet_pacemaker_restart.yaml  yum_update.yaml
major_upgrade_ceph_mon.sh                         major_upgrade_controller_pacemaker_2.sh  major_upgrade_object_storage.sh          mitaka_to_newton_ceilometer_wsgi_upgrade.pp  post_puppet_pacemaker.yaml
major_upgrade_ceph_storage.sh                     major_upgrade_controller_pacemaker_3.sh  major_upgrade_pacemaker_init.j2.yaml     pacemaker_common_functions.sh                pre_puppet_pacemaker.yaml
~~~


[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/upgrading_red_hat_openstack_platform/chap-upgrading_the_environment#sect-Major-Upgrading_the_Overcloud-Init

Comment 15 mathieu bultel 2017-04-21 07:40:13 UTC
Hi Irina,

Could you paste the deploy command please ?
Basically, this issue could appear if there is some mismatch between the path of the environment file provived, the cli could be a bit intolerant in some cases.

Comment 16 Irina Petrova 2017-04-21 09:06:25 UTC
Hi Mathieu, sure:

~~~
openstack --debug overcloud deploy \
  --templates /home/stack/templates/Isabel \
  -e /home/stack/templates/Isabel/environments/network-environment.yaml \
  -e /home/stack/templates/Isabel/environments/timezone.yaml \
  -e /home/stack/templates/Isabel/environments/scheduler_hints_env.yaml \
  -e /home/stack/templates/Isabel/environments/customIsabelConfig.yaml \
  -e /home/stack/templates/Isabel/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml \
  -e /home/stack/templates/Isabel/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \
  --control-scale 3 \
  --compute-scale 5 \
  --ceph-storage-scale 0 \
  --control-flavor baremetal \
  --ceph-storage-flavor baremetal \
  --ntp-server 192.168.xxx.xxx \
  --neutron-network-type vxlan \
  --neutron-tunnel-types vxlan
~~~

Comment 17 mathieu bultel 2017-04-21 09:27:59 UTC
Thank you Irina for details.
So i think the issue comes from the deploy argument:

the template dir is : --templates /home/stack/templates/Isabel
but the customer provide also /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 
which doesn't exist as is, because it's a jinja template.
The cli would only render the template under /home/stack/templates/Isabel and not the environment file outside.
So you have to change the path for:
/usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
to:
/home/stack/templates/Isabel/environments/major-upgrade-pacemaker-init.yaml
and it should work.

Comment 18 Irina Petrova 2017-04-21 10:11:53 UTC
Mathieu, many thanks for the prompt response.

It worked as per your suggestion.

I'm moving this BZ back to CLOSED.