Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1349492 - [upgrade] Provisioning failed with Error 403 Forbidden and unable to fetch the boot images.
Summary: [upgrade] Provisioning failed with Error 403 Forbidden and unable to fetch th...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Sachin Ghai
URL:
Whiteboard:
: 1287518 (view as bug list)
Depends On:
Blocks: 1335807
TreeView+ depends on / blocked
 
Reported: 2016-06-23 14:24 UTC by Sachin Ghai
Modified: 2021-04-06 17:57 UTC (History)
14 users (show)

Fixed In Version: pulp-rpm-2.8.3.2-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:11:45 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
console logs says failed to fetch squashfs.ks (245.86 KB, image/png)
2016-06-23 14:25 UTC, Sachin Ghai 		no flags Details
foreman-debug (1.24 MB, application/x-xz)
2016-06-23 14:30 UTC, Sachin Ghai 		no flags Details
screenshot of provisioned rhel7 host (72.94 KB, image/png)
2016-07-07 08:06 UTC, Sachin Ghai 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 1972 0 High CLOSED - CURRENTRELEASE migration 28 misses distribution symlinks 2016-06-27 17:00:20 UTC
Pulp Redmine 2056 0 High CLOSED - CURRENTRELEASE Distribution storage path migration re-links .treeinfo files incorrectly. 2016-07-18 19:30:28 UTC

Description Sachin Ghai 2016-06-23 14:24:15 UTC 
Description of problem:
I got following issue while upgrading sat6.1.9 -> sat6.2 GA snap17 on rhel7. 

    2016-06-23 09:52:21 [app] [I] Processing by HostsController#create as */*
    2016-06-23 09:52:21 [app] [I]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"gPWXf1mNZYT+6AWf1+5eOHgM1MjFIUUtCuK40X+1QUc=", "host"=>{"name"=>"hossst", "organization_id"=>"1", "location_id"=>"2", "hostgroup_id"=>"9", "compute_resource_id"=>"2", "content_facet_attributes"=>{"lifecycle_environment_id"=>"2", "content_view_id"=>"2", "id"=>"", "kickstart_repository_id"=>"47"}, "environment_id"=>"2", "content_source_id"=>"1", "openscap_proxy_id"=>"", "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"hossst", "domain_id"=>"1", "subnet_id"=>"12", "ip"=>"10.8.119.8", "managed"=>"1", "primary"=>"1", "provision"=>"1", "execution"=>"1", "virtual"=>"0", "tag"=>"", "attached_to"=>"", "compute_attributes"=>{"name"=>"nic1", "network"=>"6b5c3ecd-6738-43eb-944a-9573b9d12937"}}}, "compute_attributes"=>{"cluster"=>"00000001-0001-0001-0001-00000000022f", "template"=>"00000000-0000-0000-0000-000000000000", "cores"=>"1", "memory"=>"1073741824", "start"=>"1", "volumes_attributes"=>{"0"=>{"_delete"=>"", "size_gb"=>"8", "storage_domain"=>"af918214-e6b2-4a01-a960-73dbc06a7073", "id"=>"", "preallocate"=>"0"}}}, "architecture_id"=>"1", "operatingsystem_id"=>"1", "provision_method"=>"build", "build"=>"1", "ptable_id"=>"63", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"3-Users", "enabled"=>"1", "model_id"=>"", "comment"=>"", "overwrite"=>"false"}, "capabilities"=>"build image", "provider"=>"Ovirt", "media_selector"=>"synced_content", "bare_metal_capabilities"=>"build"}
    2016-06-23 09:52:28 [app] [I] Create DHCP reservation for hossst.satqe.lab.eng.rdu2.redhat.com-00:1a:4a:3e:a6:2b/10.8.119.8
    2016-06-23 09:52:28 [app] [I] Add DNS A record for hossst.satqe.lab.eng.rdu2.redhat.com/10.8.119.8
    2016-06-23 09:52:28 [app] [I] Add DNS PTR record for 10.8.119.8/hossst.satqe.lab.eng.rdu2.redhat.com
     
    ==> /var/log/foreman-proxy/proxy.log <==
    E, [2016-06-23T09:52:30.199918 #21122] ERROR -- : [14366] http://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/streamer/var/lib/pulp/content/distribution/ks-Red%20Hat%20Enterprise%20Linux-Server-7.2-x86_64/images/pxeboot/vmlinuz?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTAuOC4xMTkuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L2Rpc3RyaWJ1dGlvbi9rcy1SZWQgSGF0IEVudGVycHJpc2UgTGludXgtU2VydmVyLTcuMi14ODZfNjQvaW1hZ2VzL3B4ZWJvb3Qvdm1saW51eiIsICJleHBpcmF0aW9uIjogMTQ2NjY5MDAzOX0%3D;signature=Wg2zPQhaBWqPXOQjy6vKtHozRiC-VhnaDSsm4A35E-Y4FYFy-bmNIoHddzGlv2EtnaEkyERczeVbDcHuVRAvqA-7I-wru0jgzg6lEZroTmCiNbTNyk-3DXmRFLKHfNPnkoPebDahh0QW8risJOzFsqUW3XO86xE1Knx7byy2sKaMQXX5w8TmOmQHi2xHIjc0jiqow3CHmByB6aSUxVZej7BNr_fE7HPmRxxMibCecCH3eR-wq69bdx0WyIbC583X23AuXg9WJmKYo-1T8XMURW1WLgflqSR542H4yRyrgOLcgEUKo9n9Tj_Maq0XC1z1FHuM-oQWyG3stRaTH3VcXg%3D%3D:
     
    E, [2016-06-23T09:52:30.201993 #21122] ERROR -- : [14366] 2016-06-23 09:52:30 ERROR 403: Forbidden.
     
    E, [2016-06-23T09:52:30.202646 #21122] ERROR -- : [14372] http://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/streamer/var/lib/pulp/content/distribution/ks-Red%20Hat%20Enterprise%20Linux-Server-7.2-x86_64/images/pxeboot/initrd.img?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTAuOC4xMTkuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L2Rpc3RyaWJ1dGlvbi9rcy1SZWQgSGF0IEVudGVycHJpc2UgTGludXgtU2VydmVyLTcuMi14ODZfNjQvaW1hZ2VzL3B4ZWJvb3QvaW5pdHJkLmltZyIsICJleHBpcmF0aW9uIjogMTQ2NjY5MDAzOX0%3D;signature=PXxZFkJercMUWzVlKa-BH_6izRjpt-wvcqPMmyeK_u-NkGzoxXp4N0-KFE8faQfiVo22Q0V1oCKwvgWajGtJu1XdnjA7Wwtng9mVpKQ8sEBpiMH1NcwSoKA9-LoNpb1BSiF2LXJsh3WE2N328u2wm7SMqwrNjnXwGdRQ4SQ0CWtkW7YZJ8wsDeJ_eFC5JhaYTfmCsD38NbOzPvIy5tUEch-a8gu-J1W61A_6YxKH5OnTbdMZOewJFiwlY8m8OQAxiztD513qlvPh7UYVqrcsfei-nuNWOmya9_Ve-l9nOQfDtfUOTfRPDlks8gZPHfnDfposCu7wQh5auv-Vuht-cQ%3D%3D:
     
    E, [2016-06-23T09:52:30.202895 #21122] ERROR -- : [14372] 2016-06-23 09:52:30 ERROR 403: Forbidden.


Version-Release number of selected component (if applicable):
upgrading sat6.1.9 -> sat6.2 GA snap17 on rhel7. 


How reproducible:


Steps to Reproduce:
1. Create new host after upgrade ( associated hostgroup of rhel7)
2.
3.

Actual results:

Expected results:

provisioning should work

Additional info:

no firewall rules
selinux in enforcing mode
download policy set to immediate
Comment 1 Sachin Ghai 2016-06-23 14:25:31 UTC 
Created attachment 1171548 [details]
console logs says failed to fetch squashfs.ks
Comment 2 Sachin Ghai 2016-06-23 14:30:11 UTC 
Created attachment 1171549 [details]
foreman-debug
Comment 3 Brad Buckingham 2016-06-23 14:50:07 UTC 
Looking at the attached screenshot from comment 1, the file it is attempting to fetch has a broken sym link.

The broken link is:
/var/lib/pulp/published/yum/http/repos/Default_Organization/DEV/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img

Doing a find for broken links we can see many:
# find -L /var/lib/pulp/published/yum/http/repos/ -type l -! -exec test -e {} \; -print|grep LiveOS
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/rhel7/rhel7_repo/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/1.0/custom/rhel7/rhel7_repo/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/2.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/3.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/4.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_cv/1.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
/var/lib/pulp/published/yum/http/repos/Default_Organization/DEV/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
Comment 4 Brad Buckingham 2016-06-23 14:53:25 UTC 
Michael,

Can you help identify why the sym links may be broken?

Note: if you execute the command in comment 3 (without the grep), you'll see that there are far more broken links than listed in that comment.
Comment 6 pulp-infra@redhat.com 2016-06-23 17:36:07 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2016-06-23 17:36:10 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 8 Brad Buckingham 2016-06-23 17:40:12 UTC 
*** Bug 1287518 has been marked as a duplicate of this bug. ***
Comment 10 pulp-infra@redhat.com 2016-06-24 13:30:25 UTC 
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2016-06-27 17:00:21 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 12 Sachin Ghai 2016-06-30 09:05:53 UTC 
Reproducible with upgrade from sat6.1.9 -> sat62 GA snap18.1

I'm not able to provision neither rhel6 nor rhel7 host and getting 403 error while fetching boot images
Comment 17 Brad Buckingham 2016-07-01 03:05:04 UTC 
Hi Jeff,

After testing with rpms including the fix from issue 1972, I still see several files with broken sym links (all .treeinfo files).   Is this expected?

Note: prior to the Satellite/Pulp upgrade, there were no files with broken sym links.

E.g.

[root@sat61-rhel7 pulp]# find -L /var/lib/pulp/published/yum/http/repos/ -type l -! -exec test -e {} \; -print
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/capsule/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/satellite_tools/el6_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/tools_el6/custom/satellite_tools/el6_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/tools_el7/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_app_el6/custom/satellite_tools/el6_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_app_el7/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/tools_el6/1.0/custom/satellite_tools/el6_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/tools_el7/1.0/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_app_el6/1.0/custom/satellite_tools/el6_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_app_el7/1.0/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/dev/composite_app_el7/custom/satellite_tools/el7_x86_64/.treeinfo
/var/lib/pulp/published/yum/http/repos/Default_Organization/dev/composite_app_el6/custom/satellite_tools/el6_x86_64/.treeinfo
Comment 18 Jeff Ortel 2016-07-01 16:33:47 UTC 
No, the .treeinfo link should be migrated.  Looks like the migration did re-link the "<path>/.treeinfo" file but to a non-existent "<new-path>/treeinfo".  Looks like a 1 line code change upstream to do a stricter comparison.  I will file an issue upstream and get it fixed today.
Comment 19 Brad Buckingham 2016-07-01 17:21:02 UTC 
Moving the state back to ASSIGNED as we'll need the additional fix from comment 18, before it is moved over to QE.
Comment 21 pulp-infra@redhat.com 2016-07-01 22:30:20 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 22 pulp-infra@redhat.com 2016-07-01 22:30:23 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 23 Sachin Ghai 2016-07-07 08:04:10 UTC 
Verified with sat62 GA snap19.

I can provision host with snap19 without any issue.
Comment 25 Sachin Ghai 2016-07-07 08:06:18 UTC 
Created attachment 1177211 [details]
screenshot of provisioned rhel7 host
Comment 26 pulp-infra@redhat.com 2016-07-12 20:30:54 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 27 pulp-infra@redhat.com 2016-07-18 19:30:29 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 28 Bryan Kearney 2016-07-27 11:11:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501
Note You need to log in before you can comment on or make changes to this bug.