Description of problem: I got following issue while upgrading sat6.1.9 -> sat6.2 GA snap17 on rhel7. 2016-06-23 09:52:21 [app] [I] Processing by HostsController#create as */* 2016-06-23 09:52:21 [app] [I] Parameters: {"utf8"=>"✓", "authenticity_token"=>"gPWXf1mNZYT+6AWf1+5eOHgM1MjFIUUtCuK40X+1QUc=", "host"=>{"name"=>"hossst", "organization_id"=>"1", "location_id"=>"2", "hostgroup_id"=>"9", "compute_resource_id"=>"2", "content_facet_attributes"=>{"lifecycle_environment_id"=>"2", "content_view_id"=>"2", "id"=>"", "kickstart_repository_id"=>"47"}, "environment_id"=>"2", "content_source_id"=>"1", "openscap_proxy_id"=>"", "puppetclass_ids"=>[""], "managed"=>"true", "progress_report_id"=>"[FILTERED]", "type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", "type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"hossst", "domain_id"=>"1", "subnet_id"=>"12", "ip"=>"10.8.119.8", "managed"=>"1", "primary"=>"1", "provision"=>"1", "execution"=>"1", "virtual"=>"0", "tag"=>"", "attached_to"=>"", "compute_attributes"=>{"name"=>"nic1", "network"=>"6b5c3ecd-6738-43eb-944a-9573b9d12937"}}}, "compute_attributes"=>{"cluster"=>"00000001-0001-0001-0001-00000000022f", "template"=>"00000000-0000-0000-0000-000000000000", "cores"=>"1", "memory"=>"1073741824", "start"=>"1", "volumes_attributes"=>{"0"=>{"_delete"=>"", "size_gb"=>"8", "storage_domain"=>"af918214-e6b2-4a01-a960-73dbc06a7073", "id"=>"", "preallocate"=>"0"}}}, "architecture_id"=>"1", "operatingsystem_id"=>"1", "provision_method"=>"build", "build"=>"1", "ptable_id"=>"63", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"3-Users", "enabled"=>"1", "model_id"=>"", "comment"=>"", "overwrite"=>"false"}, "capabilities"=>"build image", "provider"=>"Ovirt", "media_selector"=>"synced_content", "bare_metal_capabilities"=>"build"} 2016-06-23 09:52:28 [app] [I] Create DHCP reservation for hossst.satqe.lab.eng.rdu2.redhat.com-00:1a:4a:3e:a6:2b/10.8.119.8 2016-06-23 09:52:28 [app] [I] Add DNS A record for hossst.satqe.lab.eng.rdu2.redhat.com/10.8.119.8 2016-06-23 09:52:28 [app] [I] Add DNS PTR record for 10.8.119.8/hossst.satqe.lab.eng.rdu2.redhat.com ==> /var/log/foreman-proxy/proxy.log <== E, [2016-06-23T09:52:30.199918 #21122] ERROR -- : [14366] http://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/streamer/var/lib/pulp/content/distribution/ks-Red%20Hat%20Enterprise%20Linux-Server-7.2-x86_64/images/pxeboot/vmlinuz?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTAuOC4xMTkuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L2Rpc3RyaWJ1dGlvbi9rcy1SZWQgSGF0IEVudGVycHJpc2UgTGludXgtU2VydmVyLTcuMi14ODZfNjQvaW1hZ2VzL3B4ZWJvb3Qvdm1saW51eiIsICJleHBpcmF0aW9uIjogMTQ2NjY5MDAzOX0%3D;signature=Wg2zPQhaBWqPXOQjy6vKtHozRiC-VhnaDSsm4A35E-Y4FYFy-bmNIoHddzGlv2EtnaEkyERczeVbDcHuVRAvqA-7I-wru0jgzg6lEZroTmCiNbTNyk-3DXmRFLKHfNPnkoPebDahh0QW8risJOzFsqUW3XO86xE1Knx7byy2sKaMQXX5w8TmOmQHi2xHIjc0jiqow3CHmByB6aSUxVZej7BNr_fE7HPmRxxMibCecCH3eR-wq69bdx0WyIbC583X23AuXg9WJmKYo-1T8XMURW1WLgflqSR542H4yRyrgOLcgEUKo9n9Tj_Maq0XC1z1FHuM-oQWyG3stRaTH3VcXg%3D%3D: E, [2016-06-23T09:52:30.201993 #21122] ERROR -- : [14366] 2016-06-23 09:52:30 ERROR 403: Forbidden. E, [2016-06-23T09:52:30.202646 #21122] ERROR -- : [14372] http://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/streamer/var/lib/pulp/content/distribution/ks-Red%20Hat%20Enterprise%20Linux-Server-7.2-x86_64/images/pxeboot/initrd.img?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTAuOC4xMTkuMSJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L2Rpc3RyaWJ1dGlvbi9rcy1SZWQgSGF0IEVudGVycHJpc2UgTGludXgtU2VydmVyLTcuMi14ODZfNjQvaW1hZ2VzL3B4ZWJvb3QvaW5pdHJkLmltZyIsICJleHBpcmF0aW9uIjogMTQ2NjY5MDAzOX0%3D;signature=PXxZFkJercMUWzVlKa-BH_6izRjpt-wvcqPMmyeK_u-NkGzoxXp4N0-KFE8faQfiVo22Q0V1oCKwvgWajGtJu1XdnjA7Wwtng9mVpKQ8sEBpiMH1NcwSoKA9-LoNpb1BSiF2LXJsh3WE2N328u2wm7SMqwrNjnXwGdRQ4SQ0CWtkW7YZJ8wsDeJ_eFC5JhaYTfmCsD38NbOzPvIy5tUEch-a8gu-J1W61A_6YxKH5OnTbdMZOewJFiwlY8m8OQAxiztD513qlvPh7UYVqrcsfei-nuNWOmya9_Ve-l9nOQfDtfUOTfRPDlks8gZPHfnDfposCu7wQh5auv-Vuht-cQ%3D%3D: E, [2016-06-23T09:52:30.202895 #21122] ERROR -- : [14372] 2016-06-23 09:52:30 ERROR 403: Forbidden. Version-Release number of selected component (if applicable): upgrading sat6.1.9 -> sat6.2 GA snap17 on rhel7. How reproducible: Steps to Reproduce: 1. Create new host after upgrade ( associated hostgroup of rhel7) 2. 3. Actual results: Expected results: provisioning should work Additional info: no firewall rules selinux in enforcing mode download policy set to immediate
Created attachment 1171548 [details] console logs says failed to fetch squashfs.ks
Created attachment 1171549 [details] foreman-debug
Looking at the attached screenshot from comment 1, the file it is attempting to fetch has a broken sym link. The broken link is: /var/lib/pulp/published/yum/http/repos/Default_Organization/DEV/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img Doing a find for broken links we can see many: # find -L /var/lib/pulp/published/yum/http/repos/ -type l -! -exec test -e {} \; -print|grep LiveOS /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/rhel7/rhel7_repo/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/1.0/custom/rhel7/rhel7_repo/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/2.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/3.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/rhel7_cv/4.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_cv/1.0/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img /var/lib/pulp/published/yum/http/repos/Default_Organization/DEV/rhel7_cv/content/dist/rhel/server/7/7.2/x86_64/kickstart/LiveOS/squashfs.img
Michael, Can you help identify why the sym links may be broken? Note: if you execute the command in comment 3 (without the grep), you'll see that there are far more broken links than listed in that comment.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
*** Bug 1287518 has been marked as a duplicate of this bug. ***
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Reproducible with upgrade from sat6.1.9 -> sat62 GA snap18.1 I'm not able to provision neither rhel6 nor rhel7 host and getting 403 error while fetching boot images
Hi Jeff, After testing with rpms including the fix from issue 1972, I still see several files with broken sym links (all .treeinfo files). Is this expected? Note: prior to the Satellite/Pulp upgrade, there were no files with broken sym links. E.g. [root@sat61-rhel7 pulp]# find -L /var/lib/pulp/published/yum/http/repos/ -type l -! -exec test -e {} \; -print /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/capsule/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/custom/satellite_tools/el6_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/tools_el6/custom/satellite_tools/el6_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/tools_el7/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_app_el6/custom/satellite_tools/el6_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/composite_app_el7/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/tools_el6/1.0/custom/satellite_tools/el6_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/tools_el7/1.0/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_app_el6/1.0/custom/satellite_tools/el6_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/composite_app_el7/1.0/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/dev/composite_app_el7/custom/satellite_tools/el7_x86_64/.treeinfo /var/lib/pulp/published/yum/http/repos/Default_Organization/dev/composite_app_el6/custom/satellite_tools/el6_x86_64/.treeinfo
No, the .treeinfo link should be migrated. Looks like the migration did re-link the "<path>/.treeinfo" file but to a non-existent "<new-path>/treeinfo". Looks like a 1 line code change upstream to do a stricter comparison. I will file an issue upstream and get it fixed today.
Moving the state back to ASSIGNED as we'll need the additional fix from comment 18, before it is moved over to QE.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Verified with sat62 GA snap19. I can provision host with snap19 without any issue.
Created attachment 1177211 [details] screenshot of provisioned rhel7 host
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501