Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1548230

Summary: Upgrade from 6.2 to 6.3 fails with 'Could not remove existing file'
Product: Red Hat Satellite Reporter: Craig Donnelly <cdonnell>
Component: Satellite MaintainAssignee: Kavita <kgaikwad>
Status: CLOSED ERRATA QA Contact: sthirugn <sthirugn>
Severity: high Docs Contact:
Priority: high    
Version: 6.3.0CC: apatel, arahaman, brcoca, egolov, fgarciad, inecas, kdixon, kgaikwad, ktordeur, nkathole, sthirugn
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Triaged, Upgrades, VerifiedUpstream
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-04 14:02:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Craig Donnelly 2018-02-23 00:52:33 UTC
Description of problem:
As a result of prior workarounds circulated for an issue with reading configuration files in regard to Remote Execution configuration, the foreman-maintain upgrade process from 6.2 to 6.3 can fail with 'Could not remove existing file' while attempting to convert a directory to a symlink on the filesystem.

Version-Release number of selected component (if applicable): 6.3.0


How reproducible: Requires you to have workaround in place. (Widely circulated)


Steps to Reproduce:
1. Have a 6.2.14 with REx enabled.
2. Have the following in place:
/etc/smart_proxy_dynflow_core/settings.d should be a folder and contain the following symlink:
lrwxrwxrwx. 1 root root 54 Sep 19  2016 smart_proxy_remote_execution_ssh_core.yml -> /etc/foreman-proxy/settings.d/remote_execution_ssh.yml

3. Run the foreman-maintain upgrade from 6.2.14 to 6.3

Actual results:
Installation fails with:

 Could not remove existing file
/usr/share/ruby/vendor_ruby/puppet/type/file/target.rb:43:in `mklink'
/usr/share/ruby/vendor_ruby/puppet/type/file/ensure.rb:101:in `block (2 levels) in <module:Puppet>'
/usr/share/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
/usr/share/ruby/vendor_ruby/puppet/property.rb:498:in `set'
/usr/share/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
/usr/share/ruby/vendor_ruby/puppet/type/file/ensure.rb:183:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
/usr/share/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/share/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/share/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/share/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/share/gems/gems/kafo-2.0.2/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/share/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/share/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/share/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/share/ruby/benchmark.rb:296:in `realtime'
/usr/share/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
 /Stage[main]/Foreman_proxy::Plugin::Dynflow/File[/etc/smart_proxy_dynflow_core/settings.d]/ensure: change from directory to link failed: Could not remove existing file

Expected results:
This should be changed. We could instead forcefully remove the directory, as the upgrade process will re-create it an ensure the proper symlink in that scenario (which at this point appears to be converting the symlink to instead be the 'settings.d' folder rather than the files within it).
Installation should succeed.

Additional info:
There are several KCS to workaround issues with that configuration not being read, and I believe also an errata that was released to incorporate this.

Comment 1 Craig Donnelly 2018-02-23 00:57:00 UTC
For further clarification, this can be worked around by doing the following:

# rm -rf /etc/smart_proxy_dynflow_core/settings.d

Then re-run the upgrade.

Comment 3 Brad Buckingham 2018-02-26 22:07:47 UTC
Hi Ivan,

Does this error look familiar?  Thanks!

Comment 4 Ivan Necas 2018-02-27 09:01:15 UTC
Setting this on foreman-maintain to add a check for the symlink presence on that directory and in case it's there, suggesting removing that.

Comment 5 Brad Buckingham 2018-02-27 15:25:35 UTC
Ivan, thanks for the update.

Moving to sat-maintenance based upon the update to component.

Comment 6 Satellite Program 2018-03-05 11:27:27 UTC
Upstream bug assigned to kgaikwad

Comment 7 sthirugn@redhat.com 2018-03-27 18:28:22 UTC
Verified in rubygem-foreman_maintain-0.1.5-1.el7sat.noarch

Verification steps:
# pwd
/etc/smart_proxy_dynflow_core/settings.d

# ll
total 0
lrwxrwxrwx. 1 root root 54 Mar 27 12:51 smart_proxy_remote_execution_ssh_core.yml -> /etc/foreman-proxy/settings.d/remote_execution_ssh.yml

# foreman-maintain upgrade check --target-version 6.3

--------------------------------------------------------------------------------
Check for recommended disk speed of pulp, mongodb, pgsql dir.: 
| Finished                                                                      

Disk speed : 64 MB/sec                                                [FAIL]
Slow disk detected /var/lib/pulp mounted on /dev/mapper/rhel_dhcp--10--12--211--51-root.
             Actual disk speed: 64 MB/sec
             Expected disk speed: 80 MB/sec.
--------------------------------------------------------------------------------
Check to verify remote_execution_ssh settings already exist:          [FAIL]
Settings related to remote_execution_ssh are already present under /etc/smart_proxy_dynflow_core/settings.d and 
it would conflict with the installer from the next version.
symlinks available - /etc/smart_proxy_dynflow_core/settings.d/smart_proxy_remote_execution_ssh_core.yml
--------------------------------------------------------------------------------
Continue with step [Remove existing settings.d directory before installer run.
 The next run of the installer will re-create the directory properly.]?, [y(yes), n(no), q(quit)] 
y

After the above step, I hit an issue trying to retrieve the current working directory as I was in /etc/smart_proxy_dynflow_core/settings.d which got deleted.  I created a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1561137 to track this.

I reran foreman-maintain and it ran to completion.

Comment 10 errata-xmlrpc 2018-04-04 14:02:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0633