Description of problem: When trying to add an embedded ansible git repo using SSH credentials, the following error is presented in 5.11.5 (haven't tested prior versions): Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format /var/www/miq/vmdb/app/models/git_repository.rb:161:in `rescue in handling_worktree_errors' /var/www/miq/vmdb/app/models/git_repository.rb:156:in `handling_worktree_errors' /var/www/miq/vmdb/app/models/git_repository.rb:80:in `with_worktree' /var/www/miq/vmdb/app/models/git_repository.rb:86:in `update_repo' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:114:in `playbooks_in_git_repository' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:86:in `block in sync' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction' /usr/share/ruby/monitor.rb:226:in `mon_synchronize' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/transactions.rb:210:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.1.7/lib/active_record/transactions.rb:299:in `transaction' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:83:in `sync' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:106:in `block in sync_and_notify' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/crud_common.rb:25:in `notify' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/crud_common.rb:123:in `notify' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:106:in `sync_and_notify' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:32:in `tap' /var/www/miq/vmdb/app/models/manageiq/providers/embedded_ansible/automation_manager/configuration_script_source.rb:32:in `create_in_provider' /var/www/miq/vmdb/app/models/miq_queue.rb:479:in `block in dispatch_method' /usr/share/ruby/timeout.rb:93:in `block in timeout' /usr/share/ruby/timeout.rb:33:in `block in catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:33:in `catch' /usr/share/ruby/timeout.rb:108:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:477:in `dispatch_method' /var/www/miq/vmdb/app/models/miq_queue.rb:454:in `block in deliver' /var/www/miq/vmdb/app/models/user.rb:290:in `with_user_group' /var/www/miq/vmdb/app/models/miq_queue.rb:454:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:398:in `block in start_runner_via_fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:396:in `start_runner_via_fork' /var/www/miq/vmdb/app/models/miq_worker.rb:386:in `start_runner' /var/www/miq/vmdb/app/models/miq_worker.rb:437:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:269:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:147:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_worker.rb:147:in `times' /var/www/miq/vmdb/app/models/miq_worker.rb:147:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:54:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server.rb:156:in `start' /var/www/miq/vmdb/app/models/miq_server.rb:247:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' Version-Release number of selected component (if applicable): 5.11.5.1 How reproducible: I have been able to reproduce using a vagrant environment as well as a QE sprout appliance. This does not present itself in a master build of ManageIQ, so this seems to be build/OS related, and not an issue with the code base. Steps to Reproduce: 1. Enable the EmbeddedAnsible server role (wait a minute for it to be enabled) 2. Add a SSH key credential that is associated with a github/gitlab account 3. Add a playbooks git repo that is accessible with the above credential Actual results: The error provided above is shown when attempting to sync the repository. Expected results: The repository syncs without issue. Additional info: As mentioned above, this is an error that doesn't seem to present itself with the MIQ appliance builds, but does with recent versions of CFME. This used to be working but now is not.
Please assess the impact of this issue and update the severity accordingly. Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity for a reminder on each severity's definition. If it's something like a tracker bug where it doesn't matter, please set the severity to Low.
After doing some more testing with an older version of CFME, it turns out that this seems to have been broken since at least `5.11.0.19-`, so my guess is since the change of EmbeddedAnsible's re-write in that release. I think this will warrant reducing the priority of this a bit since it has been broken, but not reported, since the changes to EmbeddedAnsible have been updated. Probably makes sense to step back and figure this out, and there probably is no immediate need to get this fixed as it hasn't been reported previously in the past 5 minor releases.
Deferring to Dennis to set the priority given the info above (just noticed the `needinfo?` I just canceled with my last comment).
5.11.0.26 uses OpenSSH 7.8 which was released on 2018-08-24 and from the release notes, "...write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014 and described in the PROTOCOL.key file in the source distribution, offers substantially better protection against offline password guessing and supports key comments in private keys. If necessary, it is possible to write old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments when generating or updating a key [with:]" ssh-keygen -p -m PEM -f ~/.ssh/id_rsa (note that this is potentially destructive, it converts to the old format, so it'd be good to make a backup first) I did try the converted key on a 5.11 and didn't see the error when I did. I'll look more into the exact version tomorrow.
Created attachment 1680698 [details] bz_1826410_replication_script.rb Thought I should also update and share a script for reproducing on an appliance: private_key = <<-SSH_PRIVATE_KEY_DATA_HERE SSH_PRIVATE_KEY_DATA_HERE worktree_params = { :clone => true, :url => 'ssh://git/NickLaMuro/manageiq-playbooks', :path => '/var/www/miq/vmdb/tmp/repo_from_test_script', :certificate_check => -> (_valid, _host) { true }, :username => 'git', :ssh_private_key => private_key } GitWorktree.new(worktree_params) You will have to supply your own SSH private key at the top, and you can use whatever repo you see fit instead of my own. I put the script in `/var/www/miq/vmdb` and ran with `bin/rails r bz_1826410_replication_script.rb`
I believe this is only broken on 5.11 as the latest 5.10 is using OpenSSH 7.4.
Gaurav, Please test and make sure that the packages are getting updated.
Verified in Version : 5.11.7.0.20200714215453_0da8a4a
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: CloudForms 5.0.7 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3358