Bug 1515667

Bryan,

I don't see any /etc/profile.d or elsewhere that a proxy environment variable is set for specific users but can you confirm whether or not HTTP_PROXY, HTTPS_PROXY, NO_PROXY envrionment variables are set for either the user who executed the playbooks or the ansible_ssh_user on the etcd hosts?

`env | grep -i _PROXY` as both the user executing the playbooks and ansible_ssh_user should show me what I'm looking for.

(In reply to Scott Dodson from comment #4)
> Bryan,
> 
> I don't see any /etc/profile.d or elsewhere that a proxy environment
> variable is set for specific users but can you confirm whether or not
> HTTP_PROXY, HTTPS_PROXY, NO_PROXY envrionment variables are set for either
> the user who executed the playbooks or the ansible_ssh_user on the etcd
> hosts?
> 
> `env | grep -i _PROXY` as both the user executing the playbooks and
> ansible_ssh_user should show me what I'm looking for.

* /etc/environment exists on every server but is 0 bytes (nothing in it)
* HTTP_PROXY and HTTPS_PROXY environment variables are set in the shell
* `env | grep -i _PROXY` as the user executing the playbooks

$ env | grep http
http_proxy=http://proxy.example.com:80
HTTP_PROXY=http://proxy.example.com:80
ftp_proxy=http://proxy.example.com:80
FTP_PROXY=http://proxy.example.com:80

* We yet don't have the output of `env | grep -i _PROXY` for the ansible_ssh_user. Will request that from the customer.

* Tried working around the issue by adding no_proxy to all of the Ansible playbooks where etcdctl is called.
 - Attached (private etcd-scheme-upgrade.log file) logs with no_proxy enabled and now fails when recreating cluster. 1st etcd restarted removes other two members leaving a cluster of 1 and fails to communicate with other two unavailable or misconfigured members.
 - Attached (private second_etcd_run.txt file) logs from a second run on a second OCP cluster that had not been upgraded and got the same error with the modified no_proxy playbooks.

How are those environment variables being set for the user? Rather than modifying the playbooks I think we should focus on figuring out how they're set and if that would potentially affect the ansible execution environment because the ansible documentation for the command module explicitly states that the command module should not be affected by the user's environment. 

One other thought, is etcdctl on the host perhaps a wrapper script? `which etcdctl` as the ansible_ssh_user would tell us.

(In reply to Scott Dodson from comment #10)
> How are those environment variables being set for the user? Rather than
> modifying the playbooks I think we should focus on figuring out how they're
> set and if that would potentially affect the ansible execution environment
> because the ansible documentation for the command module explicitly states
> that the command module should not be affected by the user's environment. 
> 
> One other thought, is etcdctl on the host perhaps a wrapper script? `which
> etcdctl` as the ansible_ssh_user would tell us.

I agree, the customer shouldn't have to modify the playbooks to get this to work. We only tried that as a workaround. The question is: why doesn't etcdctl like being run through an ssh tunnel in an Ansible playbook in an environment that has a proxy?

Here's the requested output:

[root@master01 ~]# env | grep -i _PROXY
http_proxy=http://proxy.example.com:80
FTP_PROXY=http://proxy.example.com:80
ftp_proxy=http://proxy.example.com:80
no_proxy=.example.com
HTTP_PROXY=http://proxy.example.com:80

[root@master01 ~]# which etcdctl
/usr/bin/etcdctl

Their ansible user is actually running as root and they set up ssh keys for root between masters -> all nodes. In the hosts file, ansible_ssh_user is set to root:

[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=openshift-enterprise
no_proxy="."

They added no_proxy="." recently in this dev cluster while troubleshooting the etcd schema migration issue. The no_proxy setting doesn't exist in other clusters.

I've confirmed that when the local user has HTTP_PROXY HTTPS_PROXY variables set and ansible_connection=local is used for any of the hosts affected by the playbook the playbook will fail.

Removing ansible_connection=local results in a successful migration.

Alternately, determine where those environment variables for the ansible_ssh_user are being set and comment them out for the time that the migration is being executed. They shouldn't be necessary in any form or fashion as all hosts should be co-located and shouldn't require a proxy to communicate with each other.

(In reply to Scott Dodson from comment #15)
> Alternately, determine where those environment variables for the
> ansible_ssh_user are being set and comment them out for the time that the
> migration is being executed. They shouldn't be necessary in any form or
> fashion as all hosts should be co-located and shouldn't require a proxy to
> communicate with each other.

Thank you for confirming this, Scott. The customer does have the ansible user set as 'root' and root does have those environment variables in place. I have asked the customer to confirm the ansible_connection=local setting.

I will also have them test the schema migration again after either removing ansible_connection=local or removing the environment variables from root.

(In reply to Scott Dodson from comment #14)
> I've confirmed that when the local user has HTTP_PROXY HTTPS_PROXY variables
> set and ansible_connection=local is used for any of the hosts affected by
> the playbook the playbook will fail.
> 
> Removing ansible_connection=local results in a successful migration.

They are not using the ansible_connection ansible variable in their inventory file apparently. I wonder if ansible_connection is local by default?

[user@node1 ~]$ grep ansible_connection /etc/ansible/hosts
[user@node1 ~]$

We're going to try removing those variables now instead...

The command module documentation is misleading/wrong here.  The command module does use the user's environment variables just like any other ansible module.  I've done some sleuthing in the git repository history to confirm and it looks like the documentation was added way back when in-module documentation was first added as an ansible feature and it has never matched with the module's behaviour.

Just like any other Ansible module, the command module inherits the environment variables that are set when we connect to the remote machine.  In the case of a local connection and no become method, the module inherits the same environment as the user invoking ansible.  In the case of a remote connection like ssh, the user's shell on the remote machine plays a part because the shell will source certain startup files when we open our ssh session.  In the case of become, the configuration of the become method and the startup files of the user we're becoming may also play a part.

Which shell startup files play a part depends on the shell being used (and likely, how the distribution has configured the shell).  I've written up the results of some tests on Ubuntu and CentOS in a series of three comments on this upstream bug:
https://github.com/ansible/ansible/issues/22885#issuecomment-293741361  in case you need some of those details.

I'll submit an upstream PR to correct the command module's documentation.

I can confirm that removing the proxy environment variables allowed for a successful v2 to v3 schema migration.

I'm unsure of whether the original issue was stemming from the proxy or whether it is an etcdctl problem.

The only attached cases are CLOSED and this stems from host customization outside of that which openshift-ansible is responsible for. I'd expect an admin with knowledge of their proxy configuration to have figured this out.

CLOSED WONTFIX