Do we have a specific list of services need the proxy settings? I think it would be an rfe for a specific service or it would be good to understand why they are using /etc/environment to pass this information.  I don't necessarily thing we should be bind mounting /etc/environment in as that seems to be a workaround for some missing functionality.

Hi,

In my concrete example, the customer configured proxy settings on the undercloud.

They then ran into the issue describe in https://access.redhat.com/solutions/5009791

After modifying /etc/environment on the Director, we then needed to run 'openstack undercloud install' to apply this change to the ironic containers. That's counter-intuitive and cumbersome so that's why I created this BZ.

Ideally, the proxy information should not have been passed to the ironic containers at all, I'm fine with that, too.

- Andreas

Hi,

In my concrete example, the customer configured proxy settings on the undercloud.

Then, they installed the undercloud with 'openstack undercloud install'.

They then added nodes and on node import they ran into the issue describe in https://access.redhat.com/solutions/5009791

After modifying /etc/environment on the Director, we then needed to run 'openstack undercloud install' again to apply this change to the ironic containers. That's counter-intuitive and cumbersome so that's why I created this BZ.

Ideally, the proxy information should not have been passed to the ironic containers at all, I'm fine with that, too.

- Andreas

I was trying to understand what services would need proxy access to something. At this point it seems to be that ironic itself needs the proxy to get to the redfish controller to perform actions. I don't think it makes sense to expose proxy settings to all services (as most should not be reaching out externally for anything).  I don't believe we should mount /etc/environment into all containers as a practice.  IMHO the better solution is to actually be able to set proxy information in ironic itself to handle external communications as necessary.

Hi,

I totally agree. 

One clarification: ironic did not require that a proxy be set. On the contrary. The proxy was set automatically, upon undercloud installation, from the Director's /etc/environments

And the only way to make no_proxy work within the ironic containers was by rerunning the undercloud installation

Thanks,

Andreas

So it sounds like a bad configuration in their proxy config where they should no_proxy the host actually. So I think this is an error in their proxy config and not actually a bug.  I'm not certain if we document the use of /etc/environments anywhere. If we do we should document that you must correctly no_proxy any systems that the undercloud may communicate with that do not require a proxy to get to.  This feels like a docs bug and not an actual issue.

We do document the configuration of /etc/environment for proxy. I shall review it to see if we need a line about no_proxy.

Hi Alex,

No, that's not the problem.

The customer configured a proxy setting. And they configured an "incomplete" no_proxy setting. They installed their undercloud with "openstack undercloud install". Now they continued with the documented way of running their system. Then, they ran into issues with ilo / redfish connectivity, as they passed via the proxy because their no_proxy was incomplete. Up to here, I'm totally o.k. with the process.

They opened a ticket, I saw the proxy issue, told them to update /etc/environment.  The issue persisted. I told them to reboot the undercloud as I suspected that the podman containers might not have picked up the modification to /etc/environment but to no avail.

Then, IIRC I talked to bfournier who told me that I'd have to rerun 'openstack undercloud install'. We did that, and then ironic picked up the modification to no_proxy and started working. And that's what I'm not o.k. with.  That process is counter intuitive. I'd expect either that the ironic containers directly use the system's /etc/environment --> bind mount. And restarting the containers makes ironic pick up the modified /etc/environment. Or I'd expect a dedicated variable for ironic proxy configuration (if that's even needed). What I don't expect is that we copy /etc/environment from the Director node at install time and then "freeze" that configuration, and then users have to rerun 'openstack undercloud install' to propagate that change into the containers. I find that really counter intuitive.

Hope that clarifies what I'd like to have achieved.

- Andreas

So the issue is the global nature of /etc/environment. So configuring that configures it on the *host*. Due to the nature of containers, the configuration is not shared into the running containers (as designed).  What might be counter-initiative is the fact that we copy it in at all. I'm actually uncertain where that comes in because we don't mount in /etc when handling configs and I'll have to look into it. However I disagree that the solution is to always bind mount /etc/environment into all containers. You would not accomplish the same thing if a container was running under kubernetes.  IMHO this is a doc fix.

The one specific exclusion for this is actually mistral-executor which sets /etc/environment which should be loaded on boot https://github.com/openstack/tripleo-heat-templates/blob/fffdcf0f3059a4f1146ec533f51a65442a105092/deployment/mistral/mistral-executor-container-puppet.yaml#L205

Still trying to track down how it's ending up in the container in the first place but I'm still not exactly convinced that updating /etc/environment via bind mount will help. You'd still need to at least restart the containers anyway for the running python processes to pick up the change.

So I'm currently unable to replicate the copying of /etc/environment into a container during the configuration. I'm not sure where the proxy setting information was leaking into the container.  I will continue to investigate

I found it. It looks like it is included in the the container config on creation.  So when it's started, it pulls in the Env values for the command execution.  I don't think mapping /etc/environment into the container will help this at all because the environment is controlled via Env and not what ends up in /etc/environment inside the container.

        "Config": {
            "Hostname": "director-052.hostname",
            "Domainname": "",
            "User": "root",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "http_proxy=http://hostname:8080/",
                "https_proxy=https://hostname:8080/",
                "no_proxy=127.0.0.1,172.16.0.0/16,172.17.0.0/16,172.18.0.0/16,192.168.0.0/16,10.209.107.0/26,director-052.ctlplane.localdomain",
                "HOSTNAME=director-052.vran.edn.ki.sw.ericsson.se",
                "container=oci",
                "KOLLA_BASE_DISTRO=rhel",
                "KOLLA_INSTALL_METATYPE=rhos",
                "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ",
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "TRIPLEO_CONFIG_HASH=e6c01ab90c23b5b0f6977e256644eddc",
                "LANG=en_US.UTF-8",
                "KOLLA_INSTALL_TYPE=binary",
                "KOLLA_DISTRO_PYTHON_VERSION=3.6",
                "KOLLA_BASE_ARCH=x86_64",
                "HOME=/root"
            ],


I'll keep poking at this because I can't seem to get this into my environment when testing.

It's because podman create automatically forwards the proxy items into the container on launch.  It can be disabled at podman execution time, however it's enabled by default. We'll have to expose this to make it configurable because it doesn't reset based on the current settings.  So unless you cause the containers to be rebuilt via a config change, it won't be updated.

 **--http-proxy**=*true|false*                                                   
                                                                                  
  By default proxy environment variables are passed into the container if set     
  for the Podman process.  This can be disabled by setting the `--http-proxy`     
  option to `false`.  The environment variables passed in include `http_proxy`,   
  `https_proxy`, `ftp_proxy`, `no_proxy`, and also the upper case versions of     
  those.  This option is only needed when the host system must use a proxy but    
  the container should not use any proxy.  Proxy environment variables specified  
  for the container in any other way will override the values that would have     
  been passed through from the host.  (Other ways to specify the proxy for the    
  container include passing the values with the `--env` flag, or hard coding the  
  proxy environment at container build time.)                                     
                                                                                  
  For example, to disable passing these environment variables from host to        
  container:                                                                      
                                                                                  
  `--http-proxy=false`

I too had a similar issue where the customer had to change no_proxy setting in already deployed undercloud.

But neither container restart nor undercloud install did the trick. So I had to re-spawn that container via paunch.

This needs to be reflected inside container atleast via container restart ,another options are not easy.

Maybe enforcing the --http-proxy=false in the container management (paunch or new ansible module) is the right thing to do? Not 100% sure though, since an operator might want to actually pass proxy option.
Would a new t-h-t parameter (ContainerHttpProxy: false) be ok?

Might this be related to https://bugzilla.redhat.com/show_bug.cgi?id=1682970 ?

HI,
any new advanced?. I have similar problem but openstack undercloud install don't solve my problem; container not get the new configuration (changes on env).

Closing EOL, OSP 16.0 has been retired as of Oct 27, 2020