Bug 1916070 - proxy configuration updates in /etc/environment files are not being picked up in containers correctly
Summary: proxy configuration updates in /etc/environment files are not being picked u...
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: OSP Team
QA Contact: Joe H. Rahme
URL:
Whiteboard:
: 1918408 1919200 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-14 06:11 UTC by Michele Valsecchi
Modified: 2023-09-15 18:39 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-5296 0 None None None 2022-02-05 09:08:22 UTC
Red Hat Knowledge Base (Solution) 5729391 0 None None None 2021-01-23 13:44:06 UTC

Description Michele Valsecchi 2021-01-14 06:11:01 UTC
Description of problem:

At the moment being, octavia (but not limited to) containers do not have /etc/environment bound to the same file on the host.
This means that any change to host /etc/environment (e.g. adding "no_proxy" or editing "https_proxy" ) will not be picked up by the containers even upon restarting the containers.

As mentioned in RFE[1] for RHOSP16.0 (closed as EOL), restarting the containers is not enough to force the file inside the container to be updated.
The container(s) need to be recreated. This often means to re-deploy the whole overcloud.

Version-Release number of selected component (if applicable):
- podman-1.6.4-16.module+el8.2.0+7659+b700d80e.x86_64
- openstack-tripleo-heat-templates-11.3.2-1.20200914170177.el8ost.noarch
- openstack-octavia-api-5.0.3-1.20200904153428.8c32d2e.el8ost.noarch [4]

How reproducible:
100%

Steps to Reproduce:
1. Edit the /etc/environment of overcloud nodes using ConfigurationHooks[5] (or even edit the file(s) manually)
2. Restart octavia related containers
3. Confirm that the overcloud nodes got the new /etc/environment, but the containers did not

Actual results:
No changes are applied

Expected results:
Changes to /etc/envirnoment are applied on `podman restart <container>`, without needing to re-create the containers

Additional info:

- One solution to this issue could be to include the binding to the host /etc/envirnoment as we're doing in mistral-executor [2].
  This would avoid to the operator to re-deploy the whole overcloud in order to update one single file.

- Podman seems to have been worked on in this regards [3], but that's on RHEL8.3, as per RHOSP16.1 being tied to RHEL8.2 (podman 1.6.x), we need an alternative solution.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1827224
[2] https://github.com/openstack/tripleo-heat-templates/commit/9cb715a5e766909ad8ac4d2d949cc3c141d7923f
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1682970
[4] https://catalog.redhat.com/software/containers/rhosp-rhel8/openstack-octavia-api/5de6c03bbed8bd164a0c1a2d?tag=16.1.3-5&digest=9e046a8ec9c6&architecture=amd64
[5] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/advanced_overcloud_customization/chap-configuration_hooks

Comment 1 Michele Valsecchi 2021-01-14 06:31:27 UTC
To clear any doubt, this issue is not specific to octavia containers, this is speficic to podman.
However, as I mentioned, this is already fixed in podman 2.0, and we would need a backport in order to fix this, as RHOSP16.1 is tied to RHEL82 (podman 1.6).
As per previous experiences in regards of asking backport of podman versions to minor RHEL release, I believe this is going to be easier/faster if we handle on the THT side.

Because of said above this BZ might be better handled by pidone or df.

Comment 2 Alex Schultz 2021-01-15 23:19:28 UTC
I don't believe there's a fix to be had in THT. There are other issues around environment variables when dealing with the containers because they tend to be baked in at container launch.  I think the recommendation is to not use /etc/environment at all instead of trying to make it work.  Proxy settings such as http_proxy and https_proxy have always caused problems and it would be better not to use them at all and instead implement network based transparent proxies if possible.

Comment 3 Alex Schultz 2021-01-22 00:07:03 UTC
*** Bug 1918408 has been marked as a duplicate of this bug. ***

Comment 4 Alex Stupnikov 2021-01-22 10:38:39 UTC
Hello. I would like to ask for a second look for bug #1918408 : it was marked as a duplicate, while from this bug's description it doesn't. Bug #1918408 affects single container (mistral_executor) and completely blocks introspection when proxy for HTTPs traffic is configured. This bug (#1916070) has a single workaround: to re-define containers either using deployment command or by running paunch manually. Bug #1918408 has no supported workaround: I had to tune container-startup-config manually to fix the problem.

I would like to ask to increase this bug's severity if bug #1918408 is a duplicate: it completely blocks introspection for undercloud deployments with HTTPS proxies and we could potentially see many similar cases in the future, so we need some form of fix...

Regards, Alex.

Comment 5 Alex Schultz 2021-01-22 15:00:23 UTC
It's a duplicate because container environments (where no_proxy/http_proxy/https_proxy) are inconsistent. Additionally in older version of podman they tend to get hard coded into the creation of a container.  The issue is larger in the using proxies this way is really not ideal and to be honest the containers shouldn't likely even know about the proxies.  The undercloud is a special case where it might make sense for mistral, but the overcloud containers likely shouldn't have proxy configurations.

Comment 6 Alex Stupnikov 2021-01-22 15:21:37 UTC
Hi Alex. Thank you for explaining the point. Indeed this looks like general bug here, but I am wondering what should be our next steps from support? Can we prioritize this bug (since it would be a blocker for customers using proxies on undercloud) or it is not going to be resolved soon and we need to focus on workarounds?

Comment 7 Alex Schultz 2021-01-22 15:30:28 UTC
I would recommend that users not use a global proxy but rather targeted ones specific to the application needed to prevent these types of issues.  yum/dnf can have a proxy setting that isn't global.  We need more information about which services actually need the proxy configuration because in theory the openstack services shouldn't and are breaking because customers are applying global configurations. Additionally if they are applying global configurations, what's the point of a proxy at all?

Comment 8 Alex Schultz 2021-01-22 15:52:39 UTC
*** Bug 1919200 has been marked as a duplicate of this bug. ***

Comment 9 Alex Stupnikov 2021-01-22 16:02:03 UTC
From my personal experience as sysadmin customers could have security requirements that prohibit any direct access (without proxy) from isolated "LAB" segment to their internal networks or from production deployment to the internet, or anything else. It is easier for customers to set global proxy settings for such cases: they don't need to focus on providing separate configuration to download TripleO containers (BTW, not sure if we provide such option), to download packages or to communicate with some internal platforms which belong to other segment of network.

I also would like to note that customer are not in the wrong while doing this: we officially recommend to set global ENV parameters [1]. From my personal experience, having global proxy configuration was a common thing some time ago.

From my experience with OpenStack, I can think about using Proxy to obtain container images, Ironic needing Proxy to communicate with server's management system using redfish or similar vendor-specific system (but this is uncommon scenario), about Cinder using proxy to communicate with storage's control plane (but it is irrelevant for undercloud), about customer obtaining some cloud images from Internet or downloading configuration files or extra ansible playbooks from GitHub or internal Git, about some curl calls made by some custom automation system, about some monitoring needing it, etc. I think that it is very hard to cover all scenarios here, so IMO an option to set global proxy config should be available to some extent...

[1]
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/director_installation_and_usage/index#considerations-when-running-the-undercloud-with-a-proxy

Comment 10 Alex Schultz 2021-01-22 16:05:40 UTC
There are better ways to handle proxies than using /etc/environment. If you're doing it for entire network segments a transparent proxy in the network configuration would be much better than doing it at a host level and also improves compliance.  /etc/environment is a hammer and users need scalpels but we continue to allow the hammer.  Container images can be fetched with a proxy on a single host and the process can be done with the proxy setting only used by `openstack tripleo container image prepare` rather than just setting it for all the things.  I understand the need for proxy, I question our support of less than ideal configurations.

Comment 12 Alex Stupnikov 2021-01-22 16:09:50 UTC
I agree that transparent proxy is better option. Maybe we can involve PM to adjust documentation and tell customer to consider it as #1 priority? Or open a separate bug for documentation team to do this...

Comment 13 Alex Schultz 2021-01-22 16:13:04 UTC
I know there is effort to improve the docs, so we'll see what we can improve. There is no quick fix at the moment.  If a user has a container with a bad configuration, having the deployment rebuild the container is the best way to get it to pickup the updated /etc/environment at this time.  On the undercloud it's easier because you can just stop the service, delete the container and rerun the install.  Overcloud is a bit trickier.

Comment 14 Takashi Kajinami 2021-01-23 06:58:31 UTC
IMO we should give separate thoughts for undercloud and overcloud in this case.

For overcloud IMO we don't need any global http proxy because we can assume that http proxy in overcloud node is used for the following points.
 1. registering to CDN
 2. pull packages from CDN
 3. pull containers from CDN
The (1) and (2) is covered by adding proxy definition in rhsm.conf as is described in the following KCS, and this is currently
covered by ansible-based cofngiruation of rhsm resource.
 https://access.redhat.com/solutions/65300

For (3) we would need global proxy configuration but if we use push_destination:true in container-prepare-parameters.yaml
all overcloud nodes pull container images from undercloud thus the overcloud nodes no longer requires http proxy here.
Now it is clearly described that we won't recommend push_destination: false when pulling container images from CDN
thus I think we can ignore this usage.


On the other hand, we still have a problem in undercloud. The undercloud node requires http proxy if all rpms/containers
are pulled from CDN, and we can't use specific proxy configuration but global one, since podman or some other processes
executed during image prepare workflow doesn't privide their own interface to set http proxy.
IIUC this is why we suggest using /etc/environments to set http proxy parameter.
 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/planning-your-undercloud#considerations-when-running-the-undercloud-with-a-proxy

However this causes unexepcted behavior in ironic-conductor as I reported in bz 1919200 . In addition as reported
in this bz and bz 1918408 we still have some problems with current usage depending on /etc/environments.


IIUC in undercloud services only mistral_executor requires http proxy, since it runs workflows which would invoke container prapre workflow.
I think we'd be able to implement more specific parameters for mistral-executor only so that we can inject proxy parameters for that specific container, 
and then prohibit the usage of /etc/environments (or explicitly use --env-host for the other containers)
so that the other containers doesn't get any proxy settings.

Comment 15 Takashi Kajinami 2021-01-23 07:00:41 UTC
I noticed we also need some consideration for standalone deployment process to install undercloud/standalone node.
Maybe we can pass http_proxy and the other environments when we start the standalone heat process ?

Comment 16 Alex Stupnikov 2021-01-23 13:50:41 UTC
I would like to add more context for bug #1918408 : it looks like the problem occurs when customers define environment variables in /etc/environment using quotes, which are translated by podman to container. Example:

/etc/environment:

  "no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"

podman inspect mistral_excutor Env definition contains:

  "\"no_proxy\"=\"192.168.1.1,yandex.ru,10.0.0.0/8\"",

Output of env command from container's CLI:

  "no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"



It looks like we need to tell customers to define parameters in /etc/environment without quotes and bug #1918408 would not occur.

I have created KCS https://access.redhat.com/solutions/5729391 for bug #1918408.



I am wondering if anyone has any objection to re-opening bug #1918408 as documentation bug and asking doc team to:

1. Recommend using transparent proxy by default
2. Recommend to define proxy-related variables without quotes and provide appropriate examples.

Comment 17 Dan Macpherson 2021-02-04 13:59:06 UTC
(In reply to Alex Stupnikov from comment #16)
> I would like to add more context for bug #1918408 : it looks like the
> problem occurs when customers define environment variables in
> /etc/environment using quotes, which are translated by podman to container.
> Example:
> 
> /etc/environment:
> 
>   "no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"
> 
> podman inspect mistral_excutor Env definition contains:
> 
>   "\"no_proxy\"=\"192.168.1.1,yandex.ru,10.0.0.0/8\"",
> 
> Output of env command from container's CLI:
> 
>   "no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"
> 
> 
> 
> It looks like we need to tell customers to define parameters in
> /etc/environment without quotes and bug #1918408 would not occur.
> 
> I have created KCS https://access.redhat.com/solutions/5729391 for bug
> #1918408.
> 
> 
> 
> I am wondering if anyone has any objection to re-opening bug #1918408 as
> documentation bug and asking doc team to:
> 
> 1. Recommend using transparent proxy by default
> 2. Recommend to define proxy-related variables without quotes and provide
> appropriate examples.

I have an objection on both points.

For 1., recently I revised our proxy documentation to include transparent proxy as an option. However, a couple of members of the consulting team have highlighted that for a lot of corporations they've encountered using OpenStack, a transparent proxy is not a viable solution. So it's just considered an option, just like using a system-wide proxy as an option. Unless we have a viable and supported proxy solution, I don't think we can definitively claim anything as a default. 

For 2., in general (not just in the context of RHOSP and containers) you're not meant to put quotes around environment variables when you define them, not just in /etc/environments but even when you define them on the command line. This seems less like an RHOSP-specific issue and more like a basic Linux syntax issue. I do think we should draw a line somewhere as to what's "in scope" and what's "out of scope" in the documentation. This seems like something out of scope.

I'm happy to discuss these things further if need be, so feel free to reach out to me if necessary.

Comment 18 Alex Stupnikov 2021-02-05 11:47:51 UTC
Hi Dan.

Let me share my perspective.

1. I have seen customer using transparent proxies and can remember related cases: usually customers are unable to run some overcloud commands on director node. At the same time, we have huge amount of cases for RHOSP deployments with system-wide proxy when it breaks day 2 operations or overcloud deployment. Those issues are very hard to troubleshoot and fix (and it is getting only harder as podman hardcodes env definitions and doesn't change them when "openstack undercloud install" command is executed.

I am not sure, which path should be followed to address this situation: changes in best practices, product modifications or more detailed configuration procedures for customers with system-wide proxies. But IMO we need to do something to address this since it is very hard to troubleshoot such cases: customers tend to invent new ways of breaking things.

2. I agree that customers shouldn't use quotes when defining variables in /etc/environments by design. The problem is that using quotes would work for 95% of the cases, some distros (like KDE Neon) are using quotes to define variables, and most of the customers don't know low-level specifics. I am not trying to say that we should generally provide low-level recommendations and details: our documentation would probably become unusable in that case. But IMO we should do something about this particular case (configuring proxies), so customers wouldn't be able to make ugly mistakes and support would be able to fix the issue by simply applying correct recommendations from our documentation.

That was my perspective. I am sure that other parties involved may have different one, so just saying.

Kind Regards, Alex.

Comment 19 Dan Macpherson 2021-02-05 14:26:55 UTC
Hi Alex,

Thanks for responding. Here are my thoughts:

1. Thanks for sharing your perspective on transparent proxies. It's very reasonable and I agree with a lot of what you've said. The problem is that I'm also taking into account a variety of different views that are just as reasonable. And often these different perspective conflict with on another. For example, one consultant who provided feedback on proxy configuration documentation had this to say about transparent proxies:

> I'm all good with comments -but- while transparent proxy makes it easy for us I have -yet- to see a single large corp where the proxy is implemented as a transparent proxy. In most FSI customers I've seen, the site-specific proxies are pushed through GPOs from the central AD management consoles. In such environments, the Linux 'endpoints' are 'tolerated' (sometimes whitelisted on *.redhat.com without auth to make our lives easier) but I wouldn't count on having transparent proxies available in large Corps.

So taking this into account, I've got no problem with documenting transparent proxies as an option, but I don't think I can safely say that it should be the default when there's evidence to suggest that some of our RHOSP customers do not want to use a transparent proxy, nor do I have any way of enforcing the use of transparent proxies with docs alone. The best I can do is provide all options based on a variety of sources (engineering, consulting, support, etc) and provide the limitations of each one. As I said, we need a supported and working solution before I can document a prescribed method. And this might not happen until OSP 17.0 (or possibly 16.2 if the patch gets backported).

2. I'm still not convinced this is a justification to start diving into the syntax of environment variables in the RHOSP docs. The question of whether to include quotes around the proxy environment variables themselves when defining them is really outside the scope of RHOSP configuration, especially since there are issues with the system-wide proxy configuration in the first place. And I also think that just because something would work 95% of the time, does not mean it's standard practice. For example, I checked the KDE documentation on environment variables [1] and I do not see them advocating the use of quotes around the environment variables themselves, regardless of whether it works or not. Nor do we advocate using quotes around the environment variables themselves in our documentation [2] [3]. Even Ubuntu [4] and Arch Linux [5] do not advocate using quotes around the variables themselves. And while I agree that somewhere there should be a list of guidelines for using environment variables, I don't think the RHOSP docs is the place for it.

I'm open to further discussion on this. Please let me know if anything I've said here is unreasonable.

[1] https://userbase.kde.org/Session_Environment_Variables
[2] https://www.redhat.com/sysadmin/linux-environment-variables
[3] https://access.redhat.com/solutions/157293
[4] https://help.ubuntu.com/community/EnvironmentVariables
[5] https://wiki.archlinux.org/index.php/environment_variables

Comment 20 Alex Stupnikov 2021-02-05 14:57:21 UTC
Hi Dan.

First of all, thank you for prompt follow-ups and detailed context provided.

1. I think that we are talking about different points here: I was thinking about sending the message like: "you can use transparent or system-wide proxy, but transparent one is better from support perspective" (not exactly the same, but could be summarized that way). The words could be different, but IMO this message fits documentation perfectly: it provides appropriate context for customers who are looking for advice on how to implement new infrastructure for RHOSP and describes available options for people who want to understand how RHOSP could fit into their infrastructure. I don't think that we should do anything above that.

2. I am trying to say that RHOSP users could have slightly different expectations about what is acceptable or not when following our documentation. IMO it is reasonable to add extra tips in such cases. The point about other distros was that they are actually using quotes to define variables in /etc/environment: here is an example output for my laptop [1]. I don't want to dive too deep into this discussion (if we need to provide low-level details in RHOSP documentation) because I understand your point, so wrote a follow-up just to clarify my previous message.


[1]
$ uname -a
Linux hostname 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

Comment 21 Dan Macpherson 2021-02-05 16:20:14 UTC
> 1. I think that we are talking about different points here: I was thinking
> about sending the message like: "you can use transparent or system-wide
> proxy, but transparent one is better from support perspective" (not exactly
> the same, but could be summarized that way). The words could be different,
> but IMO this message fits documentation perfectly: it provides appropriate
> context for customers who are looking for advice on how to implement new
> infrastructure for RHOSP and describes available options for people who want
> to understand how RHOSP could fit into their infrastructure. I don't think
> that we should do anything above that.

So we pretty much already do that here:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/planning-your-undercloud#considerations-when-running-the-undercloud-with-a-proxy

We list the options for client-based methods (system-wide, dnf, rhsm) and their limitations, then we talk about how the transparent proxy method alleviates the limitations associated with client-based proxy configs. What I don't want to say, though, is that trasnparent proxy is our default method. Some of our customers probably would have issues with that phrasing.

Having said that, if there is anything I'm missing, please let me know.

> 2. I am trying to say that RHOSP users could have slightly different
> expectations about what is acceptable or not when following our
> documentation. IMO it is reasonable to add extra tips in such cases. The
> point about other distros was that they are actually using quotes to define
> variables in /etc/environment: here is an example output for my laptop [1].
> I don't want to dive too deep into this discussion (if we need to provide
> low-level details in RHOSP documentation) because I understand your point,
> so wrote a follow-up just to clarify my previous message.
> 
> 
> [1]
> $ uname -a
> Linux hostname 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021
> x86_64 x86_64 x86_64 GNU/Linux
> $ cat /etc/environment
> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
> games:/usr/local/games:/snap/bin"

So the example you've provided is legitimate. Quotes are fine around the variable value and should work when passing to containers. What doesn't work is when you use quotations marks around the variable itself, which is what you outlined in comment #16:

"no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"

The above won't work. The following should work though:

no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8"

So to use your example, this should work:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

But AFAIK this won't:

"PATH"="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

My point was that I don't think it's a necessary tip to say: 'Do not put quotes around the no_proxy variable itself e.g. "no_proxy".'

Double quotes around the variable value should be legitimate though. That's how you usually pass variables content that contains strings e.g.

[dmacpher@danslaptop ~]$ MYVARIABLE="This is my variable"
[dmacpher@danslaptop ~]$ echo $MYVARIABLE
This is my variable
[dmacpher@danslaptop ~]$ MYVARIABLE=This is my variable
bash: is: command not found...

But if I'm incorrect about any of this, please let me know.

Comment 22 Alex Stupnikov 2021-02-05 16:46:54 UTC
(In reply to Dan Macpherson from comment #21)
> > 1. I think that we are talking about different points here: I was thinking
> > about sending the message like: "you can use transparent or system-wide
> > proxy, but transparent one is better from support perspective" (not exactly
> > the same, but could be summarized that way). The words could be different,
> > but IMO this message fits documentation perfectly: it provides appropriate
> > context for customers who are looking for advice on how to implement new
> > infrastructure for RHOSP and describes available options for people who want
> > to understand how RHOSP could fit into their infrastructure. I don't think
> > that we should do anything above that.
> 
> So we pretty much already do that here:
> 
> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.
> 1/html/director_installation_and_usage/planning-your-
> undercloud#considerations-when-running-the-undercloud-with-a-proxy
> 
> We list the options for client-based methods (system-wide, dnf, rhsm) and
> their limitations, then we talk about how the transparent proxy method
> alleviates the limitations associated with client-based proxy configs. What
> I don't want to say, though, is that trasnparent proxy is our default
> method. Some of our customers probably would have issues with that phrasing.
> 
> Having said that, if there is anything I'm missing, please let me know.
> 

I am not completely agree that we are telling customers that transparent proxy is better option. I may be wrong here, but from my understanding the current message is that certain types of no_proxy definitions are not supported by some application and that transparent proxy has no such limitations. At the same time, this problem is a bit deeper from technical perspective:

- current integration with podman doesn't allow dynamic updates of Env definitions
- when used by RHOSP services those env definitions are passed through multiple abstraction layers and utilized via external Python modules. As a result, it is hard to properly troubleshoot them: one need to start python CLI in appropriate container and reproduce the calls made by the actual code. And because of limitations described in documentation curl will not work, but for different reasons.

I am not saying that we need to offer something as default option (IMO it is up to PMs to address such things), but that we need to send clear message to customers who are not sure: transparent proxy is generally better.

That's just my IMO. I can provide more technical details if needed.


> > 2. I am trying to say that RHOSP users could have slightly different
> > expectations about what is acceptable or not when following our
> > documentation. IMO it is reasonable to add extra tips in such cases. The
> > point about other distros was that they are actually using quotes to define
> > variables in /etc/environment: here is an example output for my laptop [1].
> > I don't want to dive too deep into this discussion (if we need to provide
> > low-level details in RHOSP documentation) because I understand your point,
> > so wrote a follow-up just to clarify my previous message.
> > 
> > 
> > [1]
> > $ uname -a
> > Linux hostname 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021
> > x86_64 x86_64 x86_64 GNU/Linux
> > $ cat /etc/environment
> > PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
> > games:/usr/local/games:/snap/bin"
> 
> So the example you've provided is legitimate. Quotes are fine around the
> variable value and should work when passing to containers. What doesn't work
> is when you use quotations marks around the variable itself, which is what
> you outlined in comment #16:
> 
> "no_proxy"="192.168.1.1,yandex.ru,10.0.0.0/8"
> 
> The above won't work. The following should work though:
> 
> no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8"
> 
> So to use your example, this should work:
> 
> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
> games:/usr/local/games:/snap/bin"
> 
> But AFAIK this won't:
> 
> "PATH"="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
> games:/usr/local/games:/snap/bin"
> 
> My point was that I don't think it's a necessary tip to say: 'Do not put
> quotes around the no_proxy variable itself e.g. "no_proxy".'
> 
> Double quotes around the variable value should be legitimate though. That's
> how you usually pass variables content that contains strings e.g.
> 
> [dmacpher@danslaptop ~]$ MYVARIABLE="This is my variable"
> [dmacpher@danslaptop ~]$ echo $MYVARIABLE
> This is my variable
> [dmacpher@danslaptop ~]$ MYVARIABLE=This is my variable
> bash: is: command not found...
> 
> But if I'm incorrect about any of this, please let me know.

The problem is that it doesn't work the same with undercloud: customer would get no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8" defined in the env output with definition [1] in /etc/environment. I think that my original comment was a bit ambigous because "no_proxy" was also quoted. Apologies. But in customer's environment it wasn't and the problem came from quoted variable's definition.

[1]
no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8"

Comment 23 Alex Stupnikov 2021-02-05 16:49:49 UTC
Long story short, bug #1918408 was about situation when customer with record [1] in undercloud's /etc/environment got the same record in env output inside mistral_executor podman container (value contained quotes) and this didn't work the way it should.

[1]
no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8"

Comment 24 Dan Macpherson 2021-02-08 03:38:06 UTC
(In reply to Alex Stupnikov from comment #22)
> I am not completely agree that we are telling customers that transparent
> proxy is better option. I may be wrong here, but from my understanding the
> current message is that certain types of no_proxy definitions are not
> supported by some application and that transparent proxy has no such
> limitations. At the same time, this problem is a bit deeper from technical
> perspective:
> 
> - current integration with podman doesn't allow dynamic updates of Env
> definitions
> - when used by RHOSP services those env definitions are passed through
> multiple abstraction layers and utilized via external Python modules. As a
> result, it is hard to properly troubleshoot them: one need to start python
> CLI in appropriate container and reproduce the calls made by the actual
> code. And because of limitations described in documentation curl will not
> work, but for different reasons.
> 
> I am not saying that we need to offer something as default option (IMO it is
> up to PMs to address such things), but that we need to send clear message to
> customers who are not sure: transparent proxy is generally better.
> 
> That's just my IMO. I can provide more technical details if needed.

Sure, I understand what you're saying, and I personally agree that transparent proxy is a better option. But I do think that what we've current said in the docs i.e. "A transparent proxy can help overcome limitations associated with client-based proxy configuration in Red Hat OpenStack Platform." is about as far as we can go in suggesting transparent proxy as a better option. I say this because not everyone is onboard with transparent proxy being a better option. I spoke to a lot of people who filed requests on proxy support in the past and got their opinions, and there isn't a clear consensus about what the right method is. Some people are pro-transparent proxy, some people are against transparent proxy.
 
> The problem is that it doesn't work the same with undercloud: customer would
> get no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8" defined in the env output
> with definition [1] in /etc/environment. I think that my original comment
> was a bit ambigous because "no_proxy" was also quoted. Apologies. But in
> customer's environment it wasn't and the problem came from quoted variable's
> definition.
> 
> [1]
> no_proxy="192.168.1.1,yandex.ru,10.0.0.0/8"

Sorry for my misunderstanding. I saw the bad syntax in comment #16 and assumed that was the issue. I had a deeper read of the issue and it seems more like an issue affecting a couple of containers (Mistral and Octavia) rather than OpenStack containers in general. A docs fix seems like a workaround rather than fixing the actual issue, which is a lack of consistency on how our containers use environment variables. So how about this: in the limitations for the system-wide proxy I can add a bullet point that mentions there are currently issues with how certain containers read environment variables stored in /etc/environments and link to this BZ and BZ#1918408. I think that's a more accurate reflection of the issue rather than just saying "don't use quotes". And because it's a limitation, it can encourage users to use other proxy methods, especially the transparent proxy option. WDYT?

Comment 25 Alex Stupnikov 2021-02-08 11:07:14 UTC
Dan, thank you for understanding and willingness to help, this looks like a solid approach!

Comment 27 Alex Stupnikov 2021-02-25 08:07:19 UTC
Thank you very much!

Comment 28 rick.beldin@hpe.com 2021-06-18 01:16:30 UTC
Having run into this issue myself several times,  I wonder if documentation is enough. 

The contortions to get through this are significant and obscure.  The workarounds seem magical and unreliable.  I currently have somewhat limited success with deploying with the proxy turned off, except for rhsm, and then running a script to pull all of the images that normally fail when their is no proxy.   While I have 'made this work', what is needed is clear direction and guidance from Red Hat to make this feel more like a cohesive product, rather than a collection of things that sort of work together.  If podman is the blocker, then fix podman and if that can't be done in a coherent way, provide clear direction on what needs to be done if you _have_ to deploy behind a proxy.   I shudder to think how you even might attempt to do this in an air-gap environment.


Note You need to log in before you can comment on or make changes to this bug.