Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2217079 - Not possible to configure the temporary directory to be used on clients by remote execution in pull mode
Summary: Not possible to configure the temporary directory to be used on clients by re...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Remote Execution
Version: 6.12.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.14.0
Assignee: Adam Ruzicka
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-06-23 19:21 UTC by Joniel Pasqualetto
Modified: 2024-03-19 14:01 UTC (History)
14 users (show)

Fixed In Version: yggdrasil-0.2.3-1, foreman_ygg_worker-0.2.2-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-10-20 22:30:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github theforeman foreman-packaging pull 9817 0 None Merged Patch yggdrasil to pass FOREMAN_REX_WORKDIR env var to the workers 2023-09-26 14:58:22 UTC
Github theforeman foreman_ygg_worker pull 16 0 None Merged Honor FOREMAN_REX_WORKDIR 2023-09-26 14:58:23 UTC
Red Hat Issue Tracker SAT-18608 0 None None None 2023-06-29 10:39:43 UTC
Red Hat Knowledge Base (Solution) 2419861 0 None None None 2023-06-26 10:03:42 UTC

Description Joniel Pasqualetto 2023-06-23 19:21:34 UTC
Description of problem:

When using rex in pull mode, foreman_ygg_worker always use /tmp as the temporary directory for the script that will be executed.

For environments when /tmp is mounted with noexec, this is a blocker.

Currently, no configuration option is available to change this path.

Version-Release number of selected component (if applicable):

Satellite 6.12
Satellite 6.13

How reproducible:

Always, when /tmp is mounted with noexec

Steps to Reproduce:

1. Configure REX in pull mode on satellite
2. Configure a client to consume the REX queue
3. Mount /tmp on the client with "noexec"
4. Try running any job

Actual results:

~~~
 1:
/bin/sh: /tmp/ygg_rex857528743: Permission denied
   2:
/bin/sh: line 0: exec: /tmp/ygg_rex857528743: cannot execute: Permission denied
   3:
Exit status: 126
   4:
StandardError: Job execution failed
~~~

Error to execute the script, due to noexec set on /tmp. No way to modify the path

Expected results:

A way to use another directory as workdir for the worker, so jobs can run.

Additional info:

/tmp appears to be hardcoded on the foreman_ygg_worker code. See https://github.com/theforeman/foreman_ygg_worker/blob/main/src/runner.go#L62

Having a variable there would be ideal.

Comment 2 Ewoud Kohl van Wijngaarden 2023-06-26 09:11:56 UTC
Reading https://pkg.go.dev/io/ioutil#TempFile the ioutil.TempFile function is deprecated and os.CreateTemp is now the new thing. https://pkg.go.dev/os#CreateTemp states:

> If dir is the empty string, CreateTemp uses the default directory for temporary files, as returned by TempDir.

https://pkg.go.dev/os#TempDir states:

On Unix systems, it returns $TMPDIR if non-empty, else /tmp.

So that should be used instead of hardcoding /tmp. Then it can be configured using a commonly used environment variable.

Comment 4 Adam Ruzicka 2023-06-26 11:25:00 UTC
What is the expectation? Is it that it should honor the remote_working_dir tunable as described in KCS 2419861 or that one should be able to configure this on each client separately?

Comment 5 Sayan Das 2023-06-26 11:30:24 UTC
From my point of view ( or the case engineers point of view ):

* foreman_ygg_worker/yggdrasil should allow setting a different temporary directory than /tmp ( if CU has hardened their client systems ) [ Perhaps something via the config.toml ]

* At satellite side:
    * remote_working_dir should be honored by pull_mqtt mode 
    * Using the value of remote_working_dir, Something additional should happen to generate and deploy a new config.toml on the client systems and restart yggdrasild on them.

    If this has to be a manual work, we will need to document that in the same KB or somewhere in the docs itself.

Comment 6 Adam Ruzicka 2023-06-26 13:08:44 UTC
> Using the value of remote_working_dir, Something additional should happen to generate and deploy a new config.toml on the client systems and restart yggdrasild on them.

I'd do one or the other. Changing a config file on a capsule and then having to propagate the changes to all the clients, which would also mean the clients would have to modify their own configuration and restart themselves feels overly complicated and fragile. Especially when considering you don't have to do anything else for non-pull rex.

Which means if remote_working_dir should be honored then it would get sent to the client as a parameter of the job and then the client would honor it.

Comment 7 Joniel Pasqualetto 2023-06-27 12:10:14 UTC
My expectation is to have one way to configure the tmp dir used by the client.

I really like this approach of sending the remote_working_dir as a parameter of the job and the client honoring it.

Comment 8 nicholas.j.teodosio.ctr 2023-07-28 15:04:43 UTC
(In reply to Sayan Das from comment #5)
> * At satellite side:
>     * remote_working_dir should be honored by pull_mqtt mode 
>     * Using the value of remote_working_dir, Something additional should
> happen to generate and deploy a new config.toml on the client systems and
> restart yggdrasild on them.
> 
>     If this has to be a manual work, we will need to document that in the
> same KB or somewhere in the docs itself.

It wouldn't be the smoothest thing, but if pull_mqtt mode honors remote_working_dir, can't you force a re-register of the clients to update the config.toml? I know this isn't an option for everyone, just a thought.


Just to chip in my two cents this is affecting a hardened system I manage. In order to use the pull_mqtt mode currently we actually have to adjust our settings on every system we want to manage. Fixing this would be extremely helpful.

Comment 9 Ewoud Kohl van Wijngaarden 2023-07-28 15:18:41 UTC
I'm not sure remote_working_dir even makes sense. In SSH context it may make sense but with the ygg worker you have a daemon that runs persistent. I'd prefer that it respects TMPDIR and be hardened using systemd's PrivateTmp=yes. I think https://github.com/theforeman/foreman_ygg_worker/pull/13 would address at least the TMPDIR part. As for hardening I think that needs to happen in https://github.com/RedHatInsights/yggdrasil/blob/main/data/systemd/yggdrasil.service.in but I can't really oversee the implications.

Comment 10 Adam Ruzicka 2023-07-28 16:12:42 UTC
I'm afraid your patch will not really change anything, apart from us not using a now deprecated function. Yggdrasil severely reduces the environment[1] before it kicks of the workers so it won't really matter what you set the TMPDIR env var to for the service as it will not reach the worker.

[1] - https://github.com/RedHatInsights/yggdrasil/blob/0.2.0/cmd/yggd/main.go#L319-L322

Comment 15 Pavel Novotny 2023-10-17 18:46:05 UTC
Moving back to MODIFIED.

After investigation with @aruzicka it has been found out that PR https://github.com/theforeman/foreman-packaging/pull/9817/
somehow didn't make it to *downstream* yggdrasil-0.2.0-3 package. It is included in the upstream yggdrasil-0.2.0-3 though.

So far, I've verified with foreman_ygg_worker-0.2.1, that '/run' directory is used by default now, so it doesn't fail on hardened systems with noexec /tmp directory.

Waiting for updated yggdrasil package for re-verification, that also a custom directory for task execution can be used via new environment variable FOREMAN_YGG_WORKER_WORKDIR.

Comment 16 Zach Jones 2023-10-19 15:01:51 UTC
(In reply to Pavel Novotny from comment #15)
> Waiting for updated yggdrasil package for re-verification, that also a
> custom directory for task execution can be used via new environment variable
> FOREMAN_YGG_WORKER_WORKDIR.

Is this something that can come out as as it's passed QA/QC? Or does it need to wait until the next full release in 6.14?
This is an issue that affecting us in prod as we harden systems.

Comment 18 Pavel Novotny 2023-10-19 18:20:26 UTC
Verified in
foreman_ygg_worker-0.2.2-1.el8sat.x86_64
yggdrasil-0.2.3-1.el8sat.x86_64


As mentioned in comment 15, '/run' directory is now the default, so running a remote job on clients with noexec /tmp directory should not fail.

A custom temp directory can be now specified via environment variable(*) FOREMAN_YGG_WORKER_WORKDIR.
(*) environment variable in a systemd service context

Example - using custom temp directory '/tmp_custom':

$ mkdir /tmp_custom
$ mount -t tmpfs tmpfs /tmp_custom
$ echo -e "[Service]\nEnvironment=FOREMAN_YGG_WORKER_WORKDIR=/tmp_custom" > /etc/systemd/system/yggdrasild.service.d/override.conf
$ systemctl daemon-reload
$ systemctl restart yggdrasild

Running a job with `printenv` command returns:
```
   1: OLDPWD=/
   2: FOREMAN_YGG_WORKER_WORKDIR=/tmp_custom
   3: PWD=/root
   4: HOME=/root
   5: SHLVL=1
   6: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
   7: YGG_SOCKET_ADDR=unix:@yggd-dispatcher-nhpQDR
   8: _=/usr/bin/printenv
   9: Exit status: 0
```

Ensuring that the client indeed uses the custom directory, re-mounting /tmp_custom with *noexec* and running the job again now returns error:

```
   1: /bin/sh: /tmp_custom/ygg_rex2042744520: Permission denied
   2: /bin/sh: line 0: exec: /tmp_custom/ygg_rex2042744520: cannot execute: Permission denied
   3: Exit status: 126
   4: StandardError: Job execution failed
```

Comment 20 errata-xmlrpc 2023-10-20 22:30:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Satellite Client security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:5982


Note You need to log in before you can comment on or make changes to this bug.