Bug 1139148 - RFE: allow determining of pre-configured environment (Environment= and EnvironmentFile= merge-result)
Summary: RFE: allow determining of pre-configured environment (Environment= and Enviro...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-08 08:35 UTC by Pavel Raiskup
Modified: 2016-02-14 19:05 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2016-02-12 22:12:36 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Pavel Raiskup 2014-09-08 08:35:36 UTC
Some services can be configured via env variables (in .service files the
Environment= statement) and it is sometimes useful for distro-scripting to be
able to detect/use this configuration (somehow).  That way, the configuration
is on one place -> in systemd configuration files.

Currently, it seems to me that only the "systemctl show" works for us:

    $ systemctl show -p Environment postgresql
    Environment=PGPORT=5432 PGDATA=/var/lib/pgsql/data

You may similarly check for EnvironmentFile= statement.

Because the order of statements matters (and taking into account drop-in
feature, Environment=, EnvironmentFiles= (which actually beats Environment=)
without native-systemd-way of environment detection it is quite hard to detect
what env variable is going to be actually passed to service's daemon.  

The actual RFE:
Something like `systemctl show-environment postgresql` would be very nice.

Pavel

Comment 1 Lennart Poettering 2014-10-24 12:31:17 UTC
Well, this is not really that easy, as we put together the environment right after fork() and right before invoking exec() and due to a number of parameters being dynamic every process we fork might get a different set. Thus ther is no single "service environment", there are only individual ones for the variety of service processes we fork.

Comment 2 Pavel Raiskup 2014-10-24 13:00:56 UTC
For me that is not a problem because I am asking for the environment which
_would be used_ if the process was 'systemctl start'-ed _at the time_ of
show-environment execution.

Not sure whether you have some policy trying to keep the output "static",
though.  In that case, I would be OK with some 'callback' like 'execv'
substitution.  Something like:

  systemctl pretend-service-env '/usr/bin/printenv' '-O'

But I am not sure about permissions ...

Ideally I would vote for both enhancements implemented :),
Pavel

Comment 3 Jack 2015-02-17 11:41:38 UTC
It is vital to be able to interrogate the process to determine if environment variables are set.

I currently have an issue where it appears that distcc is not honoring the DISTCC_TCP_CORK=0 setting.  I am setting this in the environment file, which I know is being read, since it also sets options like "--allow NETMASK" which are visible on the command line.

I am wondering if the child processes are not inheriting this setting - and if not, why not ?

I should be able to confirm that systemd is passing that variable to the distccd daemon, but right now, I can't.

For all I can tell, systemd is failing to correctly parse the entire environment file, and so ignoring the DISTCC_TCP_CORK setting.

Comment 4 Jack 2015-02-17 11:44:01 UTC
(In reply to Jack from comment #3)
> It is vital to be able to interrogate the process to determine if
> environment variables are set.
> 
> I currently have an issue where it appears that distcc is not honoring the
> DISTCC_TCP_CORK=0 setting.  I am setting this in the environment file, which
> I know is being read, since it also sets options like "--allow NETMASK"
> which are visible on the command line.
> 
> I am wondering if the child processes are not inheriting this setting - and
> if not, why not ?
> 
> I should be able to confirm that systemd is passing that variable to the
> distccd daemon, but right now, I can't.
> 
> For all I can tell, systemd is failing to correctly parse the entire
> environment file, and so ignoring the DISTCC_TCP_CORK setting.

Actually, I suppose I can look at /proc/PID/environ.  BUT, still, it would be nice to confirm this via systemctl.

Comment 5 Lennart Poettering 2016-02-10 14:11:06 UTC
I am not sure this really belongs in systemd. The thing is that there really is no concept of a "service environment". We don't maintain any in systemd, but put together a dynamic environment block right before starting each individual process of a service, and it might be different from the next process we fork off for the service. The kernel doesn't maintain any "service environment" either. All it knows is a "process environment", which you can query with /proc/$PID/environ, andit might be different for all processes of a service (which might be various processes systemd started plus anything that forked off).

hence, I am pretty sure we shouldn't pretend there was a concept of a "service environment" when there really isn't, and hence not add a command for this in systemd. If you want to introspect processes, then there are certainly better places for such a tool than systemd, for example procps or so...

Sorry.

Comment 6 Pavel Raiskup 2016-02-10 14:25:04 UTC
This is misunderstanding -> reopening.

I'm not asking for /proc/PID/env*.  I'm asking about the setup which is
maintained in systemd's configuration files.  I do not need to care about other
variables which are part of environment.

I feel that the naming is wrong "environment" might be misleading, feel free
to choose whatever naming you want -- just to get the thing done:

I'm interested in values specified by all Environment= and EnvironmentFile=
statements of particular service.

Comment 7 Pavel Raiskup 2016-02-10 14:33:52 UTC
(In reply to Pavel Raiskup from comment #6)
> I'm interested in values specified by all Environment= and EnvironmentFile=
> statements of particular service.

I'm rather rephrasing once more..  Multiple Environment= and EnvironmentFile=
statements *define* some subset of environment.  I'm interested in values of
particular variables of particular service -- which is defined solely in
systemd's files.

Doing this manually in shell scripting is possible - but painful - because only
systemd is able to deterministically say what is the ordering of Environment=,
EnvironmentFile=, etc. among multiple files (service file, drop-ins..).  Also
the parsing of EnvironmentFile should be done by systemd only (in ideal
world), at least IMO.

Comment 8 Jóhann B. Guðmundsson 2016-02-12 11:11:00 UTC
Environment variables are supposed to be and are being set in the relevant users .profile or as custom shell script placed in /etc/profile.d and placing them in type units ( is supposed to be ) exception than a rule. 

Supporting EnvironmentFile= was a complete and utter mistaken and support for it is subjected for removal hopefully sooner rather then later in systemd. 

If it had not been for Red Hat and it's employee's then those would have already been completely removed from Fedora when I would have run the clean up process of the integration of systemd into the distribution.

As you can see environment variables are being placed all over the place and outside type units and placing them in type unit files is ( supposed to be ) rather exception than a rule so expecting or implement what you request in systemd makes little to no sense as Lennart has already stated.

Comment 9 Pavel Raiskup 2016-02-12 11:29:29 UTC
Well, reopening.  Sorry :).

Don't forget this is about Environment= too.  Is that also considered
as wrong design?

Comment 10 Jóhann B. Guðmundsson 2016-02-12 12:09:46 UTC
Daemon/services should be written in such manner they can parse configuration files on startup and those should be fixed. If there are no practical reason for not fixing it ( like for example the configuration file would end up just containing a single line ) it should as I mentioned earlier be placed in users profile or as a shell script in the profile.d/ directory.

Here is an example for a wrong thing to do for a type service unit for an java application "Environment=JAVA_HOME=/usr/java/latest" instead that lines should reside in the .profile for the user running the java application or in a shell script ( java.sh ) which resides the profile.d directory along with the java's ( JDK/JRE ) PATH.

Many are using it to do silly things like this which make absolutely no sense et all.

...

[Service]
Environment=FOO=BAR
ExecStart=/path/to/ignorance ${FOO} 

I personally have yet to come across an legit usecase that justifies the existence of Environment entries in type unit files and I would gladly here of such a case if someone actually has one.

In anycase even if such usecase exist or you disagree with what I've said you cannot reliably depend upon environment's always being placed in ( thus sourced from ) type service units files thus implementing what you request makes no sense.

Comment 11 Pavel Raiskup 2016-02-12 13:23:17 UTC
Usecase:

---

[Service]
Environment=datadir=/path/to/ignorance

---

1. Then, we need to upgrade the data directory for new service version from
   time to time

2. This might not be done by service itself, as that is risky.  So manual
   admin's action is needed - some admin's preparation is needed.

3. Let's help the admin by giving him a script helping with migration..
   because some part of migration *can* be automatized.

4. The script needs to know where the data are..

5. .. the script needs to read the contents of Environment= (but accidentally,
   as the script writer I do not know whether user uses variable or
   EnvironmentFile..).  I'm not the right guy (as service file maintainer) to
   explain to users that EnvironmentFile= was someone's mistake -- it is not
   True, and I don't really care.  I just need to know the value of $datadir
   value described here.

6. No, I can't use configuration file (as you said, ther is basically no need
   to have configured something else then the datadir).  And we expect that
   there might be more instances of this service, so at least path to datadir
   differs for each instance.

---

While you'll suggest how to do it the right way, please consider that people
hate if you change something, I hate changes too.  So the 'datadir' is
well-known variable name while it's default value is also known -- and relied
on.

I can give you more requirements and arguments, let's iterate on "how should I
do it the right systemd way".

Comment 12 Jóhann B. Guðmundsson 2016-02-12 16:38:14 UTC
People's feelings are irrelevant and updating databases is something that needs manual intervention from the database administrator which he should be able to conduct himself without the needs of some scripted mess written by incompetent individuals to assist him with.

The postgresql service unit file for systemd does not contain any Environment entries but a full path's instead as they would as the administrator started postgresql directly from the command line or in case of multiple database instance and template and specifier to separate them. 

An capable database administrator query's his postgresql database(s) for the location of the datadirectory either via psql -c "SHOW data_directory;" or via  psql -c "select setting from pg_settings where name = 'data_directory';" which either they prefer since they have better things to do that play chase down the Environment entries littered around the filesystem which may or may not exist depending how the database got started in the first place manually/systemd/upstart/legacy sysv initscript whatever. 

I suggest you start thinking like a database administrator and do the same and write the update/upgrade script which was put into your capable hands by Red Hat in such manner that it query's the database(s) themselves for the location of their data directory instead of relying on existence of environment entries in type unit files in update/upgrade scripts <sigh>

Comment 13 Pavel Raiskup 2016-02-12 19:23:42 UTC
(In reply to Jóhann B. Guðmundsson from comment #12)
> People's feelings are irrelevant and updating databases is something that
> needs manual intervention from the database administrator

I do care about user's feelings.

> The postgresql service unit file for systemd does not contain any Environment
> entries but a full path's instead as they would as the administrator started
> postgresql directly from the command line or in case of multiple database
> instance and template and specifier to separate them.

I don't get this as we have postgresql@.service template.  You suggest that
multiple instances should be started manually without systemd?

> An capable database administrator query's his postgresql database(s) for the
> location of the datadirectory either via psql -c "SHOW data_directory;" or
> via psql -c "select setting from pg_settings where name = 'data_directory';"

I don't need to check where is my datadir, but thanks for the recap.
That does not mean that I have to specify the location each time I migrate
from Fedora N to Fedora N+1.

> I suggest you start thinking like a database administrator

Thanks for suggestion.  I'm not a perfect admin probably, but with servers
I maintain I'm fine with Fedora packaging.  I'm open to suggestions, but
you should be more concrete.

> and do the same

Please elaborate here.

> and write the update/upgrade script which was put into your capable hands by
> Red Hat

You are right in this personal note -- I'm not author of this idea and I'm
one of the guys here to write/maintain that script.

I would also suggest you to think as a package maintainer.  I'm glad you look
at Fedora bugs, but then I would suggest you to try Fedora (which is, I
believe, the reason for misunderstanding here).  If you are curious, check the
`postgresql-setup upgrade` command (post-fedora-upgrade time), but this
request is not just about PostgreSQL.

To recap:  I'm just asking for feature, which would simplify things.  Feel
free to close WONTFIX as I believe the RFE is (was) clear.

Comment 14 Pavel Raiskup 2016-02-12 19:29:06 UTC
(In reply to Pavel Raiskup from comment #13)
> I would suggest you to try Fedora.  If you are curious, check the
> `postgresql-setup upgrade` command (post-fedora-upgrade time), but this
> request is not just about PostgreSQL.

This command has been working for years, before the systemd came into Fedora.
While adopting systemd in PG packaging, the command changed from `service
postgresql upgrade` to `postgresql-setup upgrade` and some other things
changed -- but the idea is the same.

Comment 15 Pavel Raiskup 2016-02-12 19:59:35 UTC
(In reply to Jóhann B. Guðmundsson from comment #12)
> People's feelings are irrelevant and updating databases is something that
> needs manual intervention from the database administrator which he should be
> able to conduct himself without the needs of some scripted mess written by
> incompetent individuals to assist him with.

Ouch, reading again I see how offensively Jóhann's comment was meant,
that's why I replied.

This RFE is still active.

Comment 16 Jóhann B. Guðmundsson 2016-02-12 22:12:36 UTC
"I'm glad you look at Fedora bugs, but then I would suggest you to try Fedora (which is, I believe, the reason for misunderstanding here). "

So here's the thing I've been using Red Hat since Red Hat linux 6  ( a.k.a Hedwic ) and I ain't talking about RHEL , administrating and maintaining Fedora,Red Hat and Centos as a profession in infrastructures and data centers since 2005, including but not limited to wide variety of postgresql implementations, dedicated close to a decade of my life contribute every free time I had to Fedora ( and making personal and professional sacrifices along that way to be able to continue to do that ) including basically building many of the processes the qa community has to day, ever since will woods was too busy playing pre upgrade with jeremy, oversaw and handled the integration of arguably the most controversial change in Linux history systemd itself which involved having to go through more or less every initscript being shipped in the distribution and migrate them, which I did up to the point Red hat employees thought they knew better than the guy that had been working on that stuff for four/five years and had successfully managed to keep himself from the urge of pouring acid into his eyes and pry them out with a fork after going through all that scripted mess but when push came to shove they failed miserably in doing it themselves and ended up instead, deciding to drop all the remaining un-migrated initscripts in the distribution ( you should have seen the announcement for that ) and I did all of that for a lousy t-shirt handful of thanks, plethora of threats including one to be outed from the project if I did not starting sucking a bag of dicks in the Red Hat desktop team so claiming that I should try fedora is something I have already done and am quite familiar with it and how Red Hat (mis)uses it. 

I have to decline your offer about start thinking like package maintainer I've already gone through packages in the hundreds ( and reports in thousands ) and I have seen how majority of them think like for something that I considerer uninteresting and a child's play to do and it ain't pretty ( and often I throughout the years I wondered how we manage to push the distribution in that shape ) but fortunately for the distribution we have few that are maintaining their components like an art form which are like diamonds in the rough when you are going through those components.

You may put whatever motion behind that response and interoperate that as  offensive or whatever currently you feel like as you read this which some would do for an individual that has reopen a bug that has been closed not once but twice but that's beside the point that the fact is this cannot be reliably implemented ( as has already been pointed out ), does not belong in systemd ( which also has already been pointed out ) and it serves no purpose listing environment values and their order since you cannot rely on environment variable in type units as a basis for anything in the first place since it would result in an implementation that would exclude manual started daemons/services, limit that implementation with systemd and the expected environment entries would have to exist in the first place in the ( upstream ) type units which in turn makes that solution not upstream-able. ( which means each downstream unit would deviate from the upstream one and that deviation would have to be maintained in a form of added patch to that component)

Feel free to create and assign your request in rhel's part of the tracker and have the rhel maintainers implement what you requested ( I'm pretty sure they will love to carry more downstream patches against systemd ) but here assigning means nothing and when a upstream project owner has said no, no means no and at that point it serves no purpose keeping the bug open hence closing for the third time...

Comment 17 Zbigniew Jędrzejewski-Szmek 2016-02-13 17:15:40 UTC
Whether EnvironmentFile is good or bad is not that relevant, as long at it is a mechanism supported by systemd. If we provide some functionality, we should provide reasonable means of introspection and debugging.

But in this case it's harder than it looks. When launching a service, the context is created dynamically (see comment #1). It *would* be possible to query the environment variables specified with Environment= and EnvironmentFiles=, but it is not the complete picture. Before execution, settings from Environment= and EnvironmentFiles= are augmented by a few other variables. Specifiers are expanded. If we provide a mechanism to query this, it shouldn't be limited to environment files.

If just expanding Environment*= is enough for you, it's easy enough to script. 'systemctl cat|grep ^Environment' will give you the settings in the right order.

For the general case we could provide something like 'systemctl mock' that would actually expand everything including specifiers and print that. I think that this would be generally useful, but it's something that needs to be discussed upstream. For example, there is a question whether this expansion can be reliably performed on systemctl side. Second question would be how the output should look. Either way, it needs some design work.

Comment 18 Zbigniew Jędrzejewski-Szmek 2016-02-13 17:22:00 UTC
https://github.com/systemd/systemd/issues/2606

Comment 19 Pavel Raiskup 2016-02-14 19:05:08 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> But in this case it's harder than it looks. When launching a service, the
> context is created dynamically (see comment #1). It *would* be possible to
> query the environment variables specified with Environment= and
> EnvironmentFiles=, but it is not the complete picture. Before execution,
> settings from Environment= and EnvironmentFiles= are augmented by a few
> other variables. Specifiers are expanded. If we provide a mechanism to query
> this, it shouldn't be limited to environment files.

Yes, if there is something else in systemd which affects the resulting
environment (e.g. specifiers), it should be in the resulting output
expanded too.
 
> If just expanding Environment*= is enough for you, it's easy enough to
> script. 'systemctl cat|grep ^Environment' will give you the settings in the
> right order.

Ah, this sounds like another good approach.  I actually parse 'systemctl show
-p Environment'.  E.g. specifiers are not expanded with `systemctl cat`, but
`systemctl show -p` does not guarantee perfect order in case there *are* used
both Environment and EnvironmentFile statements.

> For the general case we could provide something like 'systemctl mock' that
> would actually expand everything including specifiers and print that. I
> think that this would be generally useful, but it's something that needs to
> be discussed upstream. For example, there is a question whether this
> expansion can be reliably performed on systemctl side. Second question would
> be how the output should look. Either way, it needs some design work.

Right, this should be discussed upstream.  Thanks!


Note You need to log in before you can comment on or make changes to this bug.