Bug 1524833

Created attachment 1366463 [details]
freeIPA server install log

This can actually be observed in Fedora 26 container, too, and breaks a valid freeIPA use case.

Are you able to run this operation manually?

$ openssl rand -out /tmp/noise.bin 2048

If that doesn't work, there might be something specific about your system related to OpenSSL. The command works fine on my system.

As the last line of the bug description says:

"""
Please note that `docker exec freeipa-popelnice openssl rand -out /tmp/something 2048` works fine.
"""

Are you running openssl with the same user used to run pkispawn in IPA?

Perhaps someone more familiar with OpenSSL should take a look at this, but it probably has something to do with ~/.rnd file:

https://stackoverflow.com/questions/94445/using-openssl-what-does-unable-to-write-random-state-mean

pkispawn is simply calling openssl to generate a random noise.

Yes, it's run from freeIPA installer which is run as root in a container.

The same scenario seem to have been working in Fedora 25-based containers, not sure what changed.

Well, the ipa-server-install is run as root but are we sure that that's the id the openssl command is run as, invoked somehow by that pkispawn execution? I don't see noise.bin anywhere in IPA's git repository, so that seems to come completely from the pki* land, doesn't it?

pkispawn is run as a mere subprocess of the freeIPA installer so I should hope the user stays the same (root) at least for that process. Whatever pkispawn does after that with the openssl command I can't be sure and that is probably a question for Endi.

I believe noise.bin is the file that's getting generated to be a random seed for a CSR creation and thus it won't be in any repository. I'll check the ownership of the /root/.rnd file though.

There's no such file as /root/.rnd in the container. Given the link to stackoverflow, that should be probably OK as pkispawn and its subprocesses should be able to access the /root directory (HOME=/root), right?

As mentioned earlier, pkispawn is simply calling openssl to generate a random noise file (i.e. noise.bin) in a temporary folder since it's needed by certutil to generate a CSR to be signed by an external CA (installation step 1). I'm not aware of any user switching inside pkispawn. The openssl does create a /root/.rnd file on my system. It also always resets the file permission to 0600 regardless of the original permission.

Does the SELinux configuration prevent any of these operations? Have you tried running in permissive mode?

Endi,

Sorry for getting back to this this late. There don't seem to be any SELinux denials connected to this and I did try to run the container in permissive.

(In reply to Stanislav Laznicka from comment #12)
> Endi,
> 
> Sorry for getting back to this this late. There don't seem to be any SELinux
> denials connected to this and I did try to run the container in permissive.

To no avail, I forgot to add.

Some new findings -> if I inject pdb.set_trace() in freeIPA installer during docker build, and then run `pkispawn` manually by `docker exec -ti <contname> bash` at the point where it would normally be run by the installer, it succeeds. It also succeeds if I use the same trick and try to do this from python command line by importing the `run()` function from `ipapython.ipautil` and run pkispawn with it.

Weird.

Created attachment 1380037 [details]
openssl strace

Tried stracing the openssl process. There does not seem to be anything interesting in the trace to point out the error except for the openssl process returning error in the end. Attaching the strace just for the reference.

The strace also proves that the noise.bin file is indeed being written into as a root user.

Created attachment 1380067 [details]
openssl ltrace

Adding an ltrace from a different run (ends in the same way).

Tomas, Hubert, we're dealing with this issue where openssl called from an installer subprocess ends with an error: "unable to write 'random state'"

I attached an ltrace and an strace to this BZ. Do you think you'd be able to read any information about why this failure happens?

Well, app_RAND_write_file has code

    if (file == NULL)
        file = RAND_file_name(buffer, sizeof buffer);
    if (file == NULL || !RAND_write_file(file)) {
        BIO_printf(bio_err, "unable to write 'random state'\n");
        return 0;
    }

So that message (and return 0) would happen either if file == NULL (meaning RAND_file_name returned NULL) or the RAND_write_file failed.

Looking at RAND_write_file, it starts with

    i = stat(file, &sb);

and we do not see that stat, so I assume we never get into there, meaning the file is null.

The

[pid   572] getuid()                    = 0
[pid   572] geteuid()                   = 0
[pid   572] getgid()                    = 0
[pid   572] getegid()                   = 0
[pid   572] write(2, "unable to write 'random state'\n", 31) = 31

suggests the

int OPENSSL_issetugid(void)
{
    if (getuid() != geteuid())
        return 1;
    if (getgid() != getegid())
        return 1;
    return 0;
}

is called and since all those four calls return 0, the !='s are never true and so OPENSSL_issetugid returns 0. So we were in RAND_file_name and in

    if (OPENSSL_issetugid() != 0) {
        use_randfile = 0;
    } else {
        s = secure_getenv("RANDFILE");
        if (s == NULL || *s == '\0') {
            use_randfile = 0;
            s = secure_getenv("HOME");
        }
    }

the control would go through the else branch and if RANDFILE is not set, it would do

  use_randfile = 0;

and try to set path in HOME ... is HOME set in the container (in that script's execution)? Maybe openssl never tries to actually write the 'random state' and we need to figure out how to actually help it write one, help it to figure out the filename it should write.

These were done on openssl-1.1.0g-1.fc26.src.rpm.

Looking at the ltrace, both the RAND_file_name calls (not sure why there are two) end with 0, and there is never RAND_write_file called, which suggests that

    file == NULL

which suggests that RAND_file_name did not figure out what the filename should be.

https://www.openssl.org/docs/faq.html#USER2:

A possible reason is that no default filename is known because neither RANDFILE nor HOME is set. (Versions up to 0.9.6 used file ".rnd" in the current directory in this case, but this has changed with 0.9.6a.)

Not sure why we did not see the error on older Fedoras that also had openssl newer than 0.9.6 ... but maybe something else changed which made either of those environment variables unset? In any case it'd be good to verify if they are or aren't set.

I believe first of the two RAND_file_name calls tries to read it, second tries to write it (RAND_status() is called while trying to get a method to get some random data).

I tried a kind of stupid workaround where I took the file generated by `openssl rand 2048` on my fs, moved it to /opt/ipa-data-master, set proper context and added `-e RANDFILE="/data/.rnd"` to docker run arguments. It did not help, though.

Note that we run the installer from systemd service, so your environment variable value probably was not propagated to the actual process. You can check /proc/${the-pid}/environ. While you are at it, could you please check if it has or doesn't have HOME set?

Just set HOME (or RANDFILE) in ipa-server-configure-first, that should do the trick.

Just to clarify - actually ignoring the error message and exit value would also work because the random data is written to the file on the command-line, only the auxiliary random data storage in ~/.rnd is not updated.

However I will fix this to not fail this way in openssl upstream. I think the auxiliary .rnd write failure should not make the command to report failure.

Thank you Tomas for the clarification. I added a PR with an environment fix to freeipa-container repository: https://github.com/freeipa/freeipa-container/pull/184

So if we do not care about he auxiliary data, would RANDFILE=/dev/null also work, without any ill effects?

Yes, that should work as well.

(In reply to Tomas Mraz from comment #25)
> 
> However I will fix this to not fail this way in openssl upstream. I think
> the auxiliary .rnd write failure should not make the command to report
> failure.

Maybe amending the message to distinguish the situation when openssl tried to write and failed and the one when it did not even try because it did not know where to write it would help some people.

Add HOME env var to ipa-server-configure.service merged, https://github.com/freeipa/freeipa-container/commit/bd3a33adb878a823d2b20c8f1f4fcdeb063a9436

Thank you, Rob.

Tomas, I read you're performing the changes upstream and thus I presume you're tracking them as such. I am going to close this BZ as fixed but feel free to reopen pointing to openssl component if you'd like to track you progress here.

(In reply to Rob Crittenden from comment #30)
> Add HOME env var to ipa-server-configure.service merged,

Actually, in the end we went with RANDFILE=/dev/null.