It looks like your compose is somehow broken. The permissions are alright, but the private keys should be owned by the group  ssh_keys, which is probably not there during the key creation. Can you verify that?

    # grep ssh_keys /etc/group
    ssh_keys:x:999:

If the group is there, then the sshd-keygen failed to set the keys in correct way, you should see some logs from the first boot, but what else can go wrong in?

   chgrp ssh_keys /path/to/private_key

If the group is missing, it check why it is not here. It is created in spec file %pre section using

   getent group ssh_keys >/dev/null || groupadd -r ssh_keys || :


I didn't notice this behavior with composes for example in Beaker.

It could be a bug caused by rebased SELinux userspace which should be fixed in lorax - https://bugzilla.redhat.com/show_bug.cgi?id=1332147

I'm not sure when the latest lorax will be in the compose but please try the latest compose and in the mean time to relabel your filesystem:

# fixfiles onboot && reboot

- The group is there:

[root@localhost ~]# grep ssh_keys /etc/group
ssh_keys:x:999:
[root@localhost ~]# 

- Contrary to what I said in the description, changing the private keys' group by hand to ssh_keys does fix it too

- Running "fixfiles onboot && reboot" didn't fix it

I've opened this BZ because I thought I was seeing a new important issue. If you guys are certain this is rather a nightly compose problem that will be fixed in later composes, then I'd be okay with closing this as NOTABUG.

I would double check why it does not change the group during key generation. Few things to check:

 * Remove the host keys:  rm -f /etc/ssh_host_*
 * Restart the sshd service
 * Check if the keys have correct group
 * Check if there are some AVCs ?

Check once more the same with SELinux in permissive. It might really be related to some SELinux change. Any AVCs? Does it work?

If you can not reproduce it with above (after the boot sequence), you might try to boot with debug kernel argument and see what is systemd doing with these services at first boot. Or with selinux=0. Still a lot of possibilities, but probably not openssh fault.

Yes, I think it's selinux. I did the following:

# rm -f /etc/ssh/ssh_host_*
# systemctl restart sshd

Same problem (all file's group is "root"). Then checked /var/log/messages:

May 11 09:41:37 localhost systemd: Stopping OpenSSH server daemon...
May 11 09:41:37 localhost systemd: Stopping OpenSSH Server Key Generation...
May 11 09:41:37 localhost systemd: Starting OpenSSH Server Key Generation...
May 11 09:41:37 localhost sshd-keygen: Generating SSH2 RSA host key: chgrp: invalid group: ‘ssh_keys’
May 11 09:41:37 localhost kernel: type=1400 audit(1462977697.297:35): avc:  denied  { read } for  pid=10402 comm="chgrp" name="group" dev="dm-0" ino=9036568 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file
May 11 09:41:37 localhost sshd-keygen: [  OK  ]
May 11 09:41:37 localhost sshd-keygen: Generating SSH2 ECDSA host key: chgrp: invalid group: ‘ssh_keys’
May 11 09:41:37 localhost kernel: type=1400 audit(1462977697.350:36): avc:  denied  { read } for  pid=10408 comm="chgrp" name="group" dev="dm-0" ino=9036568 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file
May 11 09:41:37 localhost sshd-keygen: [  OK  ]
May 11 09:41:37 localhost sshd-keygen: Generating SSH2 ED25519 host key: chgrp: invalid group: ‘ssh_keys’
May 11 09:41:37 localhost kernel: type=1400 audit(1462977697.362:37): avc:  denied  { read } for  pid=10416 comm="chgrp" name="group" dev="dm-0" ino=9036568 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file

Then I did:

# echo 0 > /sys/fs/selinux/enforce
# rm -f /etc/ssh/ssh_host_*
# systemctl restart sshd

Works.

Yes. You are right. Thank you for checking.

Please, add a note what selinux-policy version you have compose and we can move the problem to this component.

I added Lukas into CC to have a look, if it is a regression in policy or problem related to the userspace rebase.

It is selinux-policy-3.13.1-71.el7.

Can you see any AVC denied message in /var/log/audit/audit.log when you do it? Could you please attach them?

You can use the following command to check if any AVC denied messaged occurred in last 10 minutes:

# ausearch -m avc -m user_avc -ts recent

Petr,

/var/log/audit/audit.log is empty and ausearch returns <no matches>. The avc failures are written to /var/log/messages.

I suspected the reason could be that auditd is not running. And I was right:

● auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2016-05-11 14:16:36 CDT; 5min ago
  Process: 602 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
  Process: 601 ExecStart=/sbin/auditd -n (code=exited, status=6)
 Main PID: 601 (code=exited, status=6)

May 11 14:16:36 localhost.localdomain augenrules[602]: enabled 0
May 11 14:16:36 localhost.localdomain augenrules[602]: flag 1
May 11 14:16:36 localhost.localdomain augenrules[602]: pid 0
May 11 14:16:36 localhost.localdomain augenrules[602]: rate_limit 0
May 11 14:16:36 localhost.localdomain augenrules[602]: backlog_limit 320
May 11 14:16:36 localhost.localdomain augenrules[602]: lost 13
May 11 14:16:36 localhost.localdomain augenrules[602]: backlog 0
May 11 14:16:36 localhost.localdomain systemd[1]: Failed to start Security Auditing Service.
May 11 14:16:36 localhost.localdomain systemd[1]: Unit auditd.service entered failed state.
May 11 14:16:36 localhost.localdomain systemd[1]: auditd.service failed.

Don't you want to give this a try? It's very simple to reproduce. All you have to do is to install latest RHEL7.3 on a VM and check the file permissions in /etc/ssh. I'm installing the VM with:

# virt-install -n RHEL7-ssh-test \
               --os-variant=rhel7 --vcpus=2 \
               --disk path=/var/lib/libvirt/images/rhel72.img,bus=virtio,size=10 \
               --graphics type=spice,listen=0.0.0.0 \
               --memory 2048 \
              -l <address of nightly compose>

I missed few details in the AVC. Can you check the selinux context of the group file and the keys?

    ls -Z /etc/group /etc/passwd /etc/shadow
    ls -Z /etc/ssh/ssh_host_*

From the discussion with Lukas it looks really like mislabeled /etc/group file (should have  passwd_file_t  type) and then  chgrp is failing to set correct group for the keys.

# cd /etc
# ls -Z group passwd shadow

-rw-r--r--. root root system_u:object_r:shadow_t:s0    group
-rw-r--r--. root root system_u:object_r:passwd_file_t:s0 passwd
----------. root root system_u:object_r:shadow_t:s0    shadow

# cd /etc/ssh
# ls -la -Z

drwxr-xr-x. root root system_u:object_r:etc_t:s0       .
drwxr-xr-x. root root system_u:object_r:etc_t:s0       ..
-rw-r--r--. root root system_u:object_r:etc_t:s0       moduli
-rw-r--r--. root root system_u:object_r:etc_t:s0       ssh_config
-rw-------. root root system_u:object_r:etc_t:s0       sshd_config
-rw-r-----. root root system_u:object_r:sshd_key_t:s0  ssh_host_ecdsa_key
-rw-r--r--. root root system_u:object_r:sshd_key_t:s0  ssh_host_ecdsa_key.pub
-rw-r-----. root root system_u:object_r:sshd_key_t:s0  ssh_host_ed25519_key
-rw-r--r--. root root system_u:object_r:sshd_key_t:s0  ssh_host_ed25519_key.pub
-rw-r-----. root root system_u:object_r:sshd_key_t:s0  ssh_host_rsa_key
-rw-r--r--. root root system_u:object_r:sshd_key_t:s0  ssh_host_rsa_key.pub

Hi Luiz, 

File /etc/group is mislabeled. So if you run: 
# restorecon -R -v /etc 

what is output of this command?

restorecon reset /etc/group context system_u:object_r:shadow_t:s0->system_u:object_r:passwd_file_t:s0
restorecon reset /etc/grub.d/00_tuned context system_u:object_r:usr_t:s0->system_u:object_r:etc_t:s0

Thank you. So there are two problems:

1) We have wrong label on /etc/group after installation for some reason
   (other services will probably have the same problem)
2) Fixfiles didn't fix that

Policy seems to be ok since it was fixed by the restorecon (Lukas pointed that it might not be true).

We can not reproduce it with the same compose in beaker, nor with existing updated installations. I just installed your compose in virt-manager and I see the same results as you do. There must be some difference between manual install and beaker kickstart.

But it is probably beyond my knowledge to go further in this investigation. Petr, what do you think about this?

I tried the

    fixfiles onboot && reboot

and it fixed the group on my install.

So we are left with the problem of installer (or something) setting wrong initial context of this file. We should probably get Anaconda maintainers in the loop.

Moving to anaconda component and renaming to express the actual problem. Luiz, please adjust it if I missed something.

Your description is very accurate. You guys did a great job debugging this issue.

*** This bug has been marked as a duplicate of bug 1333952 ***