Proposed as a Blocker for 35-final by Fedora user frantisekz using the blocker tracking app because:

 Not having a root account (or sudo access) available breaks a bunch of release criterions, eg. 

The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated.

This seems to be an Arm thing only. With x86_64 installation I was not able to get rid of an incomplete G-I-S in any way, which would trigger the described status.

(In reply to Lukas Ruzicka from comment #2)
> This seems to be an Arm thing only. With x86_64 installation I was not able
> to get rid of an incomplete G-I-S in any way, which would trigger the
> described status.

I haven't tested the x86_64 yet. But this occurred only when bootting directly from the image and not from the installed system.

Also, this is about initial-setup (the cli one), not G-I-S.

My leading theory is that the issue is caused by Initial Setup parsing the kickstart present on the ARM installation image (in /root/anaconda-ks.cfg), which includes the rootpw command.

This in turn satisfies the "is there an admin user ?" check:


    def check_admin_user_exists(self):
        """Reports if at least one admin user exists.
        - an unlocked root account is considered to be an admin user
        - an unlocked user account that is member of the group "wheel"
          is considered to be an admin user
        :return: if at least one admin user exists
        """
        # any root set from kickstart is fine
        if self._rootpw_seen:
            return True
        # if not set by kickstart root must not be
        # locked to be cosnidered admin
        elif self.root_password and not self.root_account_locked:
            return True

        # let's check all users
        for user in self.users:
            if not user.lock:
                if "wheel" in user.groups:
                    return True

        # no admin user found
        return False

From: https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/users/users.py#L371


In this case, _rootpw_seen indicates if the rootpw command has been seen in the input kickstart file - any usage of the rootpw command is sufficient there, including "rootpw --lock". 

The behavior is fine for the installation usecase (the user might want to deploy a machine not reachable via regular user accounts in an admin capacity) but causes an issue there for Initial Setup running on ARM in reconfig mode - the roopws command used when creating the image ends up in the kickstart Anaconda stores to the image by default. Initial Setup then sees the rootpws command and considers it sufficient for the "is there an admin ?" check, letting users setup an ARM system without any admin users.

To fix this issue, I recomend making sure the anaconda-ks.cfg file is not present on the image. This can be achieved by passing the inst.nosave=output_ks to Anaconda at image generation time (for more details see: https://anaconda-installer.readthedocs.io/en/latest/boot-options.html#inst-nosave). Can the image generation tooling running Anaconda do that ?

Alternatively making sure the rootpw command is not present in the anaconda-ks.cfg file should also fix the issue, but Anaconda currently has no provisions for filtering what command will and will not be present in the output kickstart, so some later stage of the image build process would have to do that with sed or similar.

In today's Go/No-Go meeting, we agreed: This can be fixed with a reinstall if noticed quickly and is a behavior that existed in previous releases
https://meetbot.fedoraproject.org/fedora-meeting/2021-10-21/f35-final-go_no_go-meeting.2021-10-21-17.00.log.html#l-319

Note we slipped Final today, so we do have a window to fix this if we can get a sufficiently safe fix in the next few days.

So I was hoping we could get this fixed with the slip, but I had to prioritize the blockers and it doesn't look like anyone else got to it :/

It's a bit late now, but I took a look into it, and I think we can at least in theory do what Martin suggests in Oz:

https://src.fedoraproject.org/rpms/oz/pull-request/3

submitted as a PR because I'm not 100% confident it a) would work as anticipated or b) be safe, any review appreciated.

So, the PR was merged for Rawhide yesterday, and a build was run. Can someone test an image from today's Rawhide and check whether the kickstart file is gone, and if so, whether that does fix the bug?

I'm assuming the Oz from Rawhide is used to build Rawhide images, here. If not, we would I guess need to update Oz for whatever the builders are running to see the change...

(In reply to Adam Williamson from comment #8)
> So, the PR was merged for Rawhide yesterday, and a build was run. Can
> someone test an image from today's Rawhide and check whether the kickstart
> file is gone, and if so, whether that does fix the bug?
> 


The kickstart is still included on the image. Tested Fedora-Minimal-Rawhide-20211027.n.0.aarch64.raw.

Umm. So looking more carefully, I may have patched the wrong place...though I'm not sure what the place I patched is used for, exactly. I can see the other place we might need to touch, I'll send another patch.

https://src.fedoraproject.org/rpms/oz/pull-request/4

OK, that got merged and built a couple of days ago, so once more - does it work better yet?

(In reply to Adam Williamson from comment #12)
> OK, that got merged and built a couple of days ago, so once more - does it
> work better yet?

Kickstart is still included on images from Fedora-Rawhide-20211101.n.0

OK. So, it seems we really are just using the oz package installed on the builders, and the builders are F34.

Kevin's gonna help us do a dry run here by updating Koji staging to use a scratch oz build I just did and run an image build on it, that should let us find out if the change worked. If it does, we can do an official update for F35 and F34.

So it looks like we kinda dropped the ball on this. I don't know what happened after the last comment - if we ever built that image or tested it - but the PR eventually got merged and released, and it looks like it actually breaks initial-setup completely, because it assumes the kickstart will be present:

https://bugzilla.redhat.com/show_bug.cgi?id=2057600

I'm setting this back to ASSIGNED because we've had to revert the change that was intended to fix it for now. As explained at 2057600, that change stopped initial-setup from running at all. We need to come up with a better fix for this, either by making initial-setup work without a kickstart, or figuring out a different way to fix the problem. I'm also proposing it as an F36 Beta FE - it was an FE for F36 Final, so seems logical.

+3 in https://pagure.io/fedora-qa/blocker-review/issue/636 , marking accepted.