Bug 1655329

Summary: broken options to fedora-arm-image-installer
Product: [Fedora] Fedora Reporter: Torbjorn Jansson <torbjorn>
Component: arm-image-installerAssignee: Peter Robinson <pbrobinson>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: jan.kratochvil, pbrobinson, pcfe, pwhalen, torbjorn
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: arm-image-installer-2.13-1.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-06 04:09:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rpi3 boot
none
journalctl --root=/mnt/ none

Description Torbjorn Jansson 2018-12-02 20:49:21 UTC
Description of problem:
fedora-arm-image-installer have two options that appears broken.

first one --norootpass creates an image that appears to get stuck for about 50 minutes before sometimes producing a login prompt, but it is impossible to logon.
checking systemd journal shows repeating issues starting services, and many complaints related to missing user accounts.

second issue is that --resizefs doesn't actually resize the root filesystem.
this worked fine in fedora28 version of fedora-arm-image-installer but not in fedora29


Version-Release number of selected component (if applicable):
arm-image-installer-2.5-1.fc29.noarch

How reproducible:
Yes it is 100% reproducible with my odroid hc2 and above two options specified (in addition to other options i need)


Steps to Reproduce:
Create image like this for example:
fedora-arm-image-installer --target=none --image=Fedora-Minimal-armhfp-29-1.2-sda.raw.xz --resizefs --norootpass --media=/dev/sdXXX --args "<args here like console and so on>"

odroid also needs the sd_fusing script to make a bootable image
if the uboot used does not contain this patch (this is the case with current fedora 29 uboot image):
http://git.denx.de/?p=u-boot.git;a=patch;h=9cd97c5b049a9a282dda0b1782cbb38d8cedb417
then you need to manually create a symlink so the correct dtb file is loaded.
like this:
ln -s exynos5422-odroidhc1.dtb exynos5422-odroidunknown.dtb


Actual results:
boot stuck for a very long time and never able to logon properly.

journal contains this section that repeats every 90 seconds or so
----
[ 1293.050223] systemd[1]: Failed to get initial list of names: Connection timed out
[ 1293.059841] systemd[1]: dbus.service: Main process exited, code=exited, status=1/FAILURE
[ 1293.072092] systemd[1]: dbus.service: Failed with result 'exit-code'.
[ 1293.082140] systemd[1]: sshd.service: Service RestartSec=42s expired, scheduling restart.
[ 1293.092808] systemd[1]: sshd.service: Scheduled restart job, restart counter is at 5.
[ 1293.106343] systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
[ 1293.123132] dbus-daemon[820]: dbus[820]: Unknown username "systemd-resolve" in message bus configuration file
[ 1293.139994] systemd[1]: NetworkManager.service: Service RestartSec=100ms expired, scheduling restart.
[ 1293.152811] dbus-daemon[820]: dbus[820]: Unknown username "systemd-timesync" in message bus configuration file
[ 1293.166311] dbus-daemon[820]: dbus[820]: Unknown username "systemd-network" in message bus configuration file
[ 1293.179637] dbus-daemon[820]: dbus[820]: Unknown username "polkitd" in message bus configuration file
[ 1293.191577] dbus-daemon[820]: dbus[820]: Unknown username "polkitd" in message bus configuration file
[ 1293.206002] dbus-daemon[820]: Failed to start message bus: Could not get UID and GID for username "dbus"
[ 1293.221191] systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 4.
[ 1293.235797] systemd[1]: Stopped Network Manager.
[ 1293.244930] systemd[1]: Started D-Bus System Message Bus.
[ 1293.254343] systemd[1]: Starting Network Manager...
[ 1293.262796] systemd[1]: Stopped OpenSSH server daemon.
[ 1293.272751] systemd[1]: Stopped target sshd-keygen.target.
[ 1293.281229] systemd[1]: Stopping sshd-keygen.target.
[ 1293.288891] systemd[1]: Reached target sshd-keygen.target.
[ 1293.297969] systemd[1]: Starting OpenSSH server daemon...
[ 1293.307232] systemd[1]: systemd-logind.service: Failed with result 'timeout'.
[ 1293.317644] systemd[1]: Failed to start Login Service.
[ 1293.326278] systemd[1]: systemd-logind.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[ 1293.340292] systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 11.
[ 1293.353654] systemd[1]: Stopped Login Service.
[ 1293.361281] systemd[1]: Starting Login Service...
[ 1293.371696] NetworkManager[821]: <info>  [1529667196.1274] NetworkManager (version 1.12.4-1.fc29) is starting... (after a restart)
[ 1293.386568] NetworkManager[821]: <info>  [1529667196.1283] Read config: /etc/NetworkManager/NetworkManager.conf
[ 1293.405375] systemd-logind[823]: New seat seat0. 
-----


Expected results:
a working fedora image on sd card.

Additional info:

Comment 1 Peter Robinson 2018-12-03 07:47:20 UTC
> Version-Release number of selected component (if applicable):
> arm-image-installer-2.5-1.fc29.noarch

The latest stable version is 2.8, I believe all of the above are fixed with the latest version.

Comment 2 Torbjorn Jansson 2018-12-08 13:58:06 UTC
i have re-tested with:
arm-image-installer-2.8-1.fc29.noarch

--resizefs is fixed and now correctly resizes the root partition.

but the bigger issue of --norootpass producing broken images is still there.
same as before, missing users at boot in journal.
journal is easiest seen by adding: systemd.journald.forward_to_console=1 to kernel boot command line and then use the serial port to see whats going on.

Comment 3 Jan Kratochvil 2019-06-25 08:42:19 UTC
Cannot this be a duplicate of Bug 1692903?

Comment 4 Torbjorn Jansson 2019-06-25 16:26:45 UTC
well... i think it might be the other way around regarding the duplicate or something.

all i know is that if i pass --norootpass image is broken and without it boots just fine and i have several devices using resulting image.
so i'm not so sure it is selinux related, if it was then it would not matter if i specify --norootpass or not.

for completeness yes i have selinux disabled where the image is created but i don't see how that's relevant to the image generation.
whatever --norootpass does to the image it is not working out.

Comment 5 Jan Kratochvil 2019-06-25 16:38:23 UTC
(In reply to Torbjorn Jansson from comment #4)
> all i know is that if i pass --norootpass image is broken and without it
> boots just fine
...
> for completeness yes i have selinux disabled where the image is created

It would be great if you could test a fix for this issue posted upstream today:
  https://pagure.io/arm-image-installer/pull-request/38

Comment 6 Paul Whalen 2019-06-25 19:30:35 UTC
> for completeness yes i have selinux disabled where the image is created but
> i don't see how that's relevant to the image generation.
> whatever --norootpass does to the image it is not working out.

Testing with the latest - arm-image-installer-2.12-1.fc30.noarch

I can't reproduce the issue with Fedora 30 Minimal (with no initial-setup). I also tried with SE Linux in permissive with no change. 

Command used: 

sudo arm-image-installer --target=rpi3 --image=Fedora-Minimal-armhfp-30-1.2-sda.raw.xz --resizefs --norootpass --media=/dev/sdd --args "systemd.journald.forward_to_console=1" --addconsole

To remove the root password requirement the script uses "sed -i 's/root:x:/root::/' /tmp/root/etc/passwd"

Comment 7 Paul Whalen 2019-06-25 19:32:21 UTC
Created attachment 1584429 [details]
rpi3 boot

Comment 8 Paul Whalen 2019-06-25 19:46:09 UTC
OK, reproduced with selinux disabled on the host (tsk tsk)

Comment 9 Jan Kratochvil 2019-06-26 13:18:12 UTC
Created attachment 1584820 [details]
journalctl --root=/mnt/

(In reply to Paul Whalen from comment #6)
> Testing with the latest - arm-image-installer-2.12-1.fc30.noarch

Also: arm-image-installer-2.12-1.fc30.noarch

 
> I also tried with SE Linux in permissive with no change. 

On x86_64 host system one must have: getenforce == Disabled
Is it really so? getenforce == Permissive will still build a correct ARM image, only getenforce == Disabled will break the built ARM image.


> sudo arm-image-installer --target=rpi3
> --image=Fedora-Minimal-armhfp-30-1.2-sda.raw.xz --resizefs --norootpass
> --media=/dev/sdd --args "systemd.journald.forward_to_console=1" --addconsole

I have done as root:
  arm-image-installer --target=rpi3 --image=Fedora-Minimal-armhfp-30-1.2-sda.raw.xz --norootpass --media=/dev/sdb
  (sorry it was not literally your command but I believe it would be the same)
And then it shows login screen but when I type "root" it asks for "Password:" and I cannot login with any password I try.

I do not know how to catch that console afterwards so I fsck-ed+mounted the device afterwards and typed:
  journalctl --root=/mnt/
Which I am attaching. It starts with:
Apr 12 17:18:02 localhost audit[617]: AVC avc:  denied  { read } for  pid=617 comm="sh" name="passwd" dev="sda3" ino=35796 scontext=system_u:system_r:loadkeys_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0


> To remove the root password requirement the script uses "sed -i
> 's/root:x:/root::/' /tmp/root/etc/passwd"

The problem is that the installing (=host=x86_64) system without SELinux will corrupt by that sed the SELinux context of /tmp/root/etc/passwd.

Comment 10 Jan Kratochvil 2019-06-26 13:23:01 UTC
(In reply to Paul Whalen from comment #8)
> OK, reproduced with selinux disabled on the host (tsk tsk)

Oops, OK, great; I did not have to spend time reproducing it again, sorry I did not refresh my browser window.

Comment 11 Fedora Update System 2019-06-26 18:54:20 UTC
FEDORA-2019-2dd9f78d69 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2dd9f78d69

Comment 12 Fedora Update System 2019-06-26 18:54:22 UTC
FEDORA-2019-7cd0e1fc4b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-7cd0e1fc4b

Comment 13 Fedora Update System 2019-06-27 01:41:46 UTC
arm-image-installer-2.13-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7cd0e1fc4b

Comment 14 Fedora Update System 2019-06-27 02:43:48 UTC
arm-image-installer-2.13-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2dd9f78d69

Comment 15 Fedora Update System 2019-07-06 04:09:15 UTC
arm-image-installer-2.13-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.