Bug 1949456

Summary: Cannot install TAR images from osbuild using anaconda
Product: Red Hat Enterprise Linux 8 Reporter: Terry Bowling <tbowling>
Component: osbuild-composerAssignee: Image Builder team <osbuilders>
Status: NEW --- QA Contact: Release Test Team <release-test-team>
Severity: high Docs Contact:
Priority: high    
Version: 8.3CC: akoutsou, jkonecny, jstodola, rvykydal, sbarcomb, tgunders, vponcova
Target Milestone: betaFlags: tbowling: needinfo? (tgunders)
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Anaconda bug report
none
blueprint file
none
kickstart file using liveimg directive
none
8.3 boot iso with liveimg directive creates weird rescue boot entry
none
python definition file for Image Builder to create TAR images. none

Description Terry Bowling 2021-04-14 10:25:38 UTC
Created attachment 1771849 [details]
Anaconda bug report

Description of problem:

When using the liveimg kickstart directive to install a TAR image created by Image Builder, Anaconda fails to install it properly.

The first error seen during graphical installation process is error that it could not install the bootloader.  Additional errors follow.  I created a bug report from the Anaconda interface which includes many logs and debug info.  I am attaching kickstart file and blueprint file used to recreate the issue.

Version-Release number of selected component (if applicable):

From Image Build node - latest rhel-8.3
# rpm -qa |grep -ie osbuild -e composer
osbuild-ostree-18-3.el8.noarch
composer-cli-28.14.55-2.el8.x86_64
python3-osbuild-18-3.el8.noarch
osbuild-18-3.el8.noarch
osbuild-selinux-18-3.el8.noarch
cockpit-composer-22.1-1.el8.noarch
osbuild-composer-worker-20.1-1.el8.x86_64
osbuild-composer-20.1-1.el8.x86_64

Used boot DVD ISO image for 8.3

How reproducible:

Steps to Reproduce:
1. Create a minimal rhel-8.3 TAR image using the latest osbuild and the attached blueprint.
2. Serve the attached kickstart file and TAR image from a web server.  Adjust the kickstart file to point the liveimg directive to use the TAR image on the webserver.
3. Use rhel-8.3 boot iso to boot a VM on libvirt hypervisor, passing boot options to use kickstart file from the webserver.

Actual results:
Fails to install the TAR as a system that will boot.  Anaconda reports errors.

Expected results:
The VM should reboot successfully as a nice, clean, shiny RHEL 8.3 minimal install.


Additional info:

Comment 1 Terry Bowling 2021-04-14 10:29:31 UTC
Created attachment 1771851 [details]
blueprint file

Comment 2 Terry Bowling 2021-04-14 10:30:30 UTC
Created attachment 1771852 [details]
kickstart file using liveimg directive

Comment 3 Christian Kellner 2021-04-14 10:41:41 UTC
Anaconda complains about:

pyanaconda.modules.common.errors.installation.SecurityInstallationError: /usr/sbin/authconfig is missing. Cannot setup authentication.

I wonder if that should be in the image instead of the ISO.

Comment 4 Radek Vykydal 2021-04-14 11:25:22 UTC
It needs to be in the image, perhaps by pulling in anaconda-tools. In general, the image that should be working with Anaconda has some requirements which AFAIK has not been addressed explicitly yet (for documentation at least).

See my testing blueprint in:
https://issues.redhat.com/browse/INSTALLER-2300
or perhaps a better used one by Zdenek Veleba during testing:
https://bugzilla.redhat.com/show_bug.cgi?id=1919464#c1

There is also a potential issue caused by installer package requirements being defined during installation, see
part 4. of https://docs.google.com/document/d/1_e5r1cjb2fuoh4uQNZqdywBA8048V_GhRi_f8VbP8yU/edit?ts=5fb53f77#heading=h.fm8u80iq3ve8

Comment 5 Jiri Konecny 2021-04-14 13:01:35 UTC
Also one more note, you should not use the `auth` command. It is deprecated from 8.0 see bug 1782652. Instead you should use the authselect command.

Comment 6 Terry Bowling 2021-04-14 19:40:40 UTC
(In reply to Radek Vykydal from comment #4)
> It needs to be in the image, perhaps by pulling in anaconda-tools. In
> general, the image that should be working with Anaconda has some
> requirements which AFAIK has not been addressed explicitly yet (for
> documentation at least).
> 
> See my testing blueprint in:
> https://issues.redhat.com/browse/INSTALLER-2300
> or perhaps a better used one by Zdenek Veleba during testing:
> https://bugzilla.redhat.com/show_bug.cgi?id=1919464#c1
> 
> There is also a potential issue caused by installer package requirements
> being defined during installation, see
> part 4. of
> https://docs.google.com/document/d/
> 1_e5r1cjb2fuoh4uQNZqdywBA8048V_GhRi_f8VbP8yU/edit?ts=5fb53f77#heading=h.
> fm8u80iq3ve8


Looking at my 8.3 image builder node, I do not see an anaconda-tools package I can pull into my build.  I see anaconda, anaconda-core, anaconda-dracut, anaconda-install-env-deps, and a few others that do not look like they would help.  Should I try installing some of these and see if that helps?

Would removing my user addition in the kickstart alleviate the authconfig issue?  I can define my users, ssh keys, and passwords in the image build.

Comment 7 Radek Vykydal 2021-04-15 07:18:23 UTC
(In reply to Terry Bowling from comment #6)
> (In reply to Radek Vykydal from comment #4)
> > It needs to be in the image, perhaps by pulling in anaconda-tools. In
> > general, the image that should be working with Anaconda has some
> > requirements which AFAIK has not been addressed explicitly yet (for
> > documentation at least).
> > 
> > See my testing blueprint in:
> > https://issues.redhat.com/browse/INSTALLER-2300
> > or perhaps a better used one by Zdenek Veleba during testing:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1919464#c1
> > 
> > There is also a potential issue caused by installer package requirements
> > being defined during installation, see
> > part 4. of
> > https://docs.google.com/document/d/
> > 1_e5r1cjb2fuoh4uQNZqdywBA8048V_GhRi_f8VbP8yU/edit?ts=5fb53f77#heading=h.
> > fm8u80iq3ve8
> 
> 
> Looking at my 8.3 image builder node, I do not see an anaconda-tools package
> I can pull into my build.  I see anaconda, anaconda-core, anaconda-dracut,
> anaconda-install-env-deps, and a few others that do not look like they would
> help.  Should I try installing some of these and see if that helps?

anaconda-tools is a group defined in comps:
http://download.englab.brq.redhat.com/rhel-8/rel-eng/RHEL-8/RHEL-8.3.0-RC-1.1/compose/AppStream/x86_64/os/repodata/

> Would removing my user addition in the kickstart alleviate the authconfig
> issue?  I can define my users, ssh keys, and passwords in the image build.

I don't see any user defined in kickstart in comment #2 but you are probably not meaning that one.
IIRC for automatic installation there is admin user or rootpw required to be defined in kickstart. In Bare Metal Image Deployment feature for RHEL 8.4 we intended to disable the USERS module in Anaconda for this reason assuming that user would be added in the image by Image Builder.

I am not sure removing user would fix the authconfig issue. anaconda-tools contains 2 mandatory packages - authselect-compat (with /usr/sbin/authconfig symlink) and anaconda-kdump-addon so that would be a proper fix I think.

Comment 8 Terry Bowling 2021-04-16 15:06:57 UTC
I have added all of the anaconda packages and authselect-compat to my TAR image, as well as further simplified my kickstart as shown below.  It still fails, first giving an error it cannot install the bootloader and asks if I want to preceed anyway.  If I answer yes, it completes the install, along with some configuration and seems to complete successfully.  However, after reboot, it fails to boot, stalling at the Bios screen, seemingly not finding a boot loader.

I propose that this is either a bug with how image builder is creating the image, or a bug in anaconda to use the liveimg directive with a tar image.  Requesting escalation and prioritization of this bug.

lang en_US
keyboard us
timezone America/New_York --isUtc

install
liveimg --url="http://192.168.1.5/kicks/rhel83-gold.tar.xz"

zerombr
clearpart --all --initlabel
autopart --type=plain --fstype=xfs 

reboot
text
skipx
network --bootproto=dhcp

Comment 9 Radek Vykydal 2021-04-16 15:39:50 UTC
I'd need to see the blueprint file and the anaconda logs to be able to tell more, but my guess is something is missing in the image / blueprint (kernel?). Apparently the package set required for the image installable by Anaconda needs to be documented. Or handled by packaging or comps.

Please attach the blueprint you used to create the image and /tmp/syslog gathered form the installer environment at the end of installation (you can switch to terminal/shell with Ctrl-Alt-F2) or /var/log/anaconda/journal.log from the installed system.

Comment 10 Vendula Poncova 2021-04-16 15:54:17 UTC
From anaconda.log:

16:51:31,839 INF payload.base: Installation requirements: [('package', 'authselect-compat', PayloadRequirement(id=authselect-compat, reasons=[PayloadRequirementReason(reason='Needed to support legacy authconfig kickstart command.', strong=True)])), ('package', 'firewalld', PayloadRequirement(id=firewalld, reasons=[PayloadRequirementReason(reason='Requested by the firewall kickstart command.', strong=True)])), ('package', 'chrony', PayloadRequirement(id=chrony, reasons=[PayloadRequirementReason(reason='Needed to run NTP service.', strong=True)])), ('package', 'xfsprogs', PayloadRequirement(id=xfsprogs, reasons=[PayloadRequirementReason(reason='Required to manage storage devices.', strong=True)])), ('package', 'e2fsprogs', PayloadRequirement(id=e2fsprogs, reasons=[PayloadRequirementReason(reason='Required to manage storage devices.', strong=True)])), ('package', 'grub2', PayloadRequirement(id=grub2, reasons=[PayloadRequirementReason(reason='Necessary for the bootloader configuration.', strong=True)])), ('package', 'grub2-tools', PayloadRequirement(id=grub2-tools, reasons=[PayloadRequirementReason(reason='Necessary for the bootloader configuration.', strong=True)]))]

Based on the logs, you need the following packages:

authselect-compat
firewalld
chrony
xfsprogs
e2fsprogs
grub2
grub2-tools

Or install the anaconda-tools group to make sure that all installation requirements are fulfilled:

anaconda-tools

The name of this group is misleading. Please, don't install anaconda packages. They are not needed. You also shouldn't need the anaconda-kdump-addon package. I am not sure why it is in this group, it looks like a bug to me.

Also, I think you don't install the core group. That could also cause some issues:

core

Comment 11 Terry Bowling 2021-04-20 20:57:53 UTC
I've been testing this and made some slow progress, but still have a few new issues preventing this from being useful.

Manually edited Image Builder Blueprint file to include the @core and @anaconda-tools package groups.  The UI does not allow this, so manual editing was required.  Blueprint pasted below.

Edited kickstart file as shown below.

oustanding issues:
  1. User "admin" defined in blueprint with simple password and ssh key.  However, cannot login as this user via either method.
  2. Configuring a "testuser" and root accounts with simple passwords and ssh keys, however, cannot login as either.  A "No shell" effor flashes to the screen but disappears before I can read all of the text.  Will examine the disk image later to understand better.

---------
Blueprint
---------
name = "RHEL_Gold_Image"
description = "My minimal RHEL golden image"
version = "0.0.18"
modules = []

[[packages]]
name = "bash-completion"
version = "*"

[[packages]]
name = "vim-enhanced"
version = "*"

[[groups]]
name = "anaconda-tools"

[[groups]]
name = "core"

[customizations]
hostname = "rhel-min-template"

[[customizations.user]]
name = "admin"
description = "ansible_admin"
password = "$6$3I/02Ww4rfUzaRd.$sZPJ7S... snip ...KsUgK/NHEMoRRk3x/D47."
key = "ssh-rsa AAAAB3Nz... snip ...rPf demo"
groups = ["wheel"]
[customizations.firewall]
[customizations.firewall.services]
enabled = ["ssh", "cockpit"]
[customizations.services]
enabled = ["sshd", "cockpit.socket"]

---------
Kickstart
---------
lang en_US
keyboard us
timezone America/New_York --isUtc

install
liveimg --url="http://192.168.1.5/kicks/rhel83-gold.tar.xz"

zerombr
clearpart --all --initlabel
#autopart --type=plain --fstype=xfs 
part /boot --fstype=xfs --size=1024 --asprimary
part pv.0 --fstype=lvmpv --ondisk=vda  --size=1 --grow  
#part pv.0 --fstype=lvmpv --size 1 --grow  
volgroup system --pesize=4096 pv.0 
logvol / --vgname=system --name=root --fstype=xfs --percent=80
logvol /var --vgname=system --name=var --fstype=xfs --percent=10 --grow --maxsize=5000
logvol /var/log --vgname=system --name=varlog --fstype=xfs --percent=5 --grow --maxsize=5000
logvol swap  --vgname=system --name=swap --fstype="swap" --recommended

reboot
#text
skipx
network --bootproto=dhcp
auth --passalgo=sha512 --useshadow
selinux --enforcing
#firewall --enabled --http --ssh
firstboot --disable

user --name=testuser --groups=wheel --plaintext --password=edge1234

sshkey --username=testuser "ssh-rsa AAAAB3N... snip ...7lbrPf demo"

rootpw --plaintext edge1234

sshkey --username=root "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA... snip ...7lbrPf demo"

Comment 12 Terry Bowling 2021-04-22 11:21:16 UTC
Created attachment 1774463 [details]
8.3 boot iso with liveimg directive creates weird rescue boot entry

When installing an 8.3 TAR image created by Image Builder with the 8.3 Boot ISO using liveimg kickstart directive, it creates a weird rescue boot entry as shown in this screenshot.

Comment 13 Radek Vykydal 2021-04-22 11:28:37 UTC
(In reply to Terry Bowling from comment #12)
> Created attachment 1774463 [details]
> 8.3 boot iso with liveimg directive creates weird rescue boot entry
> 
> When installing an 8.3 TAR image created by Image Builder with the 8.3 Boot
> ISO using liveimg kickstart directive, it creates a weird rescue boot entry
> as shown in this screenshot.

This is fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1919463

Comment 14 Terry Bowling 2021-04-22 14:49:58 UTC
according to @gicmo, there was a fix to added --selinux and other attributes to the tar definition beginning with https://github.com/osbuild/osbuild/commit/920f46880b316e52f4094349631c17a66a4cb602 , however, it seems it has changed further since.

I downloaded the osbuild package from the latest 8.4 nightlies and extracted and replaced the file /usr/lib/osbuild/assemblers/org.osbuild.tar in my 8.3 build node.  However, my build fails with the following errors in the logs:

Stage: org.osbuild.fix-bls
{}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system

Stage: org.osbuild.locale
{
  "language": "en_US"
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
/run/osbuild/tree/etc/locale.conf written.

Stage: org.osbuild.hostname
{
  "hostname": "rhel-min-template"
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
/run/osbuild/tree/etc/hostname written.

Stage: org.osbuild.users
{
  "users": {
    "admin": {
      "groups": [
        "wheel"
      ],
      "description": "ansible_admin",
      "password": "$6$wRWV9JxR2QLaCWJ.$.fdq50gYnHwg7/2gyx.XG/fUgSzs09dseeFDHyR7uFkGx1Wqe5zLeaoh0M.nl63nFpWhnmdRX0EXc4tJLExF9/",
      "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD37gJHuryPNTId01/A63jfqYzvrDlEDfvMgMPSscQejxVK7Wa5Nr/kNQR7EQvDC3xUR7stO0UaVFf9B/rglqGJ/LNQmAtA+F2KBYDv89vuwYHjcfSMwvNj9dvrec0V3Vxe2HscIu7bhYXCUtmhZVNi5DpIayeogOhqoJkuoSIA2x1fYbE0XBRwmwD+hkF98t9P+4rMMDTMXJ1hUZNxBaAHuq6mf5+k+nYBvJoGhNGKVrnaQH2Xd10wS1UJ08WyEpvVDmDyimKiDvRyiI5GN1ph1zn+vwtOaLqk3FHo9ogyDyjWkwDuqtz34KW+D0x70dQCJjAyG5Fmg1f/AG7lbrPf demo"
    }
  }
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system

Stage: org.osbuild.systemd
{
  "enabled_services": [
    "sshd",
    "cockpit.socket"
  ]
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Created symlink /run/osbuild/tree/etc/systemd/system/sockets.target.wants/cockpit.socket → /usr/lib/systemd/system/cockpit.socket.

Stage: org.osbuild.firewall
{
  "enabled_services": [
    "ssh",
    "cockpit"
  ]
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Adding service 'ssh' to default zone.
ALREADY_ENABLED: ssh
Adding service 'cockpit' to default zone.
ALREADY_ENABLED: cockpit
success

Stage: org.osbuild.selinux
{
  "file_contexts": "etc/selinux/targeted/contexts/files/file_contexts"
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system

Assembler org.osbuild.tar:
{
  "filename": "root.tar.xz",
  "compression": "xz"
}

Output:
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Traceback (most recent call last):
  File "/run/osbuild/lib/assemblers/org.osbuild.tar", line 109, in <module>
    args = osbuild.api.arguments()
AttributeError: module 'osbuild.api' has no attribute 'arguments'

Comment 15 Terry Bowling 2021-04-22 19:14:23 UTC
I tried to merge the selinux and similar attributes from the 8.4 nightly build into the 8.3 version of /usr/lib/osbuild/assemblers/org.osbuild.tar, but I don't have something correct so it is failing.  Below is a copy of my modifications and the errors I collected from journalctl - the osbuild logs were empty as it failed to early.

# journalctl -b -u osbuild-worker
Apr 22 10:28:00 util8vm osbuild-worker[63802]: Running job c2d7e1c8-8ad4-414c-8ba5-5c78b26e56cf
Apr 22 10:40:36 util8vm osbuild-worker[63802]: 2021/04/22 10:40:36   Job failed: running osbuild failed: exit status 1
Apr 22 10:40:36 util8vm osbuild-worker[63802]: Waiting for a new job...
Apr 22 11:13:38 util8vm systemd[1]: Stopping OSBuild Composer Worker (1)...
Apr 22 11:13:38 util8vm systemd[1]: osbuild-worker: Succeeded.
Apr 22 11:13:38 util8vm systemd[1]: Stopped OSBuild Composer Worker (1).
Apr 22 11:14:16 util8vm systemd[1]: Started OSBuild Composer Worker (1).
Apr 22 11:14:16 util8vm osbuild-worker[95933]: Waiting for a new job...
Apr 22 11:18:15 util8vm osbuild-worker[95933]: Running job 2a68cb03-cbad-4270-b545-4aaf00d00c42
Apr 22 11:18:16 util8vm osbuild-worker[95933]: Traceback (most recent call last):
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/bin/osbuild", line 11, in <module>
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     load_entry_point('osbuild==18', 'console_scripts', 'osbuild')()
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/main_cli.py", line 185, in main_cli
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     sys.exit(osbuild_cli(sys_argv=sys.argv))
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/main_cli.py", line 112, in osbuild_cli
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     res = osbuild.meta.validate(manifest, index)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/meta.py", line 472, in validate
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     schema = index.get_schema("Assembler", name)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/meta.py", line 422, in get_schema
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     info = self.get_module_info(klass, name)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/meta.py", line 398, in get_module_info
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     info = ModuleInfo.load(self.path, klass, name)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/meta.py", line 356, in load
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     return cls(klass, name, info)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib/python3.6/site-packages/osbuild/meta.py", line 283, in __init__
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     self.opts = json.loads("{" + opts + "}")
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     return _default_decoder.decode(s)
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Apr 22 11:18:16 util8vm osbuild-worker[95933]:   File "/usr/lib64/python3.6/json/decoder.py", line 355, in raw_decode
Apr 22 11:18:16 util8vm osbuild-worker[95933]:     obj, end = self.scan_once(s, idx)
Apr 22 11:18:16 util8vm osbuild-worker[95933]: json.decoder.JSONDecodeError: Expecting ',' delimiter: line 14 column 3 (char 321)
Apr 22 11:18:16 util8vm osbuild-worker[95933]: 2021/04/22 11:18:16   Job failed: error decoding osbuild output: &errors.errorString{s:"EOF"}
Apr 22 11:18:16 util8vm osbuild-worker[95933]: Waiting for a new job...


# cat /usr/lib/osbuild/assemblers/org.osbuild.tar

#!/usr/libexec/platform-python
"""
Assemble a tar archive

Assembles the tree into a tar archive named `filename`.

Uses the buildhost's `tar` command, like: `tar -cf $FILENAME -C $TREE`

If the `compression` option is given, the archive will be compressed by passing
the `--{compression}` option to `tar`. (This option is non-standard and might
not work for anything other than GNU tar.)

Known options for `compression`: "bzip2", "xz", "lzip", "lzma", "lzop", "gzip".

Note that using `compression` does not add an extension to `filename`, so the
caller is responsible for making sure that `compression` and `filename` match.

Buildhost commands used: `tar` and any named `compression` program.
"""


import json
import subprocess
import sys

SCHEMA = """
"additionalProperties": false,
"required": ["filename"],
"properties": {
  "filename": {
    "description": "Filename for tar archive",
    "type": "string"
  },
  "compression": {
    "description": "Name of compression program",
    "type": "string",
    "enum": ["bzip2", "xz", "lzip", "lzma", "lzop", "gzip"]
  }
  "acls": {
      "description": "Enable support for POSIX ACLs",
      "type": "boolean",
      "default": true
  },
  "selinux": {
      "description": "Enable support for SELinux contexts",
      "type": "boolean",
      "default": true
  },
  "xattrs": {
      "description": "Enable support for extended attributes",
      "type": "boolean",
      "default": true
  }
}
"""

def main(tree, output_dir, options):
    filename = options["filename"]
    compression = options.get("compression")

    extra_args = []
    if compression is not None:
        if compression not in {"bzip2", "xz", "lzip", "lzma", "lzop", "gzip"}:
            return 1
        extra_args.append(f"--{compression}")

    # Set environment variables for the tar operation.
    tar_env = {
        # Speed up xz by allowing it to use all CPU cores for compression.
        "XZ_OPT": "--threads 0"
    }

    # SELinux context, ACLs and extended attributes
    if options.get("acls", True):
        extra_args += ["--acls"]

    if options.get("selinux", True):
        extra_args += ["--selinux"]

    if options.get("xattrs", True):
        extra_args += ["--xattrs", "--xattrs-include", "*"]

    # Set up the tar command.
    tar_cmd = [
        "tar",
        *extra_args,
        "-cf", f"{output_dir}/{filename}",
        "-C", tree,
        "."
    ]

    # Make a tarball of the tree.
    subprocess.run(
        tar_cmd,
        stdout=subprocess.DEVNULL,
        check=True,
        env=tar_env
    )

    return 0


if __name__ == '__main__':
    args = json.load(sys.stdin)
    r = main(args["tree"], args["output_dir"], args["options"])
    sys.exit(r)

Comment 16 Terry Bowling 2021-04-23 14:49:42 UTC
Created attachment 1774852 [details]
python definition file for Image Builder to create TAR images.

This backports some changes to /usr/lib/osbuild/assemblers/org.osbuild.tar to ensure that Image Builder properly configures SELINUX contexts, POSIX ACLs, and extended attributes withing the TAR image.

Comment 17 Terry Bowling 2021-04-23 15:04:39 UTC
SUCCESS!

Using changes backported from 8.4 to /usr/lib/osbuild/assemblers/org.osbuild.tar provided in 

1. On a RHEL 8.3 Image Builder node, download the attachment from comment 16 above (https://bugzilla.redhat.com/show_bug.cgi?id=1949456#c16) and save as /usr/lib/osbuild/assemblers/org.osbuild.tar  - make a backup of the previous file first.

2. Use the sample blueprint listed below.  Not it includes adding the @core and @anaconda-tools package GROUPS, which is not yet possible using the GUI.

3. Use the sample kickstart file.  Note that the Anaconda requires either the root user to have a password set (not recommended practice in images), or a user defined for the kickstart installation to complete.  It has no awareness of the user defined in the Image Builder blueprint.  After install, one of the accounts could be removed.  The kickstart file could be placed on the same webserver directory as the TAR Image

4. Use the RHEL 8.4 Beta Boot or full DVD ISO image to kickstart the node and write the TAR image using the liveimg directive to provide the location.  Note that the 8.4 boot iso fixes a number of issues for this use case, including an errant rescue boot entry as default, and other things described above.

End result is a successful RHEL 8.3 deployment.  Presumably this would work for 8.2 or 8.1, but I have not test that.


---------
Blueprint
---------
name = "RHEL_Gold_Image"
description = "My minimal RHEL golden image"
version = "0.0.20"
modules = []

[[packages]]
name = "bash-completion"
version = "*"

[[packages]]
name = "vim-enhanced"
version = "*"

[[packages]]
name = "cockpit"
version = "*"

[[groups]]
name = "anaconda-tools"

[[groups]]
name = "core"

[customizations]
hostname = "rhel-min-template"

[[customizations.user]]
name = "admin"
description = "ansible_admin"

# edge1234
password = "$6$wRWV9JxR2QLaCWJ.$.fdq50gYnHwg7/2gyx.XG/fUgSzs09dseeFDHyR7uFkGx1Wqe5zLeaoh0M.nl63nFpWhnmdRX0EXc4tJLExF9/"

key = "ssh-rsa AAAAB3  ... snip ...  lbrPf demo"
groups = ["wheel"]

[customizations.firewall]
[customizations.firewall.services]
enabled = ["ssh", "cockpit"]
[customizations.services]
enabled = ["sshd", "cockpit.socket"]


---------
Kickstart
---------
# cat rhel83-gold.cfg 
#
# This kickstart template will use the RHEL 8.4+ Boot DVD ISO
# to automation installation of a TAR OS image created by
# RHEL 8 Image Builder.
#

lang en_US
keyboard us
timezone America/New_York --isUtc

install
liveimg --url="http://192.168.1.5/kicks/rhel83-gold.tar.xz"

zerombr
clearpart --all --initlabel
#autopart --type=plain --fstype=xfs 
part /boot --fstype=xfs --size=1024 --asprimary
part pv.0 --fstype=lvmpv --ondisk=vda  --size=1 --grow  
volgroup system --pesize=4096 pv.0 
logvol / --vgname=system --name=root --fstype=xfs --percent=80
logvol /var --vgname=system --name=var --fstype=xfs --percent=10 --grow --maxsize=5000
logvol /var/log --vgname=system --name=varlog --fstype=xfs --percent=5 --grow --maxsize=5000
logvol swap  --vgname=system --name=swap --fstype="swap" --recommended

reboot
#text
skipx
network --bootproto=dhcp
auth --passalgo=sha512 --useshadow

# Configured in Blueprint instead
# selinux --enforcing
# firstboot --disable

# Configure firewall & ervices in Blueprint instead
# firewall --enabled --http --ssh

user --name=testuser --groups=wheel --gecos="test user" --iscrypted --password="$6$wx26nHNmRxsZElqd$OSof8TucuRIh.ar6wgLc/CB67KQmrxpuiU6VFuTkxHVHw8ruBn.WsHvgYoOU7Tj5XjKbgoNYj1Fc0j/V21Ayi."

sshkey --username=testuser "ssh-rsa AAAAB3  ... snip ...  lbrPf demo"

# edge1234
#rootpw --iscrypted "$6$wx26nHNmRxsZElqd$OSof8TucuRIh.ar6wgLc/CB67KQmrxpuiU6VFuTkxHVHw8ruBn.WsHvgYoOU7Tj5XjKbgoNYj1Fc0j/V21Ayi."

#sshkey --username=root "ssh-rsa AAAAB3  ... snip ...  lbrPf demo"

Comment 18 Jiri Konecny 2021-05-03 15:42:43 UTC
Great work Terry in making this work! Do I understand it correctly we can close this bug because the requirements will go to Image Builder?

Comment 19 Terry Bowling 2021-06-23 23:23:36 UTC
@jiri thank you!  Possibly, but I want to defer to @tgunders to confirm if he agrees, understand how it applies to classic rhel and edge use cases, and post a comment linking to the github link showing it is being added there before we close this ticket.
-Terry

Comment 21 Jan Stodola 2023-02-02 12:30:43 UTC
I'm reassigning this bug to Image builder to address comment 19.