Bug 2150920

Summary: Image Builder fails with some customizations
Product: Red Hat Enterprise Linux 8 Reporter: Christophe Besson <cbesson>
Component: osbuild-composerAssignee: Brian Lane <bcl>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: medium Docs Contact: Eliane Ramos Pereira <elpereir>
Priority: medium    
Version: 8.7CC: amepatil, atodorov, bcl, elpereir, jcastran, mmatsuya, obudai, sbarcomb, thozza
Target Milestone: rcKeywords: Regression, Reproducer, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: osbuild-composer-85-1.el8 Doc Type: Bug Fix
Doc Text:
Cause: The customizations.firewall example in the documentation at - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/composing_a_customized_rhel_system_image/index#image-customizations_creating-system-images-with-composer-command-line-interface is incorrect. It should be 'ports = ["PORTS"]' not 'port = ["PORTS"]' Consequence: Users construct blueprints that will not work correctly. Fix: Update the docs. Result: Happy Customers!
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:25:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christophe Besson 2022-12-05 16:21:46 UTC
Description of problem:
The below customizations were working on RHEL 8.6: services, firewall.services, firewall. After upgrading to RHEL 8.7, the below error is printed:

# composer-cli blueprints show notworking
ERROR: Show Error: Get "http://localhost/api/v1/blueprints/info/notworking?format=toml": EOF


Version-Release number of selected component (if applicable):
osbuild-composer-62-3.el8_7

How reproducible:
Always

Steps to Reproduce:
1. Create a blueprint with
[customizations.services]
enabled = ["sshd.service", "firewalld.service"]

[customizations.firewall]
port = ["22"]

[customizations.firewall.services]
enabled = ["ssh"]

2. Push it and show it


Actual results:
golang runtime error:
Dec 05 16:21:44 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:44 POST /api/v1/blueprints/new
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:47 GET /api/v1/blueprints/info/vmware-min
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:47 http: panic serving @: runtime error: comparing uncomparable type blueprint.ServicesCustomization
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: goroutine 93 [running]:
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.(*conn).serve.func1()
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/net/http/server.go:1825 +0xbf
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: panic({0x559fb061f720, 0xc000131290})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/runtime/panic.go:844 +0x258
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).safeEncode.func1()
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:152 +0x78
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: panic({0x559fb061f720, 0xc000131290})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/runtime/panic.go:838 +0x207
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.isEmpty({0x559fb0674420?, 0xc000b82660?, 0x0?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:656 +0xf9
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct.func2({0xc00000e228?, 0x1, 0x559fb0746560?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:507 +0x23e
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct(0xc000b93518, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x1?}, 0x0)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:529 +0x266
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eMapOrStruct(0xc000b93518?, {0xc000131090?, 0x1?, 0x0?}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb0746560?}, 0x0?)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:370 +0x4e
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eTable(0xc000b93518, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb05c2c80?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:362 +0x1a5
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).encode(0x559fb05c5411?, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb05c5411?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:202 +0x399
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct.func2({0xc00000e0f0?, 0x1, 0x559fb0729bc0?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:520 +0x425
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct(0xc000b93518, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x2000?}, 0x0)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:529 +0x266
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eMapOrStruct(0xc000b93180?, {0x559fb0f1ad78?, 0x559fb062f400?, 0x559fb0729bc0?}, {0x559fb0729bc0?, 0xc0001ae160?, 0x559fb0729bc0?}, 0xc0?)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:370 +0x4e
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eTable(0xc000b93518, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x1?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:362 +0x1a5
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).encode(0xc0001ae160?, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x7fac23ee55b8?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:202 +0x399
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).safeEncode(0x559fb0729bc0?, {0x559fb0f1ad78?, 0xc000b93380?, 0x559faf654f4b?}, {0x559fb0729bc0?, 0xc0001ae160?, 0xc0001ae0b0?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:155 +0x77
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).Encode(0xc000b93518, {0x559fb0729bc0?, 0xc0001ae160?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:139 +0xc8
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).blueprintsInfoHandler(0xc00014d760, {0x559fb078e6d8?, 0xc0001f4000}, 0xc0005d0100, {0xc0005c4180?, 0x2, 0xc00037801b?})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:1462 +0xbc5
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00054eba0, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/julienschmidt/httprouter/router.go:387 +0x82b
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).ServeHTTP(0xc00014d760, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:299 +0x16a
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.serverHandler.ServeHTTP({0xc000134330?}, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100)
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/net/http/server.go:2916 +0x43b
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.(*conn).serve(0xc0003fb680, {0x559fb078f5b0, 0xc00030e090})
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/net/http/server.go:1966 +0x5d7
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: created by net/http.(*Server).Serve
Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]:         /usr/lib/golang/src/net/http/server.go:3071 +0x4db

Additional info:
- the workaround could be downgrading, but I just tried and got the following error:
Dec 05 16:39:00 localhost.localdomain osbuild-composer[2195]: cannot read state: error reading db file state: unexpected target name

Comment 2 Brian Lane 2022-12-06 17:58:31 UTC
Two unrelated issues here, the first is that you should use 'ports = ["22"]' not 'port'. That won't fix the crash though. Until that is fixed you can work around the crash by viewing the blueprint as JSON with 'composer-cli --json blueprints show notworking'.

This needs a toml library fix backported from upstream.

Comment 4 Christophe Besson 2022-12-07 11:13:52 UTC
Ah yep, port in singular was a typo.
At a first glance I thought these directives were not taken into account anymore, but it's almost cosmetic.
Hence reducing the severity to medium.

Thanks for your prompt feedback!

Comment 5 Christophe Besson 2023-06-20 10:52:29 UTC
Late note.

The doc needs to be amended, as it mentions "port" in singular whereas the code expects it in plural (as a list).
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/composing_a_customized_rhel_system_image/index#image-customizations_creating-system-images-with-composer-command-line-interface

Comment 6 Christophe Besson 2023-06-21 08:37:55 UTC
Sorry to bother you again but the doc is also wrong on the expected value...

It tells:
"""
To enable lists, you can use numeric ports, or their names from the /etc/services file.
"""

Intuitively the user can define ports with ["22", "80"] or ["22/tcp", "80/tcp"] but both forms won't work, leading to a FAILED compose without any log explaining what happened ("log is empty").
This is problematic for the support as we need to rely on strace to figure out the problem:

/tmp/worker.strace:9697  15:06:08.743339 write(1<pipe:[164644]>, "{\"type\": \"https://osbuild.org/validation-error\", \"title\": \"JSON Schema validation failed\", \"success\": false, \"errors\": [{\"message\": \"'80/tcp' does not match '.:(tcp|udp|sctp|dccp)$'\", \"path\": [\"pipelines\", 1, \"stages\", 7, \"options\", \"ports\", 0]}]}\n", 248) = 248 <0.000030>

As per /usr/lib/osbuild/stages/org.osbuild.firewall, the expected pattern seems to be a numeric port or a port range followed by a dash and the protocol (which is mandatory).
 34 SCHEMA = """
 35 "additionalProperties": false,
 36 "properties": {
 37   "ports": {
 38     "description": "Ports (or port ranges) to open",
 39     "type": "array",
 40     "items": {
 41       "type": "string",
 42       "description": "A port or port range: 'portid[-portid]:protocol'",
 43       "pattern": ".:(tcp|udp|sctp|dccp)$"
 44     }
 45   },

Thanks!

Comment 7 Brian Lane 2023-06-21 15:30:04 UTC
The documentation for blueprints is here - https://www.osbuild.org/guides/image-builder-on-premises/blueprint-reference.html#firewall
I'm not sure how we ended up with things getting out of sync.

Comment 8 Ondřej Budai 2023-07-12 09:35:51 UTC
@elpereir This seems like an issue in the documentation. May I ask you to take a look? The upstream docs should be correct about the service customization. Many thanks! :)

Comment 9 Ondřej Budai 2023-07-12 09:44:35 UTC
FTR, the crash was fixed upstream in https://github.com/osbuild/osbuild-composer/pull/3099.

Comment 18 errata-xmlrpc 2023-11-14 15:25:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (osbuild, osbuild-composer, and cockpit-composer bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:6906