Description of problem: The below customizations were working on RHEL 8.6: services, firewall.services, firewall. After upgrading to RHEL 8.7, the below error is printed: # composer-cli blueprints show notworking ERROR: Show Error: Get "http://localhost/api/v1/blueprints/info/notworking?format=toml": EOF Version-Release number of selected component (if applicable): osbuild-composer-62-3.el8_7 How reproducible: Always Steps to Reproduce: 1. Create a blueprint with [customizations.services] enabled = ["sshd.service", "firewalld.service"] [customizations.firewall] port = ["22"] [customizations.firewall.services] enabled = ["ssh"] 2. Push it and show it Actual results: golang runtime error: Dec 05 16:21:44 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:44 POST /api/v1/blueprints/new Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:47 GET /api/v1/blueprints/info/vmware-min Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: 2022/12/05 16:21:47 http: panic serving @: runtime error: comparing uncomparable type blueprint.ServicesCustomization Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: goroutine 93 [running]: Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.(*conn).serve.func1() Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/net/http/server.go:1825 +0xbf Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: panic({0x559fb061f720, 0xc000131290}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/runtime/panic.go:844 +0x258 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).safeEncode.func1() Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:152 +0x78 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: panic({0x559fb061f720, 0xc000131290}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/runtime/panic.go:838 +0x207 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.isEmpty({0x559fb0674420?, 0xc000b82660?, 0x0?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:656 +0xf9 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct.func2({0xc00000e228?, 0x1, 0x559fb0746560?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:507 +0x23e Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct(0xc000b93518, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x1?}, 0x0) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:529 +0x266 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eMapOrStruct(0xc000b93518?, {0xc000131090?, 0x1?, 0x0?}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb0746560?}, 0x0?) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:370 +0x4e Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eTable(0xc000b93518, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb05c2c80?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:362 +0x1a5 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).encode(0x559fb05c5411?, {0xc000131090, 0x1, 0x1}, {0x559fb0746560?, 0xc0003ee160?, 0x559fb05c5411?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:202 +0x399 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct.func2({0xc00000e0f0?, 0x1, 0x559fb0729bc0?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:520 +0x425 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eStruct(0xc000b93518, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x2000?}, 0x0) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:529 +0x266 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eMapOrStruct(0xc000b93180?, {0x559fb0f1ad78?, 0x559fb062f400?, 0x559fb0729bc0?}, {0x559fb0729bc0?, 0xc0001ae160?, 0x559fb0729bc0?}, 0xc0?) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:370 +0x4e Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).eTable(0xc000b93518, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x1?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:362 +0x1a5 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).encode(0xc0001ae160?, {0x559fb0f1ad78, 0x0, 0x0}, {0x559fb0729bc0?, 0xc0001ae160?, 0x7fac23ee55b8?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:202 +0x399 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).safeEncode(0x559fb0729bc0?, {0x559fb0f1ad78?, 0xc000b93380?, 0x559faf654f4b?}, {0x559fb0729bc0?, 0xc0001ae160?, 0xc0001ae0b0?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:155 +0x77 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/BurntSushi/toml.(*Encoder).Encode(0xc000b93518, {0x559fb0729bc0?, 0xc0001ae160?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/BurntSushi/toml/encode.go:139 +0xc8 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).blueprintsInfoHandler(0xc00014d760, {0x559fb078e6d8?, 0xc0001f4000}, 0xc0005d0100, {0xc0005c4180?, 0x2, 0xc00037801b?}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:1462 +0xbc5 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00054eba0, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/vendor/github.com/julienschmidt/httprouter/router.go:387 +0x82b Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).ServeHTTP(0xc00014d760, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /builddir/build/BUILD/osbuild-composer-62/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:299 +0x16a Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.serverHandler.ServeHTTP({0xc000134330?}, {0x559fb078e6d8, 0xc0001f4000}, 0xc0005d0100) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/net/http/server.go:2916 +0x43b Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: net/http.(*conn).serve(0xc0003fb680, {0x559fb078f5b0, 0xc00030e090}) Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/net/http/server.go:1966 +0x5d7 Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: created by net/http.(*Server).Serve Dec 05 16:21:47 localhost.localdomain osbuild-composer[1169]: /usr/lib/golang/src/net/http/server.go:3071 +0x4db Additional info: - the workaround could be downgrading, but I just tried and got the following error: Dec 05 16:39:00 localhost.localdomain osbuild-composer[2195]: cannot read state: error reading db file state: unexpected target name
Two unrelated issues here, the first is that you should use 'ports = ["22"]' not 'port'. That won't fix the crash though. Until that is fixed you can work around the crash by viewing the blueprint as JSON with 'composer-cli --json blueprints show notworking'. This needs a toml library fix backported from upstream.
https://github.com/osbuild/osbuild-composer/pull/3170
Ah yep, port in singular was a typo. At a first glance I thought these directives were not taken into account anymore, but it's almost cosmetic. Hence reducing the severity to medium. Thanks for your prompt feedback!
Late note. The doc needs to be amended, as it mentions "port" in singular whereas the code expects it in plural (as a list). https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/composing_a_customized_rhel_system_image/index#image-customizations_creating-system-images-with-composer-command-line-interface
Sorry to bother you again but the doc is also wrong on the expected value... It tells: """ To enable lists, you can use numeric ports, or their names from the /etc/services file. """ Intuitively the user can define ports with ["22", "80"] or ["22/tcp", "80/tcp"] but both forms won't work, leading to a FAILED compose without any log explaining what happened ("log is empty"). This is problematic for the support as we need to rely on strace to figure out the problem: /tmp/worker.strace:9697 15:06:08.743339 write(1<pipe:[164644]>, "{\"type\": \"https://osbuild.org/validation-error\", \"title\": \"JSON Schema validation failed\", \"success\": false, \"errors\": [{\"message\": \"'80/tcp' does not match '.:(tcp|udp|sctp|dccp)$'\", \"path\": [\"pipelines\", 1, \"stages\", 7, \"options\", \"ports\", 0]}]}\n", 248) = 248 <0.000030> As per /usr/lib/osbuild/stages/org.osbuild.firewall, the expected pattern seems to be a numeric port or a port range followed by a dash and the protocol (which is mandatory). 34 SCHEMA = """ 35 "additionalProperties": false, 36 "properties": { 37 "ports": { 38 "description": "Ports (or port ranges) to open", 39 "type": "array", 40 "items": { 41 "type": "string", 42 "description": "A port or port range: 'portid[-portid]:protocol'", 43 "pattern": ".:(tcp|udp|sctp|dccp)$" 44 } 45 }, Thanks!
The documentation for blueprints is here - https://www.osbuild.org/guides/image-builder-on-premises/blueprint-reference.html#firewall I'm not sure how we ended up with things getting out of sync.
@elpereir This seems like an issue in the documentation. May I ask you to take a look? The upstream docs should be correct about the service customization. Many thanks! :)
FTR, the crash was fixed upstream in https://github.com/osbuild/osbuild-composer/pull/3099.