Bug 1886149 - ostree exited with code 1 during armhfp disk creation
Summary: ostree exited with code 1 during armhfp disk creation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: ostree
Version: 33
Hardware: armhfp
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Jonathan Lebon
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
Depends On:
Blocks: IoT F33FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2020-10-07 18:40 UTC by Paul Whalen
Modified: 2020-10-10 09:37 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-10 09:37:25 UTC
Type: Bug


Attachments (Terms of Use)
syslog (1.05 MB, text/plain)
2020-10-07 18:46 UTC, Paul Whalen
no flags Details
program.log (2.03 KB, text/plain)
2020-10-07 18:47 UTC, Paul Whalen
no flags Details
anaconda.log (17.56 KB, text/plain)
2020-10-07 18:47 UTC, Paul Whalen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github ostreedev ostree pull 2211 0 None closed lib/deploy: Don't leak fd when checksumming dtbs 2020-10-14 18:41:39 UTC

Description Paul Whalen 2020-10-07 18:40:06 UTC
Description of problem:
When attempting to build an armhfp disk image in F33 iot, the installation fails:

...
Preparing deployment of fedora/devel/armhfp/iot
Deployment starting: fedora/devel/armhfp/iot
================================================================================ 
================================================================================

   The following error occurred while installing.  This is a fatal error and
   installation will be aborted.
   
   ostree ['admin', '--sysroot=/mnt/sysimage', 'deploy', '--os=fedora-iot',
   'fedora-iot:fedora/devel/armhfp/iot'] exited with code 1



Checking the logs:

00:12:22,880 INF program: Relabeling /var (no stamp file 'var/.ostree-selabeled' found)
00:12:22,887 INF program: error: Installing kernel: openat(imx6dl-yapp4-hydra.dtb): Too many open files
00:12:22,888 DBG program: Return code: 1


Version-Release number of selected component (if applicable):
ostree-2020.6-4.fc33

How reproducible:
Everytime. 


Koji task - https://koji.fedoraproject.org/koji/taskinfo?taskID=52936648

Comment 1 Paul Whalen 2020-10-07 18:46:53 UTC
Created attachment 1719820 [details]
syslog

Comment 2 Paul Whalen 2020-10-07 18:47:20 UTC
Created attachment 1719821 [details]
program.log

Comment 3 Paul Whalen 2020-10-07 18:47:48 UTC
Created attachment 1719822 [details]
anaconda.log

Comment 4 Jonathan Lebon 2020-10-07 18:55:02 UTC
It's possible the leak plugged in https://github.com/ostreedev/ostree/pull/2211 isn't the main cause of exhaustion. Would be helpful to have:
- the number of dtbs is the OSTree
- whether `ulimit -n` is lower on armhfp

Comment 5 Paul Whalen 2020-10-07 19:21:21 UTC
Proposing as a Freeze Exception for F33 final, it would be great to have this fixed and to include armhfp iot disk images in the release.

Comment 6 Jonathan Lebon 2020-10-07 20:39:39 UTC
Fixed by https://bodhi.fedoraproject.org/updates/FEDORA-2020-6cfe7b5633. As usual, the Bodhi UI doesn't want to let me attach this RHBZ to the update for some reason.

Comment 7 Fedora Update System 2020-10-08 08:21:59 UTC
FEDORA-2020-6cfe7b5633 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-6cfe7b5633

Comment 8 Peter Robinson 2020-10-08 08:24:15 UTC
I've seen that on occasion too, FYI you can add it using the cli like "bodhi updates edit --bugs 1886149 FEDORA-2020-6cfe7b5633"

Comment 9 Paul Whalen 2020-10-08 13:17:39 UTC
We pulled this in to today's compose but hit the same error.

Koji task - https://koji.fedoraproject.org/koji/taskinfo?taskID=52985114

Error visible in - https://kojipkgs.fedoraproject.org//work/tasks/5114/52985114/oz-armhfp.log

Comment 10 Jonathan Lebon 2020-10-08 14:44:27 UTC
Are we 100% sure that the OSTree in Anaconda isn't an older one? Last time this came up my conclusion was that must've been the issue: https://bugzilla.redhat.com/show_bug.cgi?id=1880499#c17.

In the logs you posted, you can see ImageFactory fetching the boot artifacts from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/armhfp/os:

Starting new HTTPS connection (1): kojipkgs.fedoraproject.org:443
https://kojipkgs.fedoraproject.org:443 "HEAD /compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/vmlinuz HTTP/1.1" 200 0
Fetching the original install media from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/vmlinuz
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org:443
https://kojipkgs.fedoraproject.org:443 "GET /compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/vmlinuz HTTP/1.1" 200 8610304
8408kB of 8408kB
Fetching the original media
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org:443
https://kojipkgs.fedoraproject.org:443 "HEAD /compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/initrd.img HTTP/1.1" 200 0
Fetching the original install media from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/initrd.img
Starting new HTTPS connection (1): kojipkgs.fedoraproject.org:443
https://kojipkgs.fedoraproject.org:443 "GET /compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/images/pxeboot/initrd.img HTTP/1.1" 200 62144616

And looking in https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/armhfp/os/Packages/o/, I see ostree-2020.6-4.fc33.armv7hl.rpm and not -5 (which is in https://bodhi.fedoraproject.org/updates/FEDORA-2020-6cfe7b5633).

Does that compose get updated daily? How can we make sure that the new ostree makes it there?

Comment 11 Peter Robinson 2020-10-08 15:13:44 UTC
(In reply to Jonathan Lebon from comment #10)
> Are we 100% sure that the OSTree in Anaconda isn't an older one? Last time
> this came up my conclusion was that must've been the issue:
> https://bugzilla.redhat.com/show_bug.cgi?id=1880499#c17.

Yep, that could be the problem

Comment 12 Adam Williamson 2020-10-08 18:19:04 UTC
+4 in the ticket, marking accepted FE.

Comment 13 Fedora Update System 2020-10-08 22:19:00 UTC
FEDORA-2020-6cfe7b5633 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-6cfe7b5633`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-6cfe7b5633

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Peter Robinson 2020-10-09 09:41:38 UTC
(In reply to Jonathan Lebon from comment #10)
> Are we 100% sure that the OSTree in Anaconda isn't an older one? Last time
> this came up my conclusion was that must've been the issue:
> https://bugzilla.redhat.com/show_bug.cgi?id=1880499#c17.

Confirmed this works on rawhide once we had a successful mainline compose!

Comment 15 Fedora Update System 2020-10-09 23:56:20 UTC
FEDORA-2020-6cfe7b5633 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 16 Adam Williamson 2020-10-10 00:03:25 UTC
The update wasn't marked to close the bug when it went stable, apparently. Does this still need to be open or can we close it now the update went stable?

Comment 17 Peter Robinson 2020-10-10 09:37:25 UTC
(In reply to Adam Williamson from comment #16)
> The update wasn't marked to close the bug when it went stable, apparently.
> Does this still need to be open or can we close it now the update went
> stable?

I edited the bodhi update using the cli and probably missed an option


Note You need to log in before you can comment on or make changes to this bug.