Bug 1793927 - Pull fails with: Image architecture mismatch: image uses "aarch64", expecting "arm64"
Summary: Pull fails with: Image architecture mismatch: image uses "aarch64", expecting...
Alias: None
Product: Fedora
Classification: Fedora
Component: imagefactory
Version: 31
Hardware: aarch64
OS: Linux
Target Milestone: ---
Assignee: Ian McLeod
QA Contact: Fedora Extras Quality Assurance
Depends On:
Blocks: ARMTracker
TreeView+ depends on / blocked
Reported: 2020-01-22 09:49 UTC by Jakub Čajka
Modified: 2020-05-09 03:12 UTC (History)
17 users (show)

Fixed In Version: imagefactory-1.1.15-2.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-05-09 03:12:14 UTC
Type: Bug

Attachments (Terms of Use)

Description Jakub Čajka 2020-01-22 09:49:54 UTC
Description of problem:

Podman fails to pull manifest listed images from registry.fp.o

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.podman pull  registry.fedoraproject.org/fedora:31

Actual results:
Trying to pull registry.fedoraproject.org/fedora:31...
  Image architecture mismatch: image uses "aarch64", expecting "arm64"
Error: error pulling image "registry.fedoraproject.org/fedora:31": unable to pull registry.fedoraproject.org/fedora:31: unable to pull image: Image architecture mismatch: image uses "aarch64", expecting "arm64"

Expected results:
Trying to pull registry.fedoraproject.org/fedora:31...
Getting image source signatures
Additional info:
This is not happening with podman-2:1.6.2-2.fc31.aarch64. So this seems as regression. This will be most likely related to the Go internals and manifest specifications, as "arm64" is way "aarch64" is called by Go compiler.
skopeo inspect  docker://registry.fedoraproject.org/fedora:31
    "Name": "registry.fedoraproject.org/fedora",
    "Digest": "sha256:3f4747a0707860060f1a1ebc0560814a53930fd8545e6017a2508bdc7824a881",
    "RepoTags": [
    "Created": "2019-10-29T05:52:30Z",
    "DockerVersion": "1.10.1",
    "Labels": {
        "license": "MIT",
        "name": "fedora",
        "vendor": "Fedora Project",
        "version": "31"
    "Architecture": "aarch64",
    "Os": "linux",
    "Layers": [
    "Env": [

Comment 1 Matthew Heon 2020-01-22 10:06:56 UTC
Tagging in Nalin and Miloslav - this looks like a c/image issue.

From the OCI image spec, the architecture field should match GOARCH, which uses aarch64, not arm64

Comment 2 Miloslav Trmač 2020-01-22 10:20:25 UTC
Yes, the image is invalid:
> $ skopeo copy docker://registry.fedoraproject.org/fedora@sha256:6050d1611e8541ee99a4cbe705fed2d29cfacf39985187f6cafe2c9e51639dd2  dir:t
> $ python -mjson.tool t/5490a83c7a4a488e24f32ada02b06a0b5fc9bd3f3ff8745ce949546b256805c2 
> {
>    "architecture": "aarch64",
>    "comment": "Created by Image Factory",
> …

(note that the manifest list already points to the image using the "arm64" architecture name, otherwise it would not be found at all:
> $ skopeo inspect --raw docker://registry.fedoraproject.org/fedora:31 | python -mjson.tool
>            "digest": "sha256:6050d1611e8541ee99a4cbe705fed2d29cfacf39985187f6cafe2c9e51639dd2",
>            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
>            "platform": {
>                "architecture": "arm64",
>                "os": "linux"

For now, the workaround would be to manually inspect the multi-arch manifest list, then refer to the architecture-specific image by digest (as done above), and use --override-arch to ~bypass the check.

Based on the “created by” comment, reassigning to Image Factory.

Comment 3 Jakub Čajka 2020-01-22 10:30:02 UTC
Yeah just been checking specs https://github.com/opencontainers/image-spec/blob/master/image-index.md#image-index-property-descriptions and it is on the registry/base image pipeline side. Also other workaround is to stay on podman-2:1.6.2-2.fc31.aarch64 for a bit.

Comment 4 Matthew Heon 2020-01-23 12:55:41 UTC
This may be resolved by https://github.com/containers/image/issues/799 (we're reverting some of the checks related to multiarch) but I'm not certain. I think that we did establish that "aarch64" is the correct string.

Comment 5 Miloslav Trmač 2020-01-23 13:05:40 UTC
No, arm64 is the correct one, per https://github.com/opencontainers/image-spec/blob/master/config.md#properties .

Comment 6 Jakub Čajka 2020-01-23 13:14:21 UTC
Assuming "arm64" is the correct value for the architecture on "aarch64"(Go(GOARCH) calls it "arm64"). IMHO the check should be relaxed to accept both(for historical reasons), but we should move to "arm64" in the metadata of images that we produce, so we are producing images that comply with current standards.

Comment 7 Jakub Čajka 2020-02-25 13:33:49 UTC
I have opened https://github.com/redhat-imaging/imagefactory/pull/434 upstream(it should resolve the issue) could somebody take a look at it please.

Comment 8 Jakub Čajka 2020-03-03 14:48:49 UTC
I have been trying to test the change in the PR, aplying this on top of the imagefactory and imagefactory-plugins. Running with the koji configs from recent job for the base container image I'm hitting this:
"2020-03-03 15:04:34,911 DEBUG imgfac.FilePersistentImageManager.FilePersistentImageManager thread(c190424e) Message: Saved metadata for image (cb401941-6aed-4f91-ac21-308d8ec8259e): {'type': 'BaseImage', 'data': '/var/lib/imagefactory/storage/cb401941-6aed-4f91-ac21-308d8ec8259e.body', 'template': "<template>\n    <name>Fedora-Container-Minimal-Base-32-20200227.n.0.aarch64</name>\n        <os>\n            <name>Fedora</name>\n            <version>22</version>\n            <arch>aarch64</arch>\n            <install type='url'>\n                <url>https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os</url>\n            </install>\n        <icicle>\n                <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>\n            </icicle>\n        </os>\n    <description>Fedora-Container-Minimal-Base-32-20200227.n.0.aarch64 OS</description>\n    <disk>\n        <size>5G</size>\n    </disk>\n</template>\n", 'icicle': None, 'percent_complete': 0, 'properties': {}, 'parameters': {'offline_icicle': 'false', 'install_script': '#version=DEVEL\n# Keyboard layouts\nkeyboard \'us\'\n# Root password\nrootpw --iscrypted --lock locked\n# Reboot after installation\nreboot\n# System timezone\ntimezone Etc/UTC --isUtc --nontp\n# Use text mode install\ntext\n# Network information\nnetwork  --bootproto=dhcp --device=link --activate\nrepo --name="koji-override-0" --baseurl=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os\n# Use network installation\nurl --url="https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os"\n\n# System bootloader configuration\nbootloader --disabled\nautopart --type=plain --nohome --noboot --noswap\n# Clear the Master Boot Record\nzerombr\n# Partition clearing information\nclearpart --all\n\n%post --logfile=/root/anaconda-post.log --erroronfail\nset -eux\n\n# Set install langs macro so that new rpms that get installed will\n# only install langs that we limit it to.\nLANG="en_US"\necho "%_install_langs $LANG" > /etc/rpm/macros.image-language-conf\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1727489\necho \'LANG="C.UTF-8"\' >  /etc/locale.conf\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1400682\necho "Import RPM GPG key"\nreleasever=$(rpm --eval \'%{fedora}\')\nrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-primary\n\necho "# fstab intentionally empty for containers" > /etc/fstab\n\n# Remove machine-id on pre generated images\nrm -f /etc/machine-id\ntouch /etc/machine-id\n\n%end\n\n%post --logfile=/root/anaconda-post.log --erroronfail\n# remove some random help txt files\nrm -fv usr/share/gnupg/help*.txt\n\n# Pruning random things\nrm usr/lib/rpm/rpm.daily\nrm -rfv usr/lib64/nss/unsupported-tools/  # unsupported\n\n# Statically linked crap\nrm -fv usr/sbin/{glibc_post_upgrade.x86_64,sln}\nln usr/bin/ln usr/sbin/sln\n\n# Remove some dnf info\nrm -rfv /var/lib/dnf\n\n# don\'t need icons\nrm -rfv /usr/share/icons/*\n\n#some random not-that-useful binaries\nrm -fv /usr/bin/pinky\n\n# we lose presets by removing /usr/lib/systemd but we do not care\nrm -rfv /usr/lib/systemd\n\n# if you want to change the timezone, bind-mount it from the host or reinstall tzdata\nrm -fv /etc/localtime\nmv /usr/share/zoneinfo/UTC /etc/localtime\nrm -rfv  /usr/share/zoneinfo\n\n# Final pruning\nrm -rfv /var/cache/* /var/log/* /tmp/*\n\n%end\n\n%post --nochroot --logfile=/mnt/sysimage/root/anaconda-post-nochroot.log --erroronfail\nset -eux\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1343138\n# Fix /run/lock breakage since it\'s not tmpfs in docker\n# This unmounts /run (tmpfs) and then recreates the files\n# in the /run directory on the root filesystem of the container\n# NOTE: run this in nochroot because "umount" does not exist in chroot\numount /mnt/sysimage/run\n# The file that specifies the /run/lock tmpfile is\n# /usr/lib/tmpfiles.d/legacy.conf, which is part of the systemd\n# rpm that isn\'t included in this image. We\'ll create the /run/lock\n# file here manually with the settings from legacy.conf\n# NOTE: chroot to run "install" because it is not in anaconda env\nchroot /mnt/sysimage install -d /run/lock -m 0755 -o root -g root\n\n\n# See: https://bugzilla.redhat.com/show_bug.cgi?id=1051816\n# NOTE: run this in nochroot because "find" does not exist in chroot\nKEEPLANG=en_US\nfor dir in locale i18n; do\n    find /mnt/sysimage/usr/share/${dir} -mindepth  1 -maxdepth 1 -type d -not \\( -name "${KEEPLANG}" -o -name POSIX \\) -exec rm -rfv {} +\ndone\n\n%end\n\n%packages --excludedocs --nocore --instLangs=en --excludeWeakdeps\nbash\ncoreutils\nfedora-release-container\nglibc-minimal-langpack\nmicrodnf\nrpm\nopenssh-server\nshadow-utils\nsssd-client\nsystemd\nutil-linux\n-dosfstools\n-e2fsprogs\n-fuse-libs\n-gnupg2-smime\n-grubby\n-kernel\n-libss\n-pinentry\n-shared-mime-info\n-trousers\n-xkeyboard-config\n\n%end\n', 'libvirt_xml': '<domain type="kvm">\n  <name>factory-build-cb401941-6aed-4f91-ac21-308d8ec8259e</name>\n  <memory>2097152</memory>\n  <currentMemory>2097152</currentMemory>\n  <uuid>fbb88ef7-a7c9-40f4-ac69-ea2dd3f4c649</uuid>\n  <clock offset="utc"/>\n  <vcpu>1</vcpu>\n  <features>\n    <acpi/>\n  </features>\n  <cpu mode="custom" match="exact">\n    <model fallback="allow">host</model>\n  </cpu>\n  <os>\n    <type arch="aarch64" machine="virt">hvm</type>\n    <boot dev="hd"/>\n    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.fd</loader>\n    <nvram template="/usr/share/AAVMF/AAVMF_VARS.fd"/>\n  </os>\n  <on_poweroff>destroy</on_poweroff>\n  <on_reboot>destroy</on_reboot>\n  <on_crash>destroy</on_crash>\n  <devices>\n    <graphics port="-1" type="vnc"/>\n    <interface type="bridge">\n      <source bridge="virbr0"/>\n      <mac address="52:54:00:99:45:51"/>\n      <model type="virtio"/>\n    </interface>\n    <input bus="usb" type="tablet"/>\n    <controller type="usb" index="0"/>\n    <input type="keyboard" bus="usb"/>\n    <serial type="pty">\n      <target port="0"/>\n    </serial>\n    <serial type="tcp">\n      <source mode="bind" host="" service="20720"/>\n      <protocol type="raw"/>\n      <target port="1"/>\n    </serial>\n    <rng model="virtio">\n      <rate bytes="1024" period="1000"/>\n      <backend model="random">/dev/random</backend>\n    </rng>\n    <disk device="disk" type="file">\n      <target dev="vda" bus="virtio"/>\n      <source file="/var/lib/imagefactory/storage/cb401941-6aed-4f91-ac21-308d8ec8259e.body"/>\n      <driver name="qemu" type="raw"/>\n    </disk>\n  </devices>\n</domain>\n'}, 'status': 'FAILED', 'identifier': 'cb401941-6aed-4f91-ac21-308d8ec8259e', 'status_detail': {'activity': 'Base Image build failed with exception.', 'error': 'download: /etc/init.d/sshd: No such file or directory'}}
2020-03-03 15:04:34,912 ERROR imgfac.Builder.Builder thread(c190424e) Message: Exception encountered in _build_image_from_template thread
2020-03-03 15:04:34,912 ERROR imgfac.Builder.Builder thread(c190424e) Message: download: /etc/init.d/sshd: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 446, in _collect_setup
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 310, in _image_ssh_setup_step_2
    startuplink = self._get_service_runlevel_link(g_handle, 'sshd')
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 156, in _get_service_runlevel_link
    lines = g_handle.cat('/etc/init.d/' + service).split("\n")
  File "/usr/lib/python3.7/site-packages/oz/GuestFSManager.py", line 183, in cat
    return self.g_handle.cat(fname)
  File "/usr/lib64/python3.7/site-packages/guestfs.py", line 1847, in cat
    r = libguestfsmod.cat(self._o, path)
RuntimeError: download: /etc/init.d/sshd: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/imgfac/Builder.py", line 132, in _build_image_from_template
    self.os_plugin.create_base_image(self, template, parameters)
  File "/usr/lib/python3.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 363, in create_base_image
    builder.base_image.icicle = self.guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python3.7/site-packages/oz/Linux.py", line 357, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, "gen_and_mod")
  File "/usr/lib/python3.7/site-packages/oz/Linux.py", line 307, in _internal_customize
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 466, in _collect_setup
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 192, in _image_ssh_teardown_step_2
    startuplink = self._get_service_runlevel_link(g_handle, 'sshd')
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 156, in _get_service_runlevel_link
    lines = g_handle.cat('/etc/init.d/' + service).split("\n")
  File "/usr/lib/python3.7/site-packages/oz/GuestFSManager.py", line 183, in cat
    return self.g_handle.cat(fname)
  File "/usr/lib64/python3.7/site-packages/guestfs.py", line 1847, in cat
    r = libguestfsmod.cat(self._o, path)
RuntimeError: download: /etc/init.d/sshd: No such file or directory
Any hints on what can be going wrong? Can someone take a look on the PR please?

Comment 9 Peter Robinson 2020-03-04 21:03:51 UTC
I've built this for Ian, I'm guessing it's going to need to be deployed to infra, which for Beta will need FBR.

Comment 10 Peter Robinson 2020-03-04 21:15:59 UTC
For reference the update is https://bodhi.fedoraproject.org/updates/FEDORA-2020-e29a47204e (for some reason bodhi doesn't want to add the RHBZ)

Comment 11 Jakub Čajka 2020-03-05 07:45:33 UTC
I just want to stress that I have not been able yet to test successfully this change/fix(AFAIK it should fix the issues) locally due to some issue in the OZ(with configs from koji). IMO it shouldn't be directly pushed to prod without testing in stg.

Comment 12 Jakub Čajka 2020-03-11 18:02:03 UTC
Sorry, have had typo in the patch. Forgot the self... PR updated. And tested locally via imagefactory --debug target_image --template=tdl-aarch64.xml --file-parameter install_script koji-f32-build-41965801-base.ks --parameter offline_icicle true docker. AFAIK should be close approximation how koji is using imagefactory. 

Could you please respin the build with the fixed patch? Thank you.

Comment 13 Peter Robinson 2020-04-30 15:33:17 UTC
Updated the patch and pushed builds for F-32+

Comment 14 Fedora Update System 2020-04-30 15:37:31 UTC
FEDORA-2020-e858471cbe has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-e858471cbe

Comment 15 Fedora Update System 2020-05-01 19:22:24 UTC
FEDORA-2020-e858471cbe has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-e858471cbe`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-e858471cbe

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2020-05-09 03:12:14 UTC
FEDORA-2020-e858471cbe has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.