I'm splitting this out from https://bugzilla.redhat.com/show_bug.cgi?id=1483278 for clarity (that bug can cover specifically the lorax change). The new 'rdma-core' source package is not built on armhfp, for what dledford assures us are good and valid reasons. However, this creates a whole ton of dependency problems. Just looking at a single library that is now built from rdma-core - libibverbs - all these things depend on it on x86_64: [root@adam vms]# dnf repoquery --whatrequires "libibverbs.so.1()(64bit)" Failed to synchronize cache for repo 'kanarip-phabricator', disabling. Last metadata expiration check: 0:00:00 ago on Tue 22 Aug 2017 01:45:14 PM PDT. ceph-base-1:12.1.3-1.fc27.x86_64 ceph-common-1:12.1.3-1.fc27.x86_64 ceph-fuse-1:12.1.3-1.fc27.x86_64 ceph-mds-1:12.1.3-1.fc27.x86_64 ceph-mgr-1:12.1.3-1.fc27.x86_64 ceph-mon-1:12.1.3-1.fc27.x86_64 ceph-osd-1:12.1.3-1.fc27.x86_64 ceph-radosgw-1:12.1.3-1.fc27.x86_64 ceph-test-1:12.1.3-1.fc27.x86_64 corosynclib-0:2.4.2-4.fc27.x86_64 dapl-0:2.1.9-4.fc26.x86_64 fio-0:2.99-3.fc27.x86_64 ga-openmpi-0:5.3b-24.fc27.x86_64 glusterfs-rdma-0:3.11.2-3.fc27.x86_64 ibacm-0:14-4.fc27.x86_64 libcephfs2-1:12.1.3-1.fc27.x86_64 libcephfs_jni1-1:12.1.3-1.fc27.x86_64 libfabric-0:1.4.2-4.fc27.x86_64 libhfi1-0:0.5-26.fc27.x86_64 libibcm-0:14-4.fc27.x86_64 libibverbs-devel-0:1.2.1-6.fc27.x86_64 libibverbs-utils-0:14-4.fc27.x86_64 libipathverbs-0:1.3-4.fc27.x86_64 libmlx4-0:1.0.6-9.fc27.x86_64 libmlx5-0:1.0.2-5.fc27.x86_64 libmthca-0:1.0.6-18.fc27.x86_64 libnes-0:1.1.4-10.fc27.x86_64 libocrdma-0:1.0.8-6.fc27.x86_64 librados-devel-1:12.1.3-1.fc27.x86_64 librados2-1:12.1.3-1.fc27.x86_64 libradosstriper1-1:12.1.3-1.fc27.x86_64 librbd1-1:12.1.3-1.fc27.x86_64 librdmacm-0:14-4.fc27.x86_64 librdmacm-utils-0:14-4.fc27.x86_64 librgw2-1:12.1.3-1.fc27.x86_64 libusnic_verbs-0:2.0.2-4.fc27.x86_64 libvma-0:8.0.1-1.fc25.x86_64 openmpi-0:2.0.2-2.fc26.x86_64 perftest-0:3.0-4.fc27.x86_64 qemu-system-aarch64-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-alpha-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-arm-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-cris-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-lm32-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-m68k-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-microblaze-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-mips-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-moxie-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-nios2-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-or1k-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-ppc-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-s390x-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-sh4-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-sparc-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-tricore-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-unicore32-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-x86-core-2:2.10.0-0.2.rc3.fc27.x86_64 qemu-system-xtensa-core-2:2.10.0-0.2.rc3.fc27.x86_64 qperf-0:0.4.9-11.fc27.x86_64 qpid-cpp-client-rdma-0:1.36.0-6.fc27.x86_64 rbd-mirror-1:12.1.3-1.fc27.x86_64 rbd-nbd-1:12.1.3-1.fc27.x86_64 rdma-core-devel-0:14-4.fc27.x86_64 scsi-target-utils-0:1.0.70-3.fc27.x86_64 srp_daemon-0:14-4.fc27.x86_64 srptools-0:1.0.3-2.fc26.x86_64 I believe most or all of those dependencies also exist on armhfp. There are, of course, many things that depend on *those* packages that are now broken. For instance, libvirt-daemon-driver-storage requires libvirt-daemon-driver-storage-rbd , which requires librbd and librados, both of which require libibverbs (which comes from rdma-core). This is just scratching the surface, I'm sure there are many others. Because of all these broken dependencies, I strongly suspect many ARM compose tasks will fail, which will likely cause the whole compose to be considered a failure. #1483278 covers one specific way we *know* this change has broken the compose (lorax explicitly pulling in 'rdma' on armhfp), but even after we fix that, I rather suspect we'll start running into many other compose problems caused by all these broken deps. This is of course a major problem given F27's compressed schedule, we're already struggling with all sorts of other problems, and having composes fail due to all these dep issues would not help at all. So we really need someone to either: 1) Just build rdma-core on armhfp again, even if it's "wrong" and won't really work, granting us some time to work through all the dependent packages and adjust them to build on armhfp without rdma-core, at which point we can disable rdma-core building on armhfp again 2) *Very quickly* identify and work through at least all the critical dep chains and adjust them all to build on armhfp without rdma-core I think a reasonable timeframe for this is, like, a couple of days, so if 2) isn't possible in that time, we'd really like you to consider doing the first part of 1) instead. Myself and probably nirik (as provenpackagers) would be happy to help out with rebuilds if necessary. Thanks! Proposing as a Beta blocker due to the high likelihood this will cause release blocking images to fail to compose.
Note for suggestion 1), this is specifically what dledford suggested could be done: <dledford> The best we can do is carry a local patch to build on arm 32 if you want. But that will highlight the problem on arm32. You will have to create defines for memory barriers that simply don't exist on arm32, so we'll have to put dummy defines in there.
I'm taking this bug. I will try to reserve an ARM machine in beaker or create an ARM virtual machine with qemu. Then I will check the deep chain which provided in #comment 1 with a x86_64 system.
Welp, I've started working on a couple of the more obvious rebuilds. qemu was just a single line change and I tested that a scratch build worked fine, so I've gone ahead and sent that for rawhide and f27. I'm also testing a scratch build for ceph; it looks like we may be able to get librados2 and librbd1 built but without dependencies on libibverbs, which would probably help with several of the dep chains that seem to run through those libs.
ceph's a bit of a beast; my scratch build started 4 hours ago and it's at like 34%. I'll check on it in the morning and send official builds if it looks good.
> <dledford> The best we can do is carry a local patch to build on arm 32 if > you want. But that will highlight the problem on arm32. You will have to > create defines for memory barriers that simply don't exist on arm32, so > we'll have to put dummy defines in there. ARMv7 has memory barriers, you'll need to check the exact details with the toolchains team as I'm not an expert there, but all variants of arm32 that we care about (ARMv7 and later) should be fine here and have had them available for some time in both the architecture and the tookchain. http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/CJAGIEIE.html
(In reply to Peter Robinson from comment #6) > > <dledford> The best we can do is carry a local patch to build on arm 32 if > > you want. But that will highlight the problem on arm32. You will have to > > create defines for memory barriers that simply don't exist on arm32, so > > we'll have to put dummy defines in there. > > ARMv7 has memory barriers, you'll need to check the exact details with the > toolchains team as I'm not an expert there, but all variants of arm32 that > we care about (ARMv7 and later) should be fine here and have had them > available for some time in both the architecture and the tookchain. > > http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/ > CJAGIEIE.html As I recall (and I'm not the best person to talk about this because when it was brought up on the upstream mailing list, I didn't know the details of arm32 that other people did), the problem is that even though ARMv7 has DMB, it is still insufficient for our needs because the architecture is specifically designed as non-coherent when it comes to PCI DMA operations and local processor cache. As such, we don't have what we need to force the proper ordering of reads of PCI DMA written memory, or writes to memory that will eventually be PCI DMA read.
We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32.
[root@calxeda-arm-soc-03 ~]# uname -r 4.12.5-300.fc26.armv7hl [root@calxeda-arm-soc-03 ~]# rpm -q --provides libibverbs libibverbs = 1.2.1-4.fc26 libibverbs(armv7hl-32) = 1.2.1-4.fc26 libibverbs.so.1 libibverbs.so.1(IBVERBS_1.0) libibverbs.so.1(IBVERBS_1.1) [root@calxeda-arm-soc-03 ~]# dnf repoquery --whatrequires 'libibverbs' Last metadata expiration check: 0:38:32 ago on Wed 23 Aug 2017 02:29:18 PM CEST. corosynclib-0:2.4.2-2.fc26.armv7hl fio-0:2.18-1.fc26.armv7hl glusterfs-rdma-0:3.10.3-1.fc26.armv7hl glusterfs-rdma-0:3.10.5-1.fc26.armv7hl ibacm-0:1.2.1-2.fc26.armv7hl libcxgb3-0:1.3.1-14.fc26.armv7hl libcxgb4-0:1.3.6-2.fc26.armv7hl libfabric-0:1.4.1-1.fc26.armv7hl libibcm-0:1.0.5-15.fc26.armv7hl libibverbs-devel-0:1.2.1-4.fc26.armv7hl libibverbs-utils-0:1.2.1-4.fc26.armv7hl libmlx4-0:1.0.6-7.fc26.armv7hl libmlx5-0:1.0.2-3.fc26.armv7hl libmthca-0:1.0.6-16.fc26.armv7hl libnes-0:1.1.4-7.fc26.armv7hl libocrdma-0:1.0.8-4.fc26.armv7hl librdmacm-0:1.1.0-4.fc26.armv7hl librdmacm-utils-0:1.1.0-4.fc26.armv7hl libusnic_verbs-0:2.0.2-2.fc26.armv7hl openmpi-0:2.0.2-2.fc26.armv7hl qemu-system-aarch64-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-aarch64-core-2:2.9.0-5.fc26.armv7hl qemu-system-alpha-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-alpha-core-2:2.9.0-5.fc26.armv7hl qemu-system-arm-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-arm-core-2:2.9.0-5.fc26.armv7hl qemu-system-cris-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-cris-core-2:2.9.0-5.fc26.armv7hl qemu-system-lm32-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-lm32-core-2:2.9.0-5.fc26.armv7hl qemu-system-m68k-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-m68k-core-2:2.9.0-5.fc26.armv7hl qemu-system-microblaze-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-microblaze-core-2:2.9.0-5.fc26.armv7hl qemu-system-mips-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-mips-core-2:2.9.0-5.fc26.armv7hl qemu-system-moxie-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-moxie-core-2:2.9.0-5.fc26.armv7hl qemu-system-nios2-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-nios2-core-2:2.9.0-5.fc26.armv7hl qemu-system-or1k-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-or1k-core-2:2.9.0-5.fc26.armv7hl qemu-system-ppc-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-ppc-core-2:2.9.0-5.fc26.armv7hl qemu-system-s390x-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-s390x-core-2:2.9.0-5.fc26.armv7hl qemu-system-sh4-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-sh4-core-2:2.9.0-5.fc26.armv7hl qemu-system-sparc-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-sparc-core-2:2.9.0-5.fc26.armv7hl qemu-system-tricore-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-tricore-core-2:2.9.0-5.fc26.armv7hl qemu-system-unicore32-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-unicore32-core-2:2.9.0-5.fc26.armv7hl qemu-system-x86-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-x86-core-2:2.9.0-5.fc26.armv7hl qemu-system-xtensa-core-2:2.9.0-1.fc26.1.armv7hl qemu-system-xtensa-core-2:2.9.0-5.fc26.armv7hl qperf-0:0.4.9-9.fc26.armv7hl qpid-cpp-client-rdma-0:1.36.0-1.fc26.armv7hl scsi-target-utils-0:1.0.55-6.fc26.armv7hl scsi-target-utils-0:1.0.70-2.fc26.armv7hl srptools-0:1.0.3-2.fc26.armv7hl [root@calxeda-arm-soc-03 ~]# cat x.sh #!/bin/bash rm -f s.txt || exit rpm_list=$(dnf repoquery --whatrequires 'libibverbs' | sed '1d') #echo $rpm_list | wc for r in ${rpm_list}; do dnf info $r | grep ^Source >> s.txt done cat s.txt | sort | uniq -c [root@calxeda-arm-soc-03 ~]# sh x.sh Last metadata expiration check: 0:44:49 ago on Wed 23 Aug 2017 02:29:18 PM CEST. 1 Source : fio-2.18-1.fc26.src.rpm 1 Source : glusterfs-3.10.3-1.fc26.src.rpm 1 Source : glusterfs-3.10.5-1.fc26.src.rpm 1 Source : ibacm-1.2.1-2.fc26.src.rpm 1 Source : libcxgb3-1.3.1-14.fc26.src.rpm 1 Source : libcxgb4-1.3.6-2.fc26.src.rpm 1 Source : libfabric-1.4.1-1.fc26.src.rpm 1 Source : libibcm-1.0.5-15.fc26.src.rpm 2 Source : libibverbs-1.2.1-4.fc26.src.rpm 1 Source : libmlx4-1.0.6-7.fc26.src.rpm 1 Source : libmlx5-1.0.2-3.fc26.src.rpm 1 Source : libmthca-1.0.6-16.fc26.src.rpm 1 Source : libnes-1.1.4-7.fc26.src.rpm 1 Source : libocrdma-1.0.8-4.fc26.src.rpm 2 Source : librdmacm-1.1.0-4.fc26.src.rpm 1 Source : libusnic_verbs-2.0.2-2.fc26.src.rpm 1 Source : openmpi-2.0.2-2.fc26.src.rpm 19 Source : qemu-2.9.0-1.fc26.1.src.rpm 19 Source : qemu-2.9.0-5.fc26.src.rpm 1 Source : qperf-0.4.9-9.fc26.src.rpm 1 Source : qpid-cpp-1.36.0-1.fc26.src.rpm 1 Source : scsi-target-utils-1.0.55-6.fc26.src.rpm 1 Source : scsi-target-utils-1.0.70-2.fc26.src.rpm 1 Source : srptools-1.0.3-2.fc26.src.rpm
We have 8 components immediately depend libibverbs. fio : fio-2.18-1.fc26.src.rpm glusterfs : glusterfs-3.10.3-1.fc26.src.rpm : glusterfs-3.10.5-1.fc26.src.rpm libfabric : libfabric-1.4.1-1.fc26.src.rpm openmpi : openmpi-2.0.2-2.fc26.src.rpm qemu : qemu-2.9.0-1.fc26.1.src.rpm : qemu-2.9.0-5.fc26.src.rpm qperf : qperf-0.4.9-9.fc26.src.rpm qpid-cpp : qpid-cpp-1.36.0-1.fc26.src.rpm scsi-target-utils : scsi-target-utils-1.0.55-6.fc26.src.rpm : scsi-target-utils-1.0.70-2.fc26.src.rpm rdma-core : ibacm-1.2.1-2.fc26.src.rpm // These packages will be retired for ARM32. : libcxgb3-1.3.1-14.fc26.src.rpm : libcxgb4-1.3.6-2.fc26.src.rpm : libibcm-1.0.5-15.fc26.src.rpm : libibverbs-1.2.1-4.fc26.src.rpm : libmlx4-1.0.6-7.fc26.src.rpm : libmlx5-1.0.2-3.fc26.src.rpm : libmthca-1.0.6-16.fc26.src.rpm : libnes-1.1.4-7.fc26.src.rpm : libocrdma-1.0.8-4.fc26.src.rpm : librdmacm-1.1.0-4.fc26.src.rpm : libusnic_verbs-2.0.2-2.fc26.src.rpm : srptools-1.0.3-2.fc26.src.rpm
(In reply to Honggang LI from comment #8) > We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported > in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from > outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32. That's not entirely true, it was originally enabled in Fedora because someone requested it, I believe it was once actually tested but I don't remember the details.
Ultimately this hasn't been communicated/coordinated at all and appears to basically be a run and dump.
Honggang: it would be better to check f27 / rawhide. You don't need to *install* them to do that; you can just pass --releasever to dnf. For instance, ceph was only set to build against rdma in commit 44c0bd8d5c6ece6c22e2f9fff1297dc082b9a0a2 , which is only on the master and f27 branches, not f26.
My scratch build of ceph looks good; we can build it with the RDMA support disabled (for 32-bit ARM) and we get librdb and librados but without the libibverbs dependency. I've now sent official builds for Rawhide and F27, but they'll take 12+ hours to run. scsi-target-utils builds against ceph (as well as directly against rdma), so we will have to wait for the ceph rebuild before rebuilding that. I'll take a look at some of the others.
(In reply to Peter Robinson from comment #11) > (In reply to Honggang LI from comment #8) > > We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported > > in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from > > outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32. > > That's not entirely true, it was originally enabled in Fedora because > someone requested it, I believe it was once actually tested but I don't > remember the details. Redhat never tested RDMA for ARM32 as don't have hardware. If you believe someone had tested this, please try your best to find and feedback the test history/info/result to us. We are happy to see that. If we can confirm RDMA ARM32 works, we will try to enable RDMA for ARM32.
(In reply to Peter Robinson from comment #12) > Ultimately this hasn't been communicated/coordinated at all and appears to > basically be a run and dump. Look at the built history of RDMA stack, the old components were built for ARM32 in F19/F20 massive building to support ARM32 as primary architecture for F20. Redhat never tested ARM32 RDMA for F20. The ARM32 packages got built just because the building/mock system was enable for ARM32 building. I never heard someone had tested it. I never saw an ARM32 RDMA bugs was filed from outside in last 6+ years. https://en.wikipedia.org/wiki/Fedora_version_history http://download-node-02.eng.bos.redhat.com/released/F-20/GOLD/Everything/ http://download-node-02.eng.bos.redhat.com/released/F-19/GOLD/Everything/ The F20 is the first release supports ARMPHF/AMR32. ----------------+-------------------------+-------------------------------+ component name | initial fedora release | first F-release enable RDMA | | with the component built| component built for ARM32 | ----------------+-------------------------+-------------------------------+ rdma | F8 (noarch) | | libibverbs | F7 | libibverbs-1.1.8-2.fc20 | libibumad | F8 | libibumad-1.3.8-2.fc19 | ----------------+-------------------------+-------------------------------+ libcxgb3 | F8 | libcxgb3-1.3.1-3.fc19 | libcxgb4 | F15 | libcxgb4-1.2.0-3.fc19 | libmlx4 | F8 | libmlx4-1.0.5-4.fc20 | libnes | F15 | libnes-1.1.3-3.fc19 | libocrdma | F24 | libocrdma-1.0.8-3.fc24 | libibcm | F8 | libibcm-1.0.5-7.fc19 | libmlx5 | F23 | libmlx5-1.0.2-2.fc23 | libusnic_verbs | F23 | libusnic_verbs-2.0.2-1.fc23 | ----------------+-------------------------+-------------------------------+ ibacm | F23 | ibacm-1.2.0-3.fc23 | srptools | F15 | srptools-0.0.4-17.fc19 | libibmad | F8 | libibmad-1.3.9-2.fc19 | librdmacm | F8 | librdmacm-1.0.17-1.fc20 | opensm | F8 | opensm-3.3.15-6.fc20 | mstflint | F15 | mstflint-1.4-9.fc19 | perftest | F15 | never built for ARM32 | dapl | F26 | never built for ARM32 | qperf | F15 | qperf-0.4.6-8.fc19 | libfabric | F21 | libfabric-1.1.0-1.fc21 | ibutils | F17 | ibutils-1.5.7-9.fc19 | infiniband-diags| F15 | infiniband-diags-1.6.1-4.fc19 | openmpi | F6 | openmpi-1.6.4-2.fc19 | fabtests | F24 | fabtests-1.3.0-3.fc24 | mpich | F18 | mpich-3.0.4-3.fc20 | libiwpm | F23 | libiwpm-1.0.3-7.fc23 | ----------------+-------------------------+-------------------------------+ libhfi1 | | libpsm2 | These RDMA components are avaiable in RHEL, but | libipathverbs | unavailable in Fedora. No ARM32 packages for them. | infinipath-psm | | opa-ff | https://apps.fedoraproject.org/packages/ | opa-fm | The build history was extracted from this website. | opa-fmgui | | libehca | | mvapichv2 | | mvapich | | mpitests | | rds-tools | | ibsim | | ----------------+---------------------------------------------------------+
----------------+-------------------------+-------------------------------+ libhfi1 | | libpsm2 | These RDMA components are avaiable in RHEL, but | libipathverbs | unavailable in Fedora. No ARM32 packages for them. | infinipath-psm | | opa-ff | https://apps.fedoraproject.org/packages/ | opa-fm | The build history was extracted from this website. | opa-fmgui | | libehca | | mvapichv2 | | mvapich | | mpitests | | rds-tools | | ibsim | | ----------------+---------------------------------------------------------+ Some of those packages are available in Fedora, but absolutely no ARM32 packages available for them.
Update on rebuilds: I've done corosync and libfabric. I also did hwloc, but I'm now building it again because it needs a Requires: changing from libibverbs%{_isa} to rdma-core-devel%{_isa}. openmpi pulls in hwloc-devel, so it needs hwloc fixing before I can build it, but it's queued up to go. ceph is still running, when it's done I'll look at scsi-target-utils. Kaleb Keithley did glusterfs yesterday.
> The F20 is the first release supports ARMPHF/AMR32. Incorrect, F-20 was when ARM was promoted to primary, the first release of Fedora on armhfp (ARMv7/hard floating point) was Fedora 15, Fedora has been built on ARM since well before Fedora 7.
(In reply to Peter Robinson from comment #19) > > The F20 is the first release supports ARMPHF/AMR32. > > Incorrect, F-20 was when ARM was promoted to primary, the first release of > Fedora on armhfp (ARMv7/hard floating point) was Fedora 15, Fedora has been > built on ARM since well before Fedora 7. OK, but the RDMA stack was enabled for ARM32 when F20 was promoted to primary. The RDMA stack got in ARM32 mostly because of that.
In reply to comment #c7 it's not that the "architecture is specifically designed" it's that the existing SoC platforms implementing it might not have the expected coherency constraints between local processor caches and PCIe inbound DMA operations. I could well believe that is true. But it's not architectural, it's implementation. It's why we have e.g. SBSA on 64-bit ARM.
OK, I've gone through all the ones I could find for this now. My status list is: Source : ceph-12.1.3-1.fc27.src.rpm DONE Source : corosync-2.4.2-4.fc27.src.rpm DONE Source : dapl-2.1.9-4.fc26.src.rpm excluded Source : fio-2.99-3.fc27.src.rpm DONE Source : ga-5.3b-24.fc27.src.rpm excluded Source : glusterfs-3.11.2-3.fc27.src.rpm DONE (kaleb) Source : libfabric-1.4.2-4.fc27.src.rpm DONE Source : libhfi1-0.5-26.fc27.src.rpm excluded Source : libipathverbs-1.3-4.fc27.src.rpm excluded Source : libmlx4-1.0.6-9.fc27.src.rpm obsoleted by rdma-core Source : libmlx5-1.0.2-5.fc27.src.rpm obsoleted by rdma-core Source : libmthca-1.0.6-18.fc27.src.rpm obsoleted by rdma-core Source : libnes-1.1.4-10.fc27.src.rpm obsoleted by rdma-core Source : libocrdma-1.0.8-6.fc27.src.rpm obsoleted by rdma-core Source : libusnic_verbs-2.0.2-4.fc27.src.rpm obsoleted by rdma-core Source : libvma-8.0.1-1.fc25.src.rpm excluded Source : openmpi-2.0.2-2.fc26.src.rpm DONE Source : perftest-3.0-4.fc27.src.rpm excluded Source : qemu-2.10.0-0.2.rc3.fc27.src.rpm DONE Source : qperf-0.4.9-11.fc27.src.rpm set to exclude Source : qpid-cpp-1.36.0-6.fc27.src.rpm in progress... Source : scsi-target-utils-1.0.70-3.fc27.src.rpm DONE Source : srptools-1.0.3-2.fc26.src.rpm obsoleted by rdma-core That qpid-cpp build is https://koji.fedoraproject.org/koji/taskinfo?taskID=21469504 . Packages marked 'excluded' are ones where 32-bit ARM is excluded by ExcludeArch or ExclusiveArch.
Note there's technically an upgrade path problem for all the packages excluded by rdma-core (or its subpackages): nothing on armv7hl excludes them, because rdma-core isn't built there. So anyone who actually has those packages installed on arm will not be able to update without --allowerasing . But if we're right about all this, that's likely not many or any people.
> . But if we're right about all this, that's likely not many or any people. There will be for things like workstation where Boxes pulls in libvirt I suspect
i guess we'll see. actually, thinking about it, the old packages might just keep lying around and no observable problems will show up...
*** Bug 1483254 has been marked as a duplicate of this bug. ***
(In reply to Adam Williamson from comment #22) > OK, I've gone through all the ones I could find for this now. My status list > is: > > Source : ceph-12.1.3-1.fc27.src.rpm DONE > Source : corosync-2.4.2-4.fc27.src.rpm DONE > Source : dapl-2.1.9-4.fc26.src.rpm excluded > Source : fio-2.99-3.fc27.src.rpm DONE > Source : ga-5.3b-24.fc27.src.rpm excluded > Source : glusterfs-3.11.2-3.fc27.src.rpm DONE (kaleb) > Source : libfabric-1.4.2-4.fc27.src.rpm DONE > Source : libhfi1-0.5-26.fc27.src.rpm excluded > Source : libipathverbs-1.3-4.fc27.src.rpm excluded > Source : libmlx4-1.0.6-9.fc27.src.rpm obsoleted by > rdma-core > Source : libmlx5-1.0.2-5.fc27.src.rpm obsoleted by > rdma-core > Source : libmthca-1.0.6-18.fc27.src.rpm obsoleted by > rdma-core > Source : libnes-1.1.4-10.fc27.src.rpm obsoleted by > rdma-core > Source : libocrdma-1.0.8-6.fc27.src.rpm obsoleted by > rdma-core > Source : libusnic_verbs-2.0.2-4.fc27.src.rpm obsoleted by > rdma-core > Source : libvma-8.0.1-1.fc25.src.rpm excluded > Source : openmpi-2.0.2-2.fc26.src.rpm DONE > Source : perftest-3.0-4.fc27.src.rpm excluded > Source : qemu-2.10.0-0.2.rc3.fc27.src.rpm DONE > Source : qperf-0.4.9-11.fc27.src.rpm set to > exclude > Source : qpid-cpp-1.36.0-6.fc27.src.rpm in > progress... > Source : scsi-target-utils-1.0.70-3.fc27.src.rpm DONE > Source : srptools-1.0.3-2.fc26.src.rpm obsoleted by > rdma-core > > That qpid-cpp build is > https://koji.fedoraproject.org/koji/taskinfo?taskID=21469504 . Packages > marked 'excluded' are ones where 32-bit ARM is excluded by ExcludeArch or > ExclusiveArch. Thanks for rebuilding those packages. The qpid-cpp building task successfully completed. Can we close this bug as all packages had been rebuilt?
Well, I found a few more, for other libraries: ibutils (looks like you're on this one) infiniband-diags opensm (looks like you got this one too) pcp (subpackage pcp-pmda-infiniband requires libibumad) srptools
(In reply to Adam Williamson from comment #28) > Well, I found a few more, for other libraries: The RDMA stack has two base libraries: libibverbs and libibumad. Those were separated packages. They had been merged into new rdma-core package. You had addressed the packages require libibverbs. So, the new issues show up are libibumad related. The good news is that libibumad is used for infiniband subnet management. So, only a few packages depend on it. And most of them are RDMA specific. > > ibutils (looks like you're on this one) > infiniband-diags > opensm (looks like you got this one too) > pcp (subpackage pcp-pmda-infiniband requires libibumad) > srptools I'm working on rebuilding opensm, ibutils, libibmad, infiniband-diags and qperf. The srptools had been replaced by rdma-core. I will check pcp.
| I will check pcp. The pcp team is looking at fixing up the PCP build fallout (BZ #1485692), we should have a build later today resolving that. cheers.
(In reply to Nathan Scott from comment #30) > | I will check pcp. > > The pcp team is looking at fixing up the PCP build fallout (BZ #1485692), we > should have a build later today resolving that. Thanks!
(In reply to Honggang LI from comment #29) > I'm working on rebuilding opensm, ibutils, libibmad, infiniband-diags and > qperf. The srptools had been replaced by rdma-core. I had rebuilt opensm, ibutils, libibmad, infiniband-diags and qperf. ARM32 had been disable for those five packages.
(In reply to Honggang LI from comment #32) > I had rebuilt opensm, ibutils, libibmad, infiniband-diags and qperf. ARM32 > had been disable for those five packages. PCP has been rebuilt now, with the affected sub-packages disabled. https://koji.fedoraproject.org/koji/taskinfo?taskID=21506896
I think we can now say this is fixed; composes are OK and I don't see any remaining rdma-core related deps in the F27 'broken deps' report (though there are a *lot* in there so it's possible I missed one or two). Thanks for all your help.