Bug 1484155 - rdma-core excluding armhfp breaks many, many dep chains
Summary: rdma-core excluding armhfp breaks many, many dep chains
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: rdma-core
Version: 27
Hardware: armhfp
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Honggang LI
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1483254 (view as bug list)
Depends On: 1485692 1485693
Blocks: TRACKER-bugs-affecting-libguestfs F27BetaBlocker 1556539
TreeView+ depends on / blocked
 
Reported: 2017-08-22 21:09 UTC by Adam Williamson
Modified: 2018-07-03 12:22 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-05 16:49:46 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1483278 'unspecified' 'CLOSED' 'lorax runtime-install tries to install ''rdma'' on armhfp, but ''rdma-core'' (which replaced it) is not built for armhf... 2019-12-09 12:47:03 UTC

Internal Links: 1483278

Description Adam Williamson 2017-08-22 21:09:54 UTC
I'm splitting this out from https://bugzilla.redhat.com/show_bug.cgi?id=1483278 for clarity (that bug can cover specifically the lorax change).

The new 'rdma-core' source package is not built on armhfp, for what dledford assures us are good and valid reasons. However, this creates a whole ton of dependency problems. Just looking at a single library that is now built from rdma-core - libibverbs - all these things depend on it on x86_64:

[root@adam vms]# dnf repoquery --whatrequires "libibverbs.so.1()(64bit)"
Failed to synchronize cache for repo 'kanarip-phabricator', disabling.
Last metadata expiration check: 0:00:00 ago on Tue 22 Aug 2017 01:45:14 PM PDT.
ceph-base-1:12.1.3-1.fc27.x86_64
ceph-common-1:12.1.3-1.fc27.x86_64
ceph-fuse-1:12.1.3-1.fc27.x86_64
ceph-mds-1:12.1.3-1.fc27.x86_64
ceph-mgr-1:12.1.3-1.fc27.x86_64
ceph-mon-1:12.1.3-1.fc27.x86_64
ceph-osd-1:12.1.3-1.fc27.x86_64
ceph-radosgw-1:12.1.3-1.fc27.x86_64
ceph-test-1:12.1.3-1.fc27.x86_64
corosynclib-0:2.4.2-4.fc27.x86_64
dapl-0:2.1.9-4.fc26.x86_64
fio-0:2.99-3.fc27.x86_64
ga-openmpi-0:5.3b-24.fc27.x86_64
glusterfs-rdma-0:3.11.2-3.fc27.x86_64
ibacm-0:14-4.fc27.x86_64
libcephfs2-1:12.1.3-1.fc27.x86_64
libcephfs_jni1-1:12.1.3-1.fc27.x86_64
libfabric-0:1.4.2-4.fc27.x86_64
libhfi1-0:0.5-26.fc27.x86_64
libibcm-0:14-4.fc27.x86_64
libibverbs-devel-0:1.2.1-6.fc27.x86_64
libibverbs-utils-0:14-4.fc27.x86_64
libipathverbs-0:1.3-4.fc27.x86_64
libmlx4-0:1.0.6-9.fc27.x86_64
libmlx5-0:1.0.2-5.fc27.x86_64
libmthca-0:1.0.6-18.fc27.x86_64
libnes-0:1.1.4-10.fc27.x86_64
libocrdma-0:1.0.8-6.fc27.x86_64
librados-devel-1:12.1.3-1.fc27.x86_64
librados2-1:12.1.3-1.fc27.x86_64
libradosstriper1-1:12.1.3-1.fc27.x86_64
librbd1-1:12.1.3-1.fc27.x86_64
librdmacm-0:14-4.fc27.x86_64
librdmacm-utils-0:14-4.fc27.x86_64
librgw2-1:12.1.3-1.fc27.x86_64
libusnic_verbs-0:2.0.2-4.fc27.x86_64
libvma-0:8.0.1-1.fc25.x86_64
openmpi-0:2.0.2-2.fc26.x86_64
perftest-0:3.0-4.fc27.x86_64
qemu-system-aarch64-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-alpha-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-arm-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-cris-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-lm32-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-m68k-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-microblaze-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-mips-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-moxie-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-nios2-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-or1k-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-ppc-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-s390x-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-sh4-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-sparc-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-tricore-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-unicore32-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-x86-core-2:2.10.0-0.2.rc3.fc27.x86_64
qemu-system-xtensa-core-2:2.10.0-0.2.rc3.fc27.x86_64
qperf-0:0.4.9-11.fc27.x86_64
qpid-cpp-client-rdma-0:1.36.0-6.fc27.x86_64
rbd-mirror-1:12.1.3-1.fc27.x86_64
rbd-nbd-1:12.1.3-1.fc27.x86_64
rdma-core-devel-0:14-4.fc27.x86_64
scsi-target-utils-0:1.0.70-3.fc27.x86_64
srp_daemon-0:14-4.fc27.x86_64
srptools-0:1.0.3-2.fc26.x86_64

I believe most or all of those dependencies also exist on armhfp. There are, of course, many things that depend on *those* packages that are now broken. For instance, libvirt-daemon-driver-storage requires libvirt-daemon-driver-storage-rbd , which requires librbd and librados, both of which require libibverbs (which comes from rdma-core). This is just scratching the surface, I'm sure there are many others.

Because of all these broken dependencies, I strongly suspect many ARM compose tasks will fail, which will likely cause the whole compose to be considered a failure. #1483278 covers one specific way we *know* this change has broken the compose (lorax explicitly pulling in 'rdma' on armhfp), but even after we fix that, I rather suspect we'll start running into many other compose problems caused by all these broken deps. This is of course a major problem given F27's compressed schedule, we're already struggling with all sorts of other problems, and having composes fail due to all these dep issues would not help at all.

So we really need someone to either:

1) Just build rdma-core on armhfp again, even if it's "wrong" and won't really work, granting us some time to work through all the dependent packages and adjust them to build on armhfp without rdma-core, at which point we can disable rdma-core building on armhfp again

2) *Very quickly* identify and work through at least all the critical dep chains and adjust them all to build on armhfp without rdma-core

I think a reasonable timeframe for this is, like, a couple of days, so if 2) isn't possible in that time, we'd really like you to consider doing the first part of 1) instead. Myself and probably nirik (as provenpackagers) would be happy to help out with rebuilds if necessary.

Thanks!

Proposing as a Beta blocker due to the high likelihood this will cause release blocking images to fail to compose.

Comment 2 Adam Williamson 2017-08-22 21:33:52 UTC
Note for suggestion 1), this is specifically what dledford suggested could be done:

<dledford> The best we can do is carry a local patch to build on arm 32 if you want.  But that will highlight the problem on arm32.  You will have to create defines for memory barriers that simply don't exist on arm32, so we'll have to put dummy defines in there.

Comment 3 Honggang LI 2017-08-23 00:35:47 UTC
I'm taking this bug. I will try to reserve an ARM machine in beaker or create an ARM virtual machine with qemu. Then I will check the deep chain which provided in  #comment 1 with a x86_64 system.

Comment 4 Adam Williamson 2017-08-23 00:42:57 UTC
Welp, I've started working on a couple of the more obvious rebuilds. qemu was just a single line change and I tested that a scratch build worked fine, so I've gone ahead and sent that for rawhide and f27. I'm also testing a scratch build for ceph; it looks like we may be able to get librados2 and librbd1 built but without dependencies on libibverbs, which would probably help with several of the dep chains that seem to run through those libs.

Comment 5 Adam Williamson 2017-08-23 05:46:26 UTC
ceph's a bit of a beast; my scratch build started 4 hours ago and it's at like 34%. I'll check on it in the morning and send official builds if it looks good.

Comment 6 Peter Robinson 2017-08-23 08:38:02 UTC
> <dledford> The best we can do is carry a local patch to build on arm 32 if
> you want.  But that will highlight the problem on arm32.  You will have to
> create defines for memory barriers that simply don't exist on arm32, so
> we'll have to put dummy defines in there.

ARMv7 has memory barriers, you'll need to check the exact details with the toolchains team as I'm not an expert there, but all variants of arm32 that we care about (ARMv7 and later) should be fine here and have had them available for some time in both the architecture and the tookchain.

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/CJAGIEIE.html

Comment 7 Doug Ledford 2017-08-23 12:45:29 UTC
(In reply to Peter Robinson from comment #6)
> > <dledford> The best we can do is carry a local patch to build on arm 32 if
> > you want.  But that will highlight the problem on arm32.  You will have to
> > create defines for memory barriers that simply don't exist on arm32, so
> > we'll have to put dummy defines in there.
> 
> ARMv7 has memory barriers, you'll need to check the exact details with the
> toolchains team as I'm not an expert there, but all variants of arm32 that
> we care about (ARMv7 and later) should be fine here and have had them
> available for some time in both the architecture and the tookchain.
> 
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/
> CJAGIEIE.html

As I recall (and I'm not the best person to talk about this because when it was brought up on the upstream mailing list, I didn't know the details of arm32 that other people did), the problem is that even though ARMv7 has DMB, it is still insufficient for our needs because the architecture is specifically designed as non-coherent when it comes to PCI DMA operations and local processor cache.  As such, we don't have what we need to force the proper ordering of reads of PCI DMA written memory, or writes to memory that will eventually be PCI DMA read.

Comment 8 Honggang LI 2017-08-23 13:04:13 UTC
We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32.

Comment 9 Honggang LI 2017-08-23 13:31:12 UTC
[root@calxeda-arm-soc-03 ~]# uname -r
4.12.5-300.fc26.armv7hl

[root@calxeda-arm-soc-03 ~]# rpm -q --provides libibverbs
libibverbs = 1.2.1-4.fc26
libibverbs(armv7hl-32) = 1.2.1-4.fc26
libibverbs.so.1
libibverbs.so.1(IBVERBS_1.0)
libibverbs.so.1(IBVERBS_1.1)


[root@calxeda-arm-soc-03 ~]# dnf repoquery --whatrequires 'libibverbs'
Last metadata expiration check: 0:38:32 ago on Wed 23 Aug 2017 02:29:18 PM CEST.
corosynclib-0:2.4.2-2.fc26.armv7hl
fio-0:2.18-1.fc26.armv7hl
glusterfs-rdma-0:3.10.3-1.fc26.armv7hl
glusterfs-rdma-0:3.10.5-1.fc26.armv7hl
ibacm-0:1.2.1-2.fc26.armv7hl
libcxgb3-0:1.3.1-14.fc26.armv7hl
libcxgb4-0:1.3.6-2.fc26.armv7hl
libfabric-0:1.4.1-1.fc26.armv7hl
libibcm-0:1.0.5-15.fc26.armv7hl
libibverbs-devel-0:1.2.1-4.fc26.armv7hl
libibverbs-utils-0:1.2.1-4.fc26.armv7hl
libmlx4-0:1.0.6-7.fc26.armv7hl
libmlx5-0:1.0.2-3.fc26.armv7hl
libmthca-0:1.0.6-16.fc26.armv7hl
libnes-0:1.1.4-7.fc26.armv7hl
libocrdma-0:1.0.8-4.fc26.armv7hl
librdmacm-0:1.1.0-4.fc26.armv7hl
librdmacm-utils-0:1.1.0-4.fc26.armv7hl
libusnic_verbs-0:2.0.2-2.fc26.armv7hl
openmpi-0:2.0.2-2.fc26.armv7hl
qemu-system-aarch64-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-aarch64-core-2:2.9.0-5.fc26.armv7hl
qemu-system-alpha-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-alpha-core-2:2.9.0-5.fc26.armv7hl
qemu-system-arm-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-arm-core-2:2.9.0-5.fc26.armv7hl
qemu-system-cris-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-cris-core-2:2.9.0-5.fc26.armv7hl
qemu-system-lm32-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-lm32-core-2:2.9.0-5.fc26.armv7hl
qemu-system-m68k-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-m68k-core-2:2.9.0-5.fc26.armv7hl
qemu-system-microblaze-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-microblaze-core-2:2.9.0-5.fc26.armv7hl
qemu-system-mips-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-mips-core-2:2.9.0-5.fc26.armv7hl
qemu-system-moxie-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-moxie-core-2:2.9.0-5.fc26.armv7hl
qemu-system-nios2-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-nios2-core-2:2.9.0-5.fc26.armv7hl
qemu-system-or1k-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-or1k-core-2:2.9.0-5.fc26.armv7hl
qemu-system-ppc-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-ppc-core-2:2.9.0-5.fc26.armv7hl
qemu-system-s390x-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-s390x-core-2:2.9.0-5.fc26.armv7hl
qemu-system-sh4-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-sh4-core-2:2.9.0-5.fc26.armv7hl
qemu-system-sparc-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-sparc-core-2:2.9.0-5.fc26.armv7hl
qemu-system-tricore-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-tricore-core-2:2.9.0-5.fc26.armv7hl
qemu-system-unicore32-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-unicore32-core-2:2.9.0-5.fc26.armv7hl
qemu-system-x86-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-x86-core-2:2.9.0-5.fc26.armv7hl
qemu-system-xtensa-core-2:2.9.0-1.fc26.1.armv7hl
qemu-system-xtensa-core-2:2.9.0-5.fc26.armv7hl
qperf-0:0.4.9-9.fc26.armv7hl
qpid-cpp-client-rdma-0:1.36.0-1.fc26.armv7hl
scsi-target-utils-0:1.0.55-6.fc26.armv7hl
scsi-target-utils-0:1.0.70-2.fc26.armv7hl
srptools-0:1.0.3-2.fc26.armv7hl 

[root@calxeda-arm-soc-03 ~]#  cat x.sh 
#!/bin/bash

rm -f s.txt || exit

rpm_list=$(dnf repoquery --whatrequires 'libibverbs' | sed '1d')
#echo $rpm_list | wc

for r in ${rpm_list}; do
	dnf info $r | grep ^Source >> s.txt
done

cat s.txt | sort | uniq -c

[root@calxeda-arm-soc-03 ~]# sh x.sh 
Last metadata expiration check: 0:44:49 ago on Wed 23 Aug 2017 02:29:18 PM CEST.
       1 Source       : fio-2.18-1.fc26.src.rpm
      1 Source       : glusterfs-3.10.3-1.fc26.src.rpm
      1 Source       : glusterfs-3.10.5-1.fc26.src.rpm
      1 Source       : ibacm-1.2.1-2.fc26.src.rpm
      1 Source       : libcxgb3-1.3.1-14.fc26.src.rpm
      1 Source       : libcxgb4-1.3.6-2.fc26.src.rpm
      1 Source       : libfabric-1.4.1-1.fc26.src.rpm
      1 Source       : libibcm-1.0.5-15.fc26.src.rpm
      2 Source       : libibverbs-1.2.1-4.fc26.src.rpm
      1 Source       : libmlx4-1.0.6-7.fc26.src.rpm
      1 Source       : libmlx5-1.0.2-3.fc26.src.rpm
      1 Source       : libmthca-1.0.6-16.fc26.src.rpm
      1 Source       : libnes-1.1.4-7.fc26.src.rpm
      1 Source       : libocrdma-1.0.8-4.fc26.src.rpm
      2 Source       : librdmacm-1.1.0-4.fc26.src.rpm
      1 Source       : libusnic_verbs-2.0.2-2.fc26.src.rpm
      1 Source       : openmpi-2.0.2-2.fc26.src.rpm
     19 Source       : qemu-2.9.0-1.fc26.1.src.rpm
     19 Source       : qemu-2.9.0-5.fc26.src.rpm
      1 Source       : qperf-0.4.9-9.fc26.src.rpm
      1 Source       : qpid-cpp-1.36.0-1.fc26.src.rpm
      1 Source       : scsi-target-utils-1.0.55-6.fc26.src.rpm
      1 Source       : scsi-target-utils-1.0.70-2.fc26.src.rpm
      1 Source       : srptools-1.0.3-2.fc26.src.rpm

Comment 10 Honggang LI 2017-08-23 13:41:45 UTC
We have 8 components immediately depend libibverbs.

      fio            : fio-2.18-1.fc26.src.rpm
      glusterfs      : glusterfs-3.10.3-1.fc26.src.rpm
                     : glusterfs-3.10.5-1.fc26.src.rpm
      libfabric      : libfabric-1.4.1-1.fc26.src.rpm
      openmpi        : openmpi-2.0.2-2.fc26.src.rpm
      qemu           : qemu-2.9.0-1.fc26.1.src.rpm
                     : qemu-2.9.0-5.fc26.src.rpm
     qperf           : qperf-0.4.9-9.fc26.src.rpm
     qpid-cpp        : qpid-cpp-1.36.0-1.fc26.src.rpm
scsi-target-utils       : scsi-target-utils-1.0.55-6.fc26.src.rpm
                     : scsi-target-utils-1.0.70-2.fc26.src.rpm

      rdma-core      : ibacm-1.2.1-2.fc26.src.rpm // These packages will be retired for ARM32.
                     : libcxgb3-1.3.1-14.fc26.src.rpm
                     : libcxgb4-1.3.6-2.fc26.src.rpm
                     : libibcm-1.0.5-15.fc26.src.rpm
                     : libibverbs-1.2.1-4.fc26.src.rpm
                     : libmlx4-1.0.6-7.fc26.src.rpm
                     : libmlx5-1.0.2-3.fc26.src.rpm
                     : libmthca-1.0.6-16.fc26.src.rpm
                     : libnes-1.1.4-7.fc26.src.rpm
                     : libocrdma-1.0.8-4.fc26.src.rpm
                     : librdmacm-1.1.0-4.fc26.src.rpm
                     : libusnic_verbs-2.0.2-2.fc26.src.rpm
                     : srptools-1.0.3-2.fc26.src.rpm

Comment 11 Peter Robinson 2017-08-23 14:52:52 UTC
(In reply to Honggang LI from comment #8)
> We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported
> in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from
> outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32.

That's not entirely true, it was originally enabled in Fedora because someone requested it, I believe it was once actually tested but I don't remember the details.

Comment 12 Peter Robinson 2017-08-23 14:54:04 UTC
Ultimately this hasn't been communicated/coordinated at all and appears to basically be a run and dump.

Comment 13 Adam Williamson 2017-08-23 15:01:09 UTC
Honggang: it would be better to check f27 / rawhide. You don't need to *install* them to do that; you can just pass --releasever to dnf. For instance, ceph was only set to build against rdma in commit 44c0bd8d5c6ece6c22e2f9fff1297dc082b9a0a2 , which is only on the master and f27 branches, not f26.

Comment 14 Adam Williamson 2017-08-23 16:37:58 UTC
My scratch build of ceph looks good; we can build it with the RDMA support disabled (for 32-bit ARM) and we get librdb and librados but without the libibverbs dependency. I've now sent official builds for Rawhide and F27, but they'll take 12+ hours to run. scsi-target-utils builds against ceph (as well as directly against rdma), so we will have to wait for the ceph rebuild before rebuilding that. I'll take a look at some of the others.

Comment 15 Honggang LI 2017-08-24 02:21:44 UTC
(In reply to Peter Robinson from comment #11)
> (In reply to Honggang LI from comment #8)
> > We don't have ARM32 hardware for RDMA test. As ARM32 RDMA never be supported
> > in RHEL/Fedora/CENTOS product, we never have a bug about ARM32 RDMA from
> > outside of Redhat. So, I'd prefer to remove RDMA stack from ARM32.
> 
> That's not entirely true, it was originally enabled in Fedora because
> someone requested it, I believe it was once actually tested but I don't
> remember the details.

Redhat never tested RDMA for ARM32 as don't have hardware. If you believe someone had tested this, please try your best to find and feedback the test history/info/result to us. We are happy to see that. If we can confirm RDMA ARM32 works, we will try to enable RDMA for ARM32.

Comment 16 Honggang LI 2017-08-24 02:37:00 UTC
(In reply to Peter Robinson from comment #12)
> Ultimately this hasn't been communicated/coordinated at all and appears to
> basically be a run and dump.

Look at the built history of RDMA stack, the old components were built for ARM32 in F19/F20 massive building to support ARM32 as primary architecture for F20. Redhat never tested ARM32 RDMA for F20.

The ARM32 packages got built just because the building/mock system was enable for ARM32 building. I never heard someone had tested it. I never saw an ARM32 RDMA bugs was filed from outside in last 6+ years.

https://en.wikipedia.org/wiki/Fedora_version_history
http://download-node-02.eng.bos.redhat.com/released/F-20/GOLD/Everything/
http://download-node-02.eng.bos.redhat.com/released/F-19/GOLD/Everything/

The F20 is the first release supports ARMPHF/AMR32.


----------------+-------------------------+-------------------------------+
component name	| initial fedora release  | first F-release enable RDMA   |
                | with the component built| component built for ARM32     |
----------------+-------------------------+-------------------------------+
rdma            | F8 (noarch)             |                               |
libibverbs      | F7                      | libibverbs-1.1.8-2.fc20       |
libibumad       | F8                      | libibumad-1.3.8-2.fc19        |
----------------+-------------------------+-------------------------------+
libcxgb3        | F8                      | libcxgb3-1.3.1-3.fc19         |
libcxgb4        | F15                     | libcxgb4-1.2.0-3.fc19         |
libmlx4         | F8                      | libmlx4-1.0.5-4.fc20          |
libnes          | F15                     | libnes-1.1.3-3.fc19           |
libocrdma       | F24                     | libocrdma-1.0.8-3.fc24        |
libibcm         | F8                      | libibcm-1.0.5-7.fc19          |
libmlx5         | F23                     | libmlx5-1.0.2-2.fc23          |
libusnic_verbs  | F23                     | libusnic_verbs-2.0.2-1.fc23   |
----------------+-------------------------+-------------------------------+
ibacm           | F23                     | ibacm-1.2.0-3.fc23            |
srptools        | F15                     | srptools-0.0.4-17.fc19        |
libibmad        | F8                      | libibmad-1.3.9-2.fc19         |
librdmacm       | F8                      | librdmacm-1.0.17-1.fc20       |
opensm          | F8                      | opensm-3.3.15-6.fc20          |
mstflint        | F15                     | mstflint-1.4-9.fc19           |
perftest        | F15                     | never built for ARM32         |
dapl            | F26                     | never built for ARM32         |
qperf           | F15                     | qperf-0.4.6-8.fc19            |
libfabric       | F21                     | libfabric-1.1.0-1.fc21        |
ibutils         | F17                     | ibutils-1.5.7-9.fc19          |
infiniband-diags| F15                     | infiniband-diags-1.6.1-4.fc19 |
openmpi         | F6                      | openmpi-1.6.4-2.fc19          |
fabtests        | F24                     | fabtests-1.3.0-3.fc24         |
mpich           | F18                     | mpich-3.0.4-3.fc20            |
libiwpm         | F23                     | libiwpm-1.0.3-7.fc23          |
----------------+-------------------------+-------------------------------+
libhfi1         |                                                         |
libpsm2         | These RDMA components are avaiable in RHEL, but         |
libipathverbs   | unavailable in Fedora. No ARM32 packages for them.      |
infinipath-psm  |                                                         |
opa-ff          | https://apps.fedoraproject.org/packages/                |
opa-fm          | The build history was extracted from this website.      |
opa-fmgui       |                                                         |
libehca         |                                                         |
mvapichv2       |                                                         |
mvapich         |                                                         |
mpitests        |                                                         |
rds-tools       |                                                         |
ibsim           |                                                         |
----------------+---------------------------------------------------------+

Comment 17 Honggang LI 2017-08-24 02:39:42 UTC
----------------+-------------------------+-------------------------------+
libhfi1         |                                                         |
libpsm2         | These RDMA components are avaiable in RHEL, but         |
libipathverbs   | unavailable in Fedora. No ARM32 packages for them.      |
infinipath-psm  |                                                         |
opa-ff          | https://apps.fedoraproject.org/packages/                |
opa-fm          | The build history was extracted from this website.      |
opa-fmgui       |                                                         |
libehca         |                                                         |
mvapichv2       |                                                         |
mvapich         |                                                         |
mpitests        |                                                         |
rds-tools       |                                                         |
ibsim           |                                                         |
----------------+---------------------------------------------------------+

Some of those packages are available in Fedora, but absolutely no ARM32 packages available for them.

Comment 18 Adam Williamson 2017-08-24 06:30:33 UTC
Update on rebuilds: I've done corosync and libfabric. I also did hwloc, but I'm now building it again because it needs a Requires: changing from libibverbs%{_isa} to rdma-core-devel%{_isa}. openmpi pulls in hwloc-devel, so it needs hwloc fixing before I can build it, but it's queued up to go. ceph is still running, when it's done I'll look at scsi-target-utils. Kaleb Keithley did glusterfs yesterday.

Comment 19 Peter Robinson 2017-08-24 09:29:33 UTC
> The F20 is the first release supports ARMPHF/AMR32.

Incorrect, F-20 was when ARM was promoted to primary, the first release of Fedora on armhfp (ARMv7/hard floating point) was Fedora 15, Fedora has been built on ARM since well before Fedora 7.

Comment 20 Honggang LI 2017-08-24 09:50:03 UTC
(In reply to Peter Robinson from comment #19)
> > The F20 is the first release supports ARMPHF/AMR32.
> 
> Incorrect, F-20 was when ARM was promoted to primary, the first release of
> Fedora on armhfp (ARMv7/hard floating point) was Fedora 15, Fedora has been
> built on ARM since well before Fedora 7.

OK, but the RDMA stack was enabled for ARM32 when F20 was promoted to primary. The RDMA stack got in ARM32 mostly because of that.

Comment 21 Jon Masters 2017-08-25 16:30:56 UTC
In reply to comment #c7 it's not that the "architecture is specifically designed" it's that the existing SoC platforms implementing it might not have the expected coherency constraints between local processor caches and PCIe inbound DMA operations. I could well believe that is true. But it's not architectural, it's implementation. It's why we have e.g. SBSA on 64-bit ARM.

Comment 22 Adam Williamson 2017-08-25 18:52:30 UTC
OK, I've gone through all the ones I could find for this now. My status list is:

Source       : ceph-12.1.3-1.fc27.src.rpm                       DONE
Source       : corosync-2.4.2-4.fc27.src.rpm                    DONE
Source       : dapl-2.1.9-4.fc26.src.rpm                        excluded
Source       : fio-2.99-3.fc27.src.rpm                          DONE
Source       : ga-5.3b-24.fc27.src.rpm                          excluded
Source       : glusterfs-3.11.2-3.fc27.src.rpm                  DONE (kaleb)
Source       : libfabric-1.4.2-4.fc27.src.rpm                   DONE
Source       : libhfi1-0.5-26.fc27.src.rpm                      excluded
Source       : libipathverbs-1.3-4.fc27.src.rpm                 excluded
Source       : libmlx4-1.0.6-9.fc27.src.rpm                     obsoleted by rdma-core
Source       : libmlx5-1.0.2-5.fc27.src.rpm                     obsoleted by rdma-core
Source       : libmthca-1.0.6-18.fc27.src.rpm                   obsoleted by rdma-core
Source       : libnes-1.1.4-10.fc27.src.rpm                     obsoleted by rdma-core
Source       : libocrdma-1.0.8-6.fc27.src.rpm                   obsoleted by rdma-core
Source       : libusnic_verbs-2.0.2-4.fc27.src.rpm              obsoleted by rdma-core
Source       : libvma-8.0.1-1.fc25.src.rpm                      excluded
Source       : openmpi-2.0.2-2.fc26.src.rpm                     DONE
Source       : perftest-3.0-4.fc27.src.rpm                      excluded
Source       : qemu-2.10.0-0.2.rc3.fc27.src.rpm                 DONE
Source       : qperf-0.4.9-11.fc27.src.rpm                      set to exclude
Source       : qpid-cpp-1.36.0-6.fc27.src.rpm                   in progress...
Source       : scsi-target-utils-1.0.70-3.fc27.src.rpm          DONE
Source       : srptools-1.0.3-2.fc26.src.rpm                    obsoleted by rdma-core

That qpid-cpp build is https://koji.fedoraproject.org/koji/taskinfo?taskID=21469504 . Packages marked 'excluded' are ones where 32-bit ARM is excluded by ExcludeArch or ExclusiveArch.

Comment 23 Adam Williamson 2017-08-25 18:54:14 UTC
Note there's technically an upgrade path problem for all the packages excluded by rdma-core (or its subpackages): nothing on armv7hl excludes them, because rdma-core isn't built there. So anyone who actually has those packages installed on arm will not be able to update without --allowerasing . But if we're right about all this, that's likely not many or any people.

Comment 24 Peter Robinson 2017-08-25 19:12:49 UTC
> . But if we're right about all this, that's likely not many or any people.

There will be for things like workstation where Boxes pulls in libvirt I suspect

Comment 25 Adam Williamson 2017-08-25 20:40:44 UTC
i guess we'll see. actually, thinking about it, the old packages might just keep lying around and no observable problems will show up...

Comment 26 Cole Robinson 2017-08-25 22:07:44 UTC
*** Bug 1483254 has been marked as a duplicate of this bug. ***

Comment 27 Honggang LI 2017-08-26 02:03:56 UTC
(In reply to Adam Williamson from comment #22)
> OK, I've gone through all the ones I could find for this now. My status list
> is:
> 
> Source       : ceph-12.1.3-1.fc27.src.rpm                       DONE
> Source       : corosync-2.4.2-4.fc27.src.rpm                    DONE
> Source       : dapl-2.1.9-4.fc26.src.rpm                        excluded
> Source       : fio-2.99-3.fc27.src.rpm                          DONE
> Source       : ga-5.3b-24.fc27.src.rpm                          excluded
> Source       : glusterfs-3.11.2-3.fc27.src.rpm                  DONE (kaleb)
> Source       : libfabric-1.4.2-4.fc27.src.rpm                   DONE
> Source       : libhfi1-0.5-26.fc27.src.rpm                      excluded
> Source       : libipathverbs-1.3-4.fc27.src.rpm                 excluded
> Source       : libmlx4-1.0.6-9.fc27.src.rpm                     obsoleted by
> rdma-core
> Source       : libmlx5-1.0.2-5.fc27.src.rpm                     obsoleted by
> rdma-core
> Source       : libmthca-1.0.6-18.fc27.src.rpm                   obsoleted by
> rdma-core
> Source       : libnes-1.1.4-10.fc27.src.rpm                     obsoleted by
> rdma-core
> Source       : libocrdma-1.0.8-6.fc27.src.rpm                   obsoleted by
> rdma-core
> Source       : libusnic_verbs-2.0.2-4.fc27.src.rpm              obsoleted by
> rdma-core
> Source       : libvma-8.0.1-1.fc25.src.rpm                      excluded
> Source       : openmpi-2.0.2-2.fc26.src.rpm                     DONE
> Source       : perftest-3.0-4.fc27.src.rpm                      excluded
> Source       : qemu-2.10.0-0.2.rc3.fc27.src.rpm                 DONE
> Source       : qperf-0.4.9-11.fc27.src.rpm                      set to
> exclude
> Source       : qpid-cpp-1.36.0-6.fc27.src.rpm                   in
> progress...
> Source       : scsi-target-utils-1.0.70-3.fc27.src.rpm          DONE
> Source       : srptools-1.0.3-2.fc26.src.rpm                    obsoleted by
> rdma-core
> 
> That qpid-cpp build is
> https://koji.fedoraproject.org/koji/taskinfo?taskID=21469504 . Packages
> marked 'excluded' are ones where 32-bit ARM is excluded by ExcludeArch or
> ExclusiveArch.

Thanks for rebuilding those packages. The qpid-cpp building task successfully completed. Can we close this bug as all packages had been rebuilt?

Comment 28 Adam Williamson 2017-08-26 03:01:44 UTC
Well, I found a few more, for other libraries:

ibutils (looks like you're on this one)
infiniband-diags
opensm (looks like you got this one too)
pcp (subpackage pcp-pmda-infiniband requires libibumad)
srptools

Comment 29 Honggang LI 2017-08-26 03:11:29 UTC
(In reply to Adam Williamson from comment #28)
> Well, I found a few more, for other libraries:

The RDMA stack has two base libraries: libibverbs and libibumad. Those were separated packages. They had been merged into new rdma-core package. You had addressed the packages require libibverbs. So, the new issues show up are libibumad related.

The good news is that libibumad is used for infiniband subnet management. So, only a few packages depend on it. And most of them are RDMA specific.

> 
> ibutils (looks like you're on this one)
> infiniband-diags
> opensm (looks like you got this one too)
> pcp (subpackage pcp-pmda-infiniband requires libibumad)
> srptools

I'm working on rebuilding opensm, ibutils, libibmad, infiniband-diags and qperf. The srptools had been replaced by rdma-core. I will check pcp.

Comment 30 Nathan Scott 2017-08-28 02:48:34 UTC
| I will check pcp.

The pcp team is looking at fixing up the PCP build fallout (BZ #1485692), we should have a build later today resolving that.

cheers.

Comment 31 Honggang LI 2017-08-28 02:55:09 UTC
(In reply to Nathan Scott from comment #30)
> | I will check pcp.
> 
> The pcp team is looking at fixing up the PCP build fallout (BZ #1485692), we
> should have a build later today resolving that.

Thanks!

Comment 32 Honggang LI 2017-08-28 02:56:40 UTC
(In reply to Honggang LI from comment #29)

> I'm working on rebuilding opensm, ibutils, libibmad, infiniband-diags and
> qperf. The srptools had been replaced by rdma-core.

I had rebuilt opensm, ibutils, libibmad, infiniband-diags and qperf. ARM32 had been disable for those five packages.

Comment 33 Nathan Scott 2017-08-28 04:50:27 UTC
(In reply to Honggang LI from comment #32)
> I had rebuilt opensm, ibutils, libibmad, infiniband-diags and qperf. ARM32
> had been disable for those five packages.

PCP has been rebuilt now, with the affected sub-packages disabled.
https://koji.fedoraproject.org/koji/taskinfo?taskID=21506896

Comment 34 Adam Williamson 2017-09-05 16:49:46 UTC
I think we can now say this is fixed; composes are OK and I don't see any remaining rdma-core related deps in the F27 'broken deps' report (though there are a *lot* in there so it's possible I missed one or two). Thanks for all your help.


Note You need to log in before you can comment on or make changes to this bug.