Bug 1271230

Summary: libssh agent support fails on big endian architectures
Product: [Fedora] Fedora Reporter: Sinny Kumari <skumari>
Component: libsshAssignee: Andreas Schneider <asn>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: asn, awilliam, dan, jan.public, karsten, klember, negativo17, pbrobinson, pschindl, pvolpe, rdieter, robatino, stefw
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64   
OS: Unspecified   
Whiteboard: RejectedBlocker AcceptedFreezeException
Fixed In Version: libssh-0.7.2-2.fc23 libssh-0.7.2-2.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-31 16:08:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 467765, 1071880, 1170820, 1170822    

Description Sinny Kumari 2015-10-13 12:34:23 UTC
Description of problem:
I tried building latest cockpit version 0.79 for ppc64 arch. Build hangs for around 1 day at the end of test and fails with message:
PASS: test-webservice 40 /web-service/resource/gzip-encoding
EXCEPTION: Timeout(86400) expired for command:
 # bash --login -c /usr/bin/rpmbuild -bb --target ppc64 --nodeps  /builddir/build/SPECS/cockpit.spec 

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 487, in do
    raise commandTimeoutExpired("Timeout(%s) expired for command:\n # %s\n%s" % (timeout, command, output))
commandTimeoutExpired: Timeout(86400) expired for command:
 # bash --login -c /usr/bin/rpmbuild -bb --target ppc64 --nodeps  /builddir/build/SPECS/cockpit.spec 

LEAVE do --> EXCEPTION RAISED

In test log, I see one of the test failing with message:

./test-sshtransport terminated with SIGABRT
FAIL: test-sshtransport 8 /ssh-transport/key-auth-message

Detailed log is available at http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2808237

Version-Release number of selected component (if applicable): cockpit-0.79-1.fc22

This issue also occur on Fedora 23 which can be verified from build log http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2808218

Comment 1 Sinny Kumari 2015-10-13 12:55:28 UTC
I complied and ran test locally on a ppc64 box to find reason of test failure. So far I found following minute details which may be useful:
Comment lines 857 and 858 of source file https://github.com/cockpit-project/cockpit/blob/0.79/src/ws/test-sshtransport.c to avoid running sub-tests key-auth-message and key-auth-failed .

After commenting them, all tests passes and F22 sub-packages build successfully too.

I will look further and will update reason of these tests failure.

Comment 2 Dan Horák 2015-10-13 13:05:54 UTC
also worth mentioning that only big endian arches seems affected, ppc64 and s390/s390x

Comment 5 Peter 2015-10-14 03:04:40 UTC
Tracked this down to a bug with libssh. While we are waiting for a fix do we want to disable key auth in cockpit on these arches. Or are we ok holding up these builds till it gets fixed.

Comment 6 Dan Horák 2015-10-14 06:54:52 UTC
(In reply to Peter from comment #5)
> Tracked this down to a bug with libssh. While we are waiting for a fix do we
> want to disable key auth in cockpit on these arches. Or are we ok holding up
> these builds till it gets fixed.

We would prefer a new cockpit release (can be just patched Fedora package) until there is fixed libssh (is the issue already tracked in separate bugzilla?). We don't have a new cockpit build for quite some time and cockpit is part of the Fedora Server flavour we produce for ppc64 and s390x.

Comment 7 Stef Walter 2015-10-14 07:05:46 UTC
Reassigning to libssh. Peter do you have more details of the bug?

Comment 8 Peter 2015-10-14 12:56:23 UTC
upstream bug report. https://red.libssh.org/issues/204

Comment 9 Sinny Kumari 2015-10-16 07:51:02 UTC
I applied provided patch for libssh (tried on both 0.7.1 and 0.7.2) https://git.libssh.org/users/asn/libssh.git/commit/?h=fix&id=dbe409ccdd59e25f5ea3639d5f4f91ed3c570a36 (additionally modifying #define bswap_32 to #define bswap_32(x) to avoid compilation error) and installed locally on ppc64 box.

Further tried building again cockpit-0.79, but I still see test failing:
FAIL: test-sshtransport 8 /ssh-transport/key-auth-message

For reference, latest build.log file is available at https://sinnykumari.fedorapeople.org/cockpit/build.log

Comment 10 Peter 2015-10-16 15:13:57 UTC
Just to be clear, the patch upstream does not fix the issue. I know it's confusing because the error message contains the word "Success" but it's still a failure.

Comment 11 Sinny Kumari 2015-10-20 07:27:32 UTC
With the latest libssh patch https://git.libssh.org/users/asn/libssh.git/commit/?h=fix&id=d60cfa24861394b1f5a8d80e26dadb41b3d3829e , cockpit-0.79 build failure has been resolved.

Successful local build log of cockpit-0.79 is available at https://sinnykumari.fedorapeople.org/cockpit/fixed_build/build.log .

I think we can now build cockpit-0.79 on big endian with newer libssh build(having "fix agent auth on big endian machines" patch included).

Comment 12 Fedora Update System 2015-10-22 09:46:19 UTC
libssh-0.7.2-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-66ba300501

Comment 13 Fedora Update System 2015-10-22 09:47:22 UTC
libssh-0.7.2-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-b242195d7f

Comment 14 Fedora Update System 2015-10-24 12:07:46 UTC
libssh-0.7.2-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update libssh'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-66ba300501

Comment 15 Fedora Blocker Bugs Application 2015-10-26 13:05:58 UTC
Proposed as a Blocker and Freeze Exception for 23-final by Fedora user karsten using the blocker tracking app because:

 This is blocking cockpit builds on bigendian secondary archs.
Requesting freeze exception as cockpit is required for the Server edition.

Comment 16 Peter Robinson 2015-10-26 15:36:47 UTC
*** Bug 1275229 has been marked as a duplicate of this bug. ***

Comment 17 Fedora Blocker Bugs Application 2015-10-26 17:23:33 UTC
Proposed as a Freeze Exception for 23-beta by Fedora user pbrobinson using the blocker tracking app because:

 The libssh breaks builds of cockpit on big endian architectures where cockpit is a core feature of the Server spins we ship. 

Proposing as a exception as it's for secondary arches

Comment 18 Petr Schindler 2015-10-26 17:25:45 UTC
Discussed at 2015-10-26 blocker review meeting: [1]. 

This bug was accepted as Final blocker: This bug doesn't qualify for blocker status, but we'd consider pulling in a tested fix despite freeze.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2015-10-26/f23-blocker-review.2015-10-26-16.00.html

Comment 19 Petr Schindler 2015-10-26 17:30:12 UTC
Sorry for the confusion. I have really a bad day. This bug was rejected as blocker but accepted as Freeze Exception.

Comment 20 Fedora Update System 2015-10-26 18:28:58 UTC
libssh-0.7.2-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update libssh'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-b242195d7f

Comment 21 Adam Williamson 2015-10-26 22:54:06 UTC
For the record, I'm leaving this out of the RC4 compose, as it's big-endian only there's no point pulling it into the compose for primary arches, all of which are little-endian. The point of the FE is to get it pushed stable, and that will happen when it has sufficient karma.

Comment 22 Peter Robinson 2015-10-26 22:56:48 UTC
(In reply to awilliam from comment #21)
> For the record, I'm leaving this out of the RC4 compose, as it's big-endian
> only there's no point pulling it into the compose for primary arches, all of
> which are little-endian. The point of the FE is to get it pushed stable, and
> that will happen when it has sufficient karma.

We need it stable for the GA f23 tag

Comment 23 Kalev Lember 2015-10-27 05:24:09 UTC
Isn't libssh on the Server DVD? I don't think it can go stable if it wasn't part of the compose because we have to make sure that the version that's on the images matches with the version that's in the stable repo.

Comment 24 Adam Williamson 2015-10-27 06:18:05 UTC
Oh, yes, sorry - you're right. I missed that, was trying to avoid changing anything potentially risky for RC4. Well, if we get to RC5 we'll pull it in. my bad!

Comment 25 Fedora Update System 2015-10-31 16:08:02 UTC
libssh-0.7.2-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 26 Fedora Update System 2015-11-01 21:49:22 UTC
libssh-0.7.2-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.