Description of problem: I am trying to update my datalog package to version 2.6 on rowhide using a fresh install of Fedora 23. I have performed all of the required steps except "fedpkg build". Some of the previous steps require that I have the correct credentials to authenticate. When I try to do the build, it stalls for quite a while and then fails saying "Could not execute build: Could not auth with koji. Login failed". I tried regenerating my credentials and this is what happens: $ fedora-packager-setup Setting up Fedora packager environment You need a client certificate from the Fedora Account System, lets get one now FAS Username: ramsdell FAS Password: Saved: /home/ramsdell/.fedora-server-ca.cert Linking: ~/.fedora-server-ca.cert to ~/.fedora-upload-ca.cert $ fedpkg build /usr/lib/python2.7/site-packages/pyrpkg/__init__.py:309: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 for (_, _, ssl_reason) in error.message: You might want to run fedora-packager-setup to regenerate SSL certificate. For more info see https://fedoraproject.org/wiki/Using_the_Koji_build_system#Fedora_Account_System_.28FAS2.29_Setup Could not execute build: Could not auth with koji. Login failed: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')] $ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Could you try running "koji hello" to see if you get the same error? If that works, you could try submitting your build directly without using fedpkg. koji build rawhide git://pkgs.fedoraproject.org/rpms/datalog?#$(git rev-parse HEAD)
"koji hello" fails too. After consulting with others in my company, we believe this problem is due to our firewall and is not a problem with fedpkg. Therefore I am closing this report. John
I'm still hitting this: autojump$ fedora-packager-setup Setting up Fedora packager environment Saved: /home/tt/.fedora-server-ca.cert autojump$ fedpkg build /usr/lib/python2.7/site-packages/pyrpkg/__init__.py:309: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 for (_, _, ssl_reason) in error.message: You might want to run fedora-packager-setup to regenerate SSL certificate. For more info see https://fedoraproject.org/wiki/Using_the_Koji_build_system#Fedora_Account_System_.28FAS2.29_Setup Could not execute build: Could not auth with koji. Login failed: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')] autojump$ koji hello Error: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')] fedpkg-1.23-2.fc25.noarch koji-1.10.1-10.fc25.noarch fedora-packager-0.5.10.7-2.fc25.noarch
Given that koji hello fails as well, I think this issue is actually not in fedpkg. Mike, do you have any suggestions for debugging this problem?
with : fedpkg-1.23-2.fc23.noarch koji-1.10.1-8.fc23.noarch fedora-packager-0.5.10.7-1.fc23.noarch #koji hello bonjour, sergiomb! You are using the hub at http://koji.fedoraproject.org/kojihub Authenticated via client certificate /home/sergio/.fedora.cert #fedora-cert -v Verifying Certificate cert expires: 2016-10-10
I agree, this certainly isn't a fedpkg issue. ``` $ fedora-cert -v Verifying Certificate cert expires: 2016-08-31 WARNING: Your cert appears in the revocation list. (' ', 'https://admin.fedoraproject.org/ca/crl.pem') ``` How do I fix this? Shall I ask someone from fedora infra?
Due to the ssl auth code in koji using a raw socket it does not work with proxy setups, you need to have direct network access, or change the code to allow a proxy to Work. reassigning to koji, but for now can you see if you can get direct network access to koji.fedoraproject.org?
I have already submitted the datalog package update by installing Fedora 23 on a machine that only uses a direct connection to the internet. So this problem is about lack of support for our internal proxy. The machine that I used for the direct connection is no long available, so testing the direct connection would require a reinstallation, so it's not easy for me to run the test you requested.
I wasn't using proxy, I was on vpn, does it make any difference? I'm trying now again, I'm on our company's wifi and it still doesn't work, same error message (no proxy, no vpn).
Ok. I'm going to try to revive that laptop and do the actual test you requested. There is a chance the disk has not been wiped yet.
So, tried with curl: ``` $ curl --cacert ~/.fedora-server-ca.cert -E ~/.fedora.cert -v https://koji.fedoraproject.org/kojihub * Trying 209.132.181.7... * Connected to koji.fedoraproject.org (209.132.181.7) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: ~/.fedora-server-ca.cert CApath: none * ALPN/NPN, server did not agree to a protocol * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: E=buildsys,CN=koji.fedoraproject.org,OU=Fedora Builders,O=Fedora Project,ST=North Carolina,C=US * start date: Apr 15 18:26:14 2014 GMT * expire date: Apr 12 18:26:14 2024 GMT * common name: koji.fedoraproject.org * issuer: E=admin,CN=Fedora Project CA,OU=Fedora Project CA,O=Fedora Project,L=Raleigh,ST=North Carolina,C=US > GET /kojihub HTTP/1.1 > Host: koji.fedoraproject.org > User-Agent: curl/7.49.1 > Accept: */* > < HTTP/1.1 405 Method Not Allowed < Date: Mon, 27 Jun 2016 12:34:17 GMT < Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 mod_wsgi/3.4 Python/2.7.5 < Allow: POST < AppTime: D=288 < AppServer: koji01.phx2.fedoraproject.org < Transfer-Encoding: chunked < Content-Type: text/plain; charset=UTF-8 < Method Not Allowed * Connection #0 to host koji.fedoraproject.org left intact This is an XML-RPC server. Only POST requests are accepted.% ``` To me this looks like it works just fine. So why does koji and `fedora-cert -v` think my client certificate is revoked?
you can only ever have one fedora cert, we revoke the old ones when you get a new one.
We found the disk and I am now able to test using a direct connection. I typed: $ koji hello It responded with some foreign language and my login name, said which hub I was using, and finally said I was authenticated with my .fedora.cert. In summary, for me, I see this problem only when using a proxy.
I moved the certificates to a machine within our firewall and tried the test. After a long wait, I received the following error message. $ koji hello Error: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')] $
Koji does not support using proxies with ssl certificate auth. You have to have direct connection. As to the revoked certificate, If youget a new cert on a different machine your older cert gets revoked. Tomas your curl call is not authenticated and so will work just fine.
Dennis, it means that I should copy the certificate across all of my machines where I want to authenticate with koji? Or is there a better way?
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle. Changing version to '25'.
Today, after "yum update" of augeas-libs, eog and firefox on F23 I saw this issue. Once I rebooted the system the issue was gone. $ fedpkg new-sources ~/git/librepo/librepo-1.7.20.tar.gz Uploading: /home/tmlcoch/git/librepo/librepo-1.7.20.tar.gz Could not execute new_sources: (35, 'Encountered end of file') $ koji hello Error: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')] $ rpm -qv fedpkg koji fedora-packager fedpkg-1.24-3.fc23.noarch koji-1.10.1-24.2.fc23.noarch fedora-packager-0.5.10.7-1.fc23.noarch $ fedora-cert -v Verifying Certificate cert expires: 2017-01-15
I had the same issue, updating to koji-1.10.1-12 helped for me - while I guess this is somehow related to OpenSSL 1.0.2g/h and the following commit there: https://github.com/openssl/openssl/commit/64193c8218540499984cd63cda41f3cd491f3f59
I've never been able to replicate the shutdown while in init problem. The problem must originate in openssl, perhaps with the commit Robert linked to (thanks Robert!). Does anyone have a clear replicator?
Not a clear reproducer, but these are my details: koji-1.9.0-5 together with openssl-1.0.2h-1 was working for me on 2016-07-05, the same constellation was no longer working at 2016-09-18 - until I updated to koji-1.10.1-12. Given my whole VM did not get any update since 2016-06-12, this can't be client-only.
koji-1.9.0-5 ??? That's a fairly new openssl and really old koji.... Which Fedora was this? (or was it epel)?
looks like I can reproduce this: [brain@x1carbon libsolv]$ rpm -q koji openssl python2-pyOpenSSL koji-1.10.1-13.fc26.noarch openssl-1.1.0b-4.fc26.x86_64 python2-pyOpenSSL-16.2.0-1.fc26.noarch [brain@x1carbon libsolv]$ fedora-cert -v Verifying Certificate cert expires: 2017-01-16 [brain@x1carbon libsolv]$ koji -d hello Traceback (most recent call last): File "/usr/bin/koji", line 7036, in <module> rv = locals()[command].__call__(options, session, args) File "/usr/bin/koji", line 6811, in handle_moshimoshi activate_session(session) File "/usr/bin/koji", line 6989, in activate_session session.ssl_login(options.cert, None, options.serverca, proxyuser=options.runas) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1929, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1979, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2124, in _callMethod self._close_connection() File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2055, in _close_connection self._connection[1].close() File "/usr/lib64/python2.7/httplib.py", line 848, in close sock.close() # close it manually... there may be other refs File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py", line 82, in close self.shutdown() File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py", line 53, in shutdown self.__dict__["conn"].shutdown() File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1503, in shutdown self._raise_ssl_error(self._ssl, result) File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1174, in _raise_ssl_error _raise_current_error() File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue raise exception_type(errors) OpenSSL.SSL.Error: [('SSL routines', 'SSL_shutdown', 'shutdown while in init')]
original backtrace is: File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2099, in _callMethod return self._sendCall(handler, headers, request) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2010, in _sendCall return self._sendOneCall(handler, headers, request) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2030, in _sendOneCall response = cnx.getresponse() File "/usr/lib64/python2.7/httplib.py", line 1136, in getresponse response.begin() File "/usr/lib64/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.7/httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib64/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/site-packages/koji/ssl/SSLConnection.py", line 140, in recv return con.recv(bufsize, flags) File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1304, in recv self._raise_ssl_error(self._ssl, result) File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1174, in _raise_ssl_error _raise_current_error() File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue raise exception_type(errors) Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert unexpected message')] Actually koji.fp.o:443 doesn't support SSLv3, but looks like koji client tries to use SSLv3....
The SSLv3 report is because the same message structure is used in TLS and SSL for the "alert" message. This latest occurrence is a result of some breakage in OpenSSL 1.1, and is tracked as bug #1393579. Unfortunately, Koji is hiding the internal SSL error because it tries to close the SSL socket, even if it never managed to connect (reconnect) fully. So perhaps it would be an idea to let the original error trickle up, as that would make it a lot easier to debug.
I have similar problem. Koji works with kerberros, but fedpkg not (do it by https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/JK267PSDD53I2KGONDLFA5D4JWYXKZTQ/): $ kinit -V hubbitus Using existing cache: persistent:1000:1000 Using principal: hubbitus Password for hubbitus: Authenticated to Kerberos v5 [pasha@hubbitus master]$ koji hello ciao, hubbitus! You are using the hub at https://koji.fedoraproject.org/kojihub Authenticated via GSSAPI [pasha@hubbitus master]$ fedpkg -v build Creating repo object from /home/pasha/SOFT/FEDORA/_my/qutim/master Initiating a koji session to https://koji.fedoraproject.org/kojihub Logging into https://koji.fedoraproject.org/kojihub with Kerberos authentication. Kerberos authentication is used, but you do not have a valid credential. Please use kinit to get credential with a principal that has realm Could not execute build: Could not login to https://koji.fedoraproject.org/kojihub Traceback (most recent call last): File "/usr/bin/fedpkg", line 16, in <module> main() File "/usr/lib/python2.7/site-packages/fedpkg/__main__.py", line 77, in main sys.exit(client.args.command()) File "/usr/lib/python2.7/site-packages/pyrpkg/cli.py", line 994, in build sets, nvr_check) File "/usr/lib/python2.7/site-packages/pyrpkg/__init__.py", line 1869, in build build_target = self.kojisession.getBuildTarget(self.target) File "/usr/lib/python2.7/site-packages/pyrpkg/__init__.py", line 217, in kojisession self.load_kojisession() File "/usr/lib/python2.7/site-packages/fedpkg/__init__.py", line 318, in load_kojisession return super(Commands, self).load_kojisession(anon) File "/usr/lib/python2.7/site-packages/pyrpkg/__init__.py", line 394, in load_kojisession self.login_koji_session(koji_config, self._kojisession) File "/usr/lib/python2.7/site-packages/pyrpkg/__init__.py", line 364, in login_koji_session raise rpkgError('Could not login to %s' % koji_config['server']) pyrpkg.errors.rpkgError: Could not login to https://koji.fedoraproject.org/kojihub
Pavel, this latest issue should be fixed with fedpkg-1.26 or higher. Since the original issue (koji with proxy) is now resolved due to our switch to Kerberos, I think this bug report can be closed. What do you think, Mike? For any issues authentication getting a Fedora Infrastructure kerberos ticket or for authenticating to koji.fedoraproject.org, the best place to request help with that would be submitting an issue at https://pagure.io/fedora-infrastructure/ .
+1 for closing , koji authentication is not the same already
ok, closing