Bug 1725437

Summary: dirmngr endlessly receives service unavailable 503 error
Product: [Fedora] Fedora Reporter: Stan King <stanley.king>
Component: gnupg2Assignee: Red Hat Crypto Team <crypto-team>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 29CC: bcl, crypto-team, tmraz, tmz
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: gnupg2-2.2.17-1.fc30 gnupg2-2.2.17-1.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-18 17:56:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
First 500 lines of dirmngr.log, debug level set to advanced. none

Description Stan King 2019-06-30 15:56:35 UTC
Created attachment 1586026 [details]
First 500 lines of dirmngr.log, debug level set to advanced.

Description of problem:

I observed an unusual amount of network traffic, and traced it to a dirmngr process.  It's receiving a 503 service unavailable error, and endlessly retries the operation, multiple times per second.

I couldn't find fixes related to this in the release notes at https://gnupg.org/download/release_notes.html, nor did I find any similar bug reports on the Redhat Bugzilla.

Version-Release number of selected component (if applicable):

gnupg2-2.2.13-1.fc29.x86_64

How reproducible:

I'm not sure what triggers the execution of dirmngr.  The system was unattended when it happened this morning.  However, when it's running, it's generally affected by this.


Steps to Reproduce:
1. wait until dirmngr starts running
2.
3.

Actual results:
Network traffic of about 10KB/second down, 5KB/second up, attempting to connect to https://keys.fedoraproject.org:443/ but receiving error 503 service unavailable each time.

Expected results:
I'm unfamiliar with the function of dirmngr, but presumably this will keep it from doing its regular job.


Additional info:

I'm attaching the first 500 lines from dirmngr.log, with debug level set to advanced.

Comment 1 Tomas Mraz 2019-07-01 08:00:04 UTC
Do you have the keys.fedoraproject.org specified in .gnupg/dirmngr.conf?

In general I would recommend filling a bug in the upstream issue tracker at https://dev.gnupg.org/ as that would speed up the investigation and fix. Unless I am able to reproduce the problem locally I would be just a messenger.

Comment 2 Stan King 2019-07-01 11:18:21 UTC
Tomas,

I do not specify keys in .gnupg/dirmngr.conf.  It's using the default value.

keys.fedoraproject.org is resolving to CNAME keys01.fedoraproject.org., A 140.211.169.207, which is the host returning error 503.  The whois command reports this IP range belongs to Network for Education and Research in Oregon (NERO), Eugene, OR.  Contact information is provided there also.

Aside from that element of the fedoraproject.org infrastructure providing an error code, the second problem is that dirmngr immediately retrying the operation that's doomed to fail, rather than delaying or trying another keys host.  The CNAME of keys01.fedoraproject.org makes me think there was an intention to also have keys02, keys03, etc. and rotate among them as is done with pool.ntp.org, but that was not done.  keys02.fedoraproject.org resolves to another IP within the same range as keys01.  There is no keys03.fedoraproject.org.  It seems strange that there's a dependency on a single IP range within the state of Oregon, given the international user base of Fedora.

My logfile indicates that it's receiving commands from an instance of /usr/bin/gpg2 that seems to have been started by thunderbird, in the middle of the night.

It is possible to run it manually from the command line.  Just type "dirmngr", and after producing some output ending in "OK Dirmngr 2.2.13 at your service", it will await commands.  The two commands you need to reproduce the behavior are

KEYSERVER --clear hkps://keys.fedoraproject.org:443

and

KS_GET -- 0x8657980D9AB51E50

after which you should see the effects of its retry loop.

Comment 3 Tomas Mraz 2019-07-01 16:20:47 UTC
OK, I've created the upstream issue:

https://dev.gnupg.org/T4600

Comment 4 Stan King 2019-07-02 00:14:50 UTC
Thank you, Tomas.  I'm sure those who use dirmngr will appreciate it.

Will you let the server continue to generate 503 until the software is fixed?  Otherwise, how will we know if the problem is repaired?  Do you have a test server than can provide error 503 on demand?

Again, thank you for reporting this problem upstream.  I'm just looking forward to verifying it when it comes downstream.

Comment 5 Tomas Mraz 2019-07-02 07:28:10 UTC
Unfortunately I do not manage the broken keyserver.

I suppose the gnupg upstream will test the fix appropriately.

Comment 6 Todd Zullinger 2019-07-13 16:36:37 UTC
It looks like the dirmngr issue was fixed in the recently released gnupg-2.2.17.

As far as keys.fedoraproject.org, it seems likely that the service will be discontinued, so anyone who has configured it in either gpg.conf or dirmngr.conf will need to adjust their configurations.

References:
https://pagure.io/fedora-infrastructure/issue/7988 (keys.fp.o is down?)
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/G4WTK5J6VU6HVOSCNSVKL54GYU7IKZNE/ (Retire keys.fedoraproject.org?)

Comment 7 Fedora Update System 2019-07-15 10:19:56 UTC
FEDORA-2019-28a3675529 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-28a3675529

Comment 8 Fedora Update System 2019-07-15 10:20:00 UTC
FEDORA-2019-2f259a6c0a has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2f259a6c0a

Comment 9 Fedora Update System 2019-07-16 00:54:12 UTC
gnupg2-2.2.17-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2f259a6c0a

Comment 10 Fedora Update System 2019-07-16 02:35:37 UTC
gnupg2-2.2.17-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-28a3675529

Comment 11 Stan King 2019-07-16 02:37:22 UTC
Unfortunately, I don't have an arrangement that can generate a 503 error to test this properly.

Comment 12 Fedora Update System 2019-07-18 17:56:11 UTC
gnupg2-2.2.17-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2019-08-06 01:55:09 UTC
gnupg2-2.2.17-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.