Bug 2144014

Summary: f37 clevis send invalid json to f36 tang
Product: [Fedora] Fedora Reporter: Barry Scott <barry>
Component: clevisAssignee: Sergio Correia <scorreia>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 37CC: amulhern, fmartine, mzeleny, npmccallum, rsroka, sarroutb, scorreia
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-08 03:02:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Barry Scott 2022-11-18 16:56:59 UTC
Description of problem:

tang reports invalid JSON when receiving request from a F37 system using clevis.
I see this on 2 systems, one server one kde plasma desktop.

Here are the logs from journalctl:

Started tangd@7-<ip-addr>:2269-<ip-addr>:39476.service - Tang Server (<ip-addr>:39476).
<ip-addr> POST /rec/<long-randon-string> JSON file (/var/db/tang/blah-sig-2022-05-29.jwk); skipping
Invalid JSON file (/var/db/tang/blah-exc-2022-05-29.jwk); skipping
 => 404 (../src/tangd.c:127)

Version-Release number of selected component (if applicable):

On keyserver:

tang-11-3.fc36.aarch64

On server/kde:

clevis-18-12.fc37.x86_64

How reproducible:

I'm guessing this will work:

Steps to Reproduce:
1. Run tang on f36
2. setup clevis on a f36 to unlocka LUKS volume
3. upgrade to f37

Actual results:

disk does not unlock
tang reports invalid JSON

Expected results:

disk unlocks

Additional info:

Comment 1 Sergio Correia 2022-11-18 22:24:31 UTC
Hello, I am still trying to reproduce this, but those invalid JSON messages look suspicious; could please run something like this in the server, to check whether the files are valid JSON files?

# jose fmt --json <full file path>

Something like this, if the paths in your example are correct:

# jose fmt --json /var/db/tang/blah-sig-2022-05-29.jwk
# jose fmt --json /var/db/tang/blah-exc-2022-05-29.jwk

(additionally, feel free to run this using any other files you find under /var/db/tang/)

You should see no output, if the files are valid JSON files, which is what we expect here; in case they fail to validate, you should see "Invalid json!" followed by some "jose fmt" usage information.

Comment 2 Barry Scott 2022-11-20 18:11:27 UTC
jose fmt --json does not report any issue for any of the files in /var/db/tang.

I use tcpdump to capture the POST request on and then use jose fmt --json to
parse the on-the-wire message. It also report no issue.

I think I am going to need to build tangd and add in debug support to see the exact
bytes that are being processed in the code. And if that is not helping I think I'll
have to add debug code to the json library.

I'm running tang on a RPi 4 and its on port 2269.

Comment 3 Barry Scott 2022-11-23 16:08:09 UTC
I added a patch to print the error details in keys.c from the call to json_load_file.

        if (strcmp(dot, pattern) == 0) {
            json_error_t error;

            /* Found a file with .jwk extension. */
            if (snprintf(filepath, PATH_MAX, "%s/%s", jwkdir, d->d_name) < 0) {
                fprintf(stderr, "Unable to prepare variable with file full path (%s); skipping\n", d->d_name);
                continue;
            }
            filepath[sizeof(filepath) - 1] = '\0';
            json_auto_t* json = json_load_file(filepath, 0, &error);
            if (!json) {
                enum json_error_code code = json_error_code(&error);
                fprintf(stderr, "Invalid JSON file (%s); skipping\n", filepath);
                fprintf(stderr, "code %d text %s line %d col %d pos %d",
                    code, error.text, error.line, error.column, error.position);
                continue;
            }

Then I see this in the journal (qqq replaces possibly sensitive info):

Nov 23 16:00:35 clef.chelsea.private tangd[12406]: 172.16.2.175 POST /rec/qqqq Invalid JSON file (/var/db/tang/qqq.jwk); skipping
Nov 23 16:00:35 clef.chelsea.private tangd[12406]: code 3 text unable to open /var/db/tang/qqq.jwk: Permission denied line -1 col -1 pos 0Invalid JSON file (/var/db/tang/qqq.jwk); skipping

So... I know what to fix now. I have no idea why the permissions changed on the upgrade.

I recommend adding the error info to tang.
Do you want a PR for this? Where should I do that? I think you are upstream maintainer.

Barry

Comment 4 Sergio Correia 2022-11-23 21:52:42 UTC
(In reply to Barry Scott from comment #3)

[...]

> So... I know what to fix now. I have no idea why the permissions changed on
> the upgrade.
> 

Thanks for the investigation, but what was this upgrade again? Were you upgrading from tang-10?

> I recommend adding the error info to tang.
> Do you want a PR for this? Where should I do that? I think you are upstream
> maintainer.
> 

Sure, if you can submit a PR at [1] it would much be appreciated. It would be very helpful to have a more specific error message for this case.

[1] https://github.com/latchset/tang

Comment 5 Barry Scott 2022-11-24 10:31:31 UTC
(In reply to Sergio Correia from comment #4)

> Thanks for the investigation, but what was this upgrade again? Were you
> upgrading from tang-10?

# dnf history info tang | grep tang
Command Line   : reinstall ./tang-11-4.fc37.aarch64.rpm
    Reinstall   tang-11-4.fc37.aarch64 @@commandline
    Reinstalled tang-11-4.fc37.aarch64 @@System
    Upgrade    tang-11-4.fc37.aarch64                                      @fedora
    Upgraded   tang-11-3.fc36.aarch64                                      @@System
Command Line   : update tang
    Upgrade  tang-11-3.fc36.aarch64 @updates
    Upgraded tang-11-2.fc36.aarch64 @@System
Command Line   : downgrade tang
    Downgrade  tang-11-2.fc36.aarch64 @fedora
    Downgraded tang-11-3.fc36.aarch64 @@System
    Upgrade       tang-11-3.fc36.aarch64                                      @updates
    Upgraded      tang-11-2.fc36.aarch64                                      @@System
Command Line   : install tang
    Install tang-11-2.fc36.aarch64           @fedora

I'm using an RPi4 as my key server. Its not full disk encrypted so I added a
use a small LUKS partition to hold the keys.

# lsblk --fs
NAME          FSTYPE      FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
mmcblk0
├─mmcblk0p1   vfat        FAT32       4F15-340F                             568.2M     5% /boot/efi
├─mmcblk0p2   ext4        1.0         ddaaada4-49ce-4a70-bf1f-19f5af8ced90  772.9M    14% /boot
├─mmcblk0p3   ext4        1.0         0cf3008c-e4cd-461e-a3d4-b882a513fba7    1.5G    59% /
├─mmcblk0p4
└─mmcblk0p5   crypto_LUKS 2           fbba9e51-bed1-450f-a991-6e9efc403d5b
  └─key-vault ext3        1.0         da052fce-5f54-4ea7-b5fa-9c14a54c112b     69M     0% /var/lib/key-vault
zram0                                                                                     [SWAP]
# ls -l /var/db/tang
lrwxrwxrwx. 1 tang tang 23 Sep  8 12:38 /var/db/tang -> /var/lib/key-vault/tang

The fix was to chown the exc and sig from root:root to tang:tang.

The upgrade to f37 sequence was:
1. upgrade f36 to latest
2. test tang working
3. system-upgrade to f37
4. test tang working
5. report this issue

> Sure, if you can submit a PR at [1] it would much be appreciated. It would
> be very helpful to have a more specific error message for this case.
> 
> [1] https://github.com/latchset/tang

Will do.

Comment 6 Barry Scott 2022-11-24 10:52:07 UTC
https://github.com/latchset/tang/pull/103

Comment 7 Fedora Update System 2022-12-07 11:50:08 UTC
FEDORA-2022-de24665e71 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-de24665e71

Comment 8 Fedora Update System 2022-12-07 11:50:35 UTC
FEDORA-2022-3c3a91e691 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3c3a91e691

Comment 9 Fedora Update System 2022-12-07 11:50:56 UTC
FEDORA-2022-92367a781f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-92367a781f

Comment 10 Fedora Update System 2022-12-08 02:10:19 UTC
FEDORA-2022-de24665e71 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-de24665e71`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-de24665e71

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2022-12-08 02:53:31 UTC
FEDORA-2022-3c3a91e691 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-3c3a91e691`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3c3a91e691

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2022-12-08 03:02:32 UTC
FEDORA-2022-92367a781f has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 Fedora Update System 2022-12-16 01:42:37 UTC
FEDORA-2022-3c3a91e691 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2022-12-16 01:56:26 UTC
FEDORA-2022-de24665e71 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.