Bug 1650883 - viewing logs from remote journals not working
Summary: viewing logs from remote journals not working
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: cockpit-session-recording
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Kirill Gliebov
QA Contact: Scott Poore
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-17 03:23 UTC by Scott Poore
Modified: 2019-06-14 00:52 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-14 00:52:56 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Scott Poore 2018-11-17 03:23:36 UTC
Description of problem:

The Cockpit Session Recording module is not showing sessions recording remote systemd journal logging.  With some review there are 3 minor issues that are blocking this feature from functioning:

The journalctl command by default does not display the remote journals on a host.  Session Recording module needs to use -m option with journalctl

The tlog user's UID may not be identical on all hosts.  the module needs to use other means of searching the journal like _EXE or _SYSLOG_IDENTIFIER. or similar so that sessions aren't missed where UID's don't match between systems.


Version-Release number of selected component (if applicable):
cockpit-session-recording-1-23.el8

How reproducible:
very

Steps to Reproduce:
On Server:
1. Setup cockpit and cockpit-session-recording
2. Setup systemd-journal-remote
On Client1:
3. Setup systemd-journal-upload to send to Server
4. Install tlog and add a user with /usr/bin/tlog-rec-session as shell
On Client2:
5. Setup systemd-journal-upload to send to Server
6. Install tlog and add a user with /usr/bin/tlog-rec-session as shell
On Server:
7. ssh localuser1@client1 and run some commands
8. ssh localuser1@client2 and run some commands
Open Browser to:
9. https://server:9090 and login as user with admin privs
10. go to Session Recording from left hand menu


Actual results:
No remote sessions were shown


Expected results:
Sessions for both client1 and client2 should be shown

Additional info:

Comment 1 Scott Poore 2018-11-17 03:40:02 UTC
FYI, marking this one as a test blocker because this was expected behavior that we want to function in RHEL8.0 GA.  Without this fixed I cannot test viewing of remote logs.

Also, Kirill already provided a partially fixed scratch build that appears to work.  I need to review more but, it looks good.

Comment 2 Kirill Gliebov 2018-11-21 11:11:42 UTC
I was trying to push to dist-git, but the push was rejected. I was using rhel-8.0.0 branch. It requires "TestBlocker" flag ACK, right?

Comment 4 Scott Poore 2018-12-12 17:19:10 UTC
Verified.

Version ::

cockpit-session-recording-1-29.el8.noarch

Results ::

rhel8-3 is setup with systemd-journal-remote

[root@rhel8-3 ~]# systemctl status systemd-journal-remote
● systemd-journal-remote.service - Journal Remote Sink Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journal-remote.service; indirect; vendor preset: di>
   Active: active (running) since Wed 2018-12-12 08:52:49 PST; 22min ago
     Docs: man:systemd-journal-remote(8)
           man:journal-remote.conf(5)
 Main PID: 1455 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 17972)
   Memory: 32.7M
   CGroup: /system.slice/systemd-journal-remote.service
           └─1455 /usr/lib/systemd/systemd-journal-remote --listen-https=-3 --output=/var/log/journal/>

Dec 12 08:52:49 rhel8-3.example.com systemd[1]: Started Journal Remote Sink Service.

rhel8-6 is setup with systemd-journal-upload

[root@rhel8-6 ~]# systemctl status systemd-journal-upload
● systemd-journal-upload.service - Journal Remote Upload Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journal-upload.service; disabled; vendor preset: di>
   Active: active (running) since Wed 2018-12-12 09:53:10 MST; 22min ago
     Docs: man:systemd-journal-upload(8)
 Main PID: 1417 (systemd-journal)
   Status: "Processing input..."
    Tasks: 1 (limit: 17972)
   Memory: 4.0M
   CGroup: /system.slice/systemd-journal-upload.service
           └─1417 /usr/lib/systemd/systemd-journal-upload --save-state

Dec 12 09:53:10 rhel8-6.example.com systemd[1]: Started Journal Remote Upload Service.

And is pointing to rhel8-3:

[root@rhel8-6 ~]# cat /etc/systemd/journal-upload.conf 
[Upload]
URL=https://rhel8-3.example.com:19532
ServerKeyFile=/etc/ssl/private/journal-upload.pem
ServerCertificateFile=/etc/ssl/certs/journal-upload.pem
TrustedCertificateFile=/etc/ssl/ca/trusted.pem

Recorded session made on rhel8-6 by ssh'ing as localuser1 to localhost.

Can see the session here:

[root@rhel8-6 ~]# journalctl _EXE=/usr/bin/tlog-rec-session --since 06:00
-- Logs begin at Fri 2018-11-16 16:08:46 MST, end at Wed 2018-12-12 10:06:36 MST. --
Dec 12 09:55:07 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:19 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:20 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:20 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:23 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:23 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:25 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:26 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:30 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:32 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:32 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:35 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:35 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:41 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 09:55:44 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">

Then I can also check remote recordings on rhel8-3:

[root@rhel8-3 ~]# journalctl -m _EXE=/usr/bin/tlog-rec-session --since 06:00
-- Logs begin at Fri 2018-11-09 12:42:11 PST, end at Wed 2018-12-12 09:18:01 PST. --
Dec 12 08:55:07 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:19 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:20 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:20 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:23 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:23 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:25 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:26 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:29 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:30 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:32 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:32 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:35 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:35 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:41 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">
Dec 12 08:55:44 rhel8-6.example.com -tlog-rec-session[1442]: {"ver":"2.2","host":"rhel8-6.example.com">

And when I login to https://rhel8-3.example.com:9090 from a browser and go to "Session Recordings" I can see and playback this recording.


Note You need to log in before you can comment on or make changes to this bug.