Bug 1383903

More info from Eric in this GH issue:

https://github.com/openshift/openshift-docs/pull/3081

(In reply to Vikram Goyal from comment #0)
> There are two tasks in this bug.
> 
> 1. In OCP 3.4, there is a new logging stack with updates to all the
> components to the latest supported versions.
> 
> Working with the devs, document what is required for the end user to upgrade
> to the latest supported versions. There might be a migration plan for
> customers for 1.0 --> 2.0 logging that will also need documenting.
> 
> Eng Trello Card:
> https://trello.com/c/hksMQXcp/175-8-upgrade-logging-stack-logging-epic-ois-
> td-cda which links of the R&D card:
> https://trello.com/c/8ekpZH5U/137-3-r-d-upgrade-logging-stack

Action items:
- Update manual EFK section, taking into consideration with advice from Trello cards. Link to relevant section: https://docs.openshift.com/container-platform/3.3/install_config/upgrading/manual_upgrades.html#manual-upgrading-efk-logging-stack
- Check for any considerations for automatic upgrades based on eric's advice in comment #1
- Test manual and automatic logging stack upgrades

> 2. Document how to use systemd journal support
> 
> Need to document the fluentd and deployer parameters use-journal,
> journal-source, and journal-read-from-head as described in
> https://github.com/openshift/origin-aggregated-logging/tree/master/
> deployer#create-deployer-configmap and
> https://github.com/openshift/origin-aggregated-logging/tree/master/
> deployer#have-fluentd-use-the-systemd-journal-as-the-log-source
> 
> Eng Trello Card:
> https://trello.com/c/4qi8GV2M/138-5-update-log-driver-to-journald-epic-ois-
> cda-logging-cda

Action items:
- Update Fluentd parameters table: https://docs.openshift.com/container-platform/3.3/install_config/aggregate_logging.html#aggregated-fluentd

@vikram and @eric - I've assessed this BZ and put together a set of AIs. Were there any other AIs required?

The only other action item I can think of is ensuring capture changes regarding the Common Data Model [1][2][3][4].

Please note that the index patterns described in the card details for [3][4] are incorrect, the correct pattern is described in [1].


Other than that, there is the Secure Forward feature and a PR for that has been merged in and tagged for 3.4 already.



[1] https://trello.com/c/Zr5G36Iy/348-3-es-plugin-changes-for-common-data-model-index-naming-logging-epic-ois-cda-cda

[2] https://trello.com/c/COsc7nrk/344-5-add-index-templates-to-elasticsearch-logging-epic-ois-cda-cda

[3] https://trello.com/c/epgJs4Zd/350-2-update-curator-for-common-data-model-index-naming-logging-epic-ois-cda-cda

[4] https://trello.com/c/PZgXm4oT/347-5-fluentd-changes-for-common-data-model-and-index-naming-epic-ois-cda-cda

(In reply to Dan Macpherson from comment #3)
> (In reply to Vikram Goyal from comment #0)
> > There are two tasks in this bug.
> > 
> > 1. In OCP 3.4, there is a new logging stack with updates to all the
> > components to the latest supported versions.
> > 
> > Working with the devs, document what is required for the end user to upgrade
> > to the latest supported versions. There might be a migration plan for
> > customers for 1.0 --> 2.0 logging that will also need documenting.
> > 
> > Eng Trello Card:
> > https://trello.com/c/hksMQXcp/175-8-upgrade-logging-stack-logging-epic-ois-
> > td-cda which links of the R&D card:
> > https://trello.com/c/8ekpZH5U/137-3-r-d-upgrade-logging-stack
> 
> Action items:
> - Update manual EFK section, taking into consideration with advice from
> Trello cards. Link to relevant section:
> https://docs.openshift.com/container-platform/3.3/install_config/upgrading/
> manual_upgrades.html#manual-upgrading-efk-logging-stack
> - Check for any considerations for automatic upgrades based on eric's advice
> in comment #1
> - Test manual and automatic logging stack upgrades
> 
> > 2. Document how to use systemd journal support
> > 
> > Need to document the fluentd and deployer parameters use-journal,
> > journal-source, and journal-read-from-head as described in
> > https://github.com/openshift/origin-aggregated-logging/tree/master/
> > deployer#create-deployer-configmap and
> > https://github.com/openshift/origin-aggregated-logging/tree/master/
> > deployer#have-fluentd-use-the-systemd-journal-as-the-log-source
> > 
> > Eng Trello Card:
> > https://trello.com/c/4qi8GV2M/138-5-update-log-driver-to-journald-epic-ois-
> > cda-logging-cda
> 
> Action items:
> - Update Fluentd parameters table:
> https://docs.openshift.com/container-platform/3.3/install_config/
> aggregate_logging.html#aggregated-fluentd
> 
> @vikram and @eric - I've assessed this BZ and put together a set of AIs.
> Were there any other AIs required?

Apart from what Eric has added this sounds really good to me. Thanks!

Progress update:

1. I tried updating the EFK stack using the MODE=upgrade method, which is pretty much the the same as the previous upgrade method. I upgraded the logging stack from Enterprise 3.3 to the latest origin. However, when I try to log into Kibana now, I get an "Unauthorized" message. Here's the log from the kibana-proxy:

10.1.0.1 - - [10/Nov/2016:19:12:42 +0000] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0"
10.1.0.1 - - [10/Nov/2016:19:12:52 +0000] "GET /auth/openshift/callback?error=access_denied&error_description=scope+denied%3A+user%3Afull HTTP/1.1" 401 - "https://openshift.danssat.net:8443/login?then=%2Foauth%2Fauthorize%3Fresponse_type%3Dcode%26redirect_uri%3Dhttps%253A%252F%252Fkibana.apps.danssat.net%252Fauth%252Fopenshift%252Fcallback%26client_id%3Dkibana-proxy" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0"

Not sure if it's something I've done wrong or if there's a step missing in the upgrade procedure. Eric, any chance you can provide some advice on what I should look for?

2. Regarding the fluentd deployer parameters (use-journal, journal-source, and journal-read-from-head), it seems these have already been added for 3.3:

https://docs.openshift.com/container-platform/3.3/install_config/aggregate_logging.html#aggregated-fluentd

@Vikram, was there anything further required for this content.

3. It seems like the main documentation required is two notes:

- A note on that you can't rollback from Elasticsearch 2.x to Elasticsearch 1.x.
- A note about the new index patterns to accommodate the Common Data Model

(In reply to Dan Macpherson from comment #6)
> Progress update:
> 
> 1. I tried updating the EFK stack using the MODE=upgrade method, which is
> pretty much the the same as the previous upgrade method. I upgraded the
> logging stack from Enterprise 3.3 to the latest origin. However, when I try

Just to confirm, is this using the same image prefix or different? Currently the deployer doesn't handle changing the image prefix for the images as part of upgrade. We only patch the version portion of our deployment object images.

> to log into Kibana now, I get an "Unauthorized" message. Here's the log from
> the kibana-proxy:
> 
> 10.1.0.1 - - [10/Nov/2016:19:12:42 +0000] "GET / HTTP/1.1" 302 0 "-"
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101
> Firefox/49.0"
> 10.1.0.1 - - [10/Nov/2016:19:12:52 +0000] "GET
> /auth/openshift/
> callback?error=access_denied&error_description=scope+denied%3A+user%3Afull
> HTTP/1.1" 401 -
> "https://openshift.danssat.net:8443/
> login?then=%2Foauth%2Fauthorize%3Fresponse_type%3Dcode%26redirect_uri%3Dhttps
> %253A%252F%252Fkibana.apps.danssat.
> net%252Fauth%252Fopenshift%252Fcallback%26client_id%3Dkibana-proxy"
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101
> Firefox/49.0"

I don't think I've seen that one before. Can you check if oauthclient/kibana-proxy has a scopeRestriction defined? Are there any errors in the deployer logs from MODE=upgrade related to being unable to (re)create any support objects?

> 
> Not sure if it's something I've done wrong or if there's a step missing in
> the upgrade procedure. Eric, any chance you can provide some advice on what
> I should look for?
> 
> 2. Regarding the fluentd deployer parameters (use-journal, journal-source,
> and journal-read-from-head), it seems these have already been added for 3.3:
> 
> https://docs.openshift.com/container-platform/3.3/install_config/
> aggregate_logging.html#aggregated-fluentd
> 
> @Vikram, was there anything further required for this content.
> 
> 3. It seems like the main documentation required is two notes:
> 
> - A note on that you can't rollback from Elasticsearch 2.x to Elasticsearch
> 1.x.
> - A note about the new index patterns to accommodate the Common Data Model

(In reply to Dan Macpherson from comment #6)
> 
> 2. Regarding the fluentd deployer parameters (use-journal, journal-source,
> and journal-read-from-head), it seems these have already been added for 3.3:
> 
> https://docs.openshift.com/container-platform/3.3/install_config/
> aggregate_logging.html#aggregated-fluentd
> 
> @Vikram, was there anything further required for this content.

If these have been documented already with examples, then that seems fair enough. Nothing more required.

> 
> 3. It seems like the main documentation required is two notes:
> 
> - A note on that you can't rollback from Elasticsearch 2.x to Elasticsearch
> 1.x.
> - A note about the new index patterns to accommodate the Common Data Model

Have created a PR:

https://github.com/openshift/openshift-docs/pull/3211

@eric -- Anything I should add to this?

Thanks @Dan, I'll review and make any comments on the PR.

This is now live but has been moved to the release notes:

https://docs.openshift.com/container-platform/3.4/release_notes/ocp_3_4_release_notes.html#ocp-34-notable-technical-changes