Bug 1288488 - [DOCS] Document s2i builder details [NEEDINFO]
[DOCS] Document s2i builder details
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation (Show other bugs)
3.0.0
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Gaurav Nelson
Wenjing Zheng
Vikram Goyal
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-04 06:40 EST by Thien-Thi Nguyen
Modified: 2018-05-08 20:30 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-02-18 18:25:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
gnelson: needinfo? (rcarvalh)


Attachments (Terms of Use)

  None (edit)
Description Thien-Thi Nguyen 2015-12-04 06:40:01 EST
What happens precisely during a Python s2i build?  How about for Ruby?  (etc)
Originally: https://github.com/openshift/openshift-docs/issues/1283
Comment 3 Thien-Thi Nguyen 2015-12-04 08:04:45 EST
In terms of source code, it looks like we want to look at:

- origin/pkg/generate/source/detector.go
- source-to-image/pkg/build/strategies/layered/layered.go

to describe how a particular "platform" is detected and how its constituent files (e.g., requirements.txt for Python) are processed.

This reference documentation should either be centralized in the main overview:

- https://docs.openshift.org/latest/using_images/s2i_images/index.html

or added to each image's overview, with the main overview describing the general detection / processing scheme.
Comment 4 Rodolfo Carvalho 2015-12-04 09:49:29 EST
@Thien-Thi, that's not what I had in mind originally.

The starting point is documenting that the Python image will automatically install project dependencies from a requirements.txt or setup.py.

It should be documented here: https://docs.openshift.org/latest/using_images/s2i_images/python.html

And the code you want to look at is here: https://github.com/openshift/sti-python/blob/master/3.4/s2i/bin/assemble

Next, look at the assemble scripts for other images and document what they are able to do, what input they consume and how.


This is not about detecting what platform to use. No Go code involved...
Comment 5 Thien-Thi Nguyen 2016-03-22 11:00:19 EDT
Hi Rodolfo,

I see Python-specific build details (i.e., `requirements.txt` and `setup.py`) are documented in:

 https://docs.openshift.com/enterprise/3.1/dev_guide/new_app.html#language-detection

but that there is no (direct) link from:

 https://docs.openshift.com/enterprise/3.1/using_images/s2i_images/python.html

to it.  So, how about we add one in the latter page's Overview?  I think the word "dependencies" would be the best place.  (I'd make an analogous change w/ the other languages.)  WDYT?
Comment 6 Rodolfo Carvalho 2016-03-22 11:52:52 EDT
It's a "coincidence" that language detection relies on files that I mentioned in https://github.com/openshift/openshift-docs/issues/1283.

The original point is to document, in a high level, what an s2i builder does.

On the OpenShift/Source-To-Image side, there's the git cloning of a repository, then the files are streamed into a container, without the .git dir, etc etc.
Then the assemble script is executed:

For Python, as an example:
https://github.com/openshift/sti-python/blob/master/2.7/s2i/bin/assemble

It will do several steps, rely on env vars, etc.

What I miss is documentation explaining how your code gets from a Git repository into a final image, and what else has happened as part of the build (download and install dependencies, etc).

Most of it is language/builder-image independent, while some is specific to each builder image, like each language has a different package management solution, a different way to list dependencies, and so on.

We can chat face-to-face tomorrow or any other time for further clarification.
Comment 7 Thien-Thi Nguyen 2016-04-04 11:31:57 EDT
Related: https://trello.com/c/WpdJNDz7
Comment 8 Rodolfo Carvalho 2016-04-04 12:10:27 EDT
@Thien-Thi the Trello link above refers to new/proposed functionality.
The relationship with this BZ is that documentation produced out of this BZ would require updates in the future with that card implemented, do you agree?

We may want to cross reference -- add a link to this BZ in that card. If you don't have access to that, I can add the link there pointing here.
Comment 9 Thien-Thi Nguyen 2016-04-04 13:19:18 EDT
Hi Rodolfo,

(In reply to Rodolfo Carvalho from comment #8)
> [...] new/proposed functionality.  The relationship with this BZ
> is that documentation produced out of this BZ would require
> updates in the future with that card implemented, do you agree?

Thanks for the clarification.  It sounds like you're saying that
card has no immediate relevance to this BZ, although in the
future, work that goes into this BZ will need to be modified once
the work for that card lands.

> We may want to [xref]

Good idea.  Done.

By the way, thanks for the interview the other day.  Here are my
notes from the session (additions/corrections welcome):

* explain flow:
*** input: code + builder image
*** builder image:
***** explicitly defined
***** auto-detected via ‘oc new-app’ (plan: link)
*** LANG-image provided assemble script capabilities
***** general flow
***** hooks for each LANG -- e.g., check requirements.txt (Python), etc
***** DOES NOT handle non-LANG-modules (cannot use yum, reason: not root)
*** user can provide assemble script
Comment 12 Gaurav Nelson 2018-01-29 20:37:37 EST
@Rodolfo

I think this is documented in the latest docs at: https://docs.openshift.com/container-platform/3.7/creating_images/s2i.html#build-process

I can surely add links to this from all s2i language docs. Can you please check?
Comment 13 Gaurav Nelson 2018-02-04 22:07:46 EST
Tracking changes in https://github.com/openshift/openshift-docs/pull/7564
Comment 15 Gaurav Nelson 2018-02-06 20:39:15 EST
I have added following section to all S2I language docs:

-------------
Build Process

S2I produces ready-to-run images by injecting source code into a container and letting the container prepare that source code for execution. It performs the following steps:

    Starts a container from the builder image.

    Downloads the application source.

    Streams the scripts and application sources into the builder image container.

    Runs the assemble script (from the builder image).

    Saves the final image.

See S2I Build Process for complete overview of the build process.

-------------

Here is the link to a rendered version doc for .Net core: https://github.com/gaurav-nelson/openshift-docs/blob/3590f20012f9fb96023650c0e8a5e90f0d68bc92/using_images/s2i_images/dot_net_core.adoc#build-process

@Wenjing Zheng
Can you please review?
Comment 16 Wenjing Zheng 2018-02-07 01:04:07 EST
The doc looks fine to me, will verify when pr is merged. Thanks!
Comment 17 openshift-github-bot 2018-02-11 06:04:19 EST
Commit pushed to master at https://github.com/openshift/openshift-docs

https://github.com/openshift/openshift-docs/commit/4dc0595534e2275bdb74eb7a794195e3f9bc1645
Merge pull request #7564 from gaurav-nelson/bug1288488-fixes

Bug#1288488 - fixes - added build process section for all s2i images
Comment 18 Gaurav Nelson 2018-02-11 18:58:59 EST
@Wenjing Zheng
I have merged it into master. 
Can you please review?
Comment 19 Wenjing Zheng 2018-02-12 04:08:33 EST
Thanks, verify now!
Comment 20 Gaurav Nelson 2018-02-18 18:25:45 EST
Changes are live at:
https://docs.openshift.com/container-platform/3.7/using_images/s2i_images/

Note You need to log in before you can comment on or make changes to this bug.