| Summary: | lxml automated library installation fails | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Software Collections | Reporter: | tamaghei | ||||||||
| Component: | rh-python35-container | Assignee: | Petr Viktorin <pviktori> | ||||||||
| Status: | CLOSED NOTABUG | QA Contact: | BaseOS QE - Apps <qe-baseos-apps> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | unspecified | CC: | aos-bugs, bparees, hhorak, tamaghei | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2016-11-02 17:50:58 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Attachments: |
|
||||||||||
|
Description
tamaghei
2016-10-31 14:35:55 UTC
Created attachment 1215833 [details]
Failure on build logs
I'm trying to reproduce this way: #> docker run -ti --rm centos/python-35-centos7 bash bash-4.2$ pip install --user 'lxml==3.6.4' and it works.. Would it be possible to provide better reproducer, please? Hi Horak, I created a minimal environment that causes the error: I created a new repo with just a requirement.txt containing lxml==3.6.4, you can have a look at the content here: https://github.com/Zerrossetto/reproducing-os3.3-lxml-bug that has been deployed in a clean os3.3 project with just the python3.5 basic container. Project id is "lxml-bug" and it's placed under the tamaghei account. I attached new logs to this bug, but what I can notice from them in respect of your reproduction attempt is that the container is rhscl/python-35-rhel7 and not centos/python-35-centos7, I'll also have a look if i can manage to run it on my local machine. The result of my new attempt is again attached to this bug report, in the meantime thank you for your help. Created attachment 1216063 [details]
New attempt in a reduced environment created for issue reproduction
I don't have os3.3 set up myself, but what I tried as well was: #> s2i build git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git registry.access.redhat.com/rhscl/python-35-rhel7:latest and it worked as well. Ben, any idea where we should look at? Can you provide the full output from your build log?
I'm also not able to recreate, here are my logs from building on a 3.3+ system using centos/python-35-centos7:
$ oc new-app docker.io/centos/python-35-centos7:latest~git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git
--> Found Docker image 358d162 (10 days old) from docker.io for
"docker.io/centos/python-35-centos7:latest"
Python 3.5
----------
Platform for building and running Python 3.5 applications
Tags: builder, python, python35, rh-python35
* An image stream will be created as "python-35-centos7:latest" that will
track the source image
* A source build using source code from
git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git will be created
* The resulting image will be pushed to image stream
"reproducing-os33-lxml-bug:latest"
* Every time "python-35-centos7:latest" changes a new build will be
triggered
* This image will be deployed in deployment config
"reproducing-os33-lxml-bug"
* Port 8080/tcp will be load balanced by service "reproducing-os33-lxml-bug"
* Other containers can access this service through the hostname
"reproducing-os33-lxml-bug"
--> Creating resources ...
imagestream "python-35-centos7" created
imagestream "reproducing-os33-lxml-bug" created
buildconfig "reproducing-os33-lxml-bug" created
deploymentconfig "reproducing-os33-lxml-bug" created
service "reproducing-os33-lxml-bug" created
--> Success
Build scheduled, use 'oc logs -f bc/reproducing-os33-lxml-bug' to track its
progress.
Run 'oc status' to view your app.
sample-app (master)$ oc logs build/reproducing-os33-lxml-bug-1
Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ...
Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt)
Author: Zerrossetto <tamaghei>
Date: Tue Nov 1 11:10:57 2016 +0100
sample-app (master)$ oc logs build/reproducing-os33-lxml-bug-1 -f
Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ...
Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt)
Author: Zerrossetto <tamaghei>
Date: Tue Nov 1 11:10:57 2016 +0100
---> Installing application source ...
---> Installing dependencies ...
You are using pip version 7.1.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting lxml==3.6.4 (from -r requirements.txt (line 1))
Downloading lxml-3.6.4.tar.gz (3.7MB)
Installing collected packages: lxml
Running setup.py install for lxml
Successfully installed lxml
Pushing image 172.30.213.111:5000/test/reproducing-os33-lxml-bug:latest ...
Pushed 0/9 layers, 10% complete
Pushed 1/9 layers, 13% complete
Pushed 2/9 layers, 24% complete
Pushed 3/9 layers, 35% complete
Pushed 4/9 layers, 46% complete
Pushed 5/9 layers, 58% complete
Pushed 6/9 layers, 68% complete
Pushed 6/9 layers, 90% complete
Pushed 7/9 layers, 95% complete
Pushed 8/9 layers, 99% complete
Pushed 9/9 layers, 100% complete
Push successful
What I managed to find out until now is: 1. Launching the command $ s2i build git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git docker.io/centos/python-35-centos7:latest reproducing-os3.3-lxml-bug on my local workstation works as expected, and a valid docker image is generated and pulled on my local docker instance. 2. Doing your command sequence on a brand new project via CLI produces the error reported initially. I also managed to find a StackOverflow entry which addresses possible insufficient memory errors at compile time (ref. http://stackoverflow.com/questions/37128813/installing-lxml-error-open-cfi-at-the-end-of-file-missing-cfi-endproc-direct), so the hypothesis could be that the remote build process fails due to insufficient memory. Anyway I currently am unable to reproduce this locally, and also this doesn't explain why Ben could build it on his project and I'm not. To me it sounds like something at a more lower level than what I could operate on. It follows the command sequence and related command outputs. $ oc delete project lxml-bug project "lxml-bug" deleted $ oc new-project lxml-bug --description="Sample project for bug report 1390236" --display-name="Sample project for S2I bug report" Already on project "lxml-bug" on server "https://api.preview.openshift.com:443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git to build a new example application in Ruby. $ oc new-app docker.io/centos/python-35-centos7:latest~git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git --> Found Docker image 358d162 (10 days old) from docker.io for "docker.io/centos/python-35-centos7:latest" Python 3.5 ---------- Platform for building and running Python 3.5 applications Tags: builder, python, python35, rh-python35 * An image stream will be created as "python-35-centos7:latest" that will track the source image * A source build using source code from git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git will be created * The resulting image will be pushed to image stream "reproducing-os33-lxml-bug:latest" * Every time "python-35-centos7:latest" changes a new build will be triggered * This image will be deployed in deployment config "reproducing-os33-lxml-bug" * Port 8080/tcp will be load balanced by service "reproducing-os33-lxml-bug" * Other containers can access this service through the hostname "reproducing-os33-lxml-bug" --> Creating resources with label app=reproducing-os33-lxml-bug ... imagestream "python-35-centos7" created imagestream "reproducing-os33-lxml-bug" created buildconfig "reproducing-os33-lxml-bug" created deploymentconfig "reproducing-os33-lxml-bug" created service "reproducing-os33-lxml-bu" created --> Success Build scheduled, use 'oc logs -f bc/reproducing-os33-lxml-bug' to track its progress. Run 'oc status' to view your app. $ oc logs -f bc/reproducing-os33-lxml-bug Pulling image "docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049" ... Pulling image "docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049" ... Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ... Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt) Author: Zerrossetto <tamaghei> Date: Tue Nov 1 11:10:57 2016 +0100 ---> Installing application source ... ---> Installing dependencies ... You are using pip version 7.1.0, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting lxml==3.6.4 (from -r requirements.txt (line 1)) Downloading lxml-3.6.4.tar.gz (3.7MB) Installing collected packages: lxml Running setup.py install for lxml Complete output from command /opt/rh/rh-python35/root/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-z8jahi3d/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-qsem1cus-record/install-record.txt --single-version-externally-managed --compile --user --prefix=: Building lxml version 3.6.4. Building without Cython. Using build configuration of libxslt 1.1.28 Building against libxml2/libxslt in the following directory: /usr/lib64 running install running build running build_py creating build creating build/lib.linux-x86_64-3.5 creating build/lib.linux-x86_64-3.5/lxml copying src/lxml/builder.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/sax.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/__init__.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.5/lxml creating build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-3.5/lxml/includes creating build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.5/lxml/html creating build/lib.linux-x86_64-3.5/lxml/isoschematron copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.5/lxml/isoschematron copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.5/lxml/includes creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 running build_ext building 'lxml.etree' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/src creating build/temp.linux-x86_64-3.5/src/lxml gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python35/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/opt/rh/rh-python35/root/usr/include/python3.5m -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -w {standard input}: Assembler messages: {standard input}:522106: Error: number of operands mismatch for `lea' {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive gcc: internal compiler error: Killed (program cc1) Please submit a full bug report, with preprocessed source if appropriate. See <http://bugzilla.redhat.com/bugzilla> for instructions. Compile failed: command 'gcc' failed with exit status 4 creating tmp cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInit0t0kl9ig.c -o tmp/xmlXPathInit0t0kl9ig.o cc tmp/xmlXPathInit0t0kl9ig.o -L/usr/lib64 -lxml2 -o a.out error: command 'gcc' failed with exit status 4 ---------------------------------------- Command "/opt/rh/rh-python35/root/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-z8jahi3d/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-qsem1cus-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-z8jahi3d/lxml error: build error: non-zero (13) exit code from docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049 How much memory does your node host have? Does your project have any default memory limits being applied to the build pods? https://docs.openshift.org/latest/dev_guide/builds.html#build-resources These are my project quotas, and I'm not setting any kind of limit in the BuildConfig, which is Openshift default by-the-book because currently I'm using and empty project with just a requirements.txt inside with lxml==3.6.4 inside. I'll dig in more with the investigations. $ oc get quota -n lxml-bug NAME AGE compute-resources 5h compute-resources-timebound 5h object-counts 5h $ oc describe quota compute-resources -n lxml-bug Name: compute-resources Namespace: lxml-bug Scopes: NotTerminating * Matches all pods that do not have an active deadline. Resource Used Hard -------- ---- ---- limits.cpu 0 4 limits.memory 0 2Gi $ oc describe quota compute-resources-timebound -n lxml-bug Name: compute-resources-timebound Namespace: lxml-bug Scopes: Terminating * Matches all pods that have an active deadline. Resource Used Hard -------- ---- ---- limits.cpu 0 3 limits.memory 0 1536Mi $ oc describe quota object-counts -n lxml-bug Name: object-counts Namespace: lxml-bug Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 2 replicationcontrollers 0 50 secrets 9 20 services 1 10 Hm, those are quotas but not limits. Does your project have a LimitRange associated with it? You might also be able to tell by dong an "oc get pod <build-pod-name> -o yaml" and see what resource limits are assigned to the pod. I finally managed to build lxml. The issue was indeed that, without any setting in the "resources" section of the related BuildConfig the resulting run-once pod came out with these settings:
resources:
limits:
cpu: '1'
memory: 512Mi
requests:
cpu: 60m
memory: 307Mi
the resolution came when I updated the BuildConfig configuration with:
resources:
limits:
memory: 1Gi
which overrided the previews defaults and led me to this scenario
CPU: 120 millicores to 2 cores
Memory: 614 MiB to 1 GiB
that was sufficient to let the job end. I personally think that having a limit higher than a default it's a bit strange.
Thank you for your support.
Great, glad we were able to sort it out. |