Hide Forgot
Description of problem: I'm trying to port on Openshift3 an application that now's running on Heroku. The build log for the application fails with the error: > Assembler messages: > Error: open CFI at the end of file; missing .cfi_endproc directive Version-Release number of selected component (if applicable): N/A How reproducible: Installing with the Python3.5:lastest image a Github repository with lxml==3.6.4 requirement Steps to Reproduce: 1. Create a pod starting from image python:3.5 2. Build a stream that has lxml==3.6.4 as a line Actual results: > Assembler messages: > Error: open CFI at the end of file; missing .cfi_endproc directive Expected results: Build completed successfully Additional info: N/A
Created attachment 1215833 [details] Failure on build logs
I'm trying to reproduce this way: #> docker run -ti --rm centos/python-35-centos7 bash bash-4.2$ pip install --user 'lxml==3.6.4' and it works.. Would it be possible to provide better reproducer, please?
Hi Horak, I created a minimal environment that causes the error: I created a new repo with just a requirement.txt containing lxml==3.6.4, you can have a look at the content here: https://github.com/Zerrossetto/reproducing-os3.3-lxml-bug that has been deployed in a clean os3.3 project with just the python3.5 basic container. Project id is "lxml-bug" and it's placed under the tamaghei account. I attached new logs to this bug, but what I can notice from them in respect of your reproduction attempt is that the container is rhscl/python-35-rhel7 and not centos/python-35-centos7, I'll also have a look if i can manage to run it on my local machine. The result of my new attempt is again attached to this bug report, in the meantime thank you for your help.
Created attachment 1216063 [details] New attempt in a reduced environment created for issue reproduction
I don't have os3.3 set up myself, but what I tried as well was: #> s2i build git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git registry.access.redhat.com/rhscl/python-35-rhel7:latest and it worked as well. Ben, any idea where we should look at?
Can you provide the full output from your build log? I'm also not able to recreate, here are my logs from building on a 3.3+ system using centos/python-35-centos7: $ oc new-app docker.io/centos/python-35-centos7:latest~git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git --> Found Docker image 358d162 (10 days old) from docker.io for "docker.io/centos/python-35-centos7:latest" Python 3.5 ---------- Platform for building and running Python 3.5 applications Tags: builder, python, python35, rh-python35 * An image stream will be created as "python-35-centos7:latest" that will track the source image * A source build using source code from git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git will be created * The resulting image will be pushed to image stream "reproducing-os33-lxml-bug:latest" * Every time "python-35-centos7:latest" changes a new build will be triggered * This image will be deployed in deployment config "reproducing-os33-lxml-bug" * Port 8080/tcp will be load balanced by service "reproducing-os33-lxml-bug" * Other containers can access this service through the hostname "reproducing-os33-lxml-bug" --> Creating resources ... imagestream "python-35-centos7" created imagestream "reproducing-os33-lxml-bug" created buildconfig "reproducing-os33-lxml-bug" created deploymentconfig "reproducing-os33-lxml-bug" created service "reproducing-os33-lxml-bug" created --> Success Build scheduled, use 'oc logs -f bc/reproducing-os33-lxml-bug' to track its progress. Run 'oc status' to view your app. sample-app (master)$ oc logs build/reproducing-os33-lxml-bug-1 Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ... Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt) Author: Zerrossetto <tamaghei> Date: Tue Nov 1 11:10:57 2016 +0100 sample-app (master)$ oc logs build/reproducing-os33-lxml-bug-1 -f Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ... Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt) Author: Zerrossetto <tamaghei> Date: Tue Nov 1 11:10:57 2016 +0100 ---> Installing application source ... ---> Installing dependencies ... You are using pip version 7.1.0, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting lxml==3.6.4 (from -r requirements.txt (line 1)) Downloading lxml-3.6.4.tar.gz (3.7MB) Installing collected packages: lxml Running setup.py install for lxml Successfully installed lxml Pushing image 172.30.213.111:5000/test/reproducing-os33-lxml-bug:latest ... Pushed 0/9 layers, 10% complete Pushed 1/9 layers, 13% complete Pushed 2/9 layers, 24% complete Pushed 3/9 layers, 35% complete Pushed 4/9 layers, 46% complete Pushed 5/9 layers, 58% complete Pushed 6/9 layers, 68% complete Pushed 6/9 layers, 90% complete Pushed 7/9 layers, 95% complete Pushed 8/9 layers, 99% complete Pushed 9/9 layers, 100% complete Push successful
What I managed to find out until now is: 1. Launching the command $ s2i build git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git docker.io/centos/python-35-centos7:latest reproducing-os3.3-lxml-bug on my local workstation works as expected, and a valid docker image is generated and pulled on my local docker instance. 2. Doing your command sequence on a brand new project via CLI produces the error reported initially. I also managed to find a StackOverflow entry which addresses possible insufficient memory errors at compile time (ref. http://stackoverflow.com/questions/37128813/installing-lxml-error-open-cfi-at-the-end-of-file-missing-cfi-endproc-direct), so the hypothesis could be that the remote build process fails due to insufficient memory. Anyway I currently am unable to reproduce this locally, and also this doesn't explain why Ben could build it on his project and I'm not. To me it sounds like something at a more lower level than what I could operate on. It follows the command sequence and related command outputs. $ oc delete project lxml-bug project "lxml-bug" deleted $ oc new-project lxml-bug --description="Sample project for bug report 1390236" --display-name="Sample project for S2I bug report" Already on project "lxml-bug" on server "https://api.preview.openshift.com:443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git to build a new example application in Ruby. $ oc new-app docker.io/centos/python-35-centos7:latest~git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git --> Found Docker image 358d162 (10 days old) from docker.io for "docker.io/centos/python-35-centos7:latest" Python 3.5 ---------- Platform for building and running Python 3.5 applications Tags: builder, python, python35, rh-python35 * An image stream will be created as "python-35-centos7:latest" that will track the source image * A source build using source code from git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git will be created * The resulting image will be pushed to image stream "reproducing-os33-lxml-bug:latest" * Every time "python-35-centos7:latest" changes a new build will be triggered * This image will be deployed in deployment config "reproducing-os33-lxml-bug" * Port 8080/tcp will be load balanced by service "reproducing-os33-lxml-bug" * Other containers can access this service through the hostname "reproducing-os33-lxml-bug" --> Creating resources with label app=reproducing-os33-lxml-bug ... imagestream "python-35-centos7" created imagestream "reproducing-os33-lxml-bug" created buildconfig "reproducing-os33-lxml-bug" created deploymentconfig "reproducing-os33-lxml-bug" created service "reproducing-os33-lxml-bu" created --> Success Build scheduled, use 'oc logs -f bc/reproducing-os33-lxml-bug' to track its progress. Run 'oc status' to view your app. $ oc logs -f bc/reproducing-os33-lxml-bug Pulling image "docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049" ... Pulling image "docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049" ... Cloning "git://github.com/Zerrossetto/reproducing-os3.3-lxml-bug.git" ... Commit: 6a16e536263bf1cc7411e5d3f23cf3b1527f620a (Added requirements.txt) Author: Zerrossetto <tamaghei> Date: Tue Nov 1 11:10:57 2016 +0100 ---> Installing application source ... ---> Installing dependencies ... You are using pip version 7.1.0, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting lxml==3.6.4 (from -r requirements.txt (line 1)) Downloading lxml-3.6.4.tar.gz (3.7MB) Installing collected packages: lxml Running setup.py install for lxml Complete output from command /opt/rh/rh-python35/root/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-z8jahi3d/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-qsem1cus-record/install-record.txt --single-version-externally-managed --compile --user --prefix=: Building lxml version 3.6.4. Building without Cython. Using build configuration of libxslt 1.1.28 Building against libxml2/libxslt in the following directory: /usr/lib64 running install running build running build_py creating build creating build/lib.linux-x86_64-3.5 creating build/lib.linux-x86_64-3.5/lxml copying src/lxml/builder.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/sax.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/__init__.py -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.5/lxml creating build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-3.5/lxml/includes creating build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.5/lxml/html copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.5/lxml/html creating build/lib.linux-x86_64-3.5/lxml/isoschematron copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.5/lxml/isoschematron copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.5/lxml copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.5/lxml/includes copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.5/lxml/includes creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 running build_ext building 'lxml.etree' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/src creating build/temp.linux-x86_64-3.5/src/lxml gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python35/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/opt/rh/rh-python35/root/usr/include/python3.5m -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -w {standard input}: Assembler messages: {standard input}:522106: Error: number of operands mismatch for `lea' {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive gcc: internal compiler error: Killed (program cc1) Please submit a full bug report, with preprocessed source if appropriate. See <http://bugzilla.redhat.com/bugzilla> for instructions. Compile failed: command 'gcc' failed with exit status 4 creating tmp cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInit0t0kl9ig.c -o tmp/xmlXPathInit0t0kl9ig.o cc tmp/xmlXPathInit0t0kl9ig.o -L/usr/lib64 -lxml2 -o a.out error: command 'gcc' failed with exit status 4 ---------------------------------------- Command "/opt/rh/rh-python35/root/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-z8jahi3d/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-qsem1cus-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-z8jahi3d/lxml error: build error: non-zero (13) exit code from docker.io/centos/python-35-centos7@sha256:66f40d07ea6ce65daff86eeaae8f0a401175773ae4a768880bae28242dc62049
How much memory does your node host have? Does your project have any default memory limits being applied to the build pods? https://docs.openshift.org/latest/dev_guide/builds.html#build-resources
These are my project quotas, and I'm not setting any kind of limit in the BuildConfig, which is Openshift default by-the-book because currently I'm using and empty project with just a requirements.txt inside with lxml==3.6.4 inside. I'll dig in more with the investigations. $ oc get quota -n lxml-bug NAME AGE compute-resources 5h compute-resources-timebound 5h object-counts 5h $ oc describe quota compute-resources -n lxml-bug Name: compute-resources Namespace: lxml-bug Scopes: NotTerminating * Matches all pods that do not have an active deadline. Resource Used Hard -------- ---- ---- limits.cpu 0 4 limits.memory 0 2Gi $ oc describe quota compute-resources-timebound -n lxml-bug Name: compute-resources-timebound Namespace: lxml-bug Scopes: Terminating * Matches all pods that have an active deadline. Resource Used Hard -------- ---- ---- limits.cpu 0 3 limits.memory 0 1536Mi $ oc describe quota object-counts -n lxml-bug Name: object-counts Namespace: lxml-bug Resource Used Hard -------- ---- ---- persistentvolumeclaims 0 2 replicationcontrollers 0 50 secrets 9 20 services 1 10
Hm, those are quotas but not limits. Does your project have a LimitRange associated with it? You might also be able to tell by dong an "oc get pod <build-pod-name> -o yaml" and see what resource limits are assigned to the pod.
I finally managed to build lxml. The issue was indeed that, without any setting in the "resources" section of the related BuildConfig the resulting run-once pod came out with these settings: resources: limits: cpu: '1' memory: 512Mi requests: cpu: 60m memory: 307Mi the resolution came when I updated the BuildConfig configuration with: resources: limits: memory: 1Gi which overrided the previews defaults and led me to this scenario CPU: 120 millicores to 2 cores Memory: 614 MiB to 1 GiB that was sufficient to let the job end. I personally think that having a limit higher than a default it's a bit strange. Thank you for your support.
Great, glad we were able to sort it out.