I cannot deploy my application because Python development header files are missing:
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g /render/project/src/.venv/include -I/opt/python-installer/Python-3.10.7/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
In file included from src/hiredis.c:1:
src/hiredis.h:4:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
The package in question is hiredis. It doesn’t have a compiled release (wheel, *.whl) for Python 3.10 and therefore pip wants to build it. How do I go about using it within my app on render.com?
Actually I am facing the same issue as you, I’ve noticed that GDAL and Spatial databases with Python running in Ubuntu (or even locally) need some additional apt-get stuff to be run first before they can be installed
I’ve managed to get them running in docker both locally and in DO Ubuntu 22.0.4, by adding those to the dockerfile to run first before the pip install -r requirements. However I was stumped by the no-logs bug. Will share the fix once it’s working!
the problem seems to be that only the header files for Python 2.7 and Python 3.7 are installed while my application uses Python 3.10.7 (which is officially supported).
Installing a package from PyPI that requires compiling works fine for the system-level Python version (in this case Python 3.7):
$ /usr/local/bin/python --version
Python 3.7.10
$ /usr/local/bin/python -m pip install --no-binary :all: hiredis
Defaulting to user installation because normal site-packages is not writeable
Collecting hiredis
Using cached hiredis-2.0.0.tar.gz (75 kB)
Preparing metadata (setup.py) ... done
Skipping wheel build for hiredis, due to binaries being disabled for it.
Installing collected packages: hiredis
Running setup.py install for hiredis ... done
Successfully installed hiredis-2.0.0
When installing a package using pip on Python 3.10, on the other hand, it fails:
$ which python
/opt/render/project/src/.venv/bin/python
$ python --version
Python 3.10.7
$ python -m pip install hiredis
[…] snip
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ivendor -I/opt/render/project/src/.venv/include -I/opt/python-installer/Python-3.10.7/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
In file included from src/hiredis.c:1:
src/hiredis.h:4:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
Is it possible to add the required header files for all supported Python versions?
Hi Jinns, one workaround is to install them through your dockerfile
# Pull base image
FROM python:3.10.5-slim-bullseye
#THIS PART INSTALLS THE SYSTEM PACKAGES
RUN apt-get update && \
apt-get install -y \
libgdal-dev \
python3-pip
Thank you Andrew_Ng for the note – unfortunately I don’t use the Dockerfile.
It would be great if Render could state which Python versions are “fully” supported i.e. include Python.h (i.e. python3-dev) – as jnns has discovered, the header’s presence depends on the specific version (Python 3.7 vs 3.10.7)
@John_B what’s the best way to deal with this? Right now, I’m manually removing psycopg2 from my requirements.txt every time after running pip freeze.
Should I open a feature request to ask for Python.h to be included with all Python versions after 3.7? Thanks!
EDIT:
Oh and actually… would Render be ready to share the Dockerfile it uses for Python deployments? That way, I/others could use it for their own Render-deployed projects, make sure that python3-dev is included, etc.
i.e. would Render let us see its equivalent to the Heroku Python buildpack? Cheers
Something has changed in the last 14 days (last time I pushed a build). I’m running Python 3.9.10 and and just tried to push a new build and now get compilation errors trying to build psycopg2 due to missing headers.
@chrisgrande Could you try a manual deploy and clear the cache - we have pushed new env images out on Friday 21st but nothing should have changed or been removed like this,
Hi folks - as a temporary fix, could you try appending CPATH=/usr/include/python3.7m to your pip install cmd; for example CPATH=/usr/include/python3.7m pip install -r requirements.txt in the Render build cmd.
I’ve replicated this build error locally and forcing the CPATH fixed the install. The new env image deploy was for an unrelated change. Our native environments have included the python3-dev package and header files (documented here Native Environments | Render · Cloud Hosting for Developers ) so I will continue to investigate why they aren’t getting used correctly.
The same problem when trying to deploy with Python 3.11.0.
Downgraded to 3.10.8 as a solution.
==> Using Python version: 3.11.0
...
Nov 23 12:32:28 AM ERROR: Failed building wheel for multidict
Nov 23 12:32:28 AM Building wheel for yarl (pyproject.toml): started
Nov 23 12:32:28 AM Building wheel for yarl (pyproject.toml): finished with status 'error'
Nov 23 12:32:28 AM error: subprocess-exited-with-error
Nov 23 12:32:28 AM
Nov 23 12:32:28 AM × Building wheel for yarl (pyproject.toml) did not run successfully.
Nov 23 12:32:28 AM │ exit code: 1
Nov 23 12:32:28 AM ╰─> [44 lines of output]
Nov 23 12:32:28 AM /tmp/pip-build-env-u48qplew/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
Nov 23 12:32:28 AM warnings.warn(msg, warning_class)
Nov 23 12:32:28 AM **********************
Nov 23 12:32:28 AM * Accelerated build *
Nov 23 12:32:28 AM **********************
Nov 23 12:32:28 AM running bdist_wheel
Nov 23 12:32:28 AM running build
Nov 23 12:32:28 AM running build_py
Nov 23 12:32:28 AM creating build
Nov 23 12:32:28 AM creating build/lib.linux-x86_64-cpython-311
Nov 23 12:32:28 AM creating build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/_url.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/_quoting_py.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/__init__.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/_quoting.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM running egg_info
Nov 23 12:32:28 AM writing yarl.egg-info/PKG-INFO
Nov 23 12:32:28 AM writing dependency_links to yarl.egg-info/dependency_links.txt
Nov 23 12:32:28 AM writing requirements to yarl.egg-info/requires.txt
Nov 23 12:32:28 AM writing top-level names to yarl.egg-info/top_level.txt
Nov 23 12:32:28 AM reading manifest file 'yarl.egg-info/SOURCES.txt'
Nov 23 12:32:28 AM reading manifest template 'MANIFEST.in'
Nov 23 12:32:28 AM warning: no previously-included files matching '*.pyc' found anywhere in distribution
Nov 23 12:32:28 AM warning: no previously-included files matching '*.cache' found anywhere in distribution
Nov 23 12:32:28 AM warning: no previously-included files found matching 'yarl/*.html'
Nov 23 12:32:28 AM warning: no previously-included files found matching 'yarl/*.so'
Nov 23 12:32:28 AM warning: no previously-included files found matching 'yarl/*.pyd'
Nov 23 12:32:28 AM no previously-included directories found matching 'docs/_build'
Nov 23 12:32:28 AM adding license file 'LICENSE'
Nov 23 12:32:28 AM writing manifest file 'yarl.egg-info/SOURCES.txt'
Nov 23 12:32:28 AM copying yarl/__init__.pyi -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/_quoting_c.pyi -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/_quoting_c.pyx -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM copying yarl/py.typed -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM running build_ext
Nov 23 12:32:28 AM building 'yarl._quoting_c' extension
Nov 23 12:32:28 AM creating build/temp.linux-x86_64-cpython-311
Nov 23 12:32:28 AM creating build/temp.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/render/project/src/.venv/include -I/opt/python-installer/Python-3.11.0/include/python3.11 -c yarl/_quoting_c.c -o build/temp.linux-x86_64-cpython-311/yarl/_quoting_c.o
Nov 23 12:32:28 AM yarl/_quoting_c.c:6:10: fatal error: Python.h: No such file or directory
Nov 23 12:32:28 AM #include "Python.h"
Nov 23 12:32:28 AM ^~~~~~~~~~
Nov 23 12:32:28 AM compilation terminated.
Nov 23 12:32:28 AM error: command '/usr/bin/gcc' failed with exit code 1
Nov 23 12:32:28 AM [end of output]
Nov 23 12:32:28 AM
Nov 23 12:32:28 AM note: This error originates from a subprocess, and is likely not a problem with pip.
Nov 23 12:32:28 AM ERROR: Failed building wheel for yarl
Nov 23 12:32:28 AM Failed to build multidict yarl
Nov 23 12:32:28 AM ERROR: Could not build wheels for multidict, yarl, which is required to install pyproject.toml-based projects
Nov 23 12:32:28 AM
Nov 23 12:32:28 AM [notice] A new release of pip available: 22.3 -> 22.3.1
Nov 23 12:32:28 AM [notice] To update, run: pip install --upgrade pip
Nov 23 12:32:29 AM ==> Build failed 😞