Python build fails. No such file: Python.h

jnns · September 13, 2022, 8:20pm

I cannot deploy my application because Python development header files are missing:

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g /render/project/src/.venv/include -I/opt/python-installer/Python-3.10.7/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
In file included from src/hiredis.c:1:
src/hiredis.h:4:10: fatal error: Python.h: No such file or directory
#include <Python.h>
         ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

The package in question is hiredis. It doesn’t have a compiled release (wheel, *.whl) for Python 3.10 and therefore pip wants to build it. How do I go about using it within my app on render.com?

In the documentation page about Native Environments you state that the python3-dev package is available. Shouldn’t it contain the required header files?

John_B · September 15, 2022, 1:32pm

Hey there, I took a look at your services and see they’ve all deployed successfully now, did you figure it out?

Regards,

John B

jnns · September 16, 2022, 8:58am

Hi John.

No, unforunately not. I just temporarily removed the packages that needed compiling form my app. I’d like to introduce them again.

Are the Python C header files expected to be present in the Native Environment?

John_B · September 22, 2022, 2:47pm

Hi there,
Yes, I’d expect them to be:

binutils \gdal-bin \libgdal-dev \libproj-dev \python3-dev \python3-setuptools \

are what we have installed.

John B

Andrew_Ng · September 24, 2022, 4:13pm

Actually I am facing the same issue as you, I’ve noticed that GDAL and Spatial databases with Python running in Ubuntu (or even locally) need some additional apt-get stuff to be run first before they can be installed

I’ve managed to get them running in docker both locally and in DO Ubuntu 22.0.4, by adding those to the dockerfile to run first before the pip install -r requirements. However I was stumped by the no-logs bug. Will share the fix once it’s working!

jnns · September 28, 2022, 8:24am

Hello @John_B

the problem seems to be that only the header files for Python 2.7 and Python 3.7 are installed while my application uses Python 3.10.7 (which is officially supported).

$ echo $PYTHON_VERSION 
3.10.7

$ lsb_release -d
Description:    Debian GNU/Linux 10 (buster)

$ apt list | grep libpython | grep dev
libpython-all-dev/now 2.7.16-1 amd64 [installed,local]
libpython-dev/now 2.7.16-1 amd64 [installed,local]
libpython2-dev/now 2.7.16-1 amd64 [installed,local]
libpython2.7-dev/now 2.7.16-2+deb10u1 amd64 [installed,local]
libpython3-dev/now 3.7.3-1 amd64 [installed,local]
libpython3.7-dev/now 3.7.3-2+deb10u3 amd64 [installed,local]

Installing a package from PyPI that requires compiling works fine for the system-level Python version (in this case Python 3.7):

$ /usr/local/bin/python --version
Python 3.7.10

$ /usr/local/bin/python -m pip install --no-binary :all: hiredis
Defaulting to user installation because normal site-packages is not writeable
Collecting hiredis
  Using cached hiredis-2.0.0.tar.gz (75 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for hiredis, due to binaries being disabled for it.
Installing collected packages: hiredis
  Running setup.py install for hiredis ... done
Successfully installed hiredis-2.0.0

When installing a package using pip on Python 3.10, on the other hand, it fails:

$ which python
/opt/render/project/src/.venv/bin/python

$ python --version
Python 3.10.7

$ python -m pip install hiredis
  […] snip
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ivendor -I/opt/render/project/src/.venv/include -I/opt/python-installer/Python-3.10.7/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
  In file included from src/hiredis.c:1:
  src/hiredis.h:4:10: fatal error: Python.h: No such file or directory
  #include <Python.h>
           ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1

Is it possible to add the required header files for all supported Python versions?

gregtechnology · September 28, 2022, 8:36pm

Same as @jnns , building psycopg2 (from my requirements.txt) under Python 3.10.7 fails for me. I am using psycopg2-binary for now. Thanks!

jnns · October 8, 2022, 12:45pm

Sorry to bump this thread and create noise but I’d really like to know what the Render team thinks about this and if this expected behaviour.

Can you provide the required header files for all Python version listed as officially supported?

Andrew_Ng · October 10, 2022, 2:33am

Hi Jinns, one workaround is to install them through your dockerfile

# Pull base image
FROM python:3.10.5-slim-bullseye

#THIS PART INSTALLS THE SYSTEM PACKAGES
RUN apt-get update && \
        apt-get install -y \
        libgdal-dev \
        python3-pip

gregtechnology · October 21, 2022, 5:10pm

Thank you Andrew_Ng for the note – unfortunately I don’t use the Dockerfile.

It would be great if Render could state which Python versions are “fully” supported i.e. include Python.h (i.e. python3-dev) – as jnns has discovered, the header’s presence depends on the specific version (Python 3.7 vs 3.10.7)

Specifying a Python Version | Render is not super clear about the supported versions – " Supported Python versions are those after 3.7.0 ."

@John_B what’s the best way to deal with this? Right now, I’m manually removing psycopg2 from my requirements.txt every time after running pip freeze.

Should I open a feature request to ask for Python.h to be included with all Python versions after 3.7? Thanks!

EDIT:

Oh and actually… would Render be ready to share the Dockerfile it uses for Python deployments? That way, I/others could use it for their own Render-deployed projects, make sure that python3-dev is included, etc.

i.e. would Render let us see its equivalent to the Heroku Python buildpack? Cheers

chrisgrande · October 27, 2022, 12:03am

Something has changed in the last 14 days (last time I pushed a build). I’m running Python 3.9.10 and and just tried to push a new build and now get compilation errors trying to build psycopg2 due to missing headers.

John_B · October 28, 2022, 10:03am

@chrisgrande Could you try a manual deploy and clear the cache - we have pushed new env images out on Friday 21st but nothing should have changed or been removed like this,

Regards,

jnns · October 28, 2022, 12:09pm

Do the new env images entail that header files are now present on all supported Python versions?

chrisgrande · October 28, 2022, 5:08pm

I tried a clear cache and build and it fails in the same way – all my services and cron jobs that are built like this are failing in the same way.

Matt_Behrens · October 29, 2022, 12:45am

Hi folks - as a temporary fix, could you try appending CPATH=/usr/include/python3.7m to your pip install cmd; for example CPATH=/usr/include/python3.7m pip install -r requirements.txt in the Render build cmd.

I’ve replicated this build error locally and forcing the CPATH fixed the install. The new env image deploy was for an unrelated change. Our native environments have included the python3-dev package and header files (documented here Native Environments | Render · Cloud Hosting for Developers ) so I will continue to investigate why they aren’t getting used correctly.

nnn · November 22, 2022, 9:19pm

The same problem when trying to deploy with Python 3.11.0.
Downgraded to 3.10.8 as a solution.

 ==> Using Python version: 3.11.0
...
Nov 23 12:32:28 AM    ERROR: Failed building wheel for multidict
Nov 23 12:32:28 AM    Building wheel for yarl (pyproject.toml): started
Nov 23 12:32:28 AM    Building wheel for yarl (pyproject.toml): finished with status 'error'
Nov 23 12:32:28 AM    error: subprocess-exited-with-error
Nov 23 12:32:28 AM  
Nov 23 12:32:28 AM    × Building wheel for yarl (pyproject.toml) did not run successfully.
Nov 23 12:32:28 AM    │ exit code: 1
Nov 23 12:32:28 AM    ╰─> [44 lines of output]
Nov 23 12:32:28 AM        /tmp/pip-build-env-u48qplew/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
Nov 23 12:32:28 AM          warnings.warn(msg, warning_class)
Nov 23 12:32:28 AM        **********************
Nov 23 12:32:28 AM        * Accelerated build *
Nov 23 12:32:28 AM        **********************
Nov 23 12:32:28 AM        running bdist_wheel
Nov 23 12:32:28 AM        running build
Nov 23 12:32:28 AM        running build_py
Nov 23 12:32:28 AM        creating build
Nov 23 12:32:28 AM        creating build/lib.linux-x86_64-cpython-311
Nov 23 12:32:28 AM        creating build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/_url.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/_quoting_py.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/__init__.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/_quoting.py -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        running egg_info
Nov 23 12:32:28 AM        writing yarl.egg-info/PKG-INFO
Nov 23 12:32:28 AM        writing dependency_links to yarl.egg-info/dependency_links.txt
Nov 23 12:32:28 AM        writing requirements to yarl.egg-info/requires.txt
Nov 23 12:32:28 AM        writing top-level names to yarl.egg-info/top_level.txt
Nov 23 12:32:28 AM        reading manifest file 'yarl.egg-info/SOURCES.txt'
Nov 23 12:32:28 AM        reading manifest template 'MANIFEST.in'
Nov 23 12:32:28 AM        warning: no previously-included files matching '*.pyc' found anywhere in distribution
Nov 23 12:32:28 AM        warning: no previously-included files matching '*.cache' found anywhere in distribution
Nov 23 12:32:28 AM        warning: no previously-included files found matching 'yarl/*.html'
Nov 23 12:32:28 AM        warning: no previously-included files found matching 'yarl/*.so'
Nov 23 12:32:28 AM        warning: no previously-included files found matching 'yarl/*.pyd'
Nov 23 12:32:28 AM        no previously-included directories found matching 'docs/_build'
Nov 23 12:32:28 AM        adding license file 'LICENSE'
Nov 23 12:32:28 AM        writing manifest file 'yarl.egg-info/SOURCES.txt'
Nov 23 12:32:28 AM        copying yarl/__init__.pyi -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/_quoting_c.pyi -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/_quoting_c.pyx -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        copying yarl/py.typed -> build/lib.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        running build_ext
Nov 23 12:32:28 AM        building 'yarl._quoting_c' extension
Nov 23 12:32:28 AM        creating build/temp.linux-x86_64-cpython-311
Nov 23 12:32:28 AM        creating build/temp.linux-x86_64-cpython-311/yarl
Nov 23 12:32:28 AM        gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/render/project/src/.venv/include -I/opt/python-installer/Python-3.11.0/include/python3.11 -c yarl/_quoting_c.c -o build/temp.linux-x86_64-cpython-311/yarl/_quoting_c.o
Nov 23 12:32:28 AM        yarl/_quoting_c.c:6:10: fatal error: Python.h: No such file or directory
Nov 23 12:32:28 AM         #include "Python.h"
Nov 23 12:32:28 AM                  ^~~~~~~~~~
Nov 23 12:32:28 AM        compilation terminated.
Nov 23 12:32:28 AM        error: command '/usr/bin/gcc' failed with exit code 1
Nov 23 12:32:28 AM        [end of output]
Nov 23 12:32:28 AM  
Nov 23 12:32:28 AM    note: This error originates from a subprocess, and is likely not a problem with pip.
Nov 23 12:32:28 AM    ERROR: Failed building wheel for yarl
Nov 23 12:32:28 AM  Failed to build multidict yarl
Nov 23 12:32:28 AM  ERROR: Could not build wheels for multidict, yarl, which is required to install pyproject.toml-based projects
Nov 23 12:32:28 AM  
Nov 23 12:32:28 AM  [notice] A new release of pip available: 22.3 -> 22.3.1
Nov 23 12:32:28 AM  [notice] To update, run: pip install --upgrade pip
Nov 23 12:32:29 AM  ==> Build failed 😞

system · December 22, 2022, 9:19pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Building a static site using Python fails	3	707	December 2, 2022
Build failed Django application	3	818	October 19, 2022
Need help with django site, can'y deploy :(	2	653	April 22, 2022
Install `python-snappy` (fatal error: snappy-c.h: No such file or directory)	2	392	January 19, 2024
Unable to build my cron job	2	553	June 21, 2023

Python build fails. No such file: Python.h

Related Topics