FastAPI with poetry deployment error

1

the build command is successfully running but it fails on the start command for some reason. Looks lke libcurand.so.10 library file is missing or not found in the system, could be related to CUDA and PyTorch dependencies, has anyone faced something similar?

Jul 7 08:30:19 PM  ==> Starting service with 'poetry run uvicorn main:app --host 0.0.0.0 --port 1000'
Jul 7 08:30:39 PM  Traceback (most recent call last):
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/torch/__init__.py", line 168, in _load_global_deps
Jul 7 08:30:39 PM      ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
Jul 7 08:30:39 PM    File "/opt/render/project/python/Python-3.11.4/lib/python3.11/ctypes/__init__.py", line 376, in __init__
Jul 7 08:30:39 PM      self._handle = _dlopen(self._name, mode)
Jul 7 08:30:39 PM                     ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM  OSError: libcurand.so.10: cannot open shared object file: No such file or directory
Jul 7 08:30:39 PM  
Jul 7 08:30:39 PM  During handling of the above exception, another exception occurred:
Jul 7 08:30:39 PM  
Jul 7 08:30:39 PM  Traceback (most recent call last):
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/bin/uvicorn", line 8, in <module>
Jul 7 08:30:39 PM      sys.exit(main())
Jul 7 08:30:39 PM               ^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
Jul 7 08:30:39 PM      return self.main(*args, **kwargs)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
Jul 7 08:30:39 PM      rv = self.invoke(ctx)
Jul 7 08:30:39 PM           ^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
Jul 7 08:30:39 PM      return ctx.invoke(self.callback, **ctx.params)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
Jul 7 08:30:39 PM      return __callback(*args, **kwargs)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/main.py", line 410, in main
Jul 7 08:30:39 PM      run(
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/main.py", line 578, in run
Jul 7 08:30:39 PM      server.run()
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
Jul 7 08:30:39 PM      return asyncio.run(self.serve(sockets=sockets))
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/python/Python-3.11.4/lib/python3.11/asyncio/runners.py", line 190, in run
Jul 7 08:30:39 PM      return runner.run(main)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/python/Python-3.11.4/lib/python3.11/asyncio/runners.py", line 118, in run
Jul 7 08:30:39 PM      return self._loop.run_until_complete(task)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/python/Python-3.11.4/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
Jul 7 08:30:39 PM      return future.result()
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
Jul 7 08:30:39 PM      config.load()
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/config.py", line 473, in load
Jul 7 08:30:39 PM      self.loaded_app = import_from_string(self.app)
Jul 7 08:30:39 PM                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
Jul 7 08:30:39 PM      module = importlib.import_module(module_str)
Jul 7 08:30:39 PM               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "/opt/render/project/python/Python-3.11.4/lib/python3.11/importlib/__init__.py", line 126, in import_module
Jul 7 08:30:39 PM      return _bootstrap._gcd_import(name[level:], package, level)
Jul 7 08:30:39 PM             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap_external>", line 940, in exec_module
Jul 7 08:30:39 PM    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
Jul 7 08:30:39 PM    File "/opt/render/project/src/main.py", line 8, in <module>
Jul 7 08:30:39 PM      from sourcing import PERSON_PATH_CS, ElasticSearchEngine, person_generator
Jul 7 08:30:39 PM    File "/opt/render/project/src/sourcing.py", line 11, in <module>
Jul 7 08:30:39 PM      import utils
Jul 7 08:30:39 PM    File "/opt/render/project/src/utils.py", line 261, in <module>
Jul 7 08:30:39 PM      from sentence_transformers import SentenceTransformer
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentence_transformers/__init__.py", line 3, in <module>
Jul 7 08:30:39 PM      from .datasets import SentencesDataset, ParallelSentencesDataset
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentence_transformers/datasets/__init__.py", line 1, in <module>
Jul 7 08:30:39 PM      from .DenoisingAutoEncoderDataset import DenoisingAutoEncoderDataset
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/sentence_transformers/datasets/DenoisingAutoEncoderDataset.py", line 1, in <module>
Jul 7 08:30:39 PM      from torch.utils.data import Dataset
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/torch/__init__.py", line 228, in <module>
Jul 7 08:30:39 PM      _load_global_deps()
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/torch/__init__.py", line 189, in _load_global_deps
Jul 7 08:30:39 PM      _preload_cuda_deps(lib_folder, lib_name)
Jul 7 08:30:39 PM    File "/opt/render/project/src/.venv/lib/python3.11/site-packages/torch/__init__.py", line 154, in _preload_cuda_deps
Jul 7 08:30:39 PM      raise ValueError(f"{lib_name} not found in the system path {sys.path}")
Jul 7 08:30:39 PM  ValueError: libcublas.so.*[0-9] not found in the system path ['', '/opt/render/project/src/.venv/bin', '/opt/render/project/python/Python-3.11.4/lib/python311.zip', '/opt/render/project/python/Python-3.11.4/lib/python3.11', '/opt/render/project/python/Python-3.11.4/lib/python3.11/lib-dynload', '/opt/render/project/src/.venv/lib/python3.11/site-packages']
2

Hi there,

I haven’t seen this specific error before, but it would be occurring because the Ubuntu CUDA packages are not included in our native environments list of packages: https://render.com/docs/native-runtimes#included-tools-and-utilities.

Your best option would be to move to using a Docker build where you have full control over the package you need for your service.

Regards,

Keith
Render Support, UTC+10 :australia:

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.