ChromeDriver is assuming that Chrome has crashed

markpro · June 27, 2023, 3:41am

I’ve been reading and testing about how to get Chrome (via Selenium) working on Render and I’m just about ready to quit. I don’t know much about Docker, so I’ve been avoiding it, but the work around is failing. I’m on the paid starter plan.

At first, I had issues getting the Driver installed, but quickly found the shell script that was posted. This then led to issues with DevTools ActivePort file doesn’t exist, but some add_arguments (the port one) fixed that. Yet, no matter what I do I cannot get selenium to run.

One thing to note, I read all the posts I could find on here about it and I decided to switched from the shell script to the pip package as it’s a lot cleaner. In both cases, the errors are the same.

Here’s my console log:

Jun 26 10:55:21 PM  ==> Starting service with 'uvicorn main:app --host 0.0.0.0 --port 10000'
Jun 26 10:55:28 PM  INFO:     Started server process [51]
Jun 26 10:55:28 PM  INFO:     Waiting for application startup.
Jun 26 10:55:28 PM  INFO:     Application startup complete.
Jun 26 10:55:28 PM  INFO:     Uvicorn running on http://0.0.0.0:10000 (Press CTRL+C to quit)
Jun 26 10:55:38 PM  Your service is live 🎉
Jun 26 10:55:39 PM  INFO:     34.82.247.67:0 - "GET / HTTP/1.1" 404 Not Found
Jun 26 10:56:32 PM  2023-06-27 02:56:32,274 loglevel=INFO   logger=endpoints.saving post_save_item() L47   post to /save_url for 'https://www.instagram.com/reel/CslboyRN041/?igshid=MTc4MmM1YmI2Ng=='
Jun 26 10:56:32 PM  
Jun 26 10:56:32 PM  2023-06-27 02:56:32,694 loglevel=INFO   logger=WDM log() L11   ====== WebDriver manager ======
Jun 26 10:56:32 PM  2023-06-27 02:56:32,841 loglevel=INFO   logger=WDM log() L11   There is no [linux64] chromedriver "latest" for browser google-chrome "None" in cache
Jun 26 10:56:32 PM  2023-06-27 02:56:32,939 loglevel=INFO   logger=WDM log() L11   Get LATEST chromedriver version for google-chrome
Jun 26 10:56:33 PM  2023-06-27 02:56:33,074 loglevel=INFO   logger=WDM log() L11   About to download new driver from https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
Jun 26 10:56:33 PM  
[WDM] - Downloading:   0%|          | 0.00/7.06M [00:00<?, ?B/s]
[WDM] - Downloading:   8%|▊         | 552k/7.06M [00:00<00:01, 5.64MB/s]
[WDM] - Downloading: 100%|██████████| 7.06M/7.06M [00:00<00:00, 38.8MB/s]
Jun 26 10:56:33 PM  2023-06-27 02:56:33,635 loglevel=INFO   logger=WDM log() L11   Driver has been saved in cache [/opt/render/.wdm/drivers/chromedriver/linux64/114.0.5735.90]
Jun 26 10:56:35 PM  Traceback (most recent call last):
Jun 26 10:56:35 PM    File "/opt/render/project/src/services/extract_link.py", line 105, in get_link_soup
Jun 26 10:56:35 PM      driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options)
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 49, in __init__
Jun 26 10:56:35 PM      super().__init__(
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 54, in __init__
Jun 26 10:56:35 PM      super().__init__(
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 206, in __init__
Jun 26 10:56:35 PM      self.start_session(capabilities)
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 291, in start_session
Jun 26 10:56:35 PM      response = self.execute(Command.NEW_SESSION, caps)["value"]
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 346, in execute
Jun 26 10:56:35 PM      self.error_handler.check_response(response)
Jun 26 10:56:35 PM    File "/opt/render/project/src/.venv/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
Jun 26 10:56:35 PM      raise exception_class(message, screen, stacktrace)
Jun 26 10:56:35 PM  selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
Jun 26 10:56:35 PM    (chrome not reachable)
Jun 26 10:56:35 PM    (The process started from chrome location /opt/render/.wdm/drivers/chromedriver is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Jun 26 10:56:35 PM  Stacktrace:
Jun 26 10:56:35 PM  #0 0x563c0482f4e3 <unknown>
Jun 26 10:56:35 PM  #1 0x563c0455ec76 <unknown>
[...]
Jun 26 10:56:35 PM  #15 0x563c04818847 <unknown>
Jun 26 10:56:35 PM  #16 0x563c04828243 <unknown>
Jun 26 10:56:35 PM  #17 0x7fbbecae6fa3 start_thread
Jun 26 10:56:35 PM  
Jun 26 10:56:35 PM  2023-06-27 02:56:35,187 loglevel=ERROR  logger=services.extract_link get_link_soup() L128  Error downloading content from URL: https://www.instagram.com/reel/CslboyRN041/?igshid=MTc4MmM1YmI2Ng==
Jun 26 10:56:35 PM  Error message: Message: unknown error: Chrome failed to start: exited abnormally.
Jun 26 10:56:35 PM    (chrome not reachable)
Jun 26 10:56:35 PM    (The process started from chrome location /opt/render/.wdm/drivers/chromedriver is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Jun 26 10:56:35 PM  Stacktrace:
Jun 26 10:56:35 PM  #0 0x563c0482f4e3 <unknown>
Jun 26 10:56:35 PM  #1 0x563c0455ec76 <unknown>
[...]
Jun 26 10:56:35 PM  #15 0x563c04818847 <unknown>
Jun 26 10:56:35 PM  #16 0x563c04828243 <unknown>
Jun 26 10:56:35 PM  #17 0x7fbbecae6fa3 start_thread
Jun 26 10:56:35 PM  
Jun 26 10:56:35 PM  2023-06-27 02:56:35,188 loglevel=ERROR  logger=endpoints.saving post_save_item() L100  Error downloading content from URL: https://www.instagram.com/reel/CslboyRN041/?igshid=MTc4MmM1YmI2Ng==
Jun 26 10:56:35 PM  INFO:     52.15.118.168:0 - "POST /save_url HTTP/1.1" 404 Not Found

And here is the related python code:

        try:
            # Set up the Selenium WebDriver
            options = webdriver.ChromeOptions()
            if ENV_NAME == 'production':
                options.binary_location = "/opt/render/.wdm/drivers/chromedriver"
            options.add_argument('--verbose')
            options.add_argument('--no-sandbox')
            options.add_argument('--headless')
            options.add_argument('--ignore-certificate-errors')
            options.add_argument('--disable-dev-shm-usage')
            options.add_argument('--remote-debugging-port=9222')
            options.add_argument('--disable-extensions')
            options.add_argument('--disable-gpu')
            options.add_argument('--user-agent={}'.format(random.choice(list(self.user_agents))))

            driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=options)
            driver.set_page_load_timeout(90)

            # Load the URL and get the page source
            driver.get(url)
            time.sleep(4)  # let load
            html = driver.page_source

I’m not sure if there’s something simple I am missing, but I hope someone can help! Cheers.
(And thank you for all the other related threads on this).

al_ps · June 27, 2023, 11:55am

Hi,

ChromeDriver isn’t Chrome itself. If you’re using a Render Native Environment, Chrome isn’t installed by default, so you’ll need to install it yourself.

As you mentioned, there are plenty of posts that cover installing Chrome on a Native Environment. The starting point is usually a script like https://gist.github.com/BigAlRender/41f4c4d87df3e25770e3db8db728443e

Alan

markpro · June 28, 2023, 3:51pm

UPDATE 2:
I reached out to some chat’s for Selenium and now I have a better understanding. In the latest version of Selenium, there is a built in driver manager that can do the heavy lifting for you. (e.g. I am using 4.10.0)

This criteria must be met:

You do in fact need to have Chrome (not the driver) itself installed and PATH set (see bash script)
You need to make sure the driver location isn’t specified in a Service class
You need to make sure no 3rd party managers are installed
There’s no drivers exist in directories included in the PATH Environment Variable

I am testing out what I learned now…

Here are the doc from Selenium:

markpro · June 28, 2023, 4:14pm

SOLUTION!!!

Okay, the above (Update 2) explains it well, but I wanted to make a mini tutorial as it seems other’s are having this issue as well.

So you first need to all the bash script to install Chrome into Render, I used the render-build.sh script linked above and stored it in my repo. (Therefore it’s executable at ./render-build.sh).

Note: I did modify this bash script to include my build commands at the bottom, in my case it was adding the following to the end of it: (as you’ll set in step the first bullet below)

pip install -r requirements.txt

Then I went into my Render to change the build and start commands:

In Dashboard > Settings > Build & Deploy > Build Command, I set it to what’s blow so it’ll run the bash script and run my added build command mentioned above.

./render-build.sh

In Dashboard > Settings > Build & Deploy > Start Command, I set it to this for FastAPI:

export PATH="${PATH}:/opt/render/project/.render/chrome/opt/google/chrome" && uvicorn main:app --host 0.0.0.0 --port 10000

Important Note on #2, I am using FastAPI so my I’m using uvicorn ..., but if you’re just running a python script than replacing everything after the && with the command will resolve it. So something like this:

export PATH="${PATH}:/opt/render/project/.render/chrome/opt/google/chrome" && python main.py

As for my Python script, the code looks something like this:

# Set up the Selenium WebDriver
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-extensions')
options.add_argument('--disable-gpu')
options.add_argument('--user-agent={}'.format(random.choice(list(self.user_agents))))

driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(90)

# Load the URL and get the page source
driver.implicitly_wait(6)
driver.get(url)
# ...

Lastly, it’s super important, if you want Selenium to manage your drives for you that the following criteria are met based on the Selenium Manager docs:

The driver location isn’t specified in a Service class (Note: my python code reflects this)
A 3rd party driver manager is not installed (Note: if you set up something previously, remove it)
No drivers exist in directories included in the PATH Environment Variable (Note: if you set up something previously remove it, note Chrome in the bash script and Chrome webdriver are different things).

Cheers

uncleschaeflit · July 26, 2023, 8:41pm

THANK YOU. Oh how I’ve struggled with this. Much love for the solution. Works perfectly. Removing the Service class was the breakthrough.

system · August 25, 2023, 8:41pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Is it possible to use selenium in render?	6	3180	January 5, 2023
Could we run a selenium service in render?	1	1587	December 16, 2020
Selenium standalone as worker	4	499	November 1, 2023
Getting Chrome in PATH	2	2087	March 3, 2023
Request Timeout - Selenium	2	330	May 11, 2023

ChromeDriver is assuming that Chrome has crashed

SOLUTION!!!

Related Topics