Selenium error: 'chromedriver' executable may have wrong permissions

The app is web scraping. I was following a few guides and videos, such as this: https://www.youtube.com/watch?v=ne3BH9-5H2o

What I eventually want is that my web app will work in the live website and the scraped data will be automatically downloaded to the user’s machine.

What I have now is that this works perfectly when run locally. I’m really not well versed in this exact thing. It is my first project with Python at all and I’ve tried many suggested solutions from Google and tried my luck with AI chatbots, but not getting anywhere.

The error from running it in a live version of the website is below.

From what I have gathered, I can either switch hosting services (seen PythonAnywhere and Railway.app as suggestions) or I can run my app in a Docker container, which I admit that I didn’t think was necessary, initially. From what I have read, Render does not support Selenium by default, hence the suggestion of Docker.

This is my output from running in a headless browser:

  File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/opt/render/project/src/chromedriver'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/googleMapsScrapingToolweb.py", line 158, in scrape
    file_path = scraper.scrape()
                ^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/googleMapsScrapingToolweb.py", line 35, in scrape
    browser = webdriver.Chrome(executable_path=chromedriver_path ,options=options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
    super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 90, in __init__
    self.service.start()
  File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/common/service.py", line 86, in start
    raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable may have wrong permissions. Please see https://chromedriver.chromium.org/home
127.0.0.1 - - [26/Apr/2024:14:49:56 +0000] "POST /scrape HTTP/1.1" 500 290 "https://googlemapsscrapingtoolweb.onrender.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0"

This is the relevant code:

class GoogleMapsScraper:
    def __init__(self, link):
        self.link = link
        self.csv_data = []
        self.elementResults = 0

    def scrape(self):
        chromedriver_path = os.path.abspath("chromedriver")
        options = webdriver.ChromeOptions()
        options.add_argument("--headless=new")
        options.add_argument("--no-sandbox")
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--disable-gpu")
        browser = webdriver.Chrome(executable_path=chromedriver_path ,options=options)
        browser.maximize_window()
        browser.get(self.link)
        try:
            WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.LQeN7.XWZjwc")))
            accept_button = browser.find_element(By.CSS_SELECTOR, ".VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.LQeN7.XWZjwc")
            accept_button.click()  # Click the accept button for Google cookies and terms
        except Exception as e:
            logging.error("Error accepting cookies:", e)
        return self._selenium_extractor(browser)

    def _selenium_extractor(self, browser):
        prev_length = 0
        logging.info("\nScraping has started. This could take a few minutes. Please do not close the browser window or click the top and move it (the script will stop if you do so).")

        while len(self._get_elements(browser)) < 1000:  # This limits the number of results per page. Google seemingly has a hard limit of 120, but 1000 ensures that it runs smoothly.
            # Acquiring elements to scrape
            logging.info(f"Number of elements: {len(self._get_elements(browser))}")
            var = len(self._get_elements(browser))
            last_element = self._get_elements(browser)[-1]
            browser.execute_script("arguments[0].scrollIntoView();", last_element)
            time.sleep(2)  # Sleep allows time for page to load
            a = self._get_elements(browser)

            try:
                if len(a) == var:
                    self.elementResults += 1
                    if self.elementResults > 20 or len(a) == prev_length:
                        break
                else:
                    self.elementResults = 0
                prev_length = len(a)
            except StaleElementReferenceException:
                continue

Could anybody please clarify my situation and help me to get on the right track? I would be absolutely and very grateful, as this should be done for somebody other than myself. Please let me know if I should write something more or if I have not followed the rules of posting here and I will correct that. This is my first post here, so please bear with me about certain aspects.

Hi Trevor,

You are correct. You’ll have the best luck doing this with a Docker container since our native environments don’t include Selenium and don’t provide the necessary permissions to install additional OS-level dependencies.

Regards,

Matt

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.