The app is web scraping. I was following a few guides and videos, such as this: https://www.youtube.com/watch?v=ne3BH9-5H2o
What I eventually want is that my web app will work in the live website and the scraped data will be automatically downloaded to the user’s machine.
What I have now is that this works perfectly when run locally. I’m really not well versed in this exact thing. It is my first project with Python at all and I’ve tried many suggested solutions from Google and tried my luck with AI chatbots, but not getting anywhere.
The error from running it in a live version of the website is below.
From what I have gathered, I can either switch hosting services (seen PythonAnywhere and Railway.app as suggestions) or I can run my app in a Docker container, which I admit that I didn’t think was necessary, initially. From what I have read, Render does not support Selenium by default, hence the suggestion of Docker.
This is my output from running in a headless browser:
File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.11/subprocess.py", line 1955, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/opt/render/project/src/chromedriver'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/googleMapsScrapingToolweb.py", line 158, in scrape
file_path = scraper.scrape()
^^^^^^^^^^^^^^^^
File "/opt/render/project/src/googleMapsScrapingToolweb.py", line 35, in scrape
browser = webdriver.Chrome(executable_path=chromedriver_path ,options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 90, in __init__
self.service.start()
File "/opt/render/project/src/.venv/lib/python3.11/site-packages/selenium/webdriver/common/service.py", line 86, in start
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable may have wrong permissions. Please see https://chromedriver.chromium.org/home
127.0.0.1 - - [26/Apr/2024:14:49:56 +0000] "POST /scrape HTTP/1.1" 500 290 "https://googlemapsscrapingtoolweb.onrender.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0"
This is the relevant code:
class GoogleMapsScraper:
def __init__(self, link):
self.link = link
self.csv_data = []
self.elementResults = 0
def scrape(self):
chromedriver_path = os.path.abspath("chromedriver")
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-gpu")
browser = webdriver.Chrome(executable_path=chromedriver_path ,options=options)
browser.maximize_window()
browser.get(self.link)
try:
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.LQeN7.XWZjwc")))
accept_button = browser.find_element(By.CSS_SELECTOR, ".VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.LQeN7.XWZjwc")
accept_button.click() # Click the accept button for Google cookies and terms
except Exception as e:
logging.error("Error accepting cookies:", e)
return self._selenium_extractor(browser)
def _selenium_extractor(self, browser):
prev_length = 0
logging.info("\nScraping has started. This could take a few minutes. Please do not close the browser window or click the top and move it (the script will stop if you do so).")
while len(self._get_elements(browser)) < 1000: # This limits the number of results per page. Google seemingly has a hard limit of 120, but 1000 ensures that it runs smoothly.
# Acquiring elements to scrape
logging.info(f"Number of elements: {len(self._get_elements(browser))}")
var = len(self._get_elements(browser))
last_element = self._get_elements(browser)[-1]
browser.execute_script("arguments[0].scrollIntoView();", last_element)
time.sleep(2) # Sleep allows time for page to load
a = self._get_elements(browser)
try:
if len(a) == var:
self.elementResults += 1
if self.elementResults > 20 or len(a) == prev_length:
break
else:
self.elementResults = 0
prev_length = len(a)
except StaleElementReferenceException:
continue
Could anybody please clarify my situation and help me to get on the right track? I would be absolutely and very grateful, as this should be done for somebody other than myself. Please let me know if I should write something more or if I have not followed the rules of posting here and I will correct that. This is my first post here, so please bear with me about certain aspects.