I have simple scraper that works locally, but I’ve been unable to get it working deployed on Render. When run, I get;
INFO -- : Spider: started: duvall
DEBUG -- : BrowserBuilder (selenium_firefox): created browser instance
DEBUG -- : BrowserBuilder (selenium_firefox): enabled before_request.delay
DEBUG -- : Browser: sleep 1 second before request...
DEBUG -- : BrowserBuilder (selenium_firefox): enabled custom user_agent
DEBUG -- : BrowserBuilder (selenium_firefox): enabled native headless_mode
INFO -- : Browser: started get request to: (url)
INFO -- : Browser: finished get request to: (url)
INFO -- : Info: visits: requests: 1, responses: 1
INFO -- : Browser: driver selenium_firefox has been destroyed
FATAL -- : Spider: stopped: {spider_name: "duvall", status: :failed, error: "#<Errno::ENOENT: No such file or directory - lsof>", environment: "development", start_time: 2025-01-10 23:59:38.599903919 +0000, stop_time: 2025-01-11 00:00:02.231782152 +0000, running_time: "23s", visits: {requests: 1, responses: 1}, items: {sent: 0, processed: 0}, events: {requests_errors: {}, drop_items_errors: {}, custom: {}}}
INFO -- : [405d1c37-1ad2-42c5-b2f1-64a59d625878] Completed 500 Internal Server Error in 23633ms (Allocations: 120617)
FATAL -- : [405d1c37-1ad2-42c5-b2f1-64a59d625878]
Where-as on my machine, it goes;
Spider: started: github_spider
DEBUG -- : BrowserBuilder (selenium_firefox): created browser instance
DEBUG -- : BrowserBuilder (selenium_firefox): enabled before_request.delay
DEBUG -- : Browser: sleep 1 second before request...
DEBUG -- : BrowserBuilder (selenium_firefox): enabled custom user_agent
DEBUG -- : BrowserBuilder (selenium_firefox): enabled native headless_mode
INFO -- : Browser: started get request to: (url)
INFO -- : Browser: finished get request to: (url)
INFO -- : Info: visits: requests: 1, responses: 1
DEBUG -- : Browser: driver.current_memory: 1022002
… and continues on from there.
I’ve seen in other help topics that it’s said Docker is the way to go with this, but I’m confused. It looks as if the Selenium browser is launching, but seems to fail when the driver’s memory is checked. FTR, I am running it with a simple form for now as I need to demo it to clients in a few days, but will be running it soon after as an API service. Should I bet looking into how to do this with Docking, or am I missing something simple? Thanks.