from selenium import webdriver

from webdriver_manager.chrome import ChromeDriverManager

from selenium.webdriver.common.by import By

from selenium.webdriver.chrome.options import Options

drivers = webdriver.Chrome(ChromeDriverManager().install())

drivers.get("https://www.google.co.in/")

drivers.find_element(By.NAME, "q").send_keys("Prabhakar Kumar Jha")

Error message

WebDriverException: Message: Service /root/.wdm/drivers/chromedriver/linux64/103.0.5060/chromedriver unexpectedly exited. Status code was: 127

Hi @Prabhakar Jha​

In the clusters library tab, please install PyPi chromedriver-binary==83.0 (or higher, probably version in the script can also be updated)

To cluster init script, please add /databricks/scripts/ selenium-install.sh , which you create using the below code.

Then, in the databricks notebook code, please use something similar to the second code below.

dbutils.fs.mkdirs("dbfs:/databricks/scripts/")
dbutils.fs.put("/databricks/scripts/selenium-install.sh","""
#!/bin/bash
apt-get update
apt-get install chromium-browser=91.0.4472.101-0ubuntu0.18.04.1 --yes
wget https://chromedriver.storage.googleapis.com/91.0.4472.101/chromedriver_linux64.zip -O /tmp/chromedriver.zip
mkdir /tmp/chromedriver
unzip /tmp/chromedriver.zip -d /tmp/chromedriver/
""", True)
display(dbutils.fs.ls("dbfs:/databricks/scripts/"))

from selenium import webdriver
chrome_driver = '/tmp/chromedriver/chromedriver'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
# chrome_options.add_argument('--disable-dev-shm-usage') 
chrome_options.add_argument('--homedir=/dbfs/tmp')
chrome_options.add_argument('--user-data-dir=/dbfs/selenium')
# prefs = {"download.default_directory":"/dbfs/tmp",
#          "download.prompt_for_download":False
# chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(executable_path=chrome_driver, options=chrome_options)

Hi Hubert,

Resolve that issue, but I got a new issue after this:

from selenium import webdriver

chrome_driver = '/tmp/chromedriver/chromedriver'

chrome_options = webdriver.ChromeOptions()

chrome_options.add_argument('--no-sandbox')

chrome_options.add_argument('--headless')

chrome_options.add_argument('--disable-dev-shm-usage')

chrome_options.add_argument('--homedir=/dbfs/tmp')

chrome_options.add_argument('--user-data-dir=/dbfs/selenium')

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

driver.get('https://www.google.com/')

But the error is:

Message: unknown error: Chrome failed to start: exited abnormally.

(unknown error: DevToolsActivePort file doesn't exist)

(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Selenium chrome driver on databricks driver On the databricks community, I see repeated problems regarding the selenium installation on the databricks... in Data Engineering 11-09-2022 Errors Using Selenium/Chromedriver in DataBricks in Data Engineering 11-01-2022 Failed to fetch archive.ubuntu in Data Engineering 10-18-2022 Using Selenium Chrome Driver in Databricks, runs the first time but fails after that in Data Engineering 08-08-2022 © Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.

Privacy Notice

| Terms of Use

| Your Privacy Choices

| Your California Privacy Rights