Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
We’re sorry but our site doesn’t work properly without JavaScript enabled. Please enable it to continue error using Selenium and Python

Ask Question
I want to build a web scraper for a builtin website.
I run the following code in jupyter notebook:
driver = webdriver.PhantomJS(executable_path="C:/Users/Downloads/phantomjs-2.1.1-windows/bin/phantomjs.exe")
driver.get("https://builtin.com/jobs")
html = driver.page_source
And I got
<noscript>
      &lt;strong&gt;
        We’re sorry but our site doesn’t work properly without JavaScript enabled. Please enable it to continue.
      &lt;/strong&gt;
    </noscript>
instead of job openings.
I tried to use webdriver.Chrome() and BeautifulSoup, I always get the same result with this website, but everything works with others.
I've run out of ideas on how to fix it. What could be the problem?
Javascript is enabled in my web browser.
I tried the code that was advised to me. It didn't work for me
  <noscript>
    <strong>
      We’re sorry but our site doesn’t work properly without JavaScript enabled. Please enable it to continue.
    </strong>
  </noscript>
...implies that possibly the co-operating useragent informed the document that it is controlled by WebDriver and subsequently an alternate code paths was be triggered during automation.
Solution
To evade the detection of Selenium driven ChromeDriver initiated google-chrome Browsing Context you can use the following argument:
--disable-blink-features=AutomationControlled
selenium4 compatible code
Your optimum code block will be:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://builtin.com/jobs")
print(driver.page_source)
driver.quit()
Console Output:
<span data-v-96c85cba="" class="sr-only">Open Search</span></button> <div data-v-96c85cba="" class="menu-item"><div data-v-11f5f671="" data-v-96c85cba=""><a data-v-fa58c90a="" data-v-11f5f671="" href="/premium/membership" class="employers">
    For Employers
  </a></div></div> <div data-v-96c85cba=""><div data-v-66b1a49a="" data-v-96c85cba="" class="profile-auth"><button data-v-66b1a49a="" class="b-oauth signup">
  </button> <span data-v-66b1a49a="" class="spl national"></span> <button data-v-66b1a49a="" class="b-oauth login">
    Log In
  </button></div></div></div></div></div></div></div> <div data-v-04d4c7c3="" class="menu-wrapper"><div data-v-04d4c7c3="" class="menu menu-desktop navigation-secondary"><ul data-v-04d4c7c3="" class="menu-container grid d-flex"><li data-v-04d4c7c3="" tabindex="0" class="menu-item"><a data-v-fa58c90a="" data-v-04d4c7c3="" href="/jobs" aria-current="page" class="menu-link nuxt-link-exact-active nuxt-link-active">
      <svg data-v-04d4c7c3="" aria-hidden="true" focusable="false" data-prefix="fas" data-icon="chevron-down" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="fa-icon svg-inline--fa fa-chevron-down fa-w-14"><path data-v-04d4c7c3="" fill="currentColor" d="M207.029 381.476L12.686 187.132c-9.373-9.373-9.373-24.569 0-33.941l22.667-22.667c9.357-9.357 24.522-9.375 33.901-.04L224 284.505l154.745-154.021c9.379-9.335 24.544-9.317 33.901.04l22.667 22.667c9.373 9.373 9.373 24.569 0 33.941L240.971 381.476c-9.373 9.372-24.569 9.372-33.942 0z" class=""></path></svg></a></li> <li data-v-04d4c7c3="" class="menu-item"><a data-v-fa58c90a="" data-v-04d4c7c3="" href="/companies" class="menu-link">Tech Companies</a></li> <li data-v-04d4c7c3="" class="menu-item"><a data-v-fa58c90a="" data-v-04d4c7c3="" href="/remote" class="menu-link">
      Remote
    </a></li> <li data-v-04d4c7c3="" tabindex="0" class="menu-item"><a data-v-fa58c90a="" data-v-04d4c7c3="" href="/tech-topics" class="menu-link">Tech Topics
      <svg data-v-04d4c7c3="" data-v-fa58c90a="" aria-hidden="true" focusable="false" data-prefix="fas" data-icon="chevron-down" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="fa-icon svg-inline--fa fa-chevron-down fa-w-14"><path data-v-04d4c7c3="" data-v-fa58c90a="" fill="currentColor" d="M207.029 381.476L12.686 187.132c-9.373-9.373-9.373-24.569 0-33.941l22.667-22.667c9.357-9.357 24.522-9.375 33.901-.04L224 284.505l154.745-154.021c9.379-9.335 24.544-9.317 33.901.04l22.667 22.667c9.373 9.373 9.373 24.569 0 33.941L240.971 381.476c-9.373 9.372-24.569 9.372-33.942 0z" class=""></path></svg></a></li> <li data-v-04d4c7c3="" tabindex="0" class="menu-item"><a data-v-fa58c90a="" data-v-04d4c7c3="" href="/salaries" class="menu-link">
      Salaries
      <svg data-v-04d4c7c3="" data-v-fa58c90a="" aria-hidden="true" focusable="false" data-prefix="fas" data-icon="chevron-down" role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="fa-icon svg-inline--fa fa-chevron-down fa-w-14"><path data-v-04d4c7c3="" data-v-fa58c90a="" fill="currentColor" d="M207.029 381.476L12.686 187.132c-9.373-9.373-9.373-24.569 0-33.941l22.667-22.667c9.357-9.357 24.522-9.375 33.901-.04L224 284.505l154.745-154.021c9.379-9.335 24.544-9.317 33.901.04l22.667 22.667c9.373 9.373 9.373 24.569 0 33.941L240.971 381.476c-9.373 9.372-24.569 9.372-33.942 0z" class=""></path></svg></a></li> <li data-v-04d4c7c3="" class="menu-item push-right">
References
You can find a couple of relevant detailed discussion in:
Evade detection of selenium automation
How to evade blocking by Walmart using Selenium Webdriver
Selenium can't open a second page
                thank you for your advice. Unfortunately, it didn't work for me (I added a screenshot to my post).
– rndm
                Dec 2, 2021 at 14:30
                @rndm The message will be there but the the HTML would be captured for sure with the mentioned option.
– undetected Selenium
                Dec 2, 2021 at 14:43
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.