用chrome headless和selenium进行下载

Question 1

我正在使用python-selenium和Chrome 59，并试图将一个简单的下载序列自动化。当我正常启动浏览器时，下载正常，但当我在无头模式下这样做时，下载就不工作了。

# Headless implementation
from selenium import webdriver
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("headless")
driver = webdriver.Chrome(chrome_options=chromeOptions)
driver.get('https://www.mockaroo.com/')
driver.find_element_by_id('download').click()
# ^^^ Download doesn't start
prefs = {"download.default_directory" : "/Users/Chetan/Desktop/"}
chromeOptions.add_argument("headless")
chromeOptions.add_experimental_option("prefs",prefs)
添加默认路径在正常实现中是可行的，但在无头版本中同样的问题依然存在。
我如何让下载在无头模式下开始？

Question 2


          
           
            是的，这是一个 "功能"，为了安全。如前所述，这里是错误讨论。
            
             https://bugs.chromium.org/p/chromium/issues/detail?id=696481
            
           
           
            在chrome 62.0.3196.0或以上版本中增加了支持，以实现下载。
           
           
            这里有一个python的实现。我不得不把这个命令添加到chromedriver的命令中。我将尝试提交一份PR，以便将来把它包含在库中。
           
           def enable_download_in_headless_chrome(self, driver, download_dir):
    # add missing support for chrome "send_command"  to selenium webdriver
    driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
    params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
    command_result = driver.execute("send_command", params)
作为参考，这里有一个小 repo 来演示如何使用。
https://github.com/shawnbutton/PythonHeadlessChrome
2020-05-01更新有评论说，这已经不起作用了。鉴于这个补丁已经有一年多了，他们很可能已经改变了底层库。

Question 3


          
           
            
             下面是一个基于Python的工作实例
             
              肖恩-巴顿的回答
             
             . I've tested this with
             
              Chromium 68.0.3440.75
             
             &
             
              chromedriver 2.38
             
            
            from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("prefs", {
  "download.default_directory": "/path/to/download/dir",
  "download.prompt_for_download": False,
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': "/path/to/download/dir"}}
command_result = driver.execute("send_command", params)
driver.get('http://download-page.url/')
driver.find_element_by_css_selector("#download_link").click()

Question 4


          
           
            
             
              这是Chrome浏览器的一项功能，以防止软件下载文件到你的电脑。不过有一个解决方法。
              
               在此阅读更多信息
              
              .
             
             
              你需要做的是通过DevTools启用它，类似这样的事情。
             
             async function setDownload () {
  const client = await CDP({tab: 'ws://localhost:9222/devtools/browser'});
  const info =  await client.send('Browser.setDownloadBehavior', {behavior : "allow", downloadPath: "/tmp/"});
  await client.close();
这就是有人在上述主题中给出的解决方案。Here is his comment.

Question 5


          
           
            
             
              
               更新的Python解决方案 -
2021年3月4日在chromedriver v88和v89上测试。
              
              
               这将允许你在无头模式下点击下载文件。
              
                  from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.options import Options
    # Instantiate headless driver
    chrome_options = Options()
    # Windows path
    chromedriver_location = 'C:\\path\\to\\chromedriver_win32\\chromedriver.exe'
    # Mac path. May have to allow chromedriver developer in os system prefs
    '/Users/path/to/chromedriver'
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_prefs = {"download.default_directory": r"C:\path\to\Downloads"} # (windows)
    chrome_options.experimental_options["prefs"] = chrome_prefs
    driver = webdriver.Chrome(chromedriver_location,options=chrome_options)
    # Download your file
    driver.get('https://www.mockaroo.com/')
    driver.find_element_by_id('download').click()

Question 6


          
           
            
             
              
               
                也许你处理的网站为浏览器返回不同的HTML页面，意味着你想要的XPath或Id在无头浏览器中可能不同。
试着在无头浏览器中下载pageSource，并将其作为HTML页面打开，看看你想要的Id或XPath。
你可以看看这个c#例子
                
                 如何在PhantomDriver中隐藏FirefoxDriver（使用Selenium）而不出现findElement函数错误？
                
                .

Question 7


          
           
            
             
              
               
                通常情况下，看到同样的东西用另一种语言写出来是多余的，但因为这个问题把我逼疯了，我希望我可以把别人从痛苦中拯救出来......所以这里是C#版本的
                
                 肖恩-巴顿的回答
                
                (用headless chrome=71.0.3578.98, chromedriver=2.45.615279, platform=Linux 4.9.125-linuxkit x86_64测试)。
               
                           var enableDownloadCommandParameters = new Dictionary<string, object>
                { "behavior", "allow" },
                { "downloadPath", downloadDirectoryPath }
            var result = ((OpenQA.Selenium.Chrome.ChromeDriver)driver).ExecuteChromeCommandWithResult("Page.setDownloadBehavior", enableDownloadCommandParameters);

Question 8


          
           
            
             
              
               
                
                 一个使用selenium-cucumber-js / selenium-webdriver的JavaScript的完整工作实例。
                
                const chromedriver = require('chromedriver');
const selenium = require('selenium-webdriver');
const command = require('selenium-webdriver/lib/command');
const chrome = require('selenium-webdriver/chrome');
module.exports = function() {
  const chromeOptions = new chrome.Options()
    .addArguments('--no-sandbox', '--headless', '--start-maximized', '--ignore-certificate-errors')
    .setUserPreferences({
      'profile.default_content_settings.popups': 0, // disable download file dialog
      'download.default_directory': '/tmp/downloads', // default file download location
      "download.prompt_for_download": false,
      'download.directory_upgrade': true,
      'safebrowsing.enabled': false,
      'plugins.always_open_pdf_externally': true,
      'plugins.plugins_disabled': ["Chrome PDF Viewer"]
    .windowSize({width: 1600, height: 1200});
  const driver = new selenium.Builder()
    .withCapabilities({
      browserName: 'chrome',
      javascriptEnabled: true,
      acceptSslCerts: true,
      path: chromedriver.path
    .setChromeOptions(chromeOptions)
    .build();
  driver.manage().window().maximize();
  driver.getSession()
    .then(session => {
      const cmd = new command.Command("SEND_COMMAND")
        .setParameter("cmd", "Page.setDownloadBehavior")
        .setParameter("params", {'behavior': 'allow', 'downloadPath': '/tmp/downloads'});
      driver.getExecutor().defineCommand("SEND_COMMAND", "POST", `/session/${session.getId()}/chromium/send_command`);
      return driver.execute(cmd);
  return driver;
关键的部分是。
  driver.getSession()
    .then(session => {
      const cmd = new command.Command("SEND_COMMAND")
        .setParameter("cmd", "Page.setDownloadBehavior")
        .setParameter("params", {'behavior': 'allow', 'downloadPath': '/tmp/downloads'});
      driver.getExecutor().defineCommand("SEND_COMMAND", "POST", `/session/${session.getId()}/chromium/send_command`);
      return driver.execute(cmd);
Tested with:
Chrome 67.0.3396.99
Chromedriver 2.36.540469
selenium-cucumber-js 1.5.12
selenium-webdriver 3.0.0

Question 9


          
           
            
             
              
               
                
                 
                  
                   以下是在Java、selenium、chromedriver和chrome v 71.x中的等效代码，其中的代码是允许保存下载的关键
额外的罐子：com.fasterxml.jackson.core，com.fasterxml.jackson.annotation，com.fasterxml.jackson.databind
                  
                  
                   System.setProperty("webdriver.chrome.driver", "C:\libraries\chromedriver.exe"）。
                  
                              String downloadFilepath = "C:\\Download";
            HashMap<String, Object> chromePreferences = new HashMap<String, Object>();
            chromePreferences.put("profile.default_content_settings.popups", 0);
            chromePreferences.put("download.prompt_for_download", "false");
            chromePreferences.put("download.default_directory", downloadFilepath);
            ChromeOptions chromeOptions = new ChromeOptions();
            chromeOptions.setBinary("C:\\pathto\\Chrome SxS\\Application\\chrome.exe");
            //ChromeOptions options = new ChromeOptions();
            //chromeOptions.setExperimentalOption("prefs", chromePreferences);
            chromeOptions.addArguments("start-maximized");
            chromeOptions.addArguments("disable-infobars");
            //HEADLESS CHROME
            **chromeOptions.addArguments("headless");**
            chromeOptions.setExperimentalOption("prefs", chromePreferences);
            DesiredCapabilities cap = DesiredCapabilities.chrome();
            cap.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
            cap.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
            **ChromeDriverService driverService = ChromeDriverService.createDefaultService();
            ChromeDriver driver = new ChromeDriver(driverService, chromeOptions);
            Map<String, Object> commandParams = new HashMap<>();
            commandParams.put("cmd", "Page.setDownloadBehavior");
            Map<String, String> params = new HashMap<>();
            params.put("behavior", "allow");
            params.put("downloadPath", downloadFilepath);
            commandParams.put("params", params);
            ObjectMapper objectMapper = new ObjectMapper();
            HttpClient httpClient = HttpClientBuilder.create().build();
            String command = objectMapper.writeValueAsString(commandParams);
            String u = driverService.getUrl().toString() + "/session/" + driver.getSessionId() + "/chromium/send_command";
            HttpPost request = new HttpPost(u);
            request.addHeader("content-type", "application/json");
            request.setEntity(new StringEntity(command));**
            try {
                httpClient.execute(request);
            } catch (IOException e2) {
                // TODO Auto-generated catch block
                e2.printStackTrace();
        //Continue using the driver for automation  
    driver.manage().window().maximize();

Question 10


          
           
            
             
              
               
                
                 
                  
                   
                    我通过使用@Shawn Button分享的变通方法和使用
                    
                     完整路径
                    
                    为 "downloadPath "参数。使用一个
                    
                     相对路径
                    
                    没有工作，给我带来了错误。
                   
                   
                    Versions:
                    

                    Chrome版本75.0.3770.100（官方版本）（32位）
                    

                    ChromeDriver 75.0.3770.90

Question 11


          
           
            
             
              
               
                
                 
                  
                   
                    Using:
                    
                     google-chrome-stable amd64 86.0.4240.111-1
                    
                    ,
                    
                     chromedriver 86.0.4240.22
                    
                    ,
                    
                     selenium 3.141.0
                    
                    
                     python 3.8.3
                    
                   
                   
                    尝试了多种建议的解决方案，没有任何东西真正对chrome headless起作用，同时我的测试网站打开了一个新的空白标签，然后数据被下载。
                   
                   
                    最后放弃了无头，实施了
                    
                     视觉效果
                    
                    和
                    
                     xvfd
                    
                    来模拟X服务器，类似的东西。
                   
                   from selenium.webdriver.chrome.options import Options # and other imports
import selenium.webdriver as webdriver
import tempfile
url = "https://really_badly_programmed_website.org"
tmp_dir = tempfile.mkdtemp(prefix="hamster_")
driver_path="/usr/bin/chromedriver"
chrome_options = Options() 
chrome_options.binary_location = "/usr/bin/google-chrome"
prefs = {'download.default_directory': tmp_dir,}
chrome_options.add_experimental_option("prefs", prefs)
with Display(backend="xvfb",size=(1920,1080),color_depth=24) as disp:
    driver = webdriver.Chrome(options=chrome_options, executable_path=driver_path)
    driver.get(url)
最后，一切都成功了，并且在临时文件夹上有了下载文件。

Question 12