Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I submit a form using the following code and i want Puppeteer to wait page load after form submit.
await page.click("button[type=submit]");
//how to wait until the new page loads before taking screenshot?
// i don't want this:
// await page.waitFor(1*1000); //← unwanted workaround
await page.screenshot({path: 'example.png'});
How to wait for page load with puppeteer?
You can wait for navigation asynchronously to avoid getting null on redirection,
await Promise.all([
page.click('button[type=submit]'),
page.waitForNavigation({waitUntil: 'networkidle2'})
This will help you if the page.click already triggers a navigation.
–
–
–
timeout <number> Maximum navigation time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. The default value can be changed by using the page.setDefaultNavigationTimeout(timeout) method.
waitUntil <string|Array<string>> When to consider navigation succeeded, defaults to load. Given an array of event strings, navigation is considered to be successful after all events have been fired. Events can be either:
load - consider navigation to be finished when the load event is fired.
domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
returns: <Promise<[?Response]>> Promise which resolves to the main resource response. In case of multiple redirects, the navigation will resolve with the response of the last redirect. In case of navigation to a different anchor or navigation due to History API usage, the navigation will resolve with null.
Readability:
You can use page.waitForNavigation() to wait for a page to navigate:
await page.waitForNavigation();
Performance:
But since page.waitForNavigation() is a shortcut for page.mainFrame().waitForNavigation(), we can use the following for a minor performance enhancement:
await page._frameManager._mainFrame.waitForNavigation();
–
–
Sometimes even using await page.waitForNavigation() will still result in a Error: Execution context was destroyed, most likely because of a navigation.
In my case, it was because the page was redirecting multiple times. The API says the default waitUntil option is Load—this required me to wait for navigation each redirect (3 times).
Using only a single instance of page.waitForNavigation with the waitUntil option networkidle2 worked well in my case:
await button.click();
await page.waitForNavigation({waitUntil: 'networkidle2'});
Finally, the API suggests using a Promise.All to prevent a race condition. I haven't needed this but provide it for completeness:
await Promise.all([button.click(), page.waitForNavigation({waitUntil:'networkidle2'})])
If all else fails, you can use page.waitForSelector as recommended on a Puppeteer github issue—or in my case, page.waitForXPath()
–
–
I know it is bit late to answer this. It may be helpful for those who are getting below exception while doing waitForNavigation.
(node:14531) UnhandledPromiseRejectionWarning: TimeoutError:
Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/user/nodejs/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21)
at -- ASYNC --
at Frame. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:111:15)
at Page.waitForNavigation (/home/user/nodejs/node_modules/puppeteer/lib/Page.js:649:49)
at Page. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:112:23)
at /home/user/nodejs/user/puppeteer/example7.js:14:12
The correct code that worked for me is as below.
await page.click('button[id=start]', {waitUntil: 'domcontentloaded'});
Similarly if you are going to a new page, code should be like
await page.goto('here goes url', {waitUntil: 'domcontentloaded'});
await page.goto(url, {
timeout: 20000,
waitUntil: ['load', 'domcontentloaded', 'networkidle0', 'networkidle2']
} catch (error) {
throw new Error("url " + url + " url not loaded -> " + error)
now you can use this with
await loadUrl(page, "https://www.google.com")
None of the above answers solved my issue. Sometimes waitForNavigation just timeout. I came up with other solution using the waitForFunction, checking if document is in ready state.
await page.waitForFunction(() => document.readyState === "complete");
await Promise.all([
page.click(selectors.submit),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
This would be the first priority to use as it waits for all network to complete and assumes it is done when you don't have more than 0 network call for 500ms.
you can also use
await page.waitForNavigation({ waitUntil: 'load' })
or else, you can use
await page.waitForResponse(response => response.ok())
this function can also be used in various places as it only allows to proceed further when all the calls are a success that is when all the response status is ok i.e (200-299)
–
Note:
If you do this inside a loop. ( scrapping page-1, click to page-2, scrapping page-2 and so on... )
await page.waitForSelector(".clickable-selector", { visible: true });
Wait for this clickable selector before doing any other scrapping on the page.
For some reason I was not able to click button (Handled an event, not in form)
<button onclick="someFunction();" class="button button2">Submit</button>
The problem was that page was rendered on server side. Thus the button didn't existed whenever I waited for input field await page.waitForSelector('button.button2')
The solution was to bind page.goto(URL) and page.waitForNavigation({ waitUntil: 'networkidle0' }) in Promise
await Promise.all([
page.goto(URL),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
console.log('page loaded')
await page.waitForSelector('button.button2')
console.log('button is here');
–
If submitting the form opens some other page, then you may just want to wait for a selector in that page. I have often had issues using page.waitForNavigation() since it's options don't really ensure we have effectively navigated to another page.
// login page
page.click("#login");
// homepage, after login
page.waitForSelector("#home", {visible: true}); // page.waitForXpath()
Of you course you can increase the wait time for the selector.
I ran into a scenario, where there was the classic POST-303-GET and an input[type=submit] was involved. It seems that in this case, the click of the button won't resolve until after the associated form's submission and redirection, so the solution was to remove the waitForNavigation, because it was executed after the redirection and thus was timing out.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
