When using Google Chrome in Headless mode to scrape websites it can often be faster to prototype interaction and data extraction commands via Developer tools. Here is a quick tutorial showing how to get started with talking to your NodeJS script.
Firstly create a script such as app.js and ensure you have the puppeteer module installed via npm.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: true, args: ['--disable-gpu'] });
const page = await browser.newPage();
// expose any variables you want to inspect to the global space
Object.assign(global, { browser, page });
// wait forever here
await browser.waitForFunction('false', { timeout: 0 });
await browser.close();
})();
On your desktop, open Google Chrome and in a new tab open chrome://inspect, then click on the Open dedicated DevTools for Node link to open a new developer tools window.
Once the dedicated developer tools window is open, confirm the IP you are using. If you are using a different IP than localhost, edit the connection box and correct the entry otherwise the default will be fine.
Now run the NodeJS script with the –inspect flag and confirm the script runs with no errors. If there are errors such as missing dependencies you will need to fix these first.
node --inspect app.js
Debugger listening on ws://127.0.0.1:9229/cc8fd8e0-2c02-4a6c-84f2-0ee9d1c5d1b6
For help see https://nodejs.org/en/docs/inspector
You may now interact with the variables browser and page
Once you see the console message, switch back to the developer tools window and you may interact with the page variable using the standard API. This can be useful for running page.goto or other commands to test data extraction.
As puppeteer returns Promises for most functions be sure to use the await command to perform actions relying on the return value.
For a comprehensive list of puppeteer interaction commands just consult the API documentation.