How To Scrape The Latest Stock Prices With Node.js And Puppeteer!

how to scrape the latest stock prices with node js and puppeeteer

First of all, you’ll need to run “npm I puppeteer” to install the puppeteer. Once the puppeteer is installed, create a new JavaScript file and need puppeteer on the first line:

Now develop the async function in which we will write our main code:


const puppeteer = require('puppeteer');


const puppeteer = require('puppeteer');

async function start() {

}
start();

Now we will start scraping by initiating a new browser and also define the URL which your web-scraper will visit.


const puppeteer = require('puppeteer');
const puppeteer = require('puppeteer');

async function start() {
  const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
  const browser = await puppeteer.launch({
    headless: false
  });
}

Then, after using the “newPage()” function, open the new page in the browser and navigate to the url we specified with the “goto()” function:


const puppeteer = require('puppeteer');

async function start() {
  const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto(url);
}

For this next step, you will need to click the current stock price and click on inspect element:

On the right side of your window, a pop-up will open, and you’ll need to locate the stock price element:

Right-click on the share value element and select “copy complete Xpath.” We’ll be able to access the stock price element in this way:

We can now add these three lines of code to our function once we obtain the Xpath of the stock price element:


var element = await page.waitForXPath("put the stock price Xpath here")
  var price = await page.evaluate(element => element.textContent, element);
  console.log(price);

The stock price element will be located using the “page.waitForXPath()” function.

The text contents of the stock price element will then be obtained by the “page.evaluate” method, which will be printed by the “console.log()” function.

Now, code will look something like this:


const puppeteer = require('puppeteer');

async function start() {
  const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto(url);
  var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]")
  var price = await page.evaluate(element => element.textContent, element);
  console.log(price);
}
start()

If you run your current code, you’ll see that when you navigate to the address you specified earlier, a pop-up appears:

To avoid this, add these two lines of code before introducing the “element” variable to your function:


var accept = ("#consent-page > div > div > div > form > div.wizard-body > div.actions.couple > button");
await page.click(accept)

This will locate the “Accept all” button and click it to create the popup go away. Now you will have a working function which will click on defined url, and will scrape the updated Tesla stock price and print in terminal.

moving one step ahead, you can write the below script:


for(var k = 1; k < 2000; k++){
      var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]")
      var price = await page.evaluate(element => element.textContent, element);
      console.log(price);
      await page.waitForTimeout(1000);

Before continuing the for loop, the “page.waitForTimeout(1000)” function will wait 1000 milliseconds (1 second).

Finally, after the for loop, add a “browser.close()” function to exit the browser and conclude your code execution:


for(var k = 1; k < 2000; k++){
const puppeteer = require('puppeteer');

async function start() {
    const url = 'https://finance.yahoo.com/quote/TSLA?p=TSLA&.tsrc=fin-srch';
    const browser = await puppeteer.launch({
      headless: false
    });  
    const page = await browser.newPage();
    await page.goto(url);
    var accept = ("#consent-page > div > div > div > form > div.wizard-body > div.actions.couple > button");
    await page.click(accept);
    for(var k = 1; k < 2000; k++){
      var element = await page.waitForXPath("/html/body/div[1]/div/div/div[1]/div/div[2]/div/div/div[5]/div/div/div/div[3]/div[1]/div[1]/span[1]");
      var price = await page.evaluate(element => element.textContent, element);
      console.log(price);
      await page.waitForTimeout(1000);
    }
    browser.close();
}
start();

For any queries, contact X-Byte Enterprise Crawling Now!

Parth Vataliya