Our goal will be to write a script that will save LeBron James' year-over-year career stats. Okay with some preliminary understanding of data formats under our belt, it's time to take a stab at scraping some real data. We'll use as our case study to learn these techniques. In this case, we'll go over a method of intercepting these API requests and work with their JSON payloads directly via a script written in Node.js. Case 1 – Using APIs DirectlyĪ very common flow that web applications use to load their data is to have JavaScript make asynchronous requests ( AJAX) to an API server (typically REST or GraphQL) and receive their data back in JSON format, which then gets rendered to the screen. Learning to read and understand this format will go a long way to helping you work with data on the web. Note these instructions were written with Chrome 78 and will likely vary slightly with different browsers. So without further adieu, let's begin with a quick primer on CSV vs JSON. We'll even try out curl and jq on the command line for a bit.
USING A WEBSCRAPER JAVASCRIPT HOW TO
I'll go through the way I investigate what is rendered on the page to figure out what to scrape, how to search through network requests to find relevant API calls, and how to automate the scraping process through scripts written in Node.js.
There are several different ways to scrape, each with their own advantages and disadvantages and I'm going to cover three of them in this article:įor each of these three cases, I'll use real websites as examples (, , and respectively) to help ground the process.
Whether you're a student, researcher, journalist, or just plain interested in some data you've found on the internet, it can be really handy to know how to automatically save this data for later analysis, a process commonly known as "scraping".
0 kommentar(er)
