diff --git a/sources/academy/platform/expert_scraping_with_apify/actors_webhooks.md b/sources/academy/platform/expert_scraping_with_apify/actors_webhooks.md index c5c7d1d4a8..e0e7c64691 100644 --- a/sources/academy/platform/expert_scraping_with_apify/actors_webhooks.md +++ b/sources/academy/platform/expert_scraping_with_apify/actors_webhooks.md @@ -20,7 +20,7 @@ Thus far, you've run Actors on the platform and written an Actor of your own, wh ## Advanced Actor overview {#advanced-actors} -In this course, we'll be working out of the Amazon scraper project from the **Web scraping basics for JavaScript devs** course. If you haven't already built that project, you can do it in [three short lessons](../../webscraping/scraping_basics_legacy/challenge/index.md). We've made a few small modifications to the project with the Apify SDK, but 99% of the code is still the same. +In this course, we'll be working out of the Amazon scraper built in the old [Web scraping basics for JavaScript devs](../../webscraping/scraping_basics_legacy/challenge/index.md) course (not the [current scraping basics course](../../webscraping/scraping_basics_javascript/index.md)). If you haven't gone through it yet, we recommend doing so - it covers the fundamentals this project is built on. If you'd rather skip straight to this course, you can use this working implementation instead: [academy-amazon-scraper](https://github.com/apify-projects/academy-amazon-scraper). Take another look at the files within your Amazon scraper project. You'll notice that there is a **Dockerfile**. Every single Actor has a Dockerfile (the Actor's **Image**) which tells Docker how to spin up a container on the Apify platform which can successfully run the Actor's code. "Apify Actors" is a serverless platform that runs multiple Docker containers. For a deeper understanding of Actor Dockerfiles, refer to the [Apify Actor Dockerfile docs](/sdk/js/docs/guides/docker-images#example-dockerfile). @@ -46,7 +46,7 @@ Prior to moving forward, please read over these resources: ## Our task {#our-task} -In this task, we'll be building on top of what we already created in the [Web scraping basics for JavaScript devs](../../webscraping/scraping_basics_legacy/challenge/index.md) course's final challenge, so keep those files safe! +In this task, we'll be building on top of the [Amazon scraper project mentioned above](#advanced-actors). Once our Amazon Actor has completed its run, we will, rather than sending an email to ourselves, call an Actor through a webhook. The Actor called will be a new Actor that we will create together, which will take the dataset ID as input, then subsequently filter through all of the results and return only the cheapest one for each product. All of the results of the Actor will be pushed to its default dataset. diff --git a/sources/academy/webscraping/scraping_basics_javascript/03_devtools_extracting_data.md b/sources/academy/webscraping/scraping_basics_javascript/03_devtools_extracting_data.md index e774b7e1d7..da431dfc19 100644 --- a/sources/academy/webscraping/scraping_basics_javascript/03_devtools_extracting_data.md +++ b/sources/academy/webscraping/scraping_basics_javascript/03_devtools_extracting_data.md @@ -81,7 +81,7 @@ In the next lesson, we'll start with our Node.js project. First we'll be figurin ### Extract the price of IKEA's most expensive artificial plant -At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/artificial-plants-flowers-20492/), use CSS selectors and HTML elements manipulation in the **Console** to extract the price of the most expensive artificial plant (sold in Sweden, as you'll be browsing their Swedish offer). Before opening DevTools, use your judgment to adjust the page to make the task as straightforward as possible. Finally, use the [`parseInt()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt) function to convert the price text into a number. +At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/artificial-plants-flowers-20492/), use CSS selectors and HTML elements manipulation in the **Console** to extract the price of the most expensive artificial plant (sold in Sweden, as you'll be browsing their Swedish offer). Before opening DevTools, use your judgment to adjust the page to make the task as straightforward as possible. Finally, use the [`parseInt()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt) function to convert the price text into a number (you may need [`replace()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace) to handle spaces).
Solution @@ -93,8 +93,8 @@ At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/a 1. Notice that the price is structured into two elements, with the integer separated from the currency, under a class named `plp-price__integer`. This structure is convenient for extracting the value. 1. In the **Console**, execute `document.querySelector('.plp-price__integer')`. This returns the element representing the first price in the listing. Since `document.querySelector()` returns the first matching element, it directly selects the most expensive plant's price. 1. Save the element in a variable by executing `price = document.querySelector('.plp-price__integer')`. - 1. Convert the price text into a number by executing `parseInt(price.textContent)`. - 1. At the time of writing, this returns `699`, meaning [699 SEK](https://www.google.com/search?q=699%20sek). + 1. Convert the price text into a number by executing `parseInt(price.textContent.replace(' ', ''))`. The price text contains spaces as thousand separators, so `.replace(' ', '')` strips them before parsing - a technique you'll explore further in the [Extracting data](./07_extracting_data.md#removing-dollar-sign-and-commas) lesson. + 1. At the time of writing, this returns `1299`, meaning [1 299 SEK](https://www.google.com/search?q=1299%20sek).
@@ -119,7 +119,7 @@ On Fandom's [Movies page](https://www.fandom.com/topics/movies), use CSS selecto ### Extract details about the first post on Guardian's F1 news -On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), use CSS selectors and HTML manipulation in the **Console** to extract details about the first post. Specifically, extract its title, lead paragraph, and URL of the associated photo. +On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), use CSS selectors and HTML manipulation in the **Console** to extract details about the first post. Specifically, extract its title, lead paragraph (if it has one), and URL of the associated photo. ![F1 news page](../scraping_basics/images/devtools-exercise-guardian2.png) @@ -132,7 +132,7 @@ On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), 1. Notice that the markup does not provide clear, reusable class names for this task. The structure uses generic tag names and randomized classes, requiring you to rely on the element hierarchy and order instead. 1. In the **Console**, execute `post = document.querySelector('#maincontent ul li')`. This returns the element representing the first post. 1. Extract the post's title by executing `post.querySelector('h3').textContent`. - 1. Extract the lead paragraph by executing `post.querySelector('span div').textContent`. + 1. Extract the lead paragraph (if it has one) by executing `post.querySelector('span div').textContent`. 1. Extract the photo URL by executing `post.querySelector('img').src`. diff --git a/sources/academy/webscraping/scraping_basics_javascript/06_locating_elements.md b/sources/academy/webscraping/scraping_basics_javascript/06_locating_elements.md index fedd418abd..b4d1f8124f 100644 --- a/sources/academy/webscraping/scraping_basics_javascript/06_locating_elements.md +++ b/sources/academy/webscraping/scraping_basics_javascript/06_locating_elements.md @@ -39,7 +39,7 @@ if (response.ok) { } ``` -Calling [`toArray()`](https://cheerio.js.org/docs/api/classes/Cheerio#toarray) converts the Cheerio selection to a standard JavaScript array. We can then loop over that array and process each selected element. +Calling [`toArray()`](https://cheerio.js.org/docs/api/classes/cheerio#toarray) converts the Cheerio selection to a standard JavaScript array. We can then loop over that array and process each selected element. Cheerio requires us to wrap each element with `$()` again before we can work with it further, and then we call `.text()`. If we run the code, it… well, it definitely prints _something_… @@ -136,7 +136,7 @@ When translated to a tree of JavaScript objects, the element with class `price` - a `span` HTML element, - a textual node representing the actual amount and possibly also white space. -We can use Cheerio's [`.contents()`](https://cheerio.js.org/docs/api/classes/Cheerio#contents) method to access individual nodes. It returns a list of nodes like this: +We can use Cheerio's [`.contents()`](https://cheerio.js.org/docs/api/classes/cheerio#contents) method to access individual nodes. It returns a list of nodes like this: ```text LoadedCheerio { @@ -197,7 +197,7 @@ if (response.ok) { } ``` -We're enjoying the fact that Cheerio selections provide utility methods for accessing items, such as [`.first()`](https://cheerio.js.org/docs/api/classes/Cheerio#first) or [`.last()`](https://cheerio.js.org/docs/api/classes/Cheerio#last). If we run the scraper now, it should print prices as only amounts: +We're enjoying the fact that Cheerio selections provide utility methods for accessing items, such as [`.first()`](https://cheerio.js.org/docs/api/classes/cheerio#first) or [`.last()`](https://cheerio.js.org/docs/api/classes/cheerio#last). If we run the scraper now, it should print prices as only amounts: ```text $ node index.js @@ -237,7 +237,7 @@ Macao, China :::tip Need a nudge? -You may want to check out Cheerio's [`.eq()`](https://cheerio.js.org/docs/api/classes/Cheerio#eq). +You may want to check out Cheerio's [`.eq()`](https://cheerio.js.org/docs/api/classes/cheerio#eq). ::: diff --git a/sources/academy/webscraping/scraping_basics_javascript/07_extracting_data.md b/sources/academy/webscraping/scraping_basics_javascript/07_extracting_data.md index 74b440b4bc..d728da6954 100644 --- a/sources/academy/webscraping/scraping_basics_javascript/07_extracting_data.md +++ b/sources/academy/webscraping/scraping_basics_javascript/07_extracting_data.md @@ -290,7 +290,7 @@ Hamilton reveals distress over ‘devastating’ groundhog accident at Canadian :::tip Need a nudge? - HTML's `time` element can have an attribute `datetime`, which [contains data in a machine-readable format](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/time), such as the ISO 8601. -- Cheerio gives you [.attr()](https://cheerio.js.org/docs/api/classes/Cheerio#attr) to access attributes. +- Cheerio gives you [.attr()](https://cheerio.js.org/docs/api/classes/cheerio#attr) to access attributes. - In JavaScript you can use an ISO 8601 string to create a [`Date`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date) object. - To get the date, you can call `.toDateString()` on `Date` objects. diff --git a/sources/academy/webscraping/scraping_basics_legacy/crawling/finding_links.md b/sources/academy/webscraping/scraping_basics_legacy/crawling/finding_links.md index d185e8b01a..3b9c03aad0 100644 --- a/sources/academy/webscraping/scraping_basics_legacy/crawling/finding_links.md +++ b/sources/academy/webscraping/scraping_basics_legacy/crawling/finding_links.md @@ -57,7 +57,7 @@ We'll start from a boilerplate that's very similar to the scraper we built in [B {Example} -Aside from importing libraries and downloading HTML, we load the HTML into Cheerio and then use it to retrieve all the `` elements. After that, we iterate over the collected links and print their `href` attributes, which we access using the [`.attr()`](https://cheerio.js.org/docs/api/classes/Cheerio#attr) method. +Aside from importing libraries and downloading HTML, we load the HTML into Cheerio and then use it to retrieve all the `` elements. After that, we iterate over the collected links and print their `href` attributes, which we access using the [`.attr()`](https://cheerio.js.org/docs/api/classes/cheerio#attr) method. When you run the above code, you'll see quite a lot of links in the terminal. Some of them may look wrong, because they don't start with the regular `https://` protocol. We'll learn what to do with them in the following lessons. diff --git a/sources/academy/webscraping/scraping_basics_python/03_devtools_extracting_data.md b/sources/academy/webscraping/scraping_basics_python/03_devtools_extracting_data.md index f864362f8a..ad21d99501 100644 --- a/sources/academy/webscraping/scraping_basics_python/03_devtools_extracting_data.md +++ b/sources/academy/webscraping/scraping_basics_python/03_devtools_extracting_data.md @@ -78,7 +78,7 @@ In the next lesson, we'll start with our Python project. First we'll be figuring ### Extract the price of IKEA's most expensive artificial plant -At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/artificial-plants-flowers-20492/), use CSS selectors and HTML elements manipulation in the **Console** to extract the price of the most expensive artificial plant (sold in Sweden, as you'll be browsing their Swedish offer). Before opening DevTools, use your judgment to adjust the page to make the task as straightforward as possible. Finally, use JavaScript's [`parseInt()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt) function to convert the price text into a number. +At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/artificial-plants-flowers-20492/), use CSS selectors and HTML elements manipulation in the **Console** to extract the price of the most expensive artificial plant (sold in Sweden, as you'll be browsing their Swedish offer). Before opening DevTools, use your judgment to adjust the page to make the task as straightforward as possible. Finally, use JavaScript's [`parseInt()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt) function to convert the price text into a number (you may need [`replace()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/replace) to handle spaces).
Solution @@ -90,8 +90,8 @@ At IKEA's [Artificial plants & flowers listing](https://www.ikea.com/se/en/cat/a 1. Notice that the price is structured into two elements, with the integer separated from the currency, under a class named `plp-price__integer`. This structure is convenient for extracting the value. 1. In the **Console**, execute `document.querySelector('.plp-price__integer')`. This returns the element representing the first price in the listing. Since `document.querySelector()` returns the first matching element, it directly selects the most expensive plant's price. 1. Save the element in a variable by executing `price = document.querySelector('.plp-price__integer')`. - 1. Convert the price text into a number by executing `parseInt(price.textContent)`. - 1. At the time of writing, this returns `699`, meaning [699 SEK](https://www.google.com/search?q=699%20sek). + 1. Convert the price text into a number by executing `parseInt(price.textContent.replace(' ', ''))`. The price text contains spaces as thousand separators, so `.replace(' ', '')` strips them before parsing - a technique you'll explore further in the [Extracting data](./07_extracting_data.md#removing-dollar-sign-and-commas) lesson. + 1. At the time of writing, this returns `1299`, meaning [1 299 SEK](https://www.google.com/search?q=1299%20sek).
@@ -116,7 +116,7 @@ On Fandom's [Movies page](https://www.fandom.com/topics/movies), use CSS selecto ### Extract details about the first post on Guardian's F1 news -On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), use CSS selectors and HTML manipulation in the **Console** to extract details about the first post. Specifically, extract its title, lead paragraph, and URL of the associated photo. +On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), use CSS selectors and HTML manipulation in the **Console** to extract details about the first post. Specifically, extract its title, lead paragraph (if it has one), and URL of the associated photo. ![F1 news page](../scraping_basics/images/devtools-exercise-guardian2.png) @@ -129,7 +129,7 @@ On the Guardian's [F1 news page](https://www.theguardian.com/sport/formulaone), 1. Notice that the markup does not provide clear, reusable class names for this task. The structure uses generic tag names and randomized classes, requiring you to rely on the element hierarchy and order instead. 1. In the **Console**, execute `post = document.querySelector('#maincontent ul li')`. This returns the element representing the first post. 1. Extract the post's title by executing `post.querySelector('h3').textContent`. - 1. Extract the lead paragraph by executing `post.querySelector('span div').textContent`. + 1. Extract the lead paragraph (if it has one) by executing `post.querySelector('span div').textContent`. 1. Extract the photo URL by executing `post.querySelector('img').src`.