Podknife - 763: Web Scraping + Reverse Engineering APIs by Syntax

Help
Help

Suggested Topics
- What is Podknife?
  
  Podknife is a curated podcast information and review site designed to be accessible in your browser from any device. We’re working to build the most useful podcast information source available by providing you with as much publicly available information about each podcast in our database as we can find and keeping it as up to date as possible.
- Can I listen to podcasts on Podknife?
  
  Yeah! There's a player at the bottom of the page.
- How can I get my podcast on Podknife?
  
  We’d love to hear about your podcast and consider it for addition. You can suggest your own by logging in and selecting the “Suggest a Podcast” link from the menu. Fill out the form and we’ll take it from there.
- How can I submit a review of a podcast?
  
  To submit a review of a podcast, go to the page of the podcast you want to review and click the "Submit Review" button below the image associated with the podcast. Clicking that button brings you to a new page where you can select a rating between 1 and 5 stars and write a brief review.
- How can I submit a review of a podcast episode?
  
  To submit a review of a podcast episode, go to the page of the podcast whose episode you want to review and find the specific episode you're looking for. Clicking on any episode reveals a "View Episode" button and clicking on that brings you to an episode-specific page where you can find more information displayed about the episode along with a "Submit Review" button. Clicking that brings you to a new page where you can select a rating between 1 and 5 stars and write a brief review of the episode.
- Is there a Podknife app?
  
  Not yet - we're keeping it in the browser at least for now. No app download required to access anything on the site.
- How can I change my display name?
  
  To change your display name, log in to Podknife and select "Account Settings" in the dropdown menu. From the Account Settings page, enter your desired new display name and press the "Save" button to confirm the change.
- How can I change my password?
  
  To change your password, log in to Podknife and select "Account Settings" in the dropdown menu. From the Account Settings page, enter your desired new password in the two fields below "Update Password" and press the "Save" button to confirm the change.
- How can I keep track of all my reviews?
  
  Within the menu, click on "My Profile", you will then be able to check all your reviews for both podcast & episode. You can also check all your favorite podcasts.
- How can I keep track of all my favorited podcasts?
  
  You can find podcasts you’ve favorited displayed on the My Profile page associated with your account (https://podknife.com/profile).
  You can share your My Profile page displaying your favorite podcasts as well as any reviews you’ve written publicly with a link in the form https://podknife.com/users/[displayname] where [displayname] is replaced by your Display Name.
- How can I favorite a podcast?
  
  On each podcast page you’ll find the outline of a star above the podcast title. Clicking this star fills it in and designates the podcast as a favorite (for logged in users) or invites the user to log in or register for an account (for users who are not logged in). You can find favorited podcasts displayed on the My Profile page associated with your account (https://podknife.com/profile).
- How can I register for a Podknife account?
  
  To register, click on the “Register” button in the top right of your browser window (or visit https://podknife.com/users/sign_up). Enter a Display Name to be associated with your account on the site (alongside any reviews you contribute, for example), an Email address to associate with the account and your password (twice for confirmation). Press the “Confirm” button and you should be all set, logged in and ready to review and favorite podcasts as you wish.
- How can I get in touch?
  
  We’d love to hear from you. You can reach us using the Feedback form (https://podknife.com/feedbacks/new) which can be found either in the footer on each page (for users who are not logged in) or in the dropdown menu (for logged in users).
  
  You can also reach us on Twitter by tweeting us @podknife (https://twitter.com/podknife) or on Facebook by messaging us from our page (https://www.facebook.com/podknife/).
- Log In
- Register
- Feedback
- Help
- Privacy Policy
- Terms of Use
© Podknife 2024

This episode currently has no reviews.

763: Web Scraping + Reverse Engineering APIs

Podcast |: Syntax - Tasty Web Development Treats
Publisher |: Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers
Media Type |: audio
Podknife tags |: Business,; Careers,; Tech News,; Technology,; Web Development
Categories Via RSS |: News,; Tech News,; Technology
Publication Date |: May 01, 2024
Episode Duration |: 00:52:33

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease.

Show Notes

00:00 Welcome to Syntax!
03:13 Brought to you by Sentry.io.
05:00 What is scraping?
Socialblade.
08:01 Examples of past scrapers.
Canadian Tire.
10:06 Cloud app downloader.
16:13 Other use cases.
16:58 Scraping 101.
17:28 Client Side.
19:08 Private API.
Proxyman.
22:40 Server rendered.
23:27 Initial state.
24:57 What format is the data in?
Google Puppeteer Extension.
27:08 Working with the DOM.
27:12 Linkedom npm package.
29:02 querySelector everything.
31:28 How to find the elements without classes.
34:08 Use XPath selectors for select by word.
34:53 Make them as flexible as you can. Classes change!
35:10 AI is good at this!
36:26 File downloading.
38:20 Working with protected routes.
Proxyman.
40:41 Programatically retrieve authentication keys because they are short-lived.
Fetch Cookie.
43:20 Deal-breakers.
Mechanical Turk.
44:58 What happened with Amazon?
Uniqlo Self-Checkout
46:42 Wes’ portable refrigerator utopia.
47:25 Sick Picks & Shameless Plugs.

This episode currently has no reviews.

Submit Review

This episode could use a review!

This episode could use a review! Have anything to say about it? Share your thoughts using the button below.

Submit Review

Show Notes

Sick Picks

Shameless Plugs

Hit us up on Socials!