Artwork

Content provided by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

1:04:49
 
Share
 

Manage episode 213574851 series 1469447
Content provided by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo’s Freelancing & Beyond — Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

3:05 - What Is the undocumented web?

  • When traditional methods don’t give you what you want
  • Wild West

6:10 - Undocumented APIS

17:34 - Faking

  • Faking User Agents
    • Sending headers from requests
    • Free Blackberry wifi on plane

23:52 - Scraping

  • What is craping?
  • Cheerio - jQuery like node package

27:30 - Automation

33:30 - Proxies

  • Need a fresh IP address
  • Need a different country
  • Side note: Canadians buying stuff in USA M4S1P1 === 41100
  • API throttling

39:00 - Tips and Tricks

  • Multiple API keys
  • diskDB - Text files as databases
  • Search Github for API keys
  • Inspect Element and Delete Overlays
  • Delete Cookiesokay

44:45 - Stories

××× SIIIIICK ××× PIIIICKS ××× Shameless Plugs Tweet us your tasty treats!
  continue reading

876 episodes

Artwork
iconShare
 
Manage episode 213574851 series 1469447
Content provided by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.

Kyle Prinsloo’s Freelancing & Beyond — Sponsor

Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

3:05 - What Is the undocumented web?

  • When traditional methods don’t give you what you want
  • Wild West

6:10 - Undocumented APIS

17:34 - Faking

  • Faking User Agents
    • Sending headers from requests
    • Free Blackberry wifi on plane

23:52 - Scraping

  • What is craping?
  • Cheerio - jQuery like node package

27:30 - Automation

33:30 - Proxies

  • Need a fresh IP address
  • Need a different country
  • Side note: Canadians buying stuff in USA M4S1P1 === 41100
  • API throttling

39:00 - Tips and Tricks

  • Multiple API keys
  • diskDB - Text files as databases
  • Search Github for API keys
  • Inspect Element and Delete Overlays
  • Delete Cookiesokay

44:45 - Stories

××× SIIIIICK ××× PIIIICKS ××× Shameless Plugs Tweet us your tasty treats!
  continue reading

876 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Quick Reference Guide

Listen to this show while you explore
Play