Share
))
This feed includes all episodes of Paul's Security Weekly, Enterprise Security Weekly, Business Security Weekly, Application Security Weekly, and Security Weekly News! Your one-stop shop for all things Security Weekly!
…
continue reading
The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”
Manage episode 213574851 series 1469447
Content provided by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Wes Bos and Scott Tolinski - Full Stack JavaScript Web Developers or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://player.fm/legal.
What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.
Kyle Prinsloo’s Freelancing & Beyond — SponsorKyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.
Freshbooks - SponsorGet a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.
Show Notes3:05 - What Is the undocumented web?
- When traditional methods don’t give you what you want
- Wild West
6:10 - Undocumented APIS
- Instagram’s private API
- Use dev tools’ XHR Request
- React Dev Tools will also have the data
- Ep 008 - Wes Bos Origin Story
- Realtor.ca - MLS Mapping
- Simon Willison’s Zeit Day Talk about XML files
- Charles Proxy App - also good for debugging requests
17:34 - Faking
- Faking User Agents
- Sending headers from requests
- Free Blackberry wifi on plane
23:52 - Scraping
- What is craping?
- Cheerio - jQuery like node package
27:30 - Automation
33:30 - Proxies
- Need a fresh IP address
- Need a different country
- Side note: Canadians buying stuff in USA M4S1P1 === 41100
- API throttling
39:00 - Tips and Tricks
- Multiple API keys
- diskDB - Text files as databases
- Search Github for API keys
- Inspect Element and Delete Overlays
- Delete Cookiesokay
44:45 - Stories
- Photo Voting
- MLS Listing
- Top Entrepreneur voting cheating
- Vanity Plate Checker
- My Hockey Team Sucks
- Vehicle auction bidding app
- youtube-dl
- Scott: Octopath Traveler & OST
- Wes 1: AvE Youtube Channel
- Wes 2: Mustie1 Youtube Channel
- Scott’s Instagram
- LevelUpTutorials Instagram
- Wes’ Instagram
- Wes’ Twitter
- Wes’ Facebook
- Scott’s Twitter
- Make sure to include @SyntaxFM in your tweets
673 episodes
