TL, DR Selenium is one of the main libraries for browser automation and web scraping. Sometimes it is painful to integrate with browsers that are installed as Snap packages, this guide gives you a few examples for a correct configuration. What are Snap packages? Snap packages is the new and preferred way to distribute applications… Continue reading Use Python Selenium with Snap browsers
Tag: Webscraping
Set your User Agent with Python Requests, Scrapy, and Selenium
TL, DR When you crawl the web to collect data you should set a User Agent that identifies you. Or one that hides the tool you are using. Here you can find how to set the User Agent in Python Requests, Scrapy, and Selenium. What is the User Agent? A User Agent is a string… Continue reading Set your User Agent with Python Requests, Scrapy, and Selenium
Scrapy: Select elements containing a specific text
TL, DR Scrapy is one of the top frameworks for web scraping in Python. Often you need to select some element where you already know part of its text. This post shows you how to select elements containing a certain text with CSS and XPATH selectors. How to select elements containing a certain text in… Continue reading Scrapy: Select elements containing a specific text
