Web scraping with Python Requests and Beautiful Soup

Python is a practical language for simple web scraping tasks. The requests library downloads web pages, while Beautiful Soup helps parse HTML and extract the data you need.

Install the dependencies

pip install requests
pip install bs4

Basic request flow

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url, timeout=15)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.get_text(strip=True))

What raise_for_status() does

This method stops the script when the server returns an HTTP error such as 404 or 500. It is better to fail clearly than to parse an invalid response.

Good scraping practices

Respect robots.txt, avoid aggressive request rates, identify your script when appropriate and cache data when possible. Web scraping should be careful, polite and legal.

When to use an API instead

If the website offers an official API, prefer it. APIs are usually more stable than HTML pages and are designed for programmatic access.