LATEST 🚀 Welcome to Upgradix by Shrash Studio – Your destination for free technology learning and career growth.
🕘 Updated:
Python

Python - Web Scraping & HTTP Requests

The requests library fetches raw HTML pages from websites over HTTP, while BeautifulSoup parses that HTML text so you can easily extract prices, titles, and structured data.

Python Web Scraping & HTTP Requests – Automating Data Extraction with Requests and BeautifulSoup

Quick Answer: The requests library fetches raw HTML pages from websites over HTTP, while BeautifulSoup parses that HTML text so you can easily extract prices, titles, and structured data.

Imagine Priya runs a grocery shop in Bengaluru and needs to check wholesale market rates across 20 supplier websites every single morning.

Opening each webpage by hand and copying prices into a spreadsheet takes over an hour every day.

How would you write a Python script to automate this entire data collection process in seconds?

1. Why Do We Need HTTP Requests and HTML Parsers?

The problem with manual browsing

We could open every supplier site in a browser and write down prices on paper.

Example 1: The manual workflow bottleneck

Manual extraction is slow, error-prone, and impossible to scale as supplier counts grow.

# Manual process flow (conceptual)
for supplier in suppliers:
    print("Open browser, type URL, search item, copy price manually.")

Expected output

Manual entry takes 60 minutes per day

This approach works for two items, but it has major operational flaws.

  1. Manual copy-pasting introduces human data entry typos.
  2. It cannot scale when checking hundreds of commodities.
  3. Prices change constantly throughout the day, requiring real-time polling.

We need a programmatic way to fetch web pages and extract the exact numbers we need.

The solution: requests and BeautifulSoup

HTTP request = a message sent from a client to a server to fetch data.

HTML parser = a tool that reads raw markup text and builds a navigable tree structure.

We use Python's requests library to download the page and BeautifulSoup to find specific elements inside it.

Automated Data Extraction Pipeline in Priya's Shop
From sending HTTP requests to extracting structured grocery prices
1. Python Requests Sends HTTP GET request to supplier website
HTML Response
2. BeautifulSoup Parser Builds parse tree and locates price tags
Extracted Data
3. Structured Output Saves clean commodity prices to CSV or database
Request Active Step Result Data

Figure 1: The automated web scraping workflow extracts clean data from raw web pages.

2. Making HTTP Requests with the `requests` Library

HTTP status code = a standard response code sent by a server indicating request success or failure (e.g., 200 for OK, 404 for Not Found).

Let us write a script to fetch a supplier page and verify the server response.

Example 2: Fetching webpage content using requests

This code sends an HTTP GET request and checks if the server responded successfully.

import requests

url = "https://example-grocery-market.com/prices"
response = requests.get(url)

if response.status_code == 200:
    print("Successfully fetched webpage!")
    print(response.text[:100]) # Print first 100 characters of HTML
else:
    print($"Failed to retrieve page. Status code: {response.status_code}")

Expected output

Successfully fetched webpage!
...

Notice how response.status_code == 200 confirms the server accepted our request.

response.text contains the entire raw HTML string of the webpage.

3. Parsing HTML and Extracting Data with BeautifulSoup

CSS selector = a pattern used to select elements on a webpage based on tag names, classes, or IDs.

Once we have the raw HTML, we pass it into BeautifulSoup to search for specific pricing elements.

Example 3: Extracting item prices using BeautifulSoup

This code parses HTML markup and extracts item titles and price values.

from bs4 import BeautifulSoup

html_content = """

Organic Basmati Rice

₹55.00
""" soup = BeautifulSoup(html_content, "html.parser") item_name = soup.find("h2", class_="item-name").text item_price = soup.find("span", class_="price").text print($"Item: {item_name} | Price: {item_price}")

Expected output

Item: Organic Basmati Rice | Price: ₹55.00

Notice how soup.find() locates specific tags using class names.

The .text property extracts clean text without surrounding HTML tags.

Comparing manual data collection with automated scraping
Manual Browsing Automated Scraping (Requests + BS4)
Takes hours of human effort daily Executes in milliseconds via script
Prone to transcription errors Consistent and perfectly repeatable
Hard to aggregate multiple sources Aggregates dozens of sites simultaneously

The important terms to remember

  • HTTP GET = a request method used to retrieve resource data from a server.
  • Status Code = a numeric code indicating HTTP request outcome (200 is success).
  • BeautifulSoup = a Python library for parsing HTML and XML documents.
  • Find / Find_all = BeautifulSoup methods for locating specific HTML tags.

Design tip: Always check response.status_code before passing HTML into BeautifulSoup to prevent parsing errors on broken links.

4. Common Mistakes to Avoid

1. Forgetting to include User-Agent headers

Web servers often block requests that lack a standard browser User-Agent header, returning 403 Forbidden errors.

Better approach: Always pass a custom headers dictionary with a valid User-Agent string in your requests.get() call.

5. Interview Questions & Answers

Question 1: What is the difference between requests and BeautifulSoup?

What the interviewer expects: A clear distinction between network communication and data parsing.

Strong answer: "The requests library handles the network layer, downloading the raw HTML from a web server over HTTP. BeautifulSoup does not make network calls; instead, it takes that downloaded HTML string and parses it into a searchable tree structure so we can extract specific elements."

What interviewers expect at different levels

  • Junior developer: Knows how to fetch pages with requests and query tags with BeautifulSoup.
  • Senior developer: Understands rate limiting, handling anti-scraping blocks, and parsing dynamic JavaScript sites with Selenium or Playwright.
  • Technical architect: Designs robust scraping pipelines with error retries, proxies, and structured database storage.

6. Practice Exercises

Beginner: Extracting article headings

Write a script using requests and BeautifulSoup to fetch a news webpage and print all

headings.

 

Expected result: Console prints all main headline titles found on the page.

Intermediate: Scraping product catalogs

Parse an HTML snippet containing multiple product cards and store names and prices in a Python list of dictionaries.

Hint: Use soup.find_all() to loop through repeating element blocks.

7. Frequently Asked Questions (FAQ)

Can BeautifulSoup scrape websites built with React or Angular?

No. BeautifulSoup only parses static HTML received from the server. If a page loads data dynamically via JavaScript, you need a headless browser tool like Playwright or Selenium.

Scraping publicly available data is generally legal, but you must respect a site's robots.txt file and avoid overloading servers with excessive request frequency.

8. Next Steps

Now that you know how to fetch and parse web data, you are ready to connect Python to relational databases and SQL in the next module.

Comments & Ratings (0)

Log in or create a student account to comment and rate this article.

Be the first to comment on this article.