README.md

Web Scraping Scripts

Scripts for extracting data from web pages.

Purpose

Web scraping scripts focus on:

• Data extraction from HTML
• Table and list parsing
• Link and text extraction
• Structured data collection
• Data export and storage

Common Patterns

Extract Text Content

from selenium.webdriver.common.by import By

# Get all paragraphs
paragraphs = driver.find_elements(By.TAG_NAME, "p")
for p in paragraphs:
    print(p.text)

# Get specific element text
header = driver.find_element(By.CLASS_NAME, "header")
print(header.text)

Extract Links

from selenium.webdriver.common.by import By

# Get all links
links = driver.find_elements(By.TAG_NAME, "a")
for link in links:
    url = link.get_attribute("href")
    text = link.text
    print(f"{text}: {url}")

Extract Table Data

from selenium.webdriver.common.by import By

# Get table
table = driver.find_element(By.TAG_NAME, "table")
rows = table.find_elements(By.TAG_NAME, "tr")

for row in rows:
    cells = row.find_elements(By.TAG_NAME, "td")
    row_data = [cell.text for cell in cells]
    print(row_data)

Using BeautifulSoup for Parsing

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get("https://example.com")

# Get page source and parse with BeautifulSoup
soup = BeautifulSoup(driver.page_source, 'html.parser')

# Extract data
headings = soup.find_all('h1')
for heading in headings:
    print(heading.text)

Recommended Packages

• selenium: Browser automation
• beautifulsoup4: HTML parsing
• pandas: Data storage and export
• requests: HTTP requests (alternative to Selenium for static pages)

Tips

1. Respect robots.txt - Check site policies
2. Use appropriate delays - Avoid overwhelming servers
3. Handle dynamic content - Use waits for JavaScript-rendered content
4. Store data efficiently - Use CSV, JSON, or database
5. Handle errors gracefully - Log issues and continue

File Naming

Use descriptive names:

• scrape_*.py for scraping scripts
• extract_*.py for data extraction
• parse_*.py for parsing logic