Simple Python Scraper Template with Proxy Rotation & Anti-Ban Delays

[Genesis Schuppe]

Hello BHW!
I've been browsing the forum for a while and noticed many guys here struggle with basic scraping blocks and proxy management. I wanted to share a clean, lightweight Python boilerplate that I use for my automation tasks.
It handles random User-Agents, rotating proxies from a list, and includes random backoff delays to prevent immediate bans.

import requests
import random
import time
from bs4 import BeautifulSoup

# List of proxies to rotate (format: 'iport' or 'userass@iport')
PROXY_LIST = [
"http://your_proxy1ort",
"http://your_proxy2ort",
"http://your_proxy3ort"
]

# List of common User-Agents to mimic different browsers
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2.1 Safari/605.1.15",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
]

def get_secure_session():
session = requests.Session()

# Set random User-Agent
session.headers.update({"User-Agent": random.choice(USER_AGENTS)})

# Set random Proxy if list is not empty
if PROXY_LIST:
proxy = random.choice(PROXY_LIST)
session.proxies = {"http": proxy, "https": proxy}

return session

def fetch_data(url):
session = get_secure_session()
try:
# Added timeout to prevent script from hanging
response = session.get(url, timeout=10)
if response.status_code == 200:
return response.text
else:
print(f"[-] Failed with status code: {response.status_code}")
return None
except requests.exceptions.RequestException as e:
print(f"[-] Connection error: {e}")
return None

# Example usage
if __name__ == "__main__":
target_url = "https://httpbin.org/ip" # Use this to test proxy rotation

print("[+] Starting request...")
html_content = fetch_data(target_url)

if html_content:
print("[+] Success! Parsing data...")
# You can add your BeautifulSoup logic here

# Random delay between 2 and 5 seconds to stay under the radar
time.sleep(random.uniform(2.5, 5.5))

Features included:
Simple HTTP session management using requests.
Randomized headers to avoid fingerprinting.
Basic try/except block to catch bad or dead proxies without crashing the script.
Feel free to copy, modify, and use it for your web scraping or data extraction tasks. Let me know if you need help adjusting this for dynamic JavaScript websites (Playwright/Selenium) — I might drop a guide on that later if you guys are interested!