使用Python爬虫获取HTML元素通常涉及以下步骤，结合BeautifulSoup和lxml等工具实现高效解析：一、核心步骤详解选择解析器BeautifulSoup：语法友好，适合结构化查询from bs4 import BeautifulSoupsoup = BeautifulSoup(html_content, 'html.parser') # 或 'lxml'/'html5lib'lxml：速度更快，支持XPathfrom lxml import etreetree = etree.HTML(html_content)获取网页内容import requestsresponse = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})response.raise_for_status() # 检查请求是否成功html_content = response.content元素定位方法BeautifulSoup：# 通过标签名查找titles = soup.find_all('h1')# 通过属性组合查找items = soup.find_all('div', class_='product', id=lambda x: x and x.startswith('item-'))lxml XPath：# 查找所有链接links = tree.xpath('//a/@href')# 条件筛选示例prices = tree.xpath('//div[@class="price" and number(.) > 100]/text()')内容提取# 获取文本（自动处理嵌套标签）text = element.get_text(strip=True)# 获取属性值href = element.get('href')# 处理嵌套元素nested = element.find('span', class_='highlight')二、进阶技巧动态内容处理from selenium import webdriverdriver = webdriver.Chrome()driver.get(url)dynamic_html = driver.page_source性能优化对BeautifulSoup指定解析器：BeautifulSoup(html, 'lxml')（需安装lxml）批量处理时复用Session对象：session = requests.Session()session.headers.update({'User-Agent': '...'})异常处理try: element = soup.find('div', id='target') if element is None: raise ValueError("Element not found")except requests.RequestException as e: print(f"Network error: {e}")三、完整示例import requestsfrom bs4 import BeautifulSoupdef scrape_product_info(url): try: # 1. 获取页面 headers = {'User-Agent': 'Mozilla/5.0'} html = requests.get(url, headers=headers, timeout=10).content # 2. 解析 soup = BeautifulSoup(html, 'lxml') # 3. 提取数据 products = [] for item in soup.select('div.product-item'): name = item.find('h3').get_text(strip=True) price = item.select_one('span.price').get_text(strip=True) link = item.find('a')['href'] products.append({'name': name, 'price': price, 'link': link}) return products except Exception as e: print(f"Scraping failed: {e}") return []# 使用示例results = scrape_product_info('https://example-store.com/products')for product in results: print(f"{product['name']} - {product['price']} ({product['link']})")四、关键注意事项合法性检查：遵守robots.txt规则设置合理的请求间隔（建议1-3秒）避免高频请求导致IP被封反爬策略应对：# 添加请求头模拟浏览器headers = { 'User-Agent': 'Mozilla/5.0', 'Accept-Language': 'en-US,en;q=0.9'}# 使用会话保持session = requests.Session()session.headers.update(headers)数据清洗：import redef clean_price(text): return float(re.sub(r'[^d.]', '', text))通过组合这些技术，可以高效稳定地提取网页中的结构化数据。对于复杂网站，建议先使用浏览器开发者工具（F12）分析DOM结构，再编写对应的解析逻辑。