python 通过requests 爬取百度搜索页面
0
悬赏园豆：20
[待解决问题]
#! python3 import requests res = requests.get('https://www.baidu.com/s?wd=python') print(res.url) print(res.headers) print(res.content)
运行结果如下:
https://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&tn=baidu&wd=python
{'Accept-Ranges': 'bytes', 'Cache-Control': 'no-cache', 'Connection': 'Keep-Alive', 'Content-Length': '227', 'Content-Type': 'text/html', 'Date': 'Tue, 22 Aug 2017 15:14:52 GMT', 'Last-Modified': 'Tue, 08 Aug 2017 12:43:00 GMT', 'P3p': 'CP=" OTI DSP COR IVA OUR IND COM "', 'Pragma': 'no-cache', 'Server': 'BWS/1.1', 'Set-Cookie': 'BD_NOT_HTTPS=1; path=/; Max-Age=300, BIDUPSID=A5F6F7E3ABB0E804ED6D9899C959B3F9; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com, PSTM=1503414892; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com', 'Strict-Transport-Security': 'max-age=0', 'X-Ua-Compatible': 'IE=Edge,chrome=1'}
b'<html>\r\n<head>\r\n\t<script>\r\n\t\tlocation.replace(location.href.replace("https://","http://"));\r\n\t</script>\r\n</head>\r\n<body>\r\n\t<noscript><meta http-equiv="refresh" content="0;url=http://www.baidu.com/"></noscript>\r\n</body>\r\n</html>'
res.content 根本就不是搜索网页,那里出错了,怎么改呀
所有回答(2)
0
有没有什么提示了
没有其他提示
0
import requests
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36",
}
res = requests.get('https://www.baidu.com/s?wd=python',headers=headers)
print(res.url)
print(res.headers)
print(res.content)
增加一个headers,给一个user-agent伪装一下自己,谢谢.
你直接用requests发送请求,百度那边收到的user-agent就是python,它知道是python的请求当不会给你返回正确结果啦!