referer设置未生效 请大佬帮忙看看问题在哪
0
[待解决问题]
我封装了一个带有异常处理的requests.get函数，在爬一个网站的时候需要在headers里面设置referer。
在具体代码中调用自己封装的函数并向函数传referer的值返回403错误，但是如果直接，requests.get(url,headers)就可以
麻烦大佬给看看代码
问题补充：
def get_resp(url, referer):
UserAgent_list = [
"Mozilla/5.0(Macintosh;U;IntelMacOSX10_6_8;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",
"Mozilla/5.0(Windows;U;WindowsNT6.1;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",
"Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0;",
"Mozilla/4.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0)",
"Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)",
"Mozilla/5.0(Macintosh;IntelMacOSX10.6;rv:2.0.1)Gecko/20100101Firefox/4.0.1",
"Mozilla/5.0(WindowsNT6.1;rv:2.0.1)Gecko/20100101Firefox/4.0.1",
"Opera/9.80(Macintosh;IntelMacOSX10.6.8;U;en)Presto/2.8.131Version/11.11",
"Opera/9.80(WindowsNT6.1;U;en)Presto/2.8.131Version/11.11",
"Mozilla/5.0(Macintosh;IntelMacOSX10_7_0)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.56Safari/535.11",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Maxthon2.0)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TencentTraveler4.0)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TheWorld)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;360SE)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;AvantBrowser)",
"Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)"]
times = 4
while times:
try:
random_user = random.choice(UserAgent_list)
headers = {
"referer": referer,
"UserAgent": random_user
}
time.sleep(random.randint(10, 300) / 100000)
resp = requests.get(url, headers=headers)
if resp.status_code == 200:
return resp
else:
print(f"{url}链接失败{resp.status_code}")
times -= 1
except Exception as e:
print(f"{url}链接失败{e}")
times -= 1
