首页 新闻 搜索 专区 学院

referer设置未生效 请大佬帮忙看看问题在哪

0
[待解决问题]

我封装了一个带有异常处理的requests.get函数,在爬一个网站的时候需要在headers里面设置referer。
在具体代码中调用自己封装的函数并向函数传referer的值返回403错误,但是如果直接,requests.get(url,headers)就可以
麻烦大佬给看看代码

问题补充:

def get_resp(url, referer):
UserAgent_list = [
"Mozilla/5.0(Macintosh;U;IntelMacOSX10_6_8;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",

    "Mozilla/5.0(Windows;U;WindowsNT6.1;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",

    "Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0;",

    "Mozilla/4.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0)",

    "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)",

    "Mozilla/5.0(Macintosh;IntelMacOSX10.6;rv:2.0.1)Gecko/20100101Firefox/4.0.1",

    "Mozilla/5.0(WindowsNT6.1;rv:2.0.1)Gecko/20100101Firefox/4.0.1",

    "Opera/9.80(Macintosh;IntelMacOSX10.6.8;U;en)Presto/2.8.131Version/11.11",

    "Opera/9.80(WindowsNT6.1;U;en)Presto/2.8.131Version/11.11",

    "Mozilla/5.0(Macintosh;IntelMacOSX10_7_0)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.56Safari/535.11",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Maxthon2.0)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TencentTraveler4.0)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TheWorld)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;360SE)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;AvantBrowser)",

    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)"]
times = 4
while times:
    try:
        random_user = random.choice(UserAgent_list)
        headers = {
            "referer": referer,
            "UserAgent": random_user
        }
        time.sleep(random.randint(10, 300) / 100000)
        resp = requests.get(url, headers=headers)
        if resp.status_code == 200:
            return resp
        else:
            print(f"{url}链接失败{resp.status_code}")
            times -= 1

    except Exception as e:
        print(f"{url}链接失败{e}")
        times -= 1
新手菜鸡111的主页 新手菜鸡111 | 菜鸟二级 | 园豆:202
提问于:2021-07-08 11:27
< >
分享
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册