首页 新闻 赞助 找找看

Java_爬虫,如何抓取Js动态生成数据的页面?

0
悬赏园豆:20 [已关闭问题] 解决于 2014-07-29 09:46

很多网站是用js或Jquery 生成数据的,到后台获取到数据以后,用 document.write()或者("#id").html="" 的方式 写到页面中,这个时候用浏览器查看源码是看不到数据的。

HttpClient是不行的,看网上说HtmlUnit,说  可以获取后台js加载完后的完整页面,但是我按照文章上说的 写了 ,都不好使。

String url = "http://xinjinqiao.tprtc.com/admin/main/flrpro.do";
try {
    WebClient webClient = new WebClient(BrowserVersion.FIREFOX_10);
    //设置webClient的相关参数
    webClient.getOptions().setJavaScriptEnabled(true);
    webClient.getOptions().setCssEnabled(false);
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());
    //webClient.getOptions().setTimeout(50000);
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    //模拟浏览器打开一个目标网址
    HtmlPage rootPage = webClient.getPage(url);
    System.out.println("为了获取js执行的数据 线程开始沉睡等待");
    Thread.sleep(3000);//主要是这个线程的等待 因为js加载也是需要时间的
    System.out.println("线程结束沉睡");
    String html = rootPage.asText();
    System.out.println(html);
} catch (Exception e) {
}


其实这段代码不好使。


求解答,其中典型的就是这个链接的页面,怎么能在java程序中获取其中的数据?

http://xinjinqiao.tprtc.com/admin/main/flrpro.do

Inszt的主页 Inszt | 初学一级 | 园豆:151
提问于:2014-07-25 17:47
< >
分享
其他回答(1)
0

可以参考下这个文档说明,很详细

http://doc.shenjianshou.cn/developmentSkills/useAJAX.html

sosozzzx | 园豆:302 (菜鸟二级) | 2016-09-08 16:53
-1

[{"id":2081,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园32号楼房产","fdate":"2017-10-31","fmoney":"256.896900","fcode":"GR2017TJ1000233-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2080,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园31号楼房产","fdate":"2017-10-31","fmoney":"306.939500","fcode":"GR2017TJ1000231-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2079,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园27号楼房产","fdate":"2017-10-31","fmoney":"281.428600","fcode":"GR2017TJ1000232-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2078,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园26号楼房产","fdate":"2017-10-31","fmoney":"281.428600","fcode":"GR2017TJ1000327-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2077,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园19号楼房产","fdate":"2017-10-31","fmoney":"226.123900","fcode":"GR2017TJ1000326-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2076,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园18号楼房产","fdate":"2017-10-31","fmoney":"226.123900","fcode":"GR2017TJ1000228-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2075,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园5号楼房产","fdate":"2017-10-31","fmoney":"306.939500","fcode":"GR2017TJ1000230-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2074,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园4号楼房产","fdate":"2017-10-31","fmoney":"256.896900","fcode":"GR2017TJ1000229-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2073,"count":67,"fname":"天津市北辰区双口镇京福公路西侧林里庄园2号楼房产","fdate":"2017-10-31","fmoney":"290.650300","fcode":"GR2017TJ1000328-3","fproducttype":"实物资产-房产","fmediadate":null},{"id":2072,"count":67,"fname":"天津市宝坻区大白庄镇锦绣香江花园玉兰园3-2-502房产","fdate":"2017-10-31","fmoney":"59.563000","fcode":"GR2017TJ1000323-3","fproducttype":"实物资产-房产","fmediadate":null}]

方得始终 | 园豆:202 (菜鸟二级) | 2017-11-09 17:02
清除回答草稿
   您需要登录以后才能回答,未注册用户请先注册