采集器的前世今生

 爬出和反爬出是矛与盾的关系,

进化史

1,java原生自带url类:url.getContent(); 

2,httpclient,

3,jsoup,htmlclean.

4,htmlunit,

5,se.

1-2只是原生http链接,

3,做了解析层面的支持,比如页面html清理,xpath支持;

4添加了js支持,这是很大的改进,不过由于各种衍生js库,支持不是很好.

   htmlunit bug比较多，开发过程发现

   browser agent 设置为chrome部分ajax无法获取，转为ie11居然就过了。

5,se,这就是走了捷径,在各分支linux server上还可添加xffb做

淘宝某js屏蔽样本:

{"rgv587_flag":"sm","url":"https://sec.taobao.com/query.htm?action=QueryAction&event_submit_do_css=ok&smApp=detailskip&smPolicy=detailskip-ifq-anti_Spider-checklogin&smCharset=GBK&smTag=MTIwLjQyLjkxLjIwNSwsZWFjNWUwNjM1N2ZmNGViMmE3MTkzNTM3Y2U1MTIzNTU%3D&smReturn=https%3A%2F%2Fajax.tbcdn.cn%2Fjson%2Fifq.htm%3Fid%3D524228313916%26sid%3D2518549420%26p%3D1%26al%3Dfalse%26ap%3D1%26ss%3D0%26free%3D0%26q%3D1%26ex%3D0%26exs%3D0%26shid%3D%26at%3Db%26ct%3D0&smSign=Gz3jbreSkNhQLqmEgkG1Bw%3D%3D"}

其实,阿里系也是有漏洞的.

转载自：http://my.oschina.net/u/856051/blog/408650