抓取防爬虫的网站信息(梁健-原创)
有的网站限制网络爬虫的抓取，例如javaeye。会出现错误提示：您的访问请求被拒绝
您可能使用了网络爬虫抓取ITeye网站页面！
ITeye网站不允许您使用网络爬虫对ITeye进行恶意的网页抓取，请您立刻停止该抓取行为！
如果您的网络爬虫不属于恶意抓取行为，希望ITeye网站允许你进行网页抓取，请和ITeye管理员联系，取得授权: webmasteriteye.com
如果您确实使用浏览器访问，但是被错误的识别为网络爬虫，请将您浏览器发送的“User Agent”信息告知我们，帮助我们解决错误: webmasteriteye.com
可以设置下链接的参数来解决。
URL url=new URL(pathString);
URLConnection con=url.openConnection();
con.setDoOutput(true);
con.setRequestProperty("User-Agent", "");
加上上面红色的一句，就可以了。
或者使用httpclient是，添加
HttpClient httpClient=new HttpClient(); httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000); httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5");
这样就可以了。
--------------------
我的部分代码参考：
private HttpClient httpClient = null;public SuNingHtmlparseUtil(String host){if(httpClient == null){ httpClient = new HttpClient();}httpClient.getHostConfiguration().setHost(host, 80, "http");httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000); httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5"); }public SuNingHtmlparseUtil(){if(httpClient == null){ httpClient = new HttpClient();}httpClient.getHostConfiguration().setHost("", 80, "http");httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(50000); httpClient.getParams().setParameter(HttpMethodParams.USER_AGENT,"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.2) Gecko/20090803 Fedora/3.5"); }public String getConnectAsString(String url,String charset) throws Exception {GetMethod get = new GetMethod(url);httpClient.executeMethod(get);if("utf-8".equals(charset.toLowerCase())){ return new String(get.getResponseBodyAsString().getBytes("iso-8859-1"),charset);}else{ return new String(get.getResponseBodyAsString().getBytes(),charset); } }
页:
[1]