爬虫模拟如何驱动搜索结果页(SERP)的分析

各大搜索引擎爬虫UA大全2020最新版

各大搜索引擎爬虫UA大全2020最新版以下是各大搜索引擎在2020年左右的爬虫User-Agent(UA)信息,包括百度、神马、360搜索、谷歌、微软Bing、搜搜、搜狗以及头条搜索等。请注意,这些信息可能会随时间发生变化,建议在实际应用中定期更新。1. 百度爬虫UA移动UA:Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,likeGecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 likeMac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)PC UA:Mozilla/5.0 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)Mozilla/5.0(compatible;Baiduspiderrender/2.0;+http://www.baidu.com/search/spider.html)百度图片:Baiduspider-image+(+http://www.baidu.com/search/spider.htm)2. 神马爬虫UAPC UA:YisouSpider移动UA:Mozilla/5.0 (Linux; U; Android 4.3; zh-CN; SCH-N719 Build/JSS15J) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 YisouSpider/1.0.0 Mobile Safari/533.13. 360搜索爬虫UAPC UA:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)移动UA:Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko)Version/4.0 Mobile Safari/534.30; 360SpiderMozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30; HaosouSpider4. 谷歌爬虫UA谷歌UA:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)谷歌图片:Googlebot-Image/1.0谷歌移动:AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari5. 微软Bing爬虫UA微软UA:Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)6. 搜搜爬虫UA搜索:Sosospider+(+http://help.soso.com/webspider.htm)搜搜图片:Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)7. 搜狗爬虫UA搜狗UA:Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Sogou Video Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)8. 头条搜索爬虫UAPC UA:Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36Android UA:Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)iOS UA:Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)IP段信息360搜索:101.226.166.* - 101.226.169.*182.118.20.* - 182.118.22.*182.118.25.*182.118.28.*182.118.35.*61.55.185.*180.153.236.*头条搜索:110.249.201.0/24110.249.202.0/24111.225.148.0/24111.225.149.0/24220.243.135.0/24220.243.136.0/24220.243.188.0/24220.243.189.0/2460.8.123.0/2460.8.151.0/24请注意,以上IP段信息仅供参考,实际使用中可能会有所变化。为了准确识别和管理搜索引擎爬虫,建议定期更新爬虫UA和IP段信息,并结合服务器日志进行综合分析。


nginx