Gitlink
/
SEOServer
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

缩小匹配爬虫ua范围

This commit is contained in:
黄心宇 2023-09-15 17:09:10 +08:00
parent 43644e83de
commit 0f23ff23a4
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -6,7 +6,7 @@
#### ng配置
```
if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|^$")
if ($http_user_agent ~* "googlebot|Mediapartners-Google|bingbot|google-structured-data-testing-tool|baiduspider|360Spider|Sogou Spider|Yahoo! Slurp China|^$")
{#判断如果是网络爬虫，转发到node服务器下
proxy_pass http://www.test.com:3000;
break;