INDEX指令告诉搜索机器人抓取该页面；
FOLLOW指令表示搜索机器人可以沿着该页面上的链接继续抓取下去；
RobotsMeta标签的缺省值是INDEX和FOLLOW，只有inktomi除外，对于它，缺省值是INDEX,NOFOLLOW。
这样，一共有四种组合：
其中
可以写成
；
可以写成
需要注意的是：上述的robots.txt和RobotsMETA标签限制搜索引擎机器人（ROBOTS）抓取站点内容的办法只是一种规则，需要搜索引擎机器人的配合才行，并不是每个ROBOTS都遵守的。
目前看来，绝大多数的搜索引擎机器人都遵守robots.txt的规则，而对于RobotsMETA标签，目前支持的并不多，但是正在逐渐增加，如著名搜索引擎GOOGLE就完全支持，而且GOOGLE还增加了一个指令“archive”，可以限制GOOGLE是否保留网页快照。例如：
表示抓取该站点中页面并沿着页面中链接抓取，但是不在GOOLGE上保留该页面的网页快照。
例子:
#robots,scram
User-agent:*
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
User-agent:Mozilla/3.01(hotwired-test/0.1)
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
User-agent:Slurp
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
User-agent:Scooter
Disallow:/cgi-bin
Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
User-agent:Ultraseek
Disallow:/cgi-bin
#Disallow:/TRANSCRIPTS
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/java
Disallow:/shockwave
Disallow:/JOBS
Disallow:/pr
Disallow:/Interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
User-agent:smallbear
Disallow:/cgi-bin
Disallow:/java
Disallow:/images
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/webmaster_logs
Disallow:/virtual
Disallow:/shockwave
Disallow:/TRANSCRIPTS
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
Disallow:/alt_index.html
User-agent:GoogleBot
Disallow:/cgi-bin
Disallow:/java
Disallow:/images
Disallow:/development
Disallow:/third
Disallow:/beta
Disallow:/webmaster_logs
Disallow:/virtual
Disallow:/shockwave
Disallow:/TRANSCRIPTS
Disallow:/newscenter
Disallow:/virtual
Disallow:/DIGEST
Disallow:/QUICKNEWS
Disallow:/SEARCH
Disallow:/alt_index.html
责任编辑:图王 QQ:470666
文章整理：西部数码--专业提供域名注册、虚拟主机服务
http://www.west263.com
以上信息与文章正文是不可分割的一部分,如果您要转载本文章,请保留以上信息，谢谢!
