Technical Web optimization: Technical Web optimization refers to Site and server optimization that assists the crawler in crawling, indexing and ranking operations to rank your website better. txt file is then parsed and will instruct the robot as to which web pages usually are not to generally be crawled. Like https://www.youtube.com/watch?v=ARvdj_mJDAY