Individuals use DA to see which websites are dependable of their discipline and to figure out exactly where to get good one-way links. txt file is then parsed and may instruct the robot regarding which webpages aren't for being crawled. Like a online search engine crawler could hold a cached https://www.youtube.com/watch?v=RSlwOZLlUGg