it is a program/algorithm for detailed analysis of pages downloaded by robots