Crawler Update Results

We used a software engine that downloaded each tgp at the same time each day for a week. After the first day, it would also locate the new hyperlinks on the page (ie the fresh listings) and download each of those pages. If that page looked like a typical gallery page, the crawler would then follow all hyperlinks on that page and record whether the result was an image or video, and how big it was.

The bottom line is that over a period of a week, our crawler-bot looked at every picture and movie from every page freshly listed on over 75 tgps. As you can imagine this was a lot of pictures and movies. The totals were: 480,370 pictures (29,430Mb) and 39,228 movies (40,810Mb).

Crawler Summary

Each tgp page contains a summary of the crawlers results for that tgp for each day. An example is:

The above shows the crawler results from the 17th-Jan compared with the 16th-Jan. The number of fresh hyperlinks on the page (when compared with the 16th-Jan download) is 101. These 101 listings were all downloaded. Then all of the content from those 101 listings was in turn downloaded. The crawler found a total of 980 pictures (ie had content type "image/*") with a total size of 60.6Mb. Similarly the crawler found 174 movies (ie had content type "video/*") with a total size of 178.2Mb.

How updates were rated.

We were not strict in this category as quantity does not equal quality. If a tgp updated each day it received an A. If its quantity of fresh pictures or fresh movies was in the top third, we gave it an A+

The table below summarizes the total number and megabytes of pictures and movies found over a week at each tgp: