1. Datasets of Normal Crawl

We consider all the YouTube videos to form a directed graph, where each video is a node in the graph. If a video b is in the related video list (first 20 only) of a video a, then there is a directed edge from a to b. Our crawler uses a breadth-first search to find videos in the graph. We define the initial set of 0-depth video IDs, which the crawler reads in to a queue at the beginning of the crawl. When processing each video, it checks the list of related videos and adds any new ones to the queue.

Given a video ID, the crawler first extracts information from the YouTube API, which contains all the meta-data except age, category and related videos. The crawler then scrapes the video's webpage to obtain the remaining information.We record the following information of a YouTube video in order; they are divided by '\t' in the data file.

Our first crawl was on February 22nd, 2007, and started with the initial set of videos from the list of "Recently Featured", "Most Viewed", "Top Rated" and "Most Discussed", for "Today", "This Week", "This Month" and "All Time", which totalled 189 unique videos on that day. The crawl went to more than four depths, finding approximately 750 thousand videos in about five days. In the following weeks we ran the the crawler every two to three days, each time defining the initial set of videos from the list of "Most Viewed", "Top Rated", and "Most Discussed", for "Today" and "This Week", which is about 200 to 300 videos. On average, the crawl finds 73 thousand distinct videos each time in less than 9 hours.

All the 35 datasets can be downloaded from here. In each package, there are: (1) "0.txt", "1.txt", "2.txt" and "3.txt" (and "4.txt" in "0222.zip"), storing the video data of depth 0, 1, 2 and 3 respectively; (2) "log.txt", a log file indicating the start and finsh time, number of videos crawled and the duration of each depth.

The new data crawled in 2008 are as followings:

2. Datasets of Updating Crawl

To study the growth trend of the video popularity, we also updated the statistics of some previously found videos. For this crawl we only retrieve the number of views for relatively new videos (uploaded after February 15th, 2007). This crawl is performed once a week from March 5th to April 16th 2007, which results in seven datasets.

The 7 datasets can be downloaded from here. In each package, there are: (1) "update.txt", storing the updated video data; it contains video ID, views, rate, ratings and comments; (2) "id.txt", containing all the video IDs to be updated; (3) "log.txt", a log file indicating the start and finish time, number of videos updated and the duration.

In 2008, we updated the statistics of 161085 videos once a week for 21 weeks. The new data are also presented here:

3. Datasets of Video File Size and Bitrate

We have also separately crawled the video file size and video bitrate information. To get the file size, the crawler retrieves the response information from the server when requesting to download the video file and extracts the information on the size of the download. Some videos have the bitrate embedded in the FLV video meta-data, which the crawler extracts after downloading the meta-data of the video file.

In "0523.zip", there are: "idlength.txt", containing the video IDs and video lengths; "size.txt", adding the video file size (integer number of Byte) into the previous information; "log.txt", a log file.

In "0628.zip", there are: "idlength.txt", containing the video IDs and video lengths; "rate.txt", adding the video bitrate (double number of kbps) into the previous information; the VBR videos result in invalid bitrate value, and we use video lengths and sizes to calculate the average bitrate; "log.txt", a log file.

In "080908sizerate.zip", there are: "idlength.txt", containing the video IDs and video lengths; "sizerate.txt", adding the video file size (integer number) and bitrate (double number of kbps) into the previous information; the VBR videos result in invalid bitrate value, and we record a "VBR"; "log.txt", a log file.

4. Datasets of User Information

We have collected the information about YouTube users. The crawler retrieves information on the number of uploaded videos and friends of each user from the YouTube API, for a total of more than 1 million users. There are: "userid.txt", containing all the username collected in previous datasets; "user.txt", containing the information of number of uploads, watches and friends in order.

In 2008, we have also re-collected the information about YouTube users, for a total of more than 2 million users. There are: "userid.txt", containing all the username collected in all normal datasets; "user.txt", containing the information of number of uploads and friends in order.

Copyright 2008 © NETSG, School of Computing Science, Simon Fraser University