Contents 1 History 2 History of Common Crawl data 3 Norvig Web Data Science Award 4 References 5 External links


History[edit] Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.[7] The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July of that year.[8] Common Crawl's archives had only included .arc files previously.[8] In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012.[9] The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."[9] In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler.[10] Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.[11]


History of Common Crawl data[edit] The following data have been collected from the official Common Crawl Blog Crawl Date Availability date Size in TB Billion of pages Comments November 2015 December 2015 151 1.82 September 2015 November 2015 106 1.32 July 2015 August 2015 145 1.81 June 2015 July 2015 131 1.67 May 2015 July 2015 159 2.05 April 2015 May 2015 168 2.11 March 2015 May 2015 124 1.64 February 2015 March 2015 145 1.9 January 2015 March 2015 139 1.82 December 2014 January 2015 160 2.08 November 2014 December 2014 135 1.95 October 2014 November 2014 254 3.7 September 2014 November 2014 220 2.8 August 2014 September 2014 200 2.8 July 2014 August 2014 266 3.6 Crawl dataset available every Month April 2014 July 2014 183 2.6 March 2014 March 2014 223 2.8 First Nutch crawl January 2014 January 2014 148 2.3 November 2013 November 2013 102 2 Data in Warc file format July 2012 July 2012 - - Data in Arc file format January 2012 January 2012 - - Public Data Set of Amazon Web Service November 2011 November 2011 40 5 first availability on Amazon


Norvig Web Data Science Award[edit] In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux.[12][13] The award is named for Peter Norvig who also chairs the judging committee for the award.[12]


References[edit] ^ Rosanna Xia (February 5, 2012). "Tech entrepreneur Gil Elbaz made it big in L.A". Los Angeles Times. Retrieved July 31, 2014.  ^ "Gil Elbaz and Common Crawl". NBC News. April 4, 2013. Retrieved July 31, 2014.  ^ "July 2015 Crawl Archive Available". Retrieved 2015-09-30.  ^ a b Tom Simonite (January 23, 2013). "A Free Database of the Entire Web May Spawn the Next Google". MIT Technology Review. Retrieved July 31, 2014.  ^ Russell Brandom (March 1, 2013). "Common Crawl: going after Google on a non-profit budget". Retrieved July 31, 2014.  ^ "Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222". This Week In Startups. January 10, 2012.  ^ Jennifer Zaino (March 13, 2012). "Common Crawl To Add New Data In Amazon Web Services Bucket". Semantic Web. Retrieved July 31, 2014.  ^ a b Jennifer Zaino (July 16, 2012). "Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable For Users To Explore". Semantic Web. Retrieved July 31, 2014.  ^ a b Jennifer Zaino (December 18, 2012). "Blekko Data Donation Is A Big Benefit To Common Crawl". Semantic Web. Retrieved July 31, 2014.  ^ Jordan Mendelson (February 20, 2014). "Common Crawl's Move to Nutch". Common Crawl. Retrieved July 31, 2014.  ^ Jordan Mendelson (November 27, 2013). "New Crawl Data Available!". Common Crawl. Retrieved July 31, 2014.  ^ a b Lisa Green (November 15, 2012). "The Norvig Web Data Science Award". Common Crawl. Retrieved July 31, 2014.  ^ "Norvig Web Data Science Award 2014". Dutch Techcentre for Life Sciences. Retrieved July 31, 2014. 


External links[edit] Wikimedia Commons has media related to Common Crawl. Common Crawl in California, United States Common Crawl GitHub Repository with the crawler, libraries and example code Common Crawl Discussion Group Common Crawl Blog Retrieved from "https://en.wikipedia.org/w/index.php?title=Common_Crawl&oldid=798032022" Categories: Internet companiesWeb archivingWeb archiving initiativesHidden categories: Commons category without a link on Wikidata


Navigation menu Personal tools Not logged inTalkContributionsCreate accountLog in Namespaces ArticleTalk Variants Views ReadEditView history More Search Navigation Main pageContentsFeatured contentCurrent eventsRandom articleDonate to WikipediaWikipedia store Interaction HelpAbout WikipediaCommunity portalRecent changesContact page Tools What links hereRelated changesUpload fileSpecial pagesPermanent linkPage informationWikidata itemCite this page Print/export Create a bookDownload as PDFPrintable version Languages Add links This page was last edited on 30 August 2017, at 15:48. Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization. Privacy policy About Wikipedia Disclaimers Contact Wikipedia Developers Cookie statement Mobile view (window.RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgPageParseReport":{"limitreport":{"cputime":"0.196","walltime":"0.256","ppvisitednodes":{"value":1041,"limit":1000000},"ppgeneratednodes":{"value":0,"limit":1500000},"postexpandincludesize":{"value":23388,"limit":2097152},"templateargumentsize":{"value":614,"limit":2097152},"expansiondepth":{"value":9,"limit":40},"expensivefunctioncount":{"value":0,"limit":500},"entityaccesscount":{"value":1,"limit":400},"timingprofile":["100.00% 233.736 1 -total"," 46.66% 109.066 1 Template:Reflist"," 31.60% 73.849 1 Template:Infobox_dot-com_company"," 27.76% 64.876 8 Template:Cite_news"," 27.60% 64.521 1 Template:Infobox"," 15.91% 37.199 1 Template:Commons_category"," 8.77% 20.488 5 Template:Cite_web"," 7.78% 18.196 1 Template:Comma_separated_entries"," 3.61% 8.433 1 Template:Commons"," 3.04% 7.107 1 Template:Sister_project"]},"scribunto":{"limitreport-timeusage":{"value":"0.093","limit":"10.000"},"limitreport-memusage":{"value":4024571,"limit":52428800}},"cachereport":{"origin":"mw1320","timestamp":"20171205164407","ttl":1900800,"transientcontent":false}}});});(window.RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgBackendResponseTime":82,"wgHostname":"mw1321"});});


Common_Crawl - Photos and All Basic Informations

Common_Crawl More Links

Types Of Business Entity501(c)(3)English LanguageSan FranciscoCaliforniaLos AngelesCaliforniaEntrepreneurshipGil ElbazPeter NorvigNova SpivackCarl MalamudKurt BollackerJoi ItoNonprofit Organization501(c) OrganizationWeb CrawlerWeb ArchivingGil ElbazPeter NorvigJoi ItoNofollowRobot Exclusion StandardAmazon Web ServicesMetadataARC (file Format)BlekkoMetadataSearch Engine OptimizationApache Software FoundationNutchWeb ARChiveBeneluxPeter NorvigHelp:CategoryCategory:Internet CompaniesCategory:Web ArchivingCategory:Web Archiving InitiativesCategory:Commons Category Without A Link On WikidataDiscussion About Edits From This IP Address [n]A List Of Edits Made From This IP Address [y]View The Content Page [c]Discussion About The Content Page [t]Edit This Page [e]Visit The Main Page [z]Guides To Browsing WikipediaFeatured Content – The Best Of WikipediaFind Background Information On Current EventsLoad A Random Article [x]Guidance On How To Use And Edit WikipediaFind Out About WikipediaAbout The Project, What You Can Do, Where To Find ThingsA List Of Recent Changes In The Wiki [r]List Of All English Wikipedia Pages Containing Links To This Page [j]Recent Changes In Pages Linked From This Page [k]Upload Files [u]A List Of All Special Pages [q]Wikipedia:AboutWikipedia:General Disclaimer



view link view link view link view link view link