Last month Yahoo! held the first Apache Hadoop Summit in Santa Clara, CA. I really wanted to go but had scheduled our family vacation to Austin, TX for that same week months before. Daniel was able to go in my place for Lookery and my friend Chris Gillett, who was on my team at Compete, also attended.

Hadoop implements Google’s MapReduce programming model to create a framework that breaks up large data into small chunks that are then processed in parallel across a cluster of commodity servers.

The framework is still at a very early stage but is already being used by Facebook, Google, Visible Measures, Yahoo, The New York Times and by us at Lookery.

Back when we started Compete in 2000 there were no Open Source options like Hadoop. Forgot about finding developers that had experience dealing with terabyte-scale data.

We ended up evaluating most of the supercomputing software that was being used mostly within government and academic settings at the time. Software like PBS, MPI, PVM, Torque and Condor where the state of the art at the time. The only option was to create our own solution for dealing with our massive clickstream “database”.

Here are the slides from the presentations Chris gave at the PyCon 2005 conference that describe some of the data processing apps we came up with at Compete.

Cool to see that the paradigms we used are being carried on with the Hadoop, PIG and HDFS projects just at a much larger scale.

Chris posted some great summaries of the Hadoop Summit on his blog. I hope he gets around to posting the summaries for the rest of the talks from that day.

Interested in Hadoop? Python? We’re looking for engineers at Lookery to work on our data processing cluster.

Learn more about working with Hadoop and BIG Data at Lookery.

I’m on vacation in Austin, Texas this week with my family but I thought I’d check-in to share the great news.

Our small team at Lookery is growing quickly and I am very happy to announce that Jay Meattle will be joining us next week.

This is Jay’s last week at Compete where he and I worked together for over 3 years. While at Compete Jay was the Product Manager for Compete.com and part of the small team responsible for creating it. Jay and I also worked together creating Bzzster.com and Shareaholic as our weekend projects while at Compete.

At Lookery Jay will be heading up Product Development for us and helping us launch some of the very exciting products we’ve been heads-down working on.

Looking forward to “bringing the thunder” at Lookery with Jay.

Compete the company I co-founded with Reed Cundiff was acquired today by Taylor Nelson Softres [LON:TNS].

Reed and I started Compete in the post-apocalyptic, post-bubble fall of 2000. It was a very strange time to start a company and an even stranger time to try and raise venture capital. Reed and I were lucky in finding an investor that was ready to stand with us during the darkest of days.

That investor was Ted Dintersmith of Charles River Ventures. One of my best experiences during my time at Compete was working with Ted. There’s nothing like working with an investor during the worst of times (2000 through 2004) to really get to know them. I learned a lot during those dark days, most of all who I could really count on. I’m happy to call Ted a friend today and when Scott and I started Lookery Ted was there again as an advisor, a friend and as our first investor.

The incredible team we put together at Compete is my proudest professional accomplishment. Like Andy said building the best team is the most important factor in building a successful company. Thanks to all of you Compete-ers for building Compete into the leader it is today.

Today Taylor Nelson Softres (TNS) announced the acquisition of Compete. TNS is a $2 billion world leader in market information and insight, headquartered in London. Together Compete and TNS are creating the world’s first and largest fully global consumer clickstream panel. Marketers will be able to turn to a single company for comprehensive insights into consumers – online and offline, behavior and attitudes around the world.

I can’t wait to see what the Compete team will be able to do with the global resources of TNS. HOORAY!

Update: Official news of the acquistion on the Compete Blog.

The power of free data APIs continues to amaze me.

It was a little less than a year ago that Compete partnered with Mashery to power it’s free data API. The results of that decision have been profound (slide 17) and continues to impress me.

This week Ask.com integrated Compete.com data into it’s search engine. It’s a project that we started working on last summer. I’m happy to see that it finally launched.

Thanks to Sven, Jay and the good folks at Mashery for making this a reality.

While I’m on the topic of APIs.. I’m spending my day, nights and weekends working on some exciting Lookery APIs we are getting to release; we’ll also be using Mashery to power our APIs.

I’m looking forward to seeing what people end up doing with our upcoming APIs.

Check out the Ask.com / Compete.com mashup.

A couple of weeks ago I commented (Go Disqus! Go!) on a post by Alexis Ohanian where he referenced Compete’s Top Moving Sites of 2007 list.

Alexis was nice enough to send me a huge bundle of Reddit tshirts and stickers once he learned that the Compete team were big Reddit fans. I’ll be dropping these off to the team at Compete next time I’m down in Boston.

Thanks Alexis!