Andy Payne has organized a Virtual/Hosting/EC2 Birds-of-a-feather meet-up for tomorrow morning (May 21st) in Waltham, MA.

We’ll be comparing notes and experiences on virtual hosting options (e.g. EC2). Thanks to Matrix Partners for hosting us at their offices.

I’ll be giving a short presentation on how we’ve been using EC2 at Lookery. I’ll post my presentation here after the meet-up tomorrow.

If you’re interested in attending let me know.

Last month Yahoo! held the first Apache Hadoop Summit in Santa Clara, CA. I really wanted to go but had scheduled our family vacation to Austin, TX for that same week months before. Daniel was able to go in my place for Lookery and my friend Chris Gillett, who was on my team at Compete, also attended.

Hadoop implements Google’s MapReduce programming model to create a framework that breaks up large data into small chunks that are then processed in parallel across a cluster of commodity servers.

The framework is still at a very early stage but is already being used by Facebook, Google, Visible Measures, Yahoo, The New York Times and by us at Lookery.

Back when we started Compete in 2000 there were no Open Source options like Hadoop. Forgot about finding developers that had experience dealing with terabyte-scale data.

We ended up evaluating most of the supercomputing software that was being used mostly within government and academic settings at the time. Software like PBS, MPI, PVM, Torque and Condor where the state of the art at the time. The only option was to create our own solution for dealing with our massive clickstream “database”.

Here are the slides from the presentations Chris gave at the PyCon 2005 conference that describe some of the data processing apps we came up with at Compete.

Cool to see that the paradigms we used are being carried on with the Hadoop, PIG and HDFS projects just at a much larger scale.

Chris posted some great summaries of the Hadoop Summit on his blog. I hope he gets around to posting the summaries for the rest of the talks from that day.

Interested in Hadoop? Python? We’re looking for engineers at Lookery to work on our data processing cluster.

Learn more about working with Hadoop and BIG Data at Lookery.

At Lookery we’ve been working at adding my favorite web server Nginx to our production stack.

If you haven’t caught the Nginx bug yet you’re missing out. Nginx is a super lightweight webserver that many use in front of their overweight Apache web servers to offload asset (images, css, js, etc) serving. It’s also a great front-end proxy to your Django or Rails stack.

A while back we settled on using Hyperic as our server monitoring and alerting platform. I spent many years using Ganglia, Munin, Cacti, Nagios and a couple of home-brewed monitoring solutions before I came across Hyperic while reading the comments on Barry’s Blog (Wordpress). Hyperic, despite being written in Java , has worked out well for us.

The only problem: Hyperic doesn’t ship a Nginx plugin and Google wasn’t able to find me one. So I asked Ashwin Phatak to create one for us. I finally got around to uploading it to Google Code tonight.

The plugin is very simple but if releasing it can save anyone some dev cycles we’ll be quite happy.

It’s the first of many code projects at Lookery we plan to open source.

Download the Hyperic Nginx Plugin Now

I’m on vacation in Austin, Texas this week with my family but I thought I’d check-in to share the great news.

Our small team at Lookery is growing quickly and I am very happy to announce that Jay Meattle will be joining us next week.

This is Jay’s last week at Compete where he and I worked together for over 3 years. While at Compete Jay was the Product Manager for Compete.com and part of the small team responsible for creating it. Jay and I also worked together creating Bzzster.com and Shareaholic as our weekend projects while at Compete.

At Lookery Jay will be heading up Product Development for us and helping us launch some of the very exciting products we’ve been heads-down working on.

Looking forward to “bringing the thunder” at Lookery with Jay.

Going to be back in my hometown NYC next week!

On Monday I’ll be speaking at the Business of API’s Conference at the Yale Club.

This is the second year for the conference, the first was hosted in San Francisco and drew an impressive crowd. This year’s speakers include Jeremy Zawodny of Yahoo, Ty Ahmad of MTV and Brad Burnham of Union Square Ventures.

On Tuesday I’ll be meeting some of our east coast angel investors for the first time. Looking forward to it!

I’ll also be interviewing some NYC engineers to join the Lookery team while down in NY. Know any great engineers? Please send them my way.