Last year I created a GreaseMonkey script called Invisible Web that notified you about the web beacons (web bugs) that the page you’re on is using to track your behavior.

Changes in this release:

• Favicons are now stored locally to increase performance.
• Updated definition for Lookery
• Updated definition for Google Analytics
• Added definition for Piwik Analytics
• Added definition for Mint
• Added definition for Facebook Beacon
• Added definition for TypePad Stats
• Added definition for Wordpress Stats

Install the Invisible Web Greasemonkey Script

Update: Invisible Web now has a Google Code Project

In my last post I compared using Amazon S3 as a CDN to other low cost alternatives. What was clear was that S3 performed badly when compared to the true CDNs. Not such a surprising outcome; S3 was not designed to be a CDN.

The real surprise to me was that CacheFly, the lowest cost CDN I tested, performed better than the much higher priced options.

My performance were pure HTTP GET based and utilized the monitoring service Pingdom. What performance monitoring services don’t monitor is how quickly a website load feels to the end-user.

A lot can be done to make a website “feel” faster most of these techniques are outlined in the book High Performance Web Sites by Steve Souders of Yahoo!. Most of the book’s content can be read for free on the Yahoo! Developer Network.

The simplest way to increase performance is to minimize requests to your server by setting proper Expires and Cache-Control headers so that your static content can be cached.

When I tested the CDN options for cacheability only Panther Express and my DIY Nginx web server EC2 servers returned proper “cacheable” headers.

S3 and CacheFly returned no “cache-friendly” headers and EdgeCast returned proper headers but the server time was inaccurate which could lead to caching issues.

I found no way to add expires headers to files hosted on CacheFly. You can add custom headers to files in S3.

By setting your Expires headers on your static files far into the future you can create a cheap CDN with S3 that can “feel” as fast a traditional CDN. You’ll need to automate this and make it part of your deployment process which ideally renames the deployed files to contain revision numbers so that you can really set those Expires headers far in the future.

At Lookery our Javascript analytics tracker is now pushing more than 250GB of bandwidth per month. This javascript file has grown a bit but is still about 6kb.

We’ve been serving that file from a Nginx webserver running on Amazon EC2 instances. Currently Amazon has 3 EC2 data centers but strangely they are all located on the East Coast of the US (Virginia). Since a lot of our users are international we needed to move that file to a CDN in order to reduce latency.

Less than a month ago we moved that single file off to CacheFly a low-priced CDN. I thought CacheFly would be an interim solution but from our performance testing they seem to be a good longer term option.

For this round of testing I tested the following serving options:

1. CacheFly
2. EdgeCast
3. Amazon S3
4. Nginx running on an Amazon EC2 Instance (DIY option)

For the next round of tests I’ll add Panther Express to the mix.

Performance testing Amazon S3 is a bit unfair but since so many people are using this as a cheap solution I thought I would test it out myself. The performance tests show that you’re much better serving your static content from a lightweight server like NGINX or using an inexpensive option like CacheFly.

For the performance tests I used Pingdom, a 3rd party performance monitoring service that we’ve been quite happy with.

The monitoring servers were geographically distributed as follows:

Summary

CacheFly performed the best but only slightly better than EdgeCast. The S3 option was the worst with the Nginx/DIY option performing just over 100 ms faster.

Details

Below are the details for a single day. I ran these tests for over 2 weeks, the results were identical to this single day.

CacheFly CDN	Nginx on an Amazon EC2 Instance (DIY)
EdgeCast CDN	Amazon S3 used as a CDN

Notes

• I also tested a second DIY option, running a Varnish cache on an Amazon EC2 instance, but for static content Nginx performed much better so I omitted the results.
• EdgeCast has an option that allows frequently used content to be served directly from RAM. My trial account did not allow me to test this option. This option would allow for even better performance, possibly matching/beating CacheFly’s performance.

So far we’re sticking with CacheFly and testing a few other options. I’ll post the Panther Express performance after our tests are complete.

Let me know if you’ve found similar results or if I should be testing any other solutions.

Andy Payne has organized a Virtual/Hosting/EC2 Birds-of-a-feather meet-up for tomorrow morning (May 21st) in Waltham, MA.

We’ll be comparing notes and experiences on virtual hosting options (e.g. EC2). Thanks to Matrix Partners for hosting us at their offices.

I’ll be giving a short presentation on how we’ve been using EC2 at Lookery. I’ll post my presentation here after the meet-up tomorrow.

If you’re interested in attending let me know.

Sonian Networks, a company that I am an advisor to, is publically launching their Sonian Archive (SA2) product this week.

Sonian is Greg Arnette’s latest creation. I met Greg at a Ruby on Rails workshop back in 2006. Greg had just sold his company Intellireach to Infocrossing and was thinking about his next big idea, specifically how to take advantage of new on-demand infrastructure services to build highly efficient IT services.

Sonian’s first product, Sonian Archive (SA2), is an on-demand message archiving service. Put simply SA2 plugs into your exisiting messaging (email++) infrastructure (MS Exchange, Google Apps, Zimbra, Lotus Notes, etc) and automatically archives all of your organization’s mailboxes offsite.

Now this isn’t simple on-demand email backup software, SA2 scales as your mail volume grows, uses U.S. Defense Department level encryption, provides compliance management tools and doesn’t move any of your data offline, all data is fully searchable at all times.

SA2’s compliance tool allows administrators (with proper permissions) to open a new investigation and search an employees email. All investigations are logged to provide an audit trail everytime anyone’s email is searched or viewed.

I love Sonian’s architecture stack. All but one monitoring machine is hosted on Amazon’s EC2 web service with all permanent storage running on Amazon’s S3 web service (just like us at Lookery). Sonian’s software stack includes Ruby, C++, Nginx (my favorite web server/proxy), MySQL, Lucene and Ruby on Rails. I prefer Python but that is still a tasty stack.

Congratulations Team Sonian!!

Free trial accounts are available this week.