Some micro-benchmarking of Lightweight HTTP Servers

Posted by Brandon Burton on November 5th, 2009 filed in blogvember, httpd, sysadmin, technology

The challenge

We were approached by a potential client who needed an infrastructure that could support a workload of delivery small (sub 1kb) text files with minimal (ideally less than 1s) latency. They had been planning to utilize a major CDN, but when they finished development and went into testing, they found the CDN could only deliver 5-20s latency. This workload was to support a live streaming event that was projected to have upwards of 300,000 concurrent users, of which at least 10-15% would be making requests against the service this client was developing.

The process

Because we knew that latency and concurrency would be our challenges, we immediately decided we should try to architect this with a lightweight HTTP server. Being that most of our experience and most of our deployments are based around Apache, and we’ve seen Apache doing very well serving static content, we kept it in the running.

A quick discussion results in us selecting the following lightweight HTTP servers to micro-benchmark with our client’s expected workload:
* Apache HTTPd
* lighttpd
* Cherokee
* ngnix

Being as we only had a week to come up with an architecture stack, implement it, load test it, and put it into production. I decide to focus my testing and if there was a “clear winner”, just go with it, trusting that we’d deploy sufficient overall capacity.

For my testing I created a 500 byte file and used the following to do my benchmarking

ab -q -c 1000 -n 100000 http://server/index.txt

The result from ab that I used for my measurements was
* mean requests per second

I had cherokee, lighttpd, apache, and nginx all on the same VM and tested each one individually.

Each web server has an OOTB configuration, just updated to serve the same Document Root and listen on a specific port.

The results

run	apache	lighttpd	cherokee	nginx
1	timed out	4283.47	timed out 70007	timed out 70007
2	1648.13	4327.95	822.44	3848.5
3	778.46	4293.63	825.27	4160.27
3	778.46	4293.63	825.27	4160.27
4	769.08	4369.24	timed out 70007	1422.01
5	timed out 104	timed out 70007	timed out 104	4238.12
6	timed out 70007	4285.23	2957.5	timed out 70007

As you can see, overall lighttpd was the clear winner, only timing out on one run, and consistently serving the most requests.

With those results in mind, we chose lighttpd and engineered a solution that ultimately handled 40,000 concurrent requests at its peak, and based on resource utilization, would have easily scaled to 150,000 concurrent requests.

In a future post I will detail the tweaks we made to lighttpd to improve the OOTB performance and how we designed the architecture we used in production.