A (small) MySQL configuration gotcha

Yesterday I ran into a small, but DUH moment with MySQL.

First, some background.  Your typical /etc/my.cnf file looks like the one below


# Default MySQL startup options

[mysqld_safe]
log-error=/var/log/mysqld.log


Well, I had intended to turn on slow query logging to assist with some performance issue analysis, so I added the following to the bottom of /etc/my.cnf


#slow query log
log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-slow-admin-statements


I added it, then didn’t come back the server for a few weeks.

When I came back I found 0 byte file. I found this odd and started doing some investigating.

After many SQL queries and much head pounding, I finally noticed I had added the lines to the of the file. This put it below the [mysqld_safe] block.

Of course that means that while the normal MySQL daemon is running, those config options will never apply.

Some quick testing on a test system confirmed this to be true and so the fix was to move those lines above the [mysqld_safe] and a scheduled restart later, we have slow query logging working.

PowerShell Link Dump

I’ve spent most of the last three days working on some pretty cool powershell account automation to “sync” accounts in MySQL and local accounts on Windows 2008 Server.

I thought I’d share some of the links I accumulated in doing so.

• http://devcentral.f5.com/weblogs/Joe/archive/2009/01/20/powershell-abcs—t-is-for-type-literals.aspx (type literals)
• http://powershell.com/cs/forums/p/1569/2121.aspx (creating a local account)
• http://support.microsoft.com/kb/968929 (downloads for posh 2.0)
• http://stackoverflow.com/questions/149191/powershell-functions-return-behavior (returning values from functions)
• http://technet.microsoft.com/en-us/magazine/2008.11.windowspowershell.aspx?pr=blog (creating functions)
• http://www.vistax64.com/powershell/56109-exit-out-powershell-script-if-block.html (exiting from an if block, used the ‘continue’ statement in my code)
• http://www.leeholmes.com/blog/MorePowerShellSyntaxHighlighting.aspx (good example of functions and implementing a custom Main function)
• http://www.westmesatech.com/misctools.html (ps1exec utility)
• http://www.hanselman.com/blog/CommentView.aspx?guid=4cc31f23-659b-495a-bae1-fa89177314a9 (run ps1 scripts in a hidden window with hstart)
• http://technet.microsoft.com/en-us/library/ee692798.aspx (Finding an item in an array)
• http://technet.microsoft.com/en-us/library/ee176828.aspx (using foreach)
• http://www.computerperformance.co.uk/powershell/powershell_if_statement.htm (powershell if statements)
• http://windowsitpro.com/article/articleid/98447/powershell-101-lesson-4.html (write-output)
• http://www.pluralsight.com/community/blogs/dan/archive/2006/10/29/41389.aspx (Using ADO.NET and MySQL in PowerShell)
• http://poshcode.org/544 (finding local group members)
• http://www.idera.com/richardsworld/post/Getting-MySql-Results-with-PowerShell.aspx (MySQL and PoSH)
• http://programming.torensma.net/2009/01/connect_powershell_to_mysql/ (Connect to MySQL)

Enjoy.

Happy Thanksgiving

Well, blogvember died a quick death. Such is life.

Happy Thanksgiving to all you American readers and have a quiet and productive day to the rest of you :)

A sort of post

Well. I was up from 3-5am tending after my daughter. She’d go back to sleep, but not me. That and busy day at work equals no real post.

Instead, you should all go read the following two blogs:

* Standalone Sysadmin
* Agile Testing

Gears in Firefox on Snow Leopard

If you are like me, and want Remember The Milk to work offline in Prism, because you need A Bit Better RTM to make RTM really usable.

Then you can use the following steps, care of http://sam.davyson.com/weblog/getting-google-gears-on-snow-leopard/

Google Gears is not yet compatible with Snow Leopard it would seem. There has been a bug report for it for months. If you go to the Google Gears website you don’t see a message that tells you this however – you instead don’t really get any advice.
How do you get to the second screenshot with the install box showing? All you need to do is add:

?platform=mac-firefox

to the URL. Or you can just click here. You can do a similar trick for Safari and download something but it does not install on Snow Leopard (will work on Leopard I’m told). The Firefox one however does work on Snow Leopard without a hitch.

Overview of the NoSQL Ecosystem

As a follow up to my NoSQL reading post on Friday. I came across this excellent overview of the NoSQL options and how they stack up, on High Scalability

Unprecedented data volumes are driving businesses to look at alternatives to the traditional relational database technology that has served us well for over thirty years. Collectively, these alternatives have become known as “NoSQL databases.”

http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosystem/#

Weekend hiatus

Between looking at houses and catching up on house work, it seems I’ve taken a weekend hiatus.

Tomorrow we’ll be back to your regularly scheduled programming.

Cheers.

Some random NoSQL/Big Data reading

I thought I’d share some semi-random nosql/big data links I have stashed in Instapaper to read or reread.

I’m very interested in playing with some of the NoSQL or post-RDBMS engines, particularly, Riak, CouchDB, and MongoDB

In no particular order or grouping, here they are:

• http://www.engineyard.com/blog/2009/mongodb-a-light-in-the-darkness-key-value-stores-part-5/
• http://www.eflorenzano.com/blog/post/my-thoughts-nosql/
• http://labs.mudynamics.com/wp-content/uploads/2009/04/icouch.html
• http://blog.oskarsson.nu/2009/06/nosql-debrief.html
• http://bitbucket.org/justin/riak/src/
• http://couchdbkit.org/
• http://archive.cloudera.com/desktop/
• http://riak.basho.com/nyc-nosql/
• http://adamblog.heroku.com/past/2009/7/6/sql_databases_dont_scale/
• http://bjclark.me/2009/08/04/nosql-if-only-it-was-that-easy/

Beyond that, if you are interested in this stuff, you should be listening to http://www.redmonk.com/cote/it-management-podcast/

I will try to follow up this post with a little more detail on my understand of the NoSQL/Big Data stuff.

Some micro-benchmarking of Lightweight HTTP Servers

The challenge

We were approached by a potential client who needed an infrastructure that could support a workload of delivery small (sub 1kb) text files with minimal (ideally less than 1s) latency. They had been planning to utilize a major CDN, but when they finished development and went into testing, they found the CDN could only deliver 5-20s latency. This workload was to support a live streaming event that was projected to have upwards of 300,000 concurrent users, of which at least 10-15% would be making requests against the service this client was developing.

The process

Because we knew that latency and concurrency would be our challenges, we immediately decided we should try to architect this with a lightweight HTTP server. Being that most of our experience and most of our deployments are based around Apache, and we’ve seen Apache doing very well serving static content, we kept it in the running.

A quick discussion results in us selecting the following lightweight HTTP servers to micro-benchmark with our client’s expected workload:
* Apache HTTPd
* lighttpd
* Cherokee
* ngnix

Being as we only had a week to come up with an architecture stack, implement it, load test it, and put it into production. I decide to focus my testing and if there was a “clear winner”, just go with it, trusting that we’d deploy sufficient overall capacity.

For my testing I created a 500 byte file and used the following to do my benchmarking

ab -q -c 1000 -n 100000 http://server/index.txt

The result from ab that I used for my measurements was
* mean requests per second

I had cherokee, lighttpd, apache, and nginx all on the same VM and tested each one individually.

Each web server has an OOTB configuration, just updated to serve the same Document Root and listen on a specific port.

The results

As you can see, overall lighttpd was the clear winner, only timing out on one run, and consistently serving the most requests.

With those results in mind, we chose lighttpd and engineered a solution that ultimately handled 40,000 concurrent requests at its peak, and based on resource utilization, would have easily scaled to 150,000 concurrent requests.

In a future post I will detail the tweaks we made to lighttpd to improve the OOTB performance and how we designed the architecture we used in production.

LISA 09 Coverage

Just a quick post tonight.

As a sysadmin the LISA conference is one of the few events I would go to every year, if I could. I was able to attend my first last year in San Diego, but sadly, this year it takes place in Baltimore. Between distance, budgets, and life, it was not an option for me.

But, that doesn’t mean I haven’t been able to follow along from Los Angeles.

I thought I’d highlight some of the ways I’ve followed along so far.

The first is, of course, Twitter. Most of the people tweeting from LISA are using the #lisa09 hashtag. So searching for this, or if you use a client like Tweetdeck, adding it as a saved search is a good way to get a minute by minute glimpse.

Matt Simmons has been posting excellent daily summaries on the Usenix Blog and I believe is an official LISA blogger. There are also other posts by authors I’m not so familiar with.

There is a live stream of the Technical Sessions, but I have not been able to watch those so far.

All in all, LISA 09 appears to be off to a great start, I’m looking forward to downloading slide decks and MP3s of talks as the conference draws to a close and am already making plans to try and attend LISA ‘10 in San Jose.