Step One: Get Harassed by Google JavaScript code.

Step Two: Agree to life without parole for uttering “Microsoft ain’t so bad.”

Step Three: Click “Run.” No, God, shut up, don’t read the instructions.

Step Four: Didn’t we already click Run?

Step Five: Cool. FINALLY we’re getting somewhere.

Step Six: You’re welcome! Do I bend over now or later?

Not-Really-A-Step: That’s a strange UserAgent string!

Sooooo, overall, not that bad. The installer itself was small, but it does it’s own download of a few megabytes of Chrome “guts” from the Google servers. The download only takes a few minutes, but that might push the patience of your end users. Next step is to test this on an unprivileged account. Any volunteers?

From what I gleaned reading blog posts during all the noise about Twitter hatin’ on Ruby and switching to Scala, they had issues with the mainstream open sores MQ software that forced them to write their own. The two issues Twitter had with existing MQs is that RabbitMQ would run out of memory and crash because it had to store all of it’s “metadata” in main memory, and ActiveMQ is a very featureful but strange pile of Java poo that can’t really handle more than a few thousand queues.

So here we have it, twitterlite, a hacky Ruby script that attempts to re-enact a scene where we use an SQLite3 database as a disk-backed message store. The script creates a fixed-size list of “actors” (or users) that can send messages to each other. It then simulates a huge flood of writes for ~140 character messages. All of the fun is wrapped up in it’s own class that manages queueing the writes and the activities of the background thread. Each actor gets it’s own table that is comprised of two columns: source and message. (NOTE: This does not model the actual Twitter MQ usage, which is for background processing; this is just for fun.) Every N seconds or after the message backlog grows to N size, the background thread is prodded to run. The background thread does the writes to database wrapped up inside transactions that will run N number of INSERTs before it commits. The SQLite database is set to do full, synchronous writes to prevent any kind of corruption or data loss on disk failure.

While there is no disk corruption risk, the trade-off for fast writes is that we have some bit of “in-flight” data that could be lost if the server was to die. In production, you would probably have two of these boxes that would be receiving replicated write requests. Even then, if the background thread is set to flush every second or after the backlog reaches 250 messages, you’re not going to be out of very much. On my Mac Pro, with 1,000 actors and 2.5 million messages, it ran at 3,482 msg/sec. The higher you set the actor count, the slower it gets, primarily because a “CREATE TABLE” is pretty expensive and it’s doing these when it runs into a table-not-found error. I did some very unscientific testing and was successful in creating a single SQLite database with 1 million tables. It performed pretty well too (no hard numbers, you’ll just have to trust me).

Obviously this does not come close to reality as there’s some reads that are going to be going on and likely blocking some writes, but from what I understand, Kestrel (Twitter’s queueing software) only hits disk if the queues grow beyond a certain size, during rare “burst” periods where they would be seeing an extraordinary amount of messages. In this scenario, there would still be plenty of messages ready in memory to satisfy any consumers.

Really though, writing your own message queue software is dumb. I’ve heard all the excuses and still don’t buy it. The Gist for my lame benchmark script is here.

We need to make this data available to a couple of different mapping systems in both our Amazon EC2 cloud environment and here in development. MapGuide’s native spatial file format, SDF, is based on SQLite and is really fast for read operations, so we keep a copy of all the SDF files we need to draw the map wherever we’re going to do it. In EC2, we have specific instances dedicated to running MapGuide that each have the full compliment of SDFs for whatever map they’re supposed to be rendering. Unfortunately, keeping a local copy of the street data (which would be >6GB in SDF format) on each instance is sort of out of question. Even if it were compressed heavily, the OS would so totally give me the hand if I expected it to cache any respectable portion of this data (along with all of the other geographic data). MapGuide really throws a little fit if the feature source files aren’t somewhere in the OS disk cache, especially if it’s churning the cache because the feature files won’t fit in RAM. It already starts a fight about my porn subscriptions, blah, blah, blah the first time it hits the ~500MB water “polygon” SDF file. 

We receive  new copies of this data on a quarterly basis, so it’s pretty much read-only. As much as PostgreSQL is my homie, I will admit that MySQL is honestly just better for this case. MyISAM, MySQL’s oft-maligned fast-at-all-costs storage format sports killer table compression and MySQL is unendingly tunable. Even though people may slam PostgreSQL for having a bogus tune from the factory, after you adjust like three (..literally) config variables you’re pretty much good to go. MySQL, on the other hand, gives you a few thousand options all over the place, which may or may not be good thing depending on your level of competency. In your typical cases, PostgreSQL, when given a decent sized shared memory cache area, will perform excellently without low-level tuning. This isn’t your Joe the Plumber case though; this is return 10,000 rows of needle in a 30 million row spatial haystack, and do it right now. I need MySQL’s redneck, badass, big block — with a quadruple Demon Carb and lots of positive signs next to imperial measurements — attitude.

To just render the streets with the right style and label, we don’t need but 3 columns (geometry, name, class), so I dropped the other 89,000. I use the fabulous OGR tools to transfer data between different formats. The OGR tools throw a primary key column in the tables by default, causing the index file to explode in size, so it’s gone as well. Just putting the data on Weight Watchers shrank the MYD file, where the table data lives, down to 3.7GB. I used myisampack to compress the table (+1,000,000 bonus points for read-only situations), which usually yields excellent compression on tables with lots of columns. This one only dropped to 3.1GB. That’s still 600MB of data that we don’t need to carry around — I’ll take it. After building the index, we had 1.9GB + 3.1GB = 5GB… from 14GB… not bad. This is probably as small as I’m going to get an indexed version of this data set.

Of course, that’s still 30 million rows over 5GB. These days we like to throw gigabytes around like they’re nothing, but that’s literally 40 billion zeros and ones. Don’t hit the print button. Even tweaking the data with the myisamchk tool, which allows you to re-order the data and index in a few different ways, will have almost no affect on a gigantic R-tree structure like that. MySQL, like PostGIS, stores it’s spatial index data in an R-tree structure so it can find stuff fast. An R-tree, like a B-tree, is organized in such a manner that it is optimized for retrieving only a small fraction of the total data in the table. It is actually faster to ignore the index for queries that need to read a large section of the table to return data or perform a huge join, a classic case which gives rise to many database forum village idiots complaining about a query that returns half their table “not using the new index they just created.”

Even when the total number of rows returned is still a fraction of the table, where the query needs something like 1,000+ rows, you’re in trouble. Each time the database server has to bravely go where it hasn’t been in a while, into a land of uncharted R-tree, it likely performs a pretty heinous random disk read every time it jumps from one part of the tree to another. In large indexes, like our nearly 2GB spatial index, which is easily 20 branches deep, this is a lot of disk seeks. To make matters worse, it’s unlikely the actual end-game table data is organized in such a fashion that we can perform a sequential scan over the disk. For those just joining the program, a random read does a disk seek, which puts the database to sleep while it waits for granny (disk head) to cross the street (magnetic platter). While the Linux disk queues act like a traffic cop and can tell granny to go home during busy traffic in many situations, there is only so long you can make granny wait before she keels over. In the seemingly minor 8-10 milliseconds it takes for granny to cross the street, the relatively coked out CPU can do like 100 million arithmetic operations — per core.  We’re still stuck doing hundreds to thousands of painfully slow disk operations just to fetch a single tile worth of street data.

So what to do? We go back to our true love — the “scaling” ingredient of choice — memory. You probably already know, as indicated by you not panicking and entering “hot chicks” into the search box before you got anywhere near this paragraph, that memory is easily a thousand times faster than your traditional magnetic hard disk in every single way. This is about the only thing even the most argumentative, foot-long-beard-having computer nerds can agree on. The almighty “memcached,” practically the most celebrated computing achievement of all time, is nothing more than a network-accessible in-memory hash table. Now that memory is like $5 per GB (or really like $20/GB if you can use in any decent quantity), it’s way chic to use lots of network-shared memory to speed up your slow ass Y Combinator-funded social networking site for people with both HIV and herpes which “just happens” to be written in the slowest web framework you could find. If you still don’t believe me, below is a simple experiment almost anyone with a computer running a recent copy of OEM-installed Windows XP, Windows Vista, Mac OS X can perform.

1. Buy a 128MB memory stick that is compatible with your computer. It will likely cost more money than the equivalent 1GB module.
2. Take all the memory out of your computer.
3. Put the 128MB memory stick in.
4. Hope your motherboard isn’t a dick about paired modules for dual channel memory access.
5. Boot up your computer.
6. Wait about thirty minutes for it to boot.
7. Try to do anything considered cool, like watch a Decemberists music video on iTunes in HD. Good luck. Your iPhone has more memory than that you faux indie motherfucker.

Like the 2009 era Republican party, the OS will flail around desperately trying to get anything done. The basic home computer now has more than 250MB of stuff loaded just to get started, 200MB of which is anti-virus software — or a complete ecosystem of porn spyware because you didn’t have it installed — and the rest being the cute buttons for the interface. It is really a spectacular site watching this drama unfold. Needless to say, you will now be thoroughly convinced that memory is faster than disk.

The OS disk cache does a pretty good job of storing files we read over and over again. For most tasks, it’s great (I want to have it’s babies, etc), but sometimes a specialized, application-specific cache performs significantly better. The application itself (in this case, MySQL) will use a different strategy to read data, one that is better adapted, and much more efficient, at reading data out of memory, namely going nuts on the random access part of Random Access Memory. In some cases, the application cache will even compress data before caching it to allow more data to get packed in there. The small computational penalty paid by uncompressing the data before it can be used far eclipses the massive cost of hitting disk.

MySQL has three such special caches: query, key, and table. The query cache stores the results from past queries in memory. This isn’t so useful for our situation because we don’t have very many repeated queries as the upstream tile cache stores the resulting fully rendered images, preventing MySQL from even getting touched. While this is a good thing, it puts ketchup water all over our burger bun. Next is the key buffer, which stores frequently accessed portions of data indexes in memory. The key buffer saves a huge amount of disk work as index traversal on a large table can often require expensive disk work than fetching the actual data that it ends up pointing to. These days we’re also smart enough to batch table data reads together and do them sequentially as much as we can, saving precious disk I/O time. Unfortunately this isn’t enough — between the various layers of abstraction and planetary alignment, it’s highly unlikely that enough of the table data is actually stored on disk in the right order to make this batching process all that useful. The table cache augments all of these by going the final step — cache even the table data in memory. Again, the lack of repeated requests means we won’t really have data “hot spots” and just plain using caching isn’t going to get us anywhere. The incoming requests are going to be totally random and all over our 3GB table. We’d be lucky to have a warm spot.

So hell with it, we cache it all, which realistically requires available main memory for every last byte of the table and it’s index. This is in Amazon EC2, so we opt for the luxurious “large” instance (OoOoO… large) at $.40/hour, with it’s 7GB of main memory to house our MySQL server. The other instances, the ones that run MapGuide and draw the map images are “medium” sized, and only have 1.7GB of main memory. Even if we setup MySQL on each of these instances, it wouldn’t even come close to the performance of our “large” MySQL dedicated instance. Even if we tied 8 EBS volumes together in a striped RAID array and had Tom Cruise perform his demon voodoo magic. Not even close. The MapGuide instance each eat up all of their available 1.7GB of memory caching the ~1GB of SDF data files we’re using to render the outgoing map images. So we’re going with dedicated instance(s) for MySQL that are configured with no query cache, a 4GB table cache to cache the 3.1GB MYD file, and a 2GB key buffer to cache the 1.9GB MYI file. This gives plenty of room for the OS to do it’s dirty work… oh yeah. Even with all of these cache resources at it’s disposal, the performance when I first start up MySQL is lackluster as I expected.

mysql> SELECT COUNT(*) FROM streets WHERE MBRIntersects(GeomFromText('Polygon((-10018150.920914
4048533.503548,-10009717.327164 4048533.503548,-10009717.327164 4056967.097298,-10018150.920914
4056967.097298,-10018150.920914 4048533.503548))'),SHAPE);
+----------+
| COUNT(*) |
+----------+
|     3779 |
+----------+
1 row in set (2.11 sec)
 

Yuck. We need some way of making MySQL go ahead and fill it’s cache coffers. I use a brute force SELECT scan and a specific LOAD INDEX command to warm the cache — a cool scalable web 2.0rgasm term for pre-loading data into the cache before anyone asks for it.

mysql> select sum(length(shape)) from streets;
+--------------------+
| sum(length(shape)) |
+--------------------+
|         3009892693 |
+--------------------+
1 row in set (1 min 7.76 sec)

mysql> load index into cache streets;
+------------------------+--------------+----------+----------+
| Table                  | Op           | Msg_type | Msg_text |
+------------------------+--------------+----------+----------+
| geography.streets      | preload_keys | status   | OK       |
+------------------------+--------------+----------+----------+
1 row in set (39.64 sec)
 

Using the closely guarded “top” command, we can see that MySQL gorges itself on RAM as I’ve instructed it to.

11557 mysql     15   0 5198m 4.9g 2.9g S    0 65.2   0:50.06 mysqld   
 

That’s 4.9GB of resident memory, just for MySQL. I clear the OS disk cache to keep myself following some iota of scientific method.

# echo 1 > /proc/sys/vm/drop_caches
 

I pulled the MBR (or bounding box) coordinates out of the development MySQL server’s log for queries against this particular table to test the performance a little bit. It now can only be described as stupid fast.

mysql> SELECT COUNT(*) FROM streets WHERE MBRIntersects(GeomFromText('Polygon((-9998307.170914
4053494.441048,-9989873.577164 4053494.441048,-9989873.577164 4061928.034798,-9998307.170914
4061928.034798,-9998307.170914 4053494.441048))'),SHAPE);
+----------+
| COUNT(*) |
+----------+
|     2866 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT COUNT(*) FROM streets WHERE MBRIntersects(GeomFromText('Polygon((-8222291.545914
4772830.378548,-8213857.952164 4772830.378548,-8213857.952164 4781263.972298,-8222291.545914
4781263.972298,-8222291.545914 4772830.378548))'),SHAPE);
+----------+
| COUNT(*) |
+----------+
|     4767 |
+----------+
1 row in set (0.02 sec)

mysql> SELECT COUNT(*) FROM streets WHERE MBRIntersects(GeomFromText('Polygon((-10018150.920914
4043572.566048,-10009717.327164 4043572.566048,-10009717.327164 4052006.159798,-10018150.920914
4052006.159798,-10018150.920914 4043572.566048))'),SHAPE);
+----------+
| COUNT(*) |
+----------+
|     3859 |
+----------+
1 row in set (0.01 sec)
 

Hell yeah I fight people! I wanted even more proof, so I threw some dirty little Ruby scripts together to run a battery of these queries. I pulled 8,000 unique MBRs from the development query log that should represent a gamut of requests from real sessions. My “torture test” uses 8 simultaneous connections, the Ruby-MySQL native compiled driver, and does SELECT COUNT(*) requests as I’m not interested in measuring the marshaling performance at either end. The script emits the number of rows returned as well as the time we waited to get a result back for each request. At the end I run a statistics script that reads the raw output and gives it a decent paying job with good insurance.

For our large EC2 instance with a huge supply of empty Red Bull cans, performance was unimpressive.

EC2 Large Instance, Cold MySQL Cache: 3m44.645s (~35 q/s — 16,463 rows/s)

=== 8000 QUERIES ===
COUNTS: avg 176 (462), med 104, dev 1010.408, 99% <5216, min 0, max 11892, sum 3698505
TIMES: avg 0.054 (0.153), med 0.005, dev 0.542, 99% <2.465, min 0.000, max 16.251, sum 1222.613
 == 0: avg 0.000 (0.004), med 0.000, dev 0.036, 99% <0.001, min 0.000, max 0.784, sum 4.116
 >0: avg 0.065 (0.173), med 0.014, dev 0.574, 99% <1.966, min 0.000, max 16.251, sum 1218.497
 <1000: avg 0.042 (0.110), med 0.004, dev 0.348, 99% <0.993, min 0.000, max 11.641, sum 667.707
 1000+: avg 0.243 (0.566), med 0.128, dev 1.207, 99% <1.775, min 0.009, max 16.251, sum 550.790

One of these requests took 16 seconds to return! Even the max time for the requests between 0 and 1000 rows was 11.641 seconds. The 99% time of 2.465 was just unacceptable. The amount of latency this adds to tile drawing times is brutal. Especially when hitting dense, urban areas at a (relatively) high scale, which can require upwards of 10,000 rows to be returned for each tile. Using iostat, a Linux tool for monitoring disk activity, I could see MySQL’s data partition getting pounded. As the concurrency climbs with these larger queries, the disk I/O starts to get spread out across multiple queries and the aggregate time is even higher. In development it curiously seemed as if it all got put into a queue anyway, as simultaneous tiles would be returned in the order they were requested.

The improvement in performance after I warm the caches is stellar — nearly 25 times the throughput.

EC2 Large Instance, Warm Cache: 9.484s (~843 q/s — 389,973 rows/s)

=== 8000 QUERIES ===
COUNTS: avg 176 (462), med 104, dev 1010.408, 99% <5216, min 0, max 11892, sum 3698505
TIMES: avg 0.003 (0.006), med 0.001, dev 0.026, 99% <0.076, min 0.000, max 0.673, sum 46.914
 == 0: avg 0.000 (0.001), med 0.000, dev 0.018, 99% <0.001, min 0.000, max 0.539, sum 1.290
 >0: avg 0.003 (0.006), med 0.001, dev 0.027, 99% <0.063, min 0.000, max 0.673, sum 45.624
 <1000: avg 0.002 (0.004), med 0.001, dev 0.019, 99% <0.029, min 0.000, max 0.673, sum 22.436
 1000+: avg 0.014 (0.024), med 0.010, dev 0.051, 99% <0.054, min 0.003, max 0.665, sum 23.188

This means we can easily back over 40 MapGuide instances with a single large, dedicated instance for MySQL before performance will even think about degrading. In reality, that’s far more than we’d have before we started another MySQL instance just for failover purposes.

I did notice an odd thing with EBS, what Amazon calls persistent data volumes — disk storage that can survive a server reboot or crash — for it’s EC2 service. EBS volumes are accessed over the local network and are attached and detached by the Xen hypervisor, which provides the virtualization layer for EC2 instances. While EC2 instances are provided with machine-local storage, the data effectively becomes lost after a reboot, so you have to either download what you need to work with every time you boot an instance or use an EBS volume. I’m using an EBS volume to store the MySQL data files for this prototyping work and I’ve noticed some strange performance patterns. It does seem as if there is a decent sized cache somewhere between my instance and the disk drive it’s attached to through the various levels of abstraction of EBS. It seems plausible that Amazon followed the el-cheapo Linux box pattern and the nodes that house EBS are just more of them, which should do their own disk caching.

# dd if=/dev/sdh of=/dev/null bs=8k count=100k
Cold: 840392704 bytes (840 MB) copied, 16.1159 s, 52.1 MB/s
Locally Cached: 838860800 bytes (839 MB) copied, 0.964485 s, 870 MB/s
Warm: 838860800 bytes (839 MB) copied, 10.8145 s, 77.6 MB/s

Next I intentionally crank up the size to about the amount of RAM of an average run-of-the-mill server and things start to slow down.

# dd if=/dev/sdh of=/dev/null bs=8k count=500k
Cold: 4194304000 bytes (4.2 GB) copied, 68.7886 s, 61.0 MB/s
Warm: 4194304000 bytes (4.2 GB) copied, 71.4547 s, 58.7 MB/s

So that sounds about right. After I saw this I had to re-run my tests and force a cache flush to make sure I had the most accurate numbers and I wasn’t just hitting some remote EBS cache boogie man. As you can see, performance of single EBS volumes is about as stellar as a cheap SATA drive.

So… what have we learned today in about 3,000 words?

• Don’t let anything piss on your girlfriend’s toilet.
• Pre-cook your data so comes out filtered and sorted without filtering and sorting.
• Fit everything you can in memory, as if that hasn’t already been said enough.
• A single box that can house all the data in RAM is faster than 20 boxes that have to hit disk.
• MySQL has a few key features that make it totally rock for big tables of read-only data.
• Use MyISAM table compression to get read-only tables as small as possible.
• EBS is an independent woman with it’s own cache and doesn’t need yours, unless there’s a sale at Aldo.
• For Web 3.0, main memory will just not be fast enough, so someone will invent the successor to memcached, cachecached, which will only store data in L2 cache.

TARBALLZ HERE!!!1 (WARNING: This unpacks in ., so it will piss you off and pollute your home directory. You probably deserve it though you self-righteous Linux-using bastard)

# apt-get install python-xml
# `echo 'oldxml' > /usr/lib/python2.5/site-packages/oldxml.pth


C:\Program Files (x86)\Autodesk\MapGuideStudio2009> CorFlags Studio.exe /32BIT+


… and it works flawlessly. Yes, it’s that easy. What a bunch of douchebags.

Squid, a caching proxy, came to the rescue. I’m not sure how they got the name “squid” for “caching proxy,” but regardless, it’s the only free/decent/stable proxy that would do what I needed. First I had to get all the routing setup so it’d be “transparent.” I created IP aliases on both the firewall and the proxy box that allowed me to transmit to-be-proxied traffic over a “black” network (192.168.80.x). The problem with simply just forwarding the packets to the proxy is that the proxy will try to respond to the connection directly back to the source host (instead of the firewall) which won’t have a clue what the proxy is talking about. It all has to be passed thru the NAT process to work correctly. For the individual host-throttling to work, we need the different hosts to be broken out, so just marking everything as sourced from the firewall’s IP was also out. The last 8 bits of of the client host directly map onto the black network (192.168.1.5 -> 192.168.80.5). At this point the proxy box can be administered via the normal network and is listening for requests on the black network.

Cool. Now we’re caching and happy, but how do we do some bandwidth management? Squid delay pools are kind of a black art. It works like a bucket of bytes, where downloads take out of the bucket and a certain amount of “bytes” are added to the bucket every second. If the bucket is 0, no transmission takes place. Of course, in another second, some bytes will be added. If you set the bytes being added to the bucket every second (reserve) to 64 kilobytes and the entire bucket is 512 kilobytes, only the net difference is actually being taken out of the bucket (download speed – bucket reserve = net difference). So for a download that’s using 128 kilobytes a second, you’re only seeing 64 kilobytes being taken out of the bucket every second. That means it’ll take 6 seconds, or 1MB of data transfer, for the bucket to be empty. By this point I wanted to find the largest bucket and dunk my head in it until the pain went away.

To add to the complexity, you can create different buckets for the entire aggregate pool or individuals. I wish I knew more about how it all worked, but after about 3 hours of tweaking the numbers, this seemed to work pretty damn well for our T1…

delay_class 1 2
delay_parameters 1 140000/160000 96000/128000
delay_initial_bucket_level 100

It was not only cloudy outside, but at work as well. An omen suspended in the lower atmosphere covered the building this morning as I gave a presentation that paired a move to the MapGuide product for IMS with the use of Amazon Web Services as a platform for our interactive mapping. I cooked up a fresh PowerPoint with some krispy graphics from Visio. I built a POC in like 2 days with MapGuide Open Source + OpenLayers and the AutoDesk Studio product running on my workstation. Showed this as well as the Studio GUI product. An hour and a half later all the haters were at bay and the boss had bug eyes. It used to be that serving up hits off our IMS servers were the costly part of hosting maps, but with MapGuide + AWS, our plans should reduce it to something like a tenth of the cost of the data licensing.

I’ve now got an AWS account on the company card and I’m giddy like a schoolgirl to mess with all of these little cute doo-dads. Seriously, how could you ask for more of a toy as a developer? I am such a sucker for this stuff. I think over the last week I’ve turned down sex to dream of the cloud. Geeked out in the worst way.

The MapGuide Open Source product is good, but it could definitely use some work just to get everything fleshed out. They’re sort of in the phase between taking a “fair” commercial product and turning into an “excellent” open source project. I’m really hoping I can carve out some time to start contributing patches. I just hope AutoDesk’s product can keep up with the fury of an open source project OR the alternative Maestro GUI gets good. With all of my forced .NET slavery, I might just lend a hand with Maestro since it’ll be a good place to start. We’ll see.

OpenLayers, is, of course, amazing. I’ve been trying to find it a place here since last year.

OpenLayers.Layer.MapGuide.prototype.TILE_PARAMS.USERNAME = "Anonymous";

Not beautiful, but works.