Without a cleanup.policy Kafka topics would eventually accumulate enough events to fill the disk and be unable to accept new events.

We need a way to tell Kafka which events are OK to purge.

The cleanup.policy configuration value controls this. It has three valid settings:

• cleanup.policy=delete - This is the default setting. This policy gives Kafka permission to occasionally remove events older than retention.ms (default: 7 days) or larger than retention.bytes (defaut: infinity).
• cleanup.policy=compact - This policy gives Kafka permission to occasionally remove events that share a key with a newer event as long as min.compaction.lag.ms (default: 0) has passed.
• cleanup.policy=compact,delete - Both settings can be used, and this gives Kafka permission to occasionally remove events using the rules for each setting. Think of this as an OR condition. Events will be removed if delete OR compact says that event can be removed. compact,delete removes the compact-only guarantee of keeping at least one event per key.

I intentionally say “permission to occasionally remove events” because there are no hard guarantees on when partition cleaning will happen. Compacted topics will not immediately remove older events with duplicate keys. delete policy topics will not instantly remove events exceeding the retention config settings. There are lots of nuances to how cleaning works.

This post details what I’ve learned about partition cleaning from using Kafka since 2016; many details are not documented elsewhere. I’ve only found the answers by digging into the source code.

Frequently Asked Questions about Compaction

These are intended to be tl;dr entries for (relatively) quick answers.

Understanding the content below the FAQ will give a more thorough understanding of why I’ve given these answers.

Should I use topic compaction?

Maybe! It depends on the requirements of the topic consumers.

• Do new consumers want to see the latest value for every key?
• Are the message keys finite (unique keys per partition no more than 10s to 100s of millions)?
• Will new events completely replace previous events with the same key? (new events are not deltas from previous events)

If the answer to those questions is “yes,” then compaction is worth considering. If the answer to any of those questions is “no,” compaction will not meet your requirements.

Can I tune compaction so there are never two events with the same key?

No. You’re looking for a database. Try PostgreSQL.

Even if all Kafka config values are tuned to constantly run compaction on a single partition (don’t do this), duplicate events with the same key will exist. Events are appended to the open segment log file on disk and do not replace previous events with the same key until the open segment is closed and the asynchronous background cleaning process selects this partition for cleaning.

With significant tuning, the lifetime of duplicates can be in the single-digit hour range. I’ve never used compaction for anything short of 24 hours.

Can I trigger compaction to run?

No, not directly.

The most direct lever would be to set the topic’s max.compaction.lag.ms to 24 hours to get it to run once a day. This config is not set by default.

I prefer this over setting a low min.cleanable.dirty.ratio (default: 0.5), as that can cause compaction to run much more frequently than necessary on fast-moving topics and cause undue load on brokers.

What is the “dirty ratio,” and what does it mean for a partition to be dirty?

Kafka has no idea how many duplicate events you have on a topic.

Kafka saves events in sequential append-only segment files on-disk for each topic partition. Segment file name are zero-padded and denote the first offset that segment holds.

The dirty ratio is the ratio of the number of bytes in uncleaned, “dirty”, segments (where duplicates might occur) to the total number of bytes for all closed segments.

Here’s an example showing how Kafka uses the dirty ratio as a proxy for how many cleanable duplicates a partition might have.

dirty ratio: 0.25 - clean

We have two previously cleaned closed segments:

• 000000.log holding unique keys between offsets 0-14,999 and is 750MiB in size. It likely does not have 15,000 events, as cleaning would have removed duplicates.
• 015000.log holds unique keys for offsets between 15,000-41,999 and is also 750MiB in size. Again, there will be gaps in offsets where duplicates were removed by cleaning.

We also have 042000.log, a 500MiB closed, immutable segment that is dirty because it has not yet been cleaned. Because it has not been cleaned, it contains all offsets between 42000 and 59999.

The currently open segment, 060000.log, is where new records are appended. It currently has 1GiB of data. It is ineligible for compaction. The bytes in this file are not part of determining the dirty ratio.

We have 750MiB + 750MiB = 1.5GiB of “clean” bytes on this partition. We have 500MiB of “dirty” bytes eligible for compaction that have not been cleaned on this partition.

This partition’s dirty ratio is 0.25 (500MiB dirty / 2iGB total closed), and (with the default min.cleanable.dirty.ratio=0.5) it is considered “clean” and will not be selected for cleaning.

But what happens when we close the open segment?

Dirty ratio: 0.5 - filthy!

Closing the open 060000.log segment has added 1GiB to the number of dirty bytes on this partition.

We now have 1.5GiB of clean bytes and 1.5GiB of dirty bytes. A 0.5 dirty ratio (1.5GiB dirty/3.0GiB total closed). This partition is now eligible for cleaning. The log cleaner thread will clean this partition on the following cleaning pass if it is the dirtiest partition on the broker.

Once cleaning completes, it could look like this:

Dirty ratio: 0.0 - squeaky clean!

Notice a couple of things about these new segments:

• The first starting segment has gone from 000000.log to 001000.log. All keys for offsets between 0 and 999 must have had new events sharing the same key get purged as duplicates. The first surviving offset on this topic is 1000.
• We went from four closed segments to two. As segments are compacted, they reconsolidate into larger segments. segment.bytes determines the maximum segment size.

What configs should I change to get better compaction?

Short answer:

• segment.ms (default: 604800000, seven days) - change to 43200000 (12 hours)
• max.compaction.lag.ms (default: infinity) - change to 86400000 (24 hours)

Changing segment.ms will close the open segment at least every 12 hours so that it is “dirty” and eligible to be compacted. If this is not changed, slow-moving topics (less than segment.bytes=1GiB of compacted data per week) will take one week to close the partition.

Changing max.compaction.lag.ms to 24 hours will make this partition eligible for compactions once per day.

Longer answer:

Most people see min.cleanable.dirty.ratio (default 0.5) and think that’s the correct config to tweak.

Changing the min.cleanable.dirty.ratio is a poor choice for almost all topics. Most people don’t understand what this does and it leads to behavior that can be hard to understand. It can also cause a partition to be cleaned far more often than is useful on fast-moving topics and can crowd out cleaning on other partitions.

I prefer setting segment.ms to 12 hours instead of 7 days, as closing a partition twice a day is not expensive. This is better than changing segment.bytes (default 1GiB of compacted data) as that can be dangerous for fast-moving topics. Setting segment.bytes to 1MiB on a topic producing gigabytes of data per day would create thousands of files on disk. If we set segment.ms to 12 hours and it only has a few megabytes of data, compaction will reconsolidate those segments back into larger segment.bytes-sized segments when compaction runs.

Setting max.compaction.lag.ms to 24 hours will make the partition eligible for cleaning once per day. It does not force the log cleaner to clean it exactly at the 24-hour mark. The log cleaner will still clean the “dirtiest” eligible partition. This setting makes the partition eligible once per day.

I published a new event with the same key as an older event, when will the older event be deleted?

With default topic settings on a slow-moving topic, duplicate events can take weeks to compact. With a tuned topic, you’re likely looking at ~24 hours before duplicates are removed.

Many things need to happen before an event is removed by compaction:

1. The event’s open log segment needs to be closed so it is no longer getting new events appended.
2. The partition must be eligible for cleaning either because max.compaction.lag.ms has elapsed or the min.cleanable.dirty.ratio for the topic has been breached.
3. The partition needs to be selected for cleaning by the log cleaner. The log cleaner always picks the “dirtiest” partition to clean, regardless of the min.cleanable.dirty.ratio.
4. The log cleaner needs enough memory to clean all unique “dirty” keys in a single pass. Otherwise, multiple cleaning passes will be necessary. With default settings, the broker can clean no more than ~5 million keys on a partition in a single pass

Another FAQ entry has suggestions for configs to consider if you’re not seeing the compaction you want.

How do I delete all events sharing a key on a compacted topic?

Publish a new event with the same key and a null body to the partition for that key. This is called a “tombstone”. Tombstones signal to the Kafka broker and consumers that the key should be removed.

If a new event is published after the tombstone with the same key, that new event will be kept by compaction.

I still see tombstones on my partition, when will they be removed?

Tombstones must be cleaned at least twice before being removed from a partition.

The first time the log cleaner sees the tombstone, it is not deleted as that could cause a race condition for consumers. If a tombstone was added to the open segment just before it is closed, then compaction runs right after it is closed. That might not give consumers enough time to see the tombstone before it is deleted.

Kafka avoids this race condition with delete.retention.ms (default 24 hours). This config value (which should be named tombstone.retention.ms IMO) determines how long the tombstone should be preserved in the logs after its first cleaning.

When the log cleaner first sees a tombstone, it writes it to the new segment along with a timestamp that it is safe to delete (cleaning time + delete.retention.ms). Subsequent cleaning passes will only remove the tombstone if it is after that timestamp.

How can I tell if compaction has failed on a partition?

Partitions can be marked as “uncleanable” if the log cleaner hits an exception while trying to clean it. I’ve seen this happen when segment file on disk is corrupted and is unable to be parsed.

The WARN-level log message will look like this:

[kafka-log-cleaner-thread-0]: Unexpected exception thrown when cleaning log Log(dir=/data/kafka/fruit-prices-0, topic=fruit-prices, partition=0, highWatermark=1256427521, lastStableOffset=1256427521, logStartOffset=1165979137, logEndOffset=1256427521). Marking its partition (fruit-prices-0) as uncleanable

As cleaning is done independently on each broker, as long as the topic has a replication factor greater than one, the broker with the uncleanable partition can be removed from the topic. A new broker will be elected as a partition replica, and it will copy the partition from another (hopefully healthy) replica.

When cleaning runs, is it guaranteed to clean all duplicates?

Nope.

There are two reasons why duplicates will not be cleaned in a single pass.

1. A newer key is in the open segment. Cleaning looks at closed segments. If the latest event is still in the open segment (with default settings, this can take seven days), the log cleaner will not see the new event.

2. The number of unique keys on a partition is more than the log cleaner can keep in memory. The log cleaner will clean the subset it can remember. The partition will then need to be selected again as the most “dirty” partition for subsequent cleaning.

Kafka brokers have config values controlling how many unique keys it can clean in a single pass.

See “Log Manager and Log Cleaner” for a lot more detail.

How can I count the number of records on a topic?

The math is easy if the topic is not compacted (cleanup.policy=delete). Take the offset of the latest record and subtract the offset of the earliest record still on the topic. No compaction has happened on this partition, so all events between those offsets are guaranteed to exist still.

There’s no quick and accurate answer if the topic is compacted (cleanup.policy=compact). You’ve got a few options:

1. Iterate over all events on the topic and count them. Iterating is the only way to get an exact count.
2. Iterate over all of the events on a partition and count them. Then multiply that by the number of partitions. You can check the size on the disk of all partitions to ensure a good distribution of events.
3. Do an exact count once, then determine the average size of events given the compression ratio. In the future, divide the current size of the partition by that known average event size for a reasonable estimate.

What signals tell if compaction cannot fully clean a partition in a single pass?

If you’ve got access to your Kafka brokers logs, look for messages like this that are a signal that the partition cannot be cleaned in a single pass:

INFO log level kafka.log.LogCleaner:

[kafka-log-cleaner-thread-2]:
  Log cleaner thread 2 cleaned log fruit-prices-12 (dirty section = [310715098, 325968881])
  25,706.8 MB of log processed in 1,554.6 seconds (16.5 MB/sec).
  Indexed 12,405.3 MB in 484.9 seconds (25.6 Mb/sec, 31.2% of total time)
  Buffer utilization: 90.0%
  Cleaned 25,706.8 MB in 1069.7 seconds (24.0 Mb/sec, 68.8% of total time)
  Start size: 25,706.8 MB (49,042,590 messages)
  End size: 13,219.4 MB (33,931,680 messages)
  48.6% size reduction (30.8% fewer messages)

DEBUG log level kafka.log.LogCleaner:

Cleaner 3: Offset map is full, 1 segments fully mapped, segment with base offset 470931255 is partially mapped

Notice the Buffer utilization: 90%? That’s our signal that we’ve hit the 0.9 default value for log.cleaner.io.buffer.load.factor. The number of unique keys on this partition exceeds what can be cleaned in a single pass.

Not having enough room in the OffsetMap for all keys in a single segment is very expensive. Each cleaning pass needs to create a map of dirty keys and then compare those keys against all clean keys to remove duplicates.

What is the maximum number of unique keys I should have per partition?

Low tens of millions with default settings to low hundreds of millions with modified cleaner settings.

With default settings, ~5 million dirty keys on a partition can be cleaned in a single pass. See “Log Manager and Log Cleaner” for more details.

How can I tell when compaction last ran on a partition?

Kafka broker logs are the best place to check.

If you don’t have access to those, but you can consume from a topic using kcat and have GNU awk installed, you could try this script. This will find the date of the earliest event where the prior offset is no longer present on the topic. This is the last place where compaction provably happened.

This script uses kcat to emit the offset (%o) and timestamp (%T) of each event on partition 0 of my-topic on mybroker.example.com:9092. If your topic has security on it, you’ll need to add additional flags to kcat to provide those credentials.

kcat -C -e -q -b "mybroker.example.com:9092" -t "my-topic" -p 0 -f '%o\t%T\n' |
  gawk'
    $1 > last_offset + 1 {
      last_gap_offset = $1
      last_gap_epoch = $2
      last_gap_row = NR
    }
    {
      last_offset = $1
      last_row = NR
    }
    END {
      print "last gap offset: ", last_gap_offset
      print "last gap row: ", last_gap_row
      print "last gap time: ", strftime("%Y-%m-%dT%H:%M:%S", substr(last_gap_epoch, 1, 10))
      print "last offset: ", last_offset
      print "last row: ", last_row
      print "contiguous ~dirty since gap: ", last_row - last_gap_row
    }
  '
last gap offset:  4156372102
last gap row:  39182297
last gap time:  2023-07-28T11:01:18
last offset:  4161793692
last row:  44603887
contiguous ~dirty rows since gap:  5421590

So this shows that the last time compaction ran on partition 0 on the lead broker was July 28th, 2023.

The contiguous ~dirty rows since gap value shows how many events are on the topic that could be dirty and eligible for compaction (once they are in a closed segment).

This script is the most accurate on fast-moving topics where duplicate events are relatively common.

Can a topic be compacted and use time-based deletion?

Yes! If the topic uses cleanup.policy=compact,delete.

The partitions on this topic will delete all records older than retention.ms (default: 7 days), even if they are the only record with that key.

Within that retention.ms period, closed segments will be compacted if the partition’s “dirty ratio” (default 0.5) threshold is surpassed. The dirty ratio is the ratio of bytes on “dirty” closed segments that might have duplicates to total bytes (clean + dirty) in closed segments.

Compaction with deletion enabled removes the guarantee of preserving at least one value per key. It adds a time horizon where even unique keys will be dropped.

compact,delete is used in situations where:

• values are time-sensitive and are not useful after some time
• downstream consumers do not need an explicit tombstone to signal deletion
• there is a periodic repopulation of all valid records on the topic

An example would be a topic with a 3-week retention.ms, and new values for the currently valid objects are published at least once per week. This allows compaction within the 3-week window but values older than that will drop off the topic.

How can I use compaction but give consumers a guaranteed time window to see all messages before they are compacted?

You’re looking for min.compaction.lag.ms (default: 0). Set this to the number of hours/days that your consumers will be guaranteed to have consumed an event.

This gives you the advantages of compaction but guarantees consumers get time to see all events.

If I have a compacted topic, can I change the number of partitions?

Yes, but you will need to clear all data on the topic first. Then change the number of partitions. Finally, replay the upstream data so that all values are on the correct partition. If you don’t have a non-Kafka source of truth for the data, you could create a job to forward all existing events to a new topic with the desired partition count.

Kafka does not have any magic around re-partitioning. Events are not shuffled as Kafka has no idea what partitioning strategy the publishers used to send messages.

If you do not do this, you will have old messages that live forever and that will never get updated. Compaction is done per-partition and does not look at events that might share the same key on other partitions. It assumes a consistent hashing strategy where all values with the same key are guaranteed to be on a single partition.

How Topic Compaction Works

Kafka stores events published to a topic on a partition (topics have 3 partitions by default). Each event has a key, a value, and optional headers. Kafka appends new events to the end of that partition’s open segment log file. Events are assigned an offset that is a unique identifier for this event. As events get appended to the end of the log, they do not share the offset or replace any previous events with the same key.

Events with the same key are guaranteed to all be stored on the same partition. All new events will have an offset value one greater than the previous event.

Compaction happens at the partition-level, not at the topic-level. If your topic somehow has the same key on multiple partitions, both partitions will keep a different event for that key (most common if the number of partitions are changed).

When compaction runs, it creates a map of the new keys on the partition paired with the highest offset of any event seen with that key. It will then scan the partition from the beginning and purge older events now known to have been replaced by a newer event.

Compaction is enabled by setting the topic’s cleanup.policy configuration value to compact.

Partition Compaction Example

Kafka stores partitions on disk in log files called “segments”. Each partition has one “open” segment where new event record batches will be appended. Each event in the record batch will be assigned a monotonically increasing offset. The first event on a partition will be at offset 0.

Here we’ve got a newly created topic, fruit-prices. It likely has multiple partitions, but we’re going to look at how events published to partition 0 are added and then compacted away.

The open segment file is 000.log (the file name denotes the first offset in the segment with some zero-padding).

The first event comes in with a key of grape and a value of $2.69. It is appended to the open segment at offset 0.

A second event is published with the key lime and a value of $0.49. It is appended to the open segment at offset 1.

A third event, with a null body, shares the grape key with our earlier event. Events with null bodies are called “tombstones” and signal to Kafka and consumers to delete all prior grape values. This event is appended to the open segment at offset 2. Notice that the previous grape value still exists at offset 0.

The next event also has the lime key, with an updated price value of $1.59. This event is again appended to the open segment at offset 3. The lime event at offset 1 still exists and will be read by consumers, but they will eventually see lime’s updated price at offset 3.

Until now, no compaction has occurred or could occur, as compaction never runs on the open segment.

A segment must be closed for it to be eligible for compaction. Segments are closed for two reasons:

1. Enough time passes. When a segment has been open for segment.ms it will be closed. The default is 604800000 (seven days).
2. Enough data has been published to the partition. When the open segment has had segment.bytes of events written, it will be closed. The default is 1073741824 bytes (1GiB). This includes the event keys, bodies, headers, and a small number of bytes for each record batch. When topic compression is used, this byte size is calculated after compression.

If you’re using a compression.type of zstd and getting 90% compression on a topic with ten partitions, you’d need to write approximately ~100GiB of data to the topic for the open segment to exceed 1GiB (default segment.bytes) and be closed in less than 7 days (default segment.ms). 1/10th of the keys are assigned to each partition, and each event is compressed to 1/10th of its original size, so we need to write ~100x the segment.bytes value within segment.ms for the open segment to be closed.

For our example, let’s say that seven days have passed since we wrote the first four events, hitting the default segment.ms threshold. The 000.log segment is closed and marked as “dirty,” as it might have duplicates. A new segment, 004.log, is opened, and a new event with the key lime is appended to the open segment with an offset of 4.

We need to talk about “dirty ratios”. Kafka does not know how many duplicate keys are on a partition. It only knows the number of uncleaned bytes in closed segments versus previously cleaned bytes in closed segments.

The dirty ratio is the ratio of bytes for dirty events in closed segments divided by the total number of bytes in closed segments. As this topic has never been cleaned, its dirty ratio is 1.0.

This partition will be selected for cleaning when its dirty ratio exceeds the 0.5 value that is the default min.cleanable.dirty.ratio.

Kafka adds this partition to the list of partitions ready to be cleaned by a log cleaner thread. The log cleaner scans the closed, dirty 000.log segment where duplicates might occur. It creates a map of 16-byte MD5 hashes of the keys and the highest offset (an 8-byte long) seen for that key within the dirty segment. Each md5(key) and offset pair takes up 24 bytes of memory in the log cleaner’s map.

md5(grape): 2
md5(lime): 3

Then the log cleaner starts reading from the beginning of the partition and creates a new “clean” segment file that keeps only the latest offsets for each key. The new segment file is called 002.log as offset 2 is the first offset it contains. This new clean segment is closed when all unique dirty events in closed segments have been processed, or when the new segment hits the segment.bytes (default 1073741824/1GiB). The new clean segment is swapped in to replace the old 000.log segment.

The grape event at offset 0 and lime at offset 1 are not copied into the new segment as their offsets are less than the highest offset the log cleaner has found in the dirty section for those keys.

Notice that the grape tombstone at offset 2 was kept. When written to 002.log, Kafka added a timestamp to the tombstone when it is safe to delete (cleaning time + delete.retention.ms). Kafka will keep tombstones (and transaction markers) to avoid race conditions for at least delete.retention.ms (default 24 hours) after the first cleaning.

Tombstones need to be cleaned at least twice before they are removed.

I believe delete.retention.ms is a confusingly named config value. It would be better with a name like “tombstone.retention.ms”. This property only controls when tombstones/transaction markers can be removed from the partition. Many engineers I’ve talked with wrongly believe it controls how long Kafka will retain all events before deletion (it does not).

We also have a lime event in the open segment at offset 4, but as the Kafka cleaner does not look at the open segment, we still have the lime event at offset 3 visible to consumers.

After compaction, the dirty ratio for the partition above is 0.0. All closed segments have been cleaned and are free of duplicates. The partition will not be cleaned again until the open partition is closed.

More time passes, and we’ve written a few more values for guava and kiwi to our 004.log open segment. After a week, segment 004.log is closed and marked “dirty”. A new segment, 008.log, is opened, and a new value for guava is appended at offset 8.

Our dirty ratio (assuming all events are the same size) is now 0.67 (two events in the previously cleaned segment/six events in all closed segments).

The 0.67 dirty ratio exceeds the min.cleanable.dirty.ratio of 0.5, and the partition is eligible to be cleaned. The cleaner scans over the dirty offsets in the newly closed segment and creates a map MD5 key hashes to the highest offsets seen for that key:

md5(lime): 4
md5(guava): 6
md5(kiwi): 7

Notice that grape is not in the map. That key only exists in the previously cleaned portion of the topic, where all keys are unique.

The log cleaner scans from the beginning of the partition and writes unique values to the new segment.

More than 24 hours (delete.retention.ms) have passed since first cleaning the grape tombstone, so it can finally be omitted from the new segment.

The new segment is named 004.log; its first offset is 4.

After cleaning, our partition again has a dirty ratio of 0.0. All closed segments are guaranteed to have unique keys.

Kafka’s append-only design with background cleaning threads allows it to read and write quickly. Writing never needs to read previous values. It only appends them. Reading can stream sequential segment files to consumers as they are already sorted by offset.

Log Cleaner Architecture and Configuration

Kafka brokers have many configuration values that impact how cleaning works.

If brokers aren’t given enough cleaning resources, your partition might never get cleaned because the cleaning time is spent on the “dirtiest” partitions.

LogManager and LogCleaner

Each broker has a single LogManger thread and zero or more LogCleaner threads.

The LogManager is responsible for:

• truncating any partitions with a cleanup.policy of delete
• creating any LogCleaner threads, LogCleaner threads handle any partitions with a cleanup.policy of compact

At startup, the LogManager checks to see if log.cleaner.enable is true (by default, it is on). If it is enabled, it will create log.cleaner.threads (default: 1) LogCleaner instances.

Topics with cleanup.policy of delete

The LogManager removes events on partitions with a cleanup.policy=delete (but does not handle those with compact or compact,delete).

On startup, it schedules a kafka-log-retention thread that will run every log.retention.check.interval.ms (default 5 minutes).

When the kafka-log-retention job executes, it will start at the oldest log segment and delete entire segments where the newest events are older than retention.ms or that bring the partition log size above retention.bytes.

Note that this is intended to be a quick operation. Segments are not split. All events must be older than the retention period for a segment to be deleted.

Topics with a cleanup.policy of compact

If log.cleaner.enable is true and log.cleaner.threads is greater than 0, the LogManager will instantiate independent LogCleaner threads that are responsible for compacting partitions with a cleanup.policy of compact.

Each LogCleaner thread runs independently, but locks ensure that two cleaning threads do not attempt to clean the same partition.

The high-level steps LogCleaners use for compacting topics are:

1. Determine which partitions are eligible for cleaning. A partition is eligible for cleaning either because:
   • the “dirty ratio” of uncleaned bytes in closed segments divided by bytes on all closed segments is compared against the min.cleanable.dirty.ratio (default 0.5). Note that this has nothing to do with the number of duplicate keys. Kafka has no idea how many duplicate keys exist in a dirty segment till it is cleaned.
   • the partition has closed segments that have not been cleaned within max.compaction.lag.ms. By default, this configuration is set to 9223372036854775807 (max long) so it is not enabled. I recommend setting this value to the cadence you’d like cleaning to happen (ex: 24 hours) rather than changing the min.cleanable.dirty.ratio. It will give much more predictable behavior.
   • if no partitions are eligible for cleaning, the thread sleeps for ~15 seconds and then checks again.
2. Pick the partition with the highest dirty ratio from the list of partitions eligible for cleaning.
   • note that this means that if one high-traffic partition is constantly the dirtiest one, it can delay the cleaning of other less dirty partitions.
3. Scan over the dirty events on the partition, add the 16-byte MD5 hash of the key and the current offset (an 8-byte Long) to the OffsetMap.
   • Each entry in the OffsetMap takes up 24 bytes of memory.
   • To prevent OutOfMemoryError exceptions, the size of the OffsetMap is constrained.
   • log.cleaner.dedupe.buffer.size (default: 134217728/128MiB) is divided equally across all LogCleaner threads. If there is one thread, it gets all 128MiB. If there are four threads, each one gets 32MiB.
   • log.cleaner.io.buffer.load.factor (default: 0.9/90%) is the maximum amount of memory used in the OffsetMap. Inserts into the OffsetMap are slower when the map approaches capacity.
   • So if we have a single cleaning thread with 128MiB of memory and a buffer load factor of 90%, we only have 115.2MiB of usable space.
   • With each entry taking 24 bytes, we can fit slightly over 5 million keys into the 115.2MiB ByteBuffer backing the OffsetMap
   • If your partition has more than 5 million unique keys on it, it will not be able to be entirely cleaned in a single pass! You can either increase the number of partitions on the topic or give the cleaner more memory.
4. Scan from beginning of the partition and write events to a new clean segment if:
   • The md5(key) is absent in the OffsetMap. This event has the key’s highest known offset that we want to keep.
   • The offset for this event is the same as the offset stored at md5(key) in the OffsetMap. This event is the newest seen with that key.
   • If the compression.type of the topic is producer (the default), maintain the compression type for this event’s record batch. Otherwise, use the compression type specified for the topic.
5. If we find a tombstone or transaction marker in the “clean” section, check the age of the event’s last cleaned timestamp. If it is greater than delete.retention.ms, delete it. Otherwise, preserve it. Tombstones must be cleaned at least twice before they’re removed from the partition.
6. If we write a maximum of segment.bytes (default: 1GiB of data using the compression.type) to the new clean segment, close it and open a new segment.
7. Swap completed segments in for the old segments and mark old segments for deletion.
8. We’re done if the cleanup.policy is just compact. If cleanup.policy=compact,delete, the LogCleaner will then delete segments using the same logic above used by the LogManager for cleanup.policy=delete.
9. GOTO #1, where the LogCleaner will recalculate the “filthiest” partition. If this partition was not fully cleaned, it could pick the same partition again.

Further Resources on Compaction

• The official Confluent Kafka documentation on compaction
• Topic Compaction, by Jun Rao, as part of Confluent’s “Kafka Internals” video series.
• KIP-58: Make Log Compaction Point Configurable
• KIP-71: Enable log compaction and deletion to co-exist
• KIP-87: Add Compaction Tombstone Flag
• KIP-280: Enhanced Log Compaction
• KIP-354: Add a Maximum Log Compaction Lag

Broker/Topic Configs

• min.insync.replicas=2 (default: 1)
• compression.type=producer (keep this default)
• create topics with a partition replication factor of 3

Producer Configs

• acks=all (default: 1)
• compression.type=lz4 (default: none)
• linger.ms=100 (default: 0, Kafka Streams uses 100 ms)
• max.request.size=4194304 (4MB, default: 1048576 - 1MB)
• batch.size=972800 - ~95% of topic’s message.max.bytes (default: 16384 - 16KB)
• buffer.memory=67108864 - increase for high-throughput apps (default: 33554432 - 32MB)
• if order matters
  • add a key to all of your messages
  • max.in.flight.requests.per.connection=1 (default: 5)

There are caveats for all of these, read on for further discussion.

Who is this post intended for?

This is the information about understanding Kafka partitioning and replication that I’ve been trying to search for the last couple of years. I finally stopped searching and read the source code.

This post assumes quite a bit of familiarity with how Kafka works at the 10k foot level. It dives into the details and describes which config settings you should be looking at when tuning your system.

How are Kafka Topics structured?

Kafka topics hold a series of ordered events (also called records). The mango record is the oldest and cherry is the newest. Any new records would be added after cherry.

This conceptual model breaks down when you learn more about Kafka. In reality, Kafka topics have a number of components to increase the reliability and throughput of each topic. These components also increase complexity and are worth learning more about if you want to be able to reason about how your data will flow through the system.

Topic Partitions

A topic is split into one or more zero-based partitions, with each partition having approximately the same fraction of records on it.

It is the Kafka producer’s job to create new records and decide which partition a record should go to. It uses a partitioning strategy to decide which partition a record should be sent to.

The default partitioner uses this algorithm:

1. if the user assigned an explicit partition, use it
2. else if the record has a key, use the key’s hash to determine the partition
3. else add the record to a batch and send it to a random partition

Default Partitioner Algorithm Details

1. If the user assigned an explicit partition, use it

If the producer has manually assigned a record to a partition, it will honor that choice. If explicit partitions are used, it is the producer’s responsibility to ensure that records are balanced across the partitions.

This is useful when you want to partition your records using something other than just the key.

Kafka uses this strategy internally on its __consumer_offsets topic that tracks the state all consumers’ group.ids. Messages are keyed on group.id + topic name + partition, but are partitioned on just the groupId (using String.hashCode()).
This allows new consumers to subscribe to a single partition that will hold all updates for their group.id.

Example command to see data on the __consumer_offsets topic

kafka-console-consumer --bootstrap-server localhost:9092 --topic __consumer_offsets \
  --formatter 'kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter' \
  --partition 0 --from-beginning

Here are two example output lines for from __consumer_offsets partition 0. It shows two records for the my-group-id-14 group id on the fruit-prices topic. One for partition 1 with a commit value at offset=5497, and another for partition 0 with a value of offset=3550.

[my-group-id-14,fruit-prices,1]::OffsetAndMetadata(offset=5497, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1619425537926, expireTimestamp=None)
[my-group-id-14,fruit-prices,0]::OffsetAndMetadata(offset=3550, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1619425553694, expireTimestamp=None)

2. else if the record has a key, use the key’s hash to determine the partition

If the record hasn’t been assigned an explicit partition, but it does have a key, then the partition is determined by calculating the murmur2 32-bit hash of the key turning that into a “positive value” with some clever bit twiddling and then modulus the number of partitions on the topic¹.

positiveValue(murmur2_32("grape")) % 3 -> partition 0
positiveValue(murmur2_32("lime")) % 3 -> partition 0
positiveValue(murmur2_32("banana")) % 3 -> partition 1
positiveValue(murmur2_32("apple")) % 3 -> partition 1
positiveValue(murmur2_32("mango")) % 3 -> partition 2
positiveValue(murmur2_32("cherry")) % 3 -> partition 2

Each record is stored with an offset number. An offset is a monotonically increasing sequence number that starts at 0. A partition will never change the offset assigned to a record and will never assign a different record to that offset.

Example: The record with the key "lime" hashes to partition 0. It was assigned an offset of 1 because it is the second record on that partition.

Adding a key to a record will ensure that all messages with the same key will be sent to the same partition. This is useful for consumers who only care about a subset of the data. It also ensures that there aren’t race conditions when updating a value or, if you’re using a compacted topic, that the latest value is preserved.

3. else add the record to a batch and send it to a random partition

If the record wasn’t assigned an explicit partition and it doesn’t have a key, it will be added to a batch of records that will be sent to a random partition.

After each record batch is sent, a new partition will be picked for the next batch (assuming there is more than one partition for the topic).

Producer Record Batching

Once the partition for a record has been determined, the producer will add that record to a record batch destined for that topic partition.

Record batches have a few benefits:

• more efficient compression, all records in the batch are compressed together
• they reduce the number of individual requests to the Kafka brokers which reduces overhead and increases throughput

Record Batch Compression

Compressing record batches trades a little bit of CPU for a significant reduction in network bandwidth and disk space usage.

Unless the values of your records are already compressed, your producers should be using compression. Start with compression.type of lz4 in your producer config. It can be worth comparing lz4 compressed size and performance with snappy and zstd with your real data.

Don’t use gzip, it is almost always strictly worse than the other options.

Even if your individual record values are already compressed, there can be advantages to using compression as the entire record (including the key, value, headers and other attributes) will be added to a record batch. This lets the compression algorithm find repeated sections across all records in the batch.

Most broker/topic configs should keep the default compression.type of producer. This lets the Kafka broker know that the producer will handle compression and it will not spend time re-compressing the record batch.

If your broker specifies a codec that is different than the producer’s codec it will re-encode the batch with the broker’s preferred codec. This can put significant burden on the brokers’ memory and CPU. It is better to offload compression to the producers if possible.

How Many Records will fit in a Batch?

Napkin math: (average compressed record’s size in bytes)/(batch.size bytes - 61 bytes)

The batch record header always takes up 61 bytes.

So if your average compressed record (including key, value, and headers) is 500 bytes and you use the batch.size of 16384, you can expect to have 32 records in the average batch.

If your average compressed record takes up 32KB, every “batch” will be a record by itself.

Batch Config Details

A producer will batch up to batch.size bytes (default 16384 - 16KB) of compressed records (minus a little overhead for the batch’s metadata) per topic partition.

When creating a new batch, Kafka will allocate a byte buffer for that batch that is the larger of batch.size (default 16384) or the uncompressed size of the record it wants to add to the batch. It then compresses the record and adds it to the batch.

When the next record comes in, if that batch hasn’t been sent yet, it will see if the existing batch has room for the record by seeing if the uncompressed record will fit into the batch. If there isn’t room, it will close that batch and allocate a new buffer with this

If there is room, it will compress and add the record to the batch and leave the batch open for new records to be added.

This means that if all of your messages are 32K uncompressed, every batch you have will have a single record in it.
If you increase the batch.size to 32K, you will still only have a single record in each batch. To get 2 or more records in a batch, your batch.size must be greater than (uncompressed message size) + (prevously compressed message size).

In testing messages that were 750KiB in size, I’ve gotten 3x the performance by increasing my batch.size from 750K to 950K because compression was allowing 2 compressed messages to fit in the batch and still seeing enough room to add one more 750K message.

Additionally, the producer will allocate a pool of buffer.memory (default: 33554432 - 32MB) bytes that is shared for all active and in-flight compressed record batches across every topic partition.

The pool of buffer.memory will be allocated in batch.size chunks that are cleared and reused by the producer for garbage collection efficiency reasons.

If a batch isn’t full it will wait up to linger.ms milliseconds (default 0) after the last message has been added to the batch before sending it. linger.ms is not enabled by default, but can help reduce the number of requests sent without much additional latency if you set it to a low value.

So if your average compressed record is 32KB, you’ll want to test out significantly increasing batch.size by
10-20x and possibly also buffer.memory depending on how many unique topic partitions you could produce to.

If you increase batch.size, you’ll want to be sure to use compression. I believe the existing setting is for compression.type of none. In my testing, larger batches left at none do far worse, but with compression on can do much better.

The maximum value of batch.size should be no larger than about 95% of message.max.bytes for the topic/broker. Values much larger than this risk being rejected by the broker and sent back to the producer to split up and resend in smaller batches. If a broker has 1MiB (1024*1024) message.max.bytes, I suggest trying batch.size=972800 (1024*950, 950KiB) to allow large batches without them being too large.

What Configs Limit the Maximum Record Size?

The producer’s max.request.size (default 1MB) config controls the maximum size of a record in uncompressed bytes².

This can be a very confusing setting, because there is a similar sounding broker / topic config called message.max.bytes that also has a default of 1MB, but message.max.bytes is what the broker will accept after compression.

This means that if you’re compressing messages on the producer and want to send compressed records as large as 1MB over the wire, you will want to increase the producer’s max.request.size by 400%+ to 4MB (this assumes your compressed messages are <25% the uncompressed size). The broker will still reject batches with compressed records that are above its message.max.bytes limit with a RecordTooLargeException.

Will a Broker ever store records in a different order than they were sent?

With the default settings? Yes, record batches on the same partition can end up in a different order than intended.

Records within a record batch will always be in the order that they were sent.

By default, a producer will retry failed requests and it also allows multiple concurrent record batches to be in-flight to a broker simultaneously.

With these settings, if record batch A fails and is retried, it could arrive after record batch B has been saved and acknowledged by the broker.

Some options for preventing this:

1. Set max.in.flight.requests.per.connection=1 to allow only one batch to be sent to a broker at a time³.
2. Set retries=0 and crash before committing if a publish fails. On restart, the messages will be tried again.
3. Keep track of the offset assigned to records in the producer’s send callback and crash before committing if they are ever out of order. On restart, the messages will be sent again.
4. Turn on idempotent messages with enable.idempotence=true or processing.guarantee=exactly_once with Kafka Streams. This adds transactions and “exactly once” delivery of messages which have their own costs.

Option #1 is probably the easiest, though it’s worth testing the effect of your system’s throughput.

Partition Leader and Replicas

For each topic partition, there will be a broker that is the partition’s “leader”, and zero or more “replicas”.

A topic has a replication factor that determines how many copies of data should be kept. A replication factor of 1 means that the records will only exist on the partition leader. If that broker crashes, the data can be lost. All replicas after the first are backup copies of the data. Their data will only be used if the leader dies, or a new leader is elected.

Data is written to, and read from, the broker that is the partition’s leader. If you try to consume directly from a replica, it will redirect you to the current leader for the partition.⁴

For topics where you want to avoid data loss, you should have a replication factor of 3 (all data on the leader + 2 replicas) and set min.insync.replicas=2 to ensure that the leader and at least one replica must acknowledge receipt of the record.

The producer should use acks=all (default: 1). This tells the producer to wait for the leader and all currently in-sync replicas to acknowledge that they’ve persisted the record. The default of 1 will only wait for the leader to acknowledge it, if the leader dies before the data is replicated those records could be lost.

Replication Example

Our fruit-prices topic has 3 partitions, a replication-factor of 3, min.insync.replicas=2 and a producer using acks=all.

Scenario 1: all replicas are in-sync with the leader

When producing a record, it will be sent to the leader (Broker 101) as well as both in-sync replicas (Brokers 100 and 104).

When consumers read a record, it will always come from the leader (Broker 101).

If you describe a kafka topic in this state you’ll see:

kafka-topics --bootstrap-server 127.0.0.1:9092 --topic user-events --describe
Topic:user-events	PartitionCount:3	ReplicationFactor:3	Configs:min.insync.replicas=2,cleanup.policy=compact,segment.bytes=1073741824,retention.ms=172800000,min.cleanable.dirty.ratio=0.5,delete.retention.ms=86400000
Topic: user-events  Partition: 0	Leader: 101	Replicas: 101,100,104	Isr: 101,100,104
Topic: user-events  Partition: 1	Leader: 104	Replicas: 104,101,102	Isr: 104,101,102
Topic: user-events  Partition: 2	Leader: 102	Replicas: 102,100,103	Isr: 102,100,103

The Isr (“In Sync Replicas”) column shows that all replicas are in-sync with the leader of the partition.

Scenario 2: only one replica is in-sync with the leader

If one broker becomes unreachable, or is lagging behind for some reason, you’re still able to produce to the topic and consume from it.

This is why having a replication factor of 3 is often desirable. The cluster can tolerate a broker being down but still maintain guarantees about not having the data in a single, vulnerable place.

The "guava" record is able to be produced to Broker 101 and replicated to Broker 100. The consumer is able to consume the "guava" record from the leader, Broker 101. Broker 104 being out of sync doesn’t stop the cluster from continuing to work.

If you describe a Kafka topic in this state it will look like this:

kafka-topics --bootstrap-server 127.0.0.1:9092 --topic user-events --describe
Topic:user-events	PartitionCount:3	ReplicationFactor:3	Configs:min.insync.replicas=2,cleanup.policy=compact,segment.bytes=1073741824,retention.ms=172800000,min.cleanable.dirty.ratio=0.5,delete.retention.ms=86400000
Topic: user-events  Partition: 0	Leader: 101	Replicas: 101,100,104	Isr: 101,100
Topic: user-events  Partition: 1	Leader: 104	Replicas: 104,101,102	Isr: 104,101,102
Topic: user-events  Partition: 2	Leader: 102	Replicas: 102,100,103	Isr: 102,100,103

The Replicas column shows that there are three replicas, but only 2 Isr (In Sync Replicas): 101 and 100. Broker 104 is a replica, but it is not in-sync. It is likely trying to catch up, but this is a signal that the broker should be looked at to see why it is not caught up if it lasts more than a minute or two.

Scenario 3: no replicas in-sync with the leader

No replicas are currently in-sync with the leader, all producers are halted because the cluster cannot meet the min.insync.replicas=2 setting.

Consumers can still retrieve any values they haven’t seen (as long as they can reach the leader), but will soon run out of events to process.

If you describe this Kafka topic you’ll see that there is only a single Isr (“In Sync Replica”), Broker 101, the leader.

kafka-topics --bootstrap-server 127.0.0.1:9092 --topic user-events --describe
Topic:user-events	PartitionCount:3	ReplicationFactor:3	Configs:min.insync.replicas=2,cleanup.policy=compact,segment.bytes=1073741824,retention.ms=172800000,min.cleanable.dirty.ratio=0.5,delete.retention.ms=86400000
Topic: user-events  Partition: 0	Leader: 101	Replicas: 101,100,104	Isr: 101
Topic: user-events  Partition: 1	Leader: 104	Replicas: 104,101,102	Isr: 104,101,102
Topic: user-events  Partition: 2	Leader: 102	Replicas: 102,100,103	Isr: 102,100,103

One way to get in this situation would be if the other replicas’ brokers crashed and new brokers were picked to be replicas for the partition. Those brokers will start streaming the existing records from the leader, but will not be “in sync” till they have replicated all the data on the leader.

The leader checks that there are enough in-sync replicas before persisting the record batch, but there is a possible race condition where it can persist to its log before discovering the outage. In this case, it will return an error to the producer so it will resend the record batch.

• curl - making requests on the command line
• jq - a command-line stream editor for JSON
• ganda (with a little awk) - this post

What is ganda?

ganda is an open-source golang app on github. It was designed to be able to do up to millions of parallel HTTP/HTTPS requests in just a few minutes from a single box.

It expects to be piped a stream of urls that it will request in parallel:

cat file_of_urls | ganda

or if you’re averse to useless uses of cat:

ganda < file_of_urls

(for the rest of the post, I’m going to use cat, even though it is “useless” as I find it clearer and easier to switch to some other utility like head or grep if I don’t want a full file.)

Why ganda and not curl?

I created ganda to fill what I saw as a gap in command-line tooling. We saw in part 1 of this series how curl can be used to make individual requests which could then be saved or piped into tools such as jq.

curl knows how to request a single url, but what if we want to make more than a single request and process all of the results? Well, you can use a utility like xargs to create many instances of curl:

time seq 10000 |\
 awk '{printf "http://localhost:1323/%s\n", $1}' |\
 xargs -I {} -P 8 curl -s {} > /dev/null

This creates a sequence of numbers from 1 to 10,000 and pipes them to awk to create the urls http://localhost:1323/1 to http://localhost:1323/10000. Those urls are then piped to xargs which uses 8 parallel threads to do curl requests to a local server running an echoserver process running on port 1323. We throw away the results by piping to /dev/null for this test.

echoserver simply echos back the path it is passed, so it returns responses from 1 through 10000.

On my 2-core, 3 Ghz mac-mini it takes slightly over 2 minutes, for xargs and curl to make those 10k requests:

seq 10000  0.01s user 0.00s system 84% cpu 0.010 total
awk '{printf "http://localhost:1323/%s\n", $1}'  0.02s user 0.00s system 0% cpu 44.851 total
xargs -I {} -P 8 curl -s {} > /dev/null  110.73s user 84.53s system 237% cpu 1:22.25 total

if we instead pipe those urls to ganda:

time seq 10000 |\
 awk '{printf "http://localhost:1323/%s\n", $1}' |\
 ganda -s > /dev/null

It is able to make all 10,000 requests in 1 second:

seq 10000  0.00s user 0.00s system 87% cpu 0.008 total
awk '{printf "http://localhost:1323/%s\n", $1}'  0.02s user 0.00s system 2% cpu 0.866 total
ganda -s > /dev/null  0.85s user 0.37s system 106% cpu 1.150 total

The one advantage that curl does have over ganda is that it is likely already installed on your system. If you have to make hundreds (or more) requests, you’ll save time by installing ganda first.

Installing ganda

On Macs, you can use homebrew to install the latest version:

brew tap tednaleid/homebrew-ganda
brew install ganda

On linux, you can download a binary from the releases page and put it in your path.

or compile it from source if you have the golang toolchain available:

go get -u github.com/tednaleid/ganda

How to use ganda

ganda wants to be piped urls. You can create a list of urls as a pre-step. This creates /tmp/list_of_urls with 3 httpbin urls and then pipes them to ganda:

cat << EOF > /tmp/list_of_urls
http://httpbin.org/anything/1
http://httpbin.org/anything/2
http://httpbin.org/anything/3
EOF

cat /tmp/list_of_urls | ganda

the output of this is:

Response: 200 http://httpbin.org/anything/3
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/1.1"
  },
  "json": null,
  "method": "GET",
  "url": "http://httpbin.org/anything/3"
}
Response: 200 http://httpbin.org/anything/1
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/1.1"
  },
  "json": null,
  "method": "GET",
  "url": "http://httpbin.org/anything/1"
}
Response: 200 http://httpbin.org/anything/2
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/1.1"
  },
  "json": null,
  "method": "GET",
  "url": "http://httpbin.org/anything/2"
}

As you can see, the results are not guaranteed to come back in the same order that they were requested (on this run they came back in 3, 1, 2 order). The Response: <STATUS_CODE> url line is written to stderr and the body of the response is written to stdout. This lets us use a tool like jq to process the results:

cat /tmp/list_of_urls | ganda | jq -c '[.method, .url]'

Response: 200 http://httpbin.org/anything/2
["GET","http://httpbin.org/anything/2"]
Response: 200 http://httpbin.org/anything/1
["GET","http://httpbin.org/anything/1"]
Response: 200 http://httpbin.org/anything/3
["GET","http://httpbin.org/anything/3"]

The Response line can be silenced with the -s flag so that you only see the info piped to stdout:

cat /tmp/list_of_urls | ganda -s | jq -c '[.method, .url]'

["GET","http://httpbin.org/anything/1"]
["GET","http://httpbin.org/anything/2"]
["GET","http://httpbin.org/anything/3"]

Using awk to construct urls in a pipeline to ganda

Say I have a file of IDs (and other values that vary in the url):

cat << EOF > /tmp/list_of_id_and_param
1000 foo
2000 bar
3000 baz
EOF

I can use awk to construct urls out of that file:

cat /tmp/list_of_id_and_param |\
  awk '{printf "http://httpbin.org/anything/%s?value=%s\n", $1, $2}'

outputs:

http://httpbin.org/anything/1000?value=foo
http://httpbin.org/anything/2000?value=bar
http://httpbin.org/anything/3000?value=baz

Instead of creating a file out of that, I’ll just pipe it directly to ganda and then process the results:

cat /tmp/list_of_id_and_param |\
  awk '{printf "http://httpbin.org/anything/%s?value=%s\n", $1, $2}' |\
  ganda -s |\
  jq -c "[.url, .args]"

results in:

["http://httpbin.org/anything/2000?value=bar",{"value":"bar"}]
["http://httpbin.org/anything/3000?value=baz",{"value":"baz"}]
["http://httpbin.org/anything/1000?value=foo",{"value":"foo"}]

Workers and throttling

WARNING ganda used without some care can easily send more requests than many servers can handle.

Please don’t abuse it or free services like httpbin.org with thousands of requests. Use it on your own servers however you like.

By default, ganda uses 30 parallel workers (goroutines that listen to a channel of incoming requests). This is the maximum number of requests that will be made in parallel. You can adjust the number of workers with the -W flag, using -W 1 will make ganda use a single worker and a single request at a time:

cat /tmp/list_of_urls | ganda -W 1 | grep url
Response: 200 http://httpbin.org/anything/1
  "url": "http://httpbin.org/anything/1"
Response: 200 http://httpbin.org/anything/2
  "url": "http://httpbin.org/anything/2"
Response: 200 http://httpbin.org/anything/3
  "url": "http://httpbin.org/anything/3"

You can also throttle the number of requests that are made per second with the -t <max per second> flag. This throttles it to 1 request per second, even though it is using 30 workers:

time cat /tmp/list_of_urls | ganda -W 30 -t 1 > /dev/null
Response: 200 http://httpbin.org/anything/1
Response: 200 http://httpbin.org/anything/2
Response: 200 http://httpbin.org/anything/3
cat /tmp/list_of_urls  0.00s user 0.00s system 75% cpu 0.004 total
ganda -W 30 -t 1 > /dev/null  0.01s user 0.01s system 0% cpu 3.131 total

Saving results to individual files

If you want to save the response data rather than using them in a pipe, you can use the -o <directory> flag to specify a directory where the individual results should be saved:

cat /tmp/list_of_urls | ganda -o /tmp/output
Response: 200 http://httpbin.org/anything/1 -> /tmp/output/http-httpbin-org-anything-1
Response: 200 http://httpbin.org/anything/2 -> /tmp/output/http-httpbin-org-anything-2
Response: 200 http://httpbin.org/anything/3 -> /tmp/output/http-httpbin-org-anything-3

It will transform the url into a filename within that directory and save the results:

cat /tmp/output/http-httpbin-org-anything-1
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept-Encoding": "gzip",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Go-http-client/1.1"
  },
  "json": null,
  "method": "GET",
  "url": "http://httpbin.org/anything/1"
}

If you will be saving more than a few thousand results to files, you likely want to save them in multiple subdirectories otherwise your operating system will be cranky when you ls that directory. ganda can hash the url and put the response in a subdirectory with the -S <subdir length> flag, use 2 for more than about 5k urls, and 4 for more than 5 million urls:

cat /tmp/list_of_urls | ganda -o /tmp/output -S 2
Response: 200 http://httpbin.org/anything/2 -> /tmp/output/64/http-httpbin-org-anything-2
Response: 200 http://httpbin.org/anything/1 -> /tmp/output/99/http-httpbin-org-anything-1
Response: 200 http://httpbin.org/anything/3 -> /tmp/output/b6/http-httpbin-org-anything-3

Notice that the files are now in subdirectories within the /tmp/output output directory.

Authentication with headers

Like curl, ganda supports the -H <header> flags to send headers with each request:

cat /tmp/list_of_urls | ganda -H "Authorization: Bearer <mytoken>" | grep -E "Authorization|url"
Response: 200 http://httpbin.org/anything/1
    "Authorization": "Bearer <mytoken>",
  "url": "http://httpbin.org/anything/1"
Response: 200 http://httpbin.org/anything/3
    "Authorization": "Bearer <mytoken>",
  "url": "http://httpbin.org/anything/3"
Response: 200 http://httpbin.org/anything/2
    "Authorization": "Bearer <mytoken>",
  "url": "http://httpbin.org/anything/2"

Using other request methods

By default, ganda makes GET requests. You can use the -X <method> flag to use another request type, ex:

cat /tmp/list_of_urls | ganda -X POST | grep -E "method|url"
Response: 200 http://httpbin.org/anything/2
  "method": "POST",
  "url": "http://httpbin.org/anything/2"
Response: 200 http://httpbin.org/anything/1
  "method": "POST",
  "url": "http://httpbin.org/anything/1"
Response: 200 http://httpbin.org/anything/3
  "method": "POST",
  "url": "http://httpbin.org/anything/3"

adding data to the body of the request

ganda allows sending a templated request body via the -d flag. If you want the same thing sent with every request you can use a literal string:

cat /tmp/list_of_urls | ganda -X POST -d "foo=bar" | grep -E 'data|url'
Response: 200 http://httpbin.org/anything/1
  "data": "foo=bar",
  "url": "http://httpbin.org/anything/1"
Response: 200 http://httpbin.org/anything/2
  "data": "foo=bar",
  "url": "http://httpbin.org/anything/2"
Response: 200 http://httpbin.org/anything/3
  "data": "foo=bar",
  "url": "http://httpbin.org/anything/3"

but ganda also allows you to parameterize the template body if you’d like to vary the body that gets posted with each request. It will replace each %s in the body template with a space-delimited values after the urls that it is streamed, so with the file:

cat << EOF > /tmp/list_of_urls_and_values
http://httpbin.org/anything/1 bar 111
http://httpbin.org/anything/2 baz 222
http://httpbin.org/anything/3 qux 333
EOF

We have each url followed by values that should be used in the body that we POST to that url:

cat /tmp/list_of_urls_and_values | ganda -X POST -d '{"first": "%s", "second": "%s"}' | grep -E 'data|url'
Response: 200 http://httpbin.org/anything/3
  "data": "{\"first\": \"qux\", \"second\": \"333\"}",
  "url": "http://httpbin.org/anything/3"
Response: 200 http://httpbin.org/anything/2
  "data": "{\"first\": \"baz\", \"second\": \"222\"}",
  "url": "http://httpbin.org/anything/2"
Response: 200 http://httpbin.org/anything/1
  "data": "{\"first\": \"bar\", \"second\": \"111\"}",
  "url": "http://httpbin.org/anything/1"

This can be very useful for hitting things like GraphQL services.

When to use ganda

I tend to use ganda for anything beyond a single url, but still use curl for singular urls as it is more portable. It is useful any time I want to make a number of requests and perform an analysis across urls.

Unix tools are best when they do “one thing well” and ganda’s whole purpose is to make lots and lots of http/https requests. It composes really well with tools like jq, sort, uniq and grep and enables many ad-hoc analysis possibilities when you have access to an HTTP endpoint but not direct access to a database.

• curl - making requests on the command line
• jq - this post
• ganda (with a little awk) - using ganda to make many requests

What is jq?

jq is a command-line JSON stream editor. It’s like awk for JSON.

It is one of my favorite command line tools, it is great at one thing, parsing JSON and composes well with other command line utilities (curl, cat, awk, etc).

It has an excellent manual on the web (most of which is also in the man jq page).

There is also an online “playground” called jqplay where you can try out jq input with expressions to see what the output would be without installing.

Installation

Quick install instructions:

OSX, use homebrew

brew install jq

Debian/Ubuntu Linux:

sudo apt-get install jq

Fedora:

sudo dnf install jq

Full instructions, including downloadable binaries for all platforms, are on the jq site.

jq basics

simple value extraction

jq has a concise mini-programming language for selecting and transforming JSON. The most basic expression is the . path selector, which simply selects the root of the current document. By itself it will work as a pretty-printer:

echo '{"foo": "bar", "baz": [1, 2, 3]}' | jq '.'

{
  "foo": "bar",
  "baz": [
    1,
    2,
    3
  ]
}

You can add the name of a field to narrow what jq will select:

echo '{"foo": "bar", "baz": [1, 2, 3]}' | jq '.foo'

"bar"

A nested value can be accessed with dot-notation, just like many programming languages:

echo '{"a": {"b": {"c": {"d": "value"}}}}' | jq '.a.b.c.d'

"value"

Output Options

Compact Output (reverse-pretty printing) with -c

This example uses a heredoc to give jq pretty-printed JSON and having it compact it to a single line:

input:

cat <<EOF | jq -c '.'
{
  "foo": "bar",
  "baz": [
    1,
    2,
    3
  ]
}
EOF

output:

{"foo":"bar","baz":[1,2,3]}

Raw String Output with -r

By default, if the result is a string, jq will emit it with double-quotes around it:

echo '{"foo": "bar", "baz": [1, 2, 3]}' | jq '.foo'

"bar"

If you want those quotes stripped off (maybe for piping to another command), you can use -r for raw string output:

echo '{"foo": "bar", "baz": [1, 2, 3]}' | jq -r '.foo'

bar

Output Sorted JSON Keys with -S

If you want the JSON that is emitted to have sorted keys (maybe for diffing/comparing to other output), you

echo '{"z": "fourth", "a": "first", "y": "third", "b": "second"}' | jq -S '.'

{
  "a": "first",
  "b": "second",
  "y": "third",
  "z": "fourth"
}

Formatting/Constructing Output

jq’s selectors (like . and .foo) will emit whatever matches their expression, but those selectors can be surrounded with valid JSON to create remixed JSON output.

Creating JSON Objects

You can also create remixed JSON objects from your input. Here we are moving the fields in the user object to the top-level:

cat <<EOF | jq '{first: .user.first, last: .user.last, username: .username}'
{
  "user": {
    "first": "Ted",
    "last": "Naleid"
  },
  "username": "tnaleid"
}
EOF

{
  "first": "Ted",
  "last": "Naleid",
  "username": "tnaleid"
}

Creating Arrays

if you want to turn a couple of JSON object values into an array, just use [] in your expression:

echo '{"first": "Ted", "last": "Naleid"}' | jq -c '[.first]'

["Ted"]

You can use multiple selectors in your expression to have multiple values in your output:

echo '{"first": "Ted", "last": "Naleid"}' | jq -c '[.first, .last]'

["Ted","Naleid"]

Other Selectors

Array Selectors

Each element in an array can be emitted with [], this will “unwrap” the array:

echo '[1, 2, 3]' | jq '.[]'

1
2
3

It can also be done on arrays that are nested:

echo '{"ids":[1, 2, 3]}' | jq '.ids[]'

1
2
3

If you know you want a particular index or slice of an array, you can pass values to the [].

To get the first element out of an array pass a 0 (it is zero-based):

echo '[1, 2, 3]' | jq '.[0]'

1

Negative offsets can be used to get elements from the end of an array whose length you are unsure of:

echo '[1, 2, 3, 4, 5, 6, 7]' | jq '.[-1]'

7

echo '[1, 2, 3, 4, 5, 6, 7]' | jq '.[-3]'

5

Array slices can be created by giving the start and end element you want, here we get the “tail” of the array by getting the 2nd element (1 as it is zero-based) to the last element in the array:

echo '[1, 2, 3, 4, 5, 6, 7]' | jq -c '.[1:-1]'

[2,3,4,5,6]

If you want specific values, you can specify those with comma separated indexes:

echo '[1, 2, 3, 4, 5, 6, 7]' | jq '.[1, 3, 5]'

2
4
6

keys Selector

If you want to a list all of the keys in an object, use the keys selector:

echo '{"z": "fourth", "a": "first", "y": "third", "b": "second"}' | jq -c 'keys'

["a","b","y","z"]

Composing Filters with |, , and ()

Just like many shells, jq has a “pipe” mechanism that allows the results of a selector to be piped to another jq function. Every result from the expression on the left of the | is fed into the expression on the right.

The expression .a.b.c is equivalent to the expression .a | .b | .c:

echo '{"a": {"b": {"c": "value"}}}' | jq '.a.b.c'

"value"

echo '{"a": {"b": {"c": "value"}}}' | jq '.a | .b | .c'

"value"

A , is used to separate filters, and the results so far will be composed with each filter:

echo '{"a": {"b": {"c1": "value", "c2": "other"}}}' | jq '.a | .b | .c1, .c2'

"value"
"other"

Parenthesis can be used to group filters together:

echo '{"a1": {"b": {"c": "value"}}, "a2": "other"}' | jq '(.a1 | .b | .c), .a2'

"value"
"other"

using select to filter results

If you only want particular values, you can give a boolean expression to select and it will return only the matches:

echo '[1, 2, 3, 4]' | jq '.[] | select(. > 2)'

3
4

It returns the full matching object that it was given:

echo '[{"a": 1}, {"a": 2}, {"a": 3}, {"a": 4}]' | jq -c '.[] | select(.a > 2)'

{"a":3}
{"a":4}

This can be useful if the thing you’re testing isn’t the actual result you want:

echo '[{"a": 1, "b": "bad"}, {"a": 4, "b": "good"}]' | jq -r '.[] | select(.a > 2) | .b'

good

Using ? for values that might not exist

Some expressions will work fine with some input:

echo '{"ids": [{"id": 123}, {"id": 456}]}' | jq '.ids[].id'

123
456

But will blow up if the chain of values doesn’t exist:

echo '{}' | jq '.ids[].id'

jq: error (at <stdin>:1): Cannot iterate over null (null)

This can be gotten around using the ? to tell jq that the value might be null and to ignore it:

echo '{}' | jq '.ids[]?.id'

Values with Varying Types

Sometimes, when an API has a breaking change, a value that was a string has become an integer, if you want to ensure that you only find things that are integers, use the type in a select:

echo '[{"id": 123}, {"id": "456"}]' | jq '.[].id | select(type == "string")'

"456"

echo '[{"id": 123}, {"id": "456"}]' | jq '.[].id | select(type == "number")'

123

You can use | pipes inside the select as part of your boolean expression. This will modify what select evaluates, but will not modify what gets selected:

echo '[{"id": 123}, {"id": "456"}]' | jq '.[] | select(.id | type == "number")'

{
  "id": 123
}

Valid types are number, boolean, array, object, null, string. There are also shortcut versions of these (and others) using the plural of the type:

echo '[{"id": 123}, {"id": "456"}]' | jq '.[].id | numbers'

123

.. Recursive Decent Value Selector

The .. selector will recursively descend down the currently selection and emit each:

echo '{"a": {"b": {"c": "value"}}}' | jq '.a | ..'

{
  "b": {
    "c": "value"
  }
}
{
  "c": "value"
}
"value"

This is most useful for combining with the select which can be used to filter output and search for matching structures:

cat <<EOF | jq -c '.. | select(.target? != null)'
{
  "object": {
     "target": "fromobject"
  },
  "values": [
    {
      "target": "fromarray"
    },
    {
      "ignored": "omitted"
    }
  ]
}
EOF

{"target":"fromobject"}
{"target":"fromarray"}

Create JSON from stdin that isn’t JSON

use -R to do raw input and then you can do this:

seq 3 | jq -R '{id: ., body: ({id: .} | @json)}'
{
  "id": "1",
  "body": "{\"id\":\"1\"}"
}
{
  "id": "2",
  "body": "{\"id\":\"2\"}"
}
{
  "id": "3",
  "body": "{\"id\":\"3\"}"
}

For more advanced versions where you want to potentially split up the arguments, you can use -Rn and input/inputs

seq 3 | jq -Rn 'inputs | {id: .}'
{
  "id": "1"
}
{
  "id": "2"
}
{
  "id": "3"
}

Working with Dates

An ISO-8601 date string can be turned into a unix time:

echo '{"created_date_time":"2017-08-17T19:02:51Z"}' | jq '.created_date_time | fromdate'

1502996571

given a stream of events that have created_date_time field like this:

{"id":"169","created_date_time":"2016-01-01T00:00:00Z"}
{"id":"170","created_date_time":"2017-08-17T19:02:51Z"}
{"id":"433","created_date_time":"2018-01-01T00:00:00Z"}

If we want to find all the ones that are less than or equal to “2017-08-17T19:02:51Z”, we can use this:

cat <<EOF | jq -c 'select((.created_date_time | fromdate) <= ("2017-08-17T19:02:51Z" | fromdate))'
{"id":"169","created_date_time":"2016-01-01T00:00:00Z"}
{"id":"170","created_date_time":"2017-08-17T19:02:51Z"}
{"id":"433","created_date_time":"2018-01-01T00:00:00Z"}
EOF

{"id":"169","created_date_time":"2016-01-01T00:00:00Z"}
{"id":"170","created_date_time":"2017-08-17T19:02:51Z"}

You can also use variables in jq like this to reuse a value:

cat <<EOF | jq -c '(.created_date_time | fromdate) as $created | select($created <= 1512996571 and $created > 1500000000)'
{"id":"169","created_date_time":"2016-01-01T00:00:00Z"}
{"id":"170","created_date_time":"2017-08-17T19:02:51Z"}
{"id":"433","created_date_time":"2018-01-01T00:00:00Z"}
EOF

{"id":"170","created_date_time":"2017-08-17T19:02:51Z"}

Debugging complex expressions with debug

Using debug in your pipeline expression will emit what the current value of something is and can be useful for figuring out what is causing errors:

This example shows us why the select that is looking for numbers we are picking 123 but not "456":

echo '[{"id": 123}, {"id": "456"}]' | jq '.[] | select(.id | debug | type == "number")'

["DEBUG:",123]
{
  "id": 123
}
["DEBUG:","456"]

Output in CSV/TSV format

It can be useful to emit values other than JSON. The first step is to emit the results as an array:

cat <<EOF | jq '[.id, .value]'
{"id":"169","value": 1}
{"id":"170","value": 2}
{"id":"433","value": 3}
EOF

[
  "169",
  1
]
[
  "170",
  2
]
[
  "433",
  3
]

Then, pipe that array either through the @tsv or @csv formatter to get valid output (also use -r to get the “raw” output):

cat <<EOF | jq -r '[.id, .value] | @csv'
{"id":"169","value": 1}
{"id":"170","value": 2}
{"id":"433","value": 3}
EOF

"169",1
"170",2
"433",3

cat <<EOF | jq -r '[.id, .value] | @tsv'
{"id":"169","value": 1}
{"id":"170","value": 2}
{"id":"433","value": 3}
EOF

169	1
170	2
433	3

I’ve often turned jq output into tab-separated output for later processing with other unix command tools like awk, sort, uniq etc

Working with Objects That Want to be Arrays

Sometimes, you’re given an object that has keys and values that you want to work with as if they are an array of key/value pairs, the to_entries operator can help:

cat <<EOF | jq -r '.states | to_entries[] | [.key, .value.at] | @tsv'
{
  "states": {
    "running": {
      "at": "2017-12-14T18:08:11Z"
    },
    "stopped": {
      "at": "2018-01-01T00:08:11Z"
    }
  }
}
EOF

running	2017-12-14T18:08:11Z
stopped	2018-01-01T00:08:11Z

Further Resources

There are many features and uses of jq that I haven’t documented here, I’ve only scratched the surface of what it can do. Here are some further links for reading:

• jq manual
• Cookbook
• Advanced Topics
• Avoiding Pitfalls

Using jq across many http requests

If you want to see how you can map over many http requests and process the results with jq check out the next blog post in this series.

• curl - this post
• jq - a command-line stream editor for JSON
• ganda (with a little awk) - using ganda to make many requests

What is curl?

curl is a command-line app that lets you make requests (http, as well as a variety of other types) and view/save the results.

It has pretty good docs on the web and an extensive man curl page.

Example using httpbin (a great service that I’ll use for a number of examples):

curl "http://httpbin.org/uuid"
{
  "uuid": "339d635c-e5d0-4f5a-afac-0ee954ed508a"
}

Why use curl?

I use curl because it is almost everywhere. It is on my work laptop, home desktop, every machine I ssh into, and every box of every developer that I work with. Its interface is a little crufty, but if you learn a few simple options you’ll be able to use anywhere.

There are friendier command-line alternatives like httpie and resty. I don’t use those because they are likely to only be where I’ve manually installed them. Other developers won’t have them, so sending them commands via slack has barriers to usage and understanding (“What’s httpie? Oh, I don’t have that installed… ")

Why not postman? Because it isn’t everywhere, it doesn’t compose easily with other commands (like jq), and it isn’t easy to share with others via slack chat.

I can see why those tools are useful for others, but this is why I use curl over those alternatives.

curl usage

curl is used on the command-line, so I always use double-quotes to encode my url so that I do not have to escape any special characters (like ? and &) that have other meanings in the shell.

curl "http://httpbin.org/response-headers?key=val"
{
  "Content-Type": "application/json",
  "key": "val"
}

Seeing verbose output for debugging

It is often useful to use the -v (verbose) flag to see metadata about what is being sent/recieved by curl:

curl -v "http://httpbin.org/uuid"
*   Trying 54.243.202.193...
* TCP_NODELAY set
* Connected to httpbin.org (54.243.202.193) port 80 (#0)
> GET /uuid HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: keep-alive
< Server: meinheld/0.6.1
< Date: Sun, 26 Nov 2017 19:14:47 GMT
< Content-Type: application/json
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< X-Powered-By: Flask
< X-Processed-Time: 0.000504970550537
< Content-Length: 53
< Via: 1.1 vegur
<
{
  "uuid": "78c816c8-857b-4726-9070-65e182e3c8c7"
}
* Connection #0 to host httpbin.org left intact

HTTP Methods (POST, PUT, DELETE) and sending form data

The default http method in curl is GET:

curl "http://httpbin.org/post"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

You can change the method with the -X parameter and post form data fields with -F:

curl -X POST -F "field=value" "http://httpbin.org/post"
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "field": "value"
  },
  ...

POST data can be sent with -d either inline:

curl -X POST -d '{"field":"value"}' "http://httpbin.org/post"

or as a file using a @ before the filename:

echo '{"field":"value"}' > postme.json
curl -X POST -d @postme.json "http://httpbin.org/post"

You can make a HEAD request with the -I flag, this can be useful in a script for checking if website is up:

curl -I "http://httpbin.org/uuid"
HTTP/1.1 200 OK
Connection: keep-alive
Server: meinheld/0.6.1
Date: Sun, 26 Nov 2017 19:12:25 GMT
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Powered-By: Flask
X-Processed-Time: 0.000476121902466
Content-Length: 53
Via: 1.1 vegur

Headers

Headers (like authorization tokens/basic auth) can be sent with -H:

curl -H "Authorization: Bearer <mytoken>" "http://httpbin.org/headers"
{
  "headers": {
    "Accept": "*/*",
    "Authorization": "Bearer <mytoken>",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  }
}

Cookies

If the service you are hitting uses cookies and you need to make an authorization request first followed by using the saved cookie in further requests, you can use the -c parameter to write the cookies and then -b to use them in subsequent requests (-L follows 302 redirects which are sometimes sent in auth requests):

curl -c cookie.txt -L "http://httpbin.org/cookies/set?key=value"
{
  "cookies": {
    "key": "value"
  }
}

You can re-use that cookie file on subsequent requests with -b

curl -b cookie.txt "http://httpbin.org/cookies"
{
  "cookies": {
    "key": "value"
  }
}

HTTPS SSL/TLS

HTTPS endpoint certificate can be easily ignored (if you know you don’t care about cert verification) with -k switch for quick manual commands. If you want to use and verify certificates, that is a longer topic that is covered in the official docs.

Custom request timing output

There are a number of additional output settings that you can use. One I’ve found useful when debugging slow connections is to see the various timings that curl keeps track of when making a request.

If you have a curl template file at /tmp/curl-format.txt like this:

   namelookup:  %{time_namelookup}\n
      connect:  %{time_connect}\n
   appconnect:  %{time_appconnect}\n
  pretransfer:  %{time_pretransfer}\n
     redirect:  %{time_redirect}\n
starttransfer:  %{time_starttransfer}\n
   time_total:  %{time_total}\n

You can use the -w switch to emit timing for a request (the @ at the beginning of the property tells curl to load a file rather that inline formatting):

curl -w "@/tmp/curl-format.txt" "http://httpbin.org/uuid"
{
  "uuid": "63846b2b-7502-45c4-b578-4bb04ba0c0b9"
}
   namelookup:  0.065418
      connect:  0.114471
   appconnect:  0.000000
  pretransfer:  0.114569
     redirect:  0.000000
starttransfer:  0.165436
   time_total:  0.165513

This can also be done inline (which is great for copy/paste commands, but is a bit uglier):

curl -w '   namelookup: %{time_namelookup}\n      connect: %{time_connect}\n   appconnect: %{time_appconnect}\n  pretransfer: %{time_pretransfer}\n     redirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\n        total: %{time_total}\n\n' "http://httpbin.org/uuid"
{
  "uuid": "fcec0ce4-5374-403e-9786-1ebabaadb3b0"
}
   namelookup: 0.004505
      connect: 0.050844
   appconnect: 0.000000
  pretransfer: 0.050907
     redirect: 0.000000
starttransfer: 0.102053
        total: 0.102116

Saving output to a file

You can just redirect stdout to a file, when curl is not sending the response to stdout, it adds a progress indicator:

curl "http://httpbin.org/uuid" > output.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    53  100    53    0     0    417      0 --:--:-- --:--:-- --:--:--   420


cat output.json
{
  "uuid": "856320f6-2d90-4bf2-800c-edaef59eec57"
}

You can suppress the progress indicator with -s:

curl -s "http://httpbin.org/uuid" > output.json

cat output.json
{
  "uuid": "a4ce5649-6bb2-49c0-af93-e97f731c7483"
}

You can also use the -o (output) flag:

curl -s -o output.json "http://httpbin.org/uuid"


cat output.json
{
  "uuid": "1246c5ac-2906-4998-aee7-6f5a63b6da66"
}

If you don’t care about the output (but still want to request it, maybe to look at verbose output with -v or other metadata with -w), you can just send it to /dev/null:

curl -s -w '%{http_code}' -o /dev/null "http://httpbin.org/uuid"
200%

Composing curl with other commands using pipes

By default, curl output is sent to stdout which makes it easy to compose with other commands using the | operator:

curl -s -H "Authorization: Bearer <mytoken>" "http://httpbin.org/headers" |\
       grep Authorization

    "Authorization": "Bearer <mytoken>",

For JSON output, grep is often a bad tool to use to find the information you want, the next blog post details how jq can be used to parse and manipulate JSON.

The documentation (and blog posts and Dan Woods’ excellent Learning Ratpack) all discuss another benefit of Ratpack: serialized execution of asynchronous code.

Even though I’d read about Ratpack’s serial execution model, I had not fully internalized the consequences of that feature of Ratpack until I dug in for myself. My previous async programming had been NodeJS and Scala-based and I was using that as my mental model for how Ratpack would behave.

In those other systems, “async” and “parallel” were mostly interchangeable. If you need to make 3 async GET requests, you map over the urls and fire off the async requests. All of the requests are likely sent before any of them have responded.

Ratpack doesn’t work this way. If you make 3 async GET requests, Ratpack will wait for the first one to be completed before sending off the second one.

To understand why this is, we need to discuss some of the details of Ratpack’s architecture.

In Ratpack, Work is Done on One of Two Thread Pools

1. “Compute” Thread Pool

This thread pool is where all requests are handled, and are where all async, non-blocking code in your app executes. Under the covers, it is a Netty epoll EventLoopGroup, so it is very fast as long as you don’t run any blocking operations on it (on non-linux boxes it uses NIO instead of epoll).

The compute thread pool size by default is 2 * # of CPUs. Though you can easily change it with a config value:

ratpack {
    serverConfig {
        threads 8
        …

2. “Blocking” Thread Pool

The Blocking thread pool is unbounded in size (till you run out of memory). It uses a cached thread pool so that it can re-use previously threads when they are available again.

No work is done on a blocking thread unless you explicitly ask. Any blocking operations (like ones using traditional database drivers or file IO), should wrap their call in a Blocking method.

Blocking methods create a Java 8 CompletableFuture that is registered to notify a Ratpack Promise running on the current compute thread when the Future is completed.

It can be useful when testing async code to print out the Thread.currentThread().name to understand which thread your code is running on.

This simple Ratpack app uses the compute and blocking thread pools (full app):

  handlers {
    all { Context context ->

      println "A. Original compute thread: ${Thread.currentThread().name}"

      Blocking.exec { ->
        context.render "hello from blocking" // pretend blocking work
        println "B. Blocking thread : ${Thread.currentThread().name}"
      }

      println "C. Original compute thread: ${Thread.currentThread().name}"
    }
  }

Which prints

A. Original compute thread: ratpack-compute-1-2
C. Original compute thread: ratpack-compute-1-2
B. Blocking thread : ratpack-blocking-3-1

This Blocking code is executed after the original thread, Ratpack detects that no response has yet been rendered in the original thread and that work has been scheduled on a thread in the blocking pool that it needs to register a callback for.

This code will always print A … C … B, serial behavior is guaranteed by Ratpack.

Requests are Processed in a Pipeline of Async Execution Segments

When a Ratpack app starts, it creates an ExecController which is in charge of running all the Execution segments during request processing.

If you do not have any asynchronous calls, each request will run in a single execution segment that runs on a compute thread.

If you do have asynchronous calls (including blocking calls which become asynchronous via Blocking), the request is processed in multiple execution segments, each of which is encapsulated in a Ratpack Promise. (full app)

  handlers {
    all { Context context ->

      println "A. Original compute thread: ${Thread.currentThread().name}"

      Promise.async { downstream ->
        println "B. Promise thread : ${Thread.currentThread().name}"
        downstream.success("hello from async promise")
      }.then { result ->
        context.render result
      }

      println "C. Original compute thread: ${Thread.currentThread().name}"
    }
  }

The output shows that the async Promise runs after the original handler code, but it’s execution stays on the same compute thread:

A. Original compute thread: ratpack-compute-1-2
C. Original compute thread: ratpack-compute-1-2
B. Promise thread : ratpack-compute-1-2

Registering an ExecInterceptor Lets You See the Segments of Execution

Ratpack allows you to register an ExecInterceptor to view the segments of execution (and create metrics).

If we create this ExecInterceptor that captures time at the execution and segment level:

public class LoggingExecInterceptor implements ExecInterceptor {
  @Override
  void intercept(Execution execution, ExecInterceptor.ExecType execType, Block executionSegment) throws Exception {
    ExecutionTimer timer = ExecutionTimer.startExecutionSegment(execution)
    try {
      executionSegment.execute()
    } finally {
      println "${Thread.currentThread().name} - $timer - ${execType}"
    }
  }
}

and register it in this app:

  bindings {
    bindInstance(new LoggingExecInterceptor())
  }
  handlers {
    all { Context context ->
      final String executionId = context.get(ExecutionTimer).id.toString()

      println "${Thread.currentThread().name} - $executionId - A. Original compute thread"
      context.render "hello from compute"
    }
  }

You’ll see interceptor output with one COMPUTE thread println because we did not have any async or blocking work in our app, all of the work is done in a single execute segment:

ratpack-compute-1-4 - 7a265c2b-82b7-4c23-9f0d-92130fff5c26 - A. Original compute thread
ratpack-compute-1-4 - 7a265c2b-82b7-4c23-9f0d-92130fff5c26 - segment time: 1 execution time: 1ms - COMPUTE

Adding a blocking call to our app (full app):

   handlers {
    all { Context context ->
      final String executionId = context.get(ExecutionTimer).id.toString()

      println "${Thread.currentThread().name} - $executionId - A. Original compute thread"

      Blocking.exec { ->
        context.render "hello from blocking" // pretend blocking work
        println "${Thread.currentThread().name} - $executionId - B. Blocking thread"
      }

      println "${Thread.currentThread().name} - $executionId - C. Original compute thread"
    }
  }

gives us this output:

ratpack-compute-1-6 - f04d95cd-2043-47ae-8fc7-0600085eb399 - A. Original compute thread
ratpack-compute-1-6 - f04d95cd-2043-47ae-8fc7-0600085eb399 - C. Original compute thread
ratpack-compute-1-6 - f04d95cd-2043-47ae-8fc7-0600085eb399 - segment time: 0 execution time: 0ms - COMPUTE
ratpack-blocking-3-1 - f04d95cd-2043-47ae-8fc7-0600085eb399 - B. Blocking thread
ratpack-blocking-3-1 - f04d95cd-2043-47ae-8fc7-0600085eb399 - segment time: 1 execution time: 1ms - BLOCKING
ratpack-compute-1-6 - f04d95cd-2043-47ae-8fc7-0600085eb399 - segment time: 0 execution time: 1ms - COMPUTE

Notice that it adds an extra trailing COMPUTE execution after the BLOCKING one? Ratpack registered our Blocking call to notify an execution segment Promise on our original thread (ratpack-compute-1-6) when it was complete.

That kind of monitoring is how Ratpack knows when an execution is finished. If you spawn your own threads outside of a Promise, Ratpack has no idea that your work exists and you won’t get the behavior that you’re probably looking for.

Parallelized Code Must Notify the Original Compute Thread

Normally, running your non-blocking async work in serial fashion on the same compute thread is fast enough.

If you really want something to run in parallel, you can ask for that work to be scheduled on a different compute thread, but you have to notify the original thread that the work is complete (full app):

  handlers {
    all { Context context ->
      final String executionId = context.get(ExecutionTimer).id.toString()

      println "${Thread.currentThread().name} - $executionId - A. Original compute thread"

      Promise.async({ Downstream downstream ->
        println "${Thread.currentThread().name} - $executionId - B1. Inside async promise, same thread still"

        // ask for an execution to be scheduled on another compute thread
        Execution.fork().start({ forkedExec ->
          println "${Thread.currentThread().name} - $executionId - C. Forked work on another thread"
          downstream.success("hello from fork")
        })

        println "${Thread.currentThread().name} - $executionId - B2. After fork().start()"

      }).then { result ->
        println "${Thread.currentThread().name} - $executionId - D. `then` notifies original compute thread"
        context.render result
      }
    }
  }

The output shows that the original compute thread has an execution segment that runs last. It is notified of the work that was done on that other thread by the call to downstream.success:

ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - A. Original compute thread
ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - B1. Inside async promise, same thread still
ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - B2. After fork().start()
ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - segment time: 1 execution time: 1ms - COMPUTE
ratpack-compute-1-7 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - C. Forked work on another thread
ratpack-compute-1-7 - 83505b78-455a-47c1-8012-486b163d587f - segment time: 0 execution time: 0ms - COMPUTE
ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - D. `then` notifies original compute thread
ratpack-compute-1-6 - edd2b6d3-54f2-43cf-87af-41fc6752cde5 - segment time: 1 execution time: 2ms - COMPUTE

Parallelizing Promise Streams

There isn’t much built-in syntax sugar for working with parallelism using Promises, partially because many apps don’t need it. As of this blog post, there’s an open issue on github to make this better in future versions of Ratpack.

If you need to parallelize your request handling right now, your best option is to use the RxJava integration. This makes RxJava Observables work on top of Ratpack’s Execution model.

RxJava/Promise Streams are Processed in Serial by Default

All work that ratpack does within an execution is on the same thread, and the work is fully serial. This has implications if you’re trying to do something like make a microservice that makes HTTP requests to multiple back-end services for each request it receives.

Your requests will all be made one after the other, even though you are using fully non-blocking http APIs.

Demonstrating this takes a bit of setup. Here is stub embedded ratpack app that our application under test will use as a back-end service.

Each GET request to http://localhost:<port>/:sleepFor> will sleep and then return back to the caller. We sleep on a Blocking thread so we don’t hold up our compute threads as sleep is blocking!

EmbeddedApp stubApp = GroovyEmbeddedApp.of {
  handlers {
    get(":sleepFor") {
      Integer sleepFor = context.pathTokens['sleepFor'].toInteger() ?: 1
      Blocking.exec { ->
        println "Stub Sleep App GET Request, sleep for: $sleepFor seconds"
        sleep(sleepFor * 1000)
        context.render sleepFor.toString()
      }
    }
  }
}

Our application under test will have an Observable stream of 3 URIs that will each do a non-blocking, async call to our stub sleep application above.

It will then collect the results from each request and render out a response to the original caller to the app (full app):

  handlers {
    all { Context context ->
      HttpClient httpClient = context.get(HttpClient)
      final String executionId = context.get(ExecutionTimer).id.toString()

      // create a List of URIs to the stub app above that will ask it to sleep
      // for N seconds before returning the number of seconds it was asked to sleep
      final List<URI> REQUEST_SLEEP_URIS = [3, 2, 1].collect {
        URI.create("http://${stubApp.address.host}:${stubApp.address.port}/${it}")
      }

      println "${Thread.currentThread().name} - $executionId - A. Original compute thread"

      // Iterate over all uris, make async http request for each and collect the results to render out
      Observable.from(REQUEST_SLEEP_URIS) // stream of URIs
        .flatMap { uri ->
          println "${Thread.currentThread().name} - $executionId - B. GET: $uri"
          RxRatpack.observe(httpClient.get(uri))  // async http request to "sleep" service
        }
        .map { it.body.text } // get the body text for each http result
        .toList()             // collect into a single list and then subscribe to it
        .subscribe({ List<String> responses ->
          println "${Thread.currentThread().name} - $executionId - C. Subscribe final result"
          context.render responses.join(", ")
        })
    }
  }

We’re asking for the requests in REQUEST_SLEEP_URIS to each sleep 3, 2, and 1 seconds before returning results. We can see from the output that it took slightly over 6 seconds (3+2+1) for our request to be fulfilled, and that stub app did not get the 2nd request till the execution segment for the first request had been completed.

ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - A. Original compute thread
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - B. GET: http://localhost:50735/3
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - B. GET: http://localhost:50735/2
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - B. GET: http://localhost:50735/1
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - segment time: 2 execution time: 2ms - COMPUTE
Stub Sleep App GET Request, sleep for: 3 seconds
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - segment time: 0 execution time: 3016ms - COMPUTE
Stub Sleep App GET Request, sleep for: 2 seconds
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - segment time: 1 execution time: 5024ms - COMPUTE
Stub Sleep App GET Request, sleep for: 1 seconds
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - C. Subscribe final result
ratpack-compute-1-4 - 64949f0f-6010-4eb0-abd7-b655769809e7 - segment time: 1 execution time: 6029ms - COMPUTE

Also notice that all work in the app under test was done on the same COMPUTE thread: ratpack-compute-1-4.

This kind of behavior is a good default for Ratpack to have as it makes things very predictable and easy to reason about. There are cases though where you might really need additional performance for a single request.

Parallelism Must be Explicitly Requested

If you want your reactive stream to be processed in parallel, but the work is still async non-blocking work, you can add the forkEach and bindExec methods into your stream.

forkEach will schedule each observable value to be run on the next available compute thread.

bindExec works like a thread “join” operation. It converts the stream into a Ratpack Promise and then back into an observable. This brings processing of that value back to the original thread. If you don’t include an explicit bindExec, Ratpack will take care of bringing the execution back to the main thread for the subscriber automatically.

If we add forkEach and bindExec into our stream from above (full app):

Observable.from(REQUEST_SLEEP_URIS) 
  .forkEach()           // <-- run in parallel on different compute thread
  .flatMap { uri ->
    println "${Thread.currentThread().name} - $executionId - B. GET: $uri"
    RxRatpack.observe(httpClient.get(uri))  
  }
  .map { it.body.text } 
  .bindExec()           // <-- bind forked thread results to original compute thread
  .toList()             
  .subscribe({ List<String> responses ->
    println "${Thread.currentThread().name} - $executionId - C. Subscribe final result"
    context.render responses.join(", ")
  })

You’ll see that our request time reduces to slightly over 3 seconds, the longest sleep time that we were using:

ratpack-compute-1-7 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - A. Original compute thread
ratpack-compute-1-7 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - segment time: 2 execution time: 2ms - COMPUTE
ratpack-compute-1-8 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - B. GET: http://localhost:51763/3
ratpack-compute-1-9 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - B. GET: http://localhost:51763/2
ratpack-compute-1-10 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - B. GET: http://localhost:51763/1
ratpack-compute-1-9 - 41bb05d8-82b8-45bc-8f2b-b3f9330ee61a - segment time: 2 execution time: 2ms - COMPUTE
ratpack-compute-1-8 - 9d0a9729-641a-416b-bd1b-cf04e1aa16b1 - segment time: 2 execution time: 2ms - COMPUTE
ratpack-compute-1-10 - cd2b08e4-625c-48a3-8141-227c6b496ae4 - segment time: 2 execution time: 2ms - COMPUTE
Stub Sleep App GET Request, sleep for: 3 seconds
Stub Sleep App GET Request, sleep for: 2 seconds
Stub Sleep App GET Request, sleep for: 1 seconds
ratpack-compute-1-10 - cd2b08e4-625c-48a3-8141-227c6b496ae4 - segment time: 1 execution time: 1061ms - COMPUTE
ratpack-compute-1-9 - 41bb05d8-82b8-45bc-8f2b-b3f9330ee61a - segment time: 0 execution time: 2060ms - COMPUTE
ratpack-compute-1-7 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - C. Subscribe final result
ratpack-compute-1-8 - 9d0a9729-641a-416b-bd1b-cf04e1aa16b1 - segment time: 1 execution time: 3037ms - COMPUTE
ratpack-compute-1-7 - 6537dfd0-732a-4599-b82c-7f48bf1c5a42 - segment time: 1 execution time: 3038ms - COMPUTE

We’ve gone from 4 execution segments in the original (serial) execution to 8 execution segments (3 more for forking each URI onto the new compute threads, and one for collecting the returned results).

Asking for parallel execution of your streams means that a single request could be handled more quickly, but you are likely reducing the number of transactions per second that your app can handle.

You shouldn’t parallelize your code without first running performance and load tests to determine that you get an actual boost.

You are also giving up some of the ordering guarantees that Ratpack gives you as a default, it can make your code harder to reason about, but only within the forked part of the stream.

Other Notes About RxJava/Ratpack

If you’re familiar with RxJava, you might have seen information on using a Scheduler along with the scheduleOn and observeOn methods. We don’t have direct access to a scheduler of ratpack’s compute/blocking thread pools so we can’t use these methods to get our work done in parallel. Currently, forkEach/bindExec is the best way to get your Observable code to run in parallel.

Understanding Ratpack Executions for Yourself

If you’re new to async/non-blocking programming, there will be a bit of a learning curve. Even if you’ve worked with previous frameworks, every framework has it’s own behavior that you need to learn, and Ratpack is no different. I’ve linked to full running groovy scripts for each of the sample applications above. I think the best way to internalize how Ratpack works is to dive in and play around with some examples for yourself.

Hopefully this post has helped given you some tools and places to start exploring for yourself.

I’d also highly recommend the Ratpack project on GitHub, I’ve gotten a huge amount of help from Ratpack team members as well as others in the community. Simply lurking there has been extremely valuable, and I’ve always gotten a great response to getting my questions answered.

I tend to be a hands-on, visual learner, so I looked around for a nice wallpaper to help me learn and retain the panoply of vim movement commands, but all I was able to find were simple lists of commands, so I decided to whip my own version up.

It shows all of the default movement commands, each command is placed relative to the center of the image and are ordered based on how far your cursor will likely travel. All of these are active in vim’s “normal” mode (though I believe most if not all of them will also work in “visual” mode.

I also find it useful to set my MacVim window to be slightly transparent so that I can see the shortcuts through the window if I need to. In MacVim 7.3 you need to turn on the Advanced->”Use experimental renderer” option. Then you can “:set transp=20” to make it 20% transparent (which feels right to me, but you might want to move it up/down depending on your preferences).

You can download the full size (1900x1200) (or 2560x1600) image for yourself.

I’ve also got the original OmniGraffle file that I used to create it checked in to a GitHub repo if anyone feels like remixing it or adding their own shortcuts or customizations to it.

A number of interesting techniques have popped out of this that I think would be useful for others doing similar types of importing using GORM/Hibernate/MySQL.

Background

For background, the imports in our app happen a few times a week, and range in size from less than a hundred up to a few hundred thousand new records.

It is also possible for us to get updates on previous data that we’ve imported, if a record in the import matches a record in our database, we want to update the existing record rather than inserting a duplicate.

We had a desire to use a Grails service to do the importing as we could leverage our existing data model, plus, there are some special cases where the import logic can get a bit hairy and having access to all of our domain methods is nice.

If you’re doing much more frequent imports, with heavy performance requirements using millions to hundreds of millions of rows, you’ll likely want to look at another tool more specifically targeted for batch/ETL processes or writing something directly in Java. For smaller jobs, you might be surprised how far you can get with groovy and grails.

Example Naive Import Application

To demonstrate the techniques, I’ve created a small test application. This simple app only has Books. It imports 100,000 randomly generated books in a single job (importing a Library of Books). If our database already has a book with that ISBN and edition number, we want to update the title of previous book rather than creating a new book.

Here are the setup files that won’t change as we tweak our application.

The Domain class:

// domain/Book.groovy
class Book {
    String title
    String isbn
    Integer edition
}

The BookController that calls the service:

// controller/BookController.groovy
class BookController {

    def bookService
	
    def index = { 
        def library = new Library()
        def startTime = System.currentTimeMillis()		
        bookService.importBooksInLibrary(library)	
        render "time: ${(startTime - System.currentTimeMillis())/1000} seconds"
    }
}

And a Library class that’s the source of our books (don’t worry too much about the details of the Library class, just know that it’s the source of the fields that we’ll be importing. For each book, it simply returns a map of the field values that should be saved or updated for the Book):

class Library implements Iterator {
    def startTime = System.currentTimeMillis()
    def lastBatchStarted = startTime
    def BOOKS_IN_LIBRARY = 100000
    def currentBook = 1
    def random = new Random(12345) // random generator with defined seed
	
    Iterator iterator() { return this as Iterator }
    boolean hasNext() { currentBook <= BOOKS_IN_LIBRARY }
    void remove() { }
    def next() {
        if (! (currentBook % 100) ) printStatus()
	
        return [
            title: "Book ${currentBook++}",
            isbn: randomIsbn(), 
            edition: randomEdition()
        ]
    }
	
    def randomIsbn() {
        // one of 50,000 random isbn numbers
        return "isbn-${random.nextInt(50000)}"
    }
	
    def randomEdition() {
        // first through tenth editions
        return random.nextInt(9) + 1
    }
	
    def printStatus() {
        def batchEnded = System.currentTimeMillis()
        def seconds = (batchEnded-lastBatchStarted)/1000
        def total = (batchEnded-startTime)/1000
        println "Library Book $currentBook, last batch: ${seconds}s, total: ${total}s"
        lastBatchStarted = batchEnded
    }
}

The Library class is a proxy for whatever the source of your import info is (a .csv/excel file that you’re parsing through, a webservice, something posted to your server, etc.).

Also, to help see how we affect timing, every 100 books the Library prints out how long it took to process the last 100 books.

Here is the naive BookService that iterates through the Library that the BookController gives it. For each book it receives a map of the books values. It does a findByIsbnAndEdition() to see if the book already exists. If it does, it updates it, otherwise it will create a new book:

class BookService {
    def importBooksInLibrary(library) {		
        library.each { Map bookValueMap ->
            updateOrInsertBook(bookValueMap)
        }		
    }

    def updateOrInsertBook(Map bookValueMap) {
        def title = bookValueMap.title
        def isbn = bookValueMap.isbn
        def edition = bookValueMap.edition
        def existingBook = Book.findByIsbnAndEdition(isbn, edition)
		
        if (existingBook) { // just update title
            existingBook.title = title
            existingBook.save()
        } else { 
            new Book(bookValueMap).save()
        }
    }
}	

When you run this code, the first thing you notice is how slow it is. The first 1000 records aren’t that bad. On my old laptop they take about 7 seconds to insert. After that things get slower and slower, books 10,000 to 11,000 take 20 seconds to insert. Importing gets progressively worse as time goes on.

Yikes! It takes 2 hours and 36 minutes to import 100k records? Over 3 minutes for the last thousand books to get imported? That can’t be right.

As it runs, if you open up your activity monitor/task manager/top application, you’ll also see that Grails is running very hot but also that MySQL’s CPU usage is slowly going up over time.

MySQL Database Performance Tweaks

Start with the obvious: Database Indexes

If you’ve done much database work, you know how much indexes can improve the speed of a query. But how do you know what queries need indexes?

In some situations, like our example above, it can be fairly obvious. We’re calling Book.findByIsbnAndEdition(). Chances are we need an index on the “book” table that includes the “isbn” and “edition” fields:

CREATE INDEX `idx_book_isbn_edition` ON `book`(`isbn`, `edition`);

What if it isn't so obvious which queries need an index?

There are a number of techniques available to us through grails to see what kind of SQL is getting executed.

Turn on grails SQL logging

Within grails, you can turn on SQL logging by putting this in your Config.groovy’s log4j closure:

trace 'org.hibernate.SQL'

Install the p6spy plugin and JDBC SQL profiler

The p6spy grails plugin inserts a proxy between your database driver classes and Grails. Because of this, it knows the full query, including the filled in query parameters and some timing information.

This is useful by itself, but you can have the p6spy plugin log it’s messages to a log4j socket appender that talks to the JDBC SQL Profiler which will aggregate all of the information and help you decide where you’ve got slow queries and need indexes.

Mike Hugo has a great post on using the p6spy plugin with the JDBC SQL profiler.

Turn on the MySQL Slow Query Log

If you’re using a recent version of MySQL, there’s an easy, built-in way to see which queries are slow and which queries aren’t using indexes: the slow query log.

To turn it on, you simply need to add a few lines to your mysql cnf file (by default on OSX just create a file at /etc/my.cnf and it will get loaded):

[mysqld]
log-slow-queries
#Log all queries > long_query_time seconds, default 10 seconds
long_query_time=1

This will automatically create a log file in your mysql “data” directory called host_name-slow.log that contains all queries taking longer than 1 second.

You can optionally add

log-queries-not-using-indexes		

And this will also log those queries that don’t have an index (but don’t take longer than 1 second to execute).

MySQL also comes with a perl script to parse through the slow log file and show you the number of times each query is slow with some additional statistics. To run it, you need to add basedir to your my.cnf file. The basedir value just holds the location of mysql’s data directory.

basedir=/usr/local/mysql	

Then run the script:

mysqldumpslow

These steps should get you a long way towards figuring out which tables need indexes.

Turn on the MySQL General Query log

MySQL has an additional query log called the general query log. It logs everything that comes into MySQL and is extremely noisy. One benefit of it though is that it gives you the full query with all parameters, so it’s more useful than turning on grails SQL trace log and potentially a little quicker to set up than the p6spy plugin.

You can enable it by putting this in your my.cnf:

general_log=1

Suggested Reading for MySQL indexes

If you haven’t created many indexes before, I highly suggest reading this post on indexes on the mysql high performance blog. The executive summary is that multi-key indexes tailored to your query have a performance edge over individual key indexes on each of the fields. If you’re executing a query enough times, it might be worth creating a multi-key index like we have above.

MySQL Database Engines

If MySQL is still your performance bottleneck, or if you’re finding that everything else comes to a stop when you’re running your batch import, check which storage engine you’re using.

For grails applications, I’d prefer InnoDB (or Falcon) to the default MyISAM. MyISAM is good for read-only situations where you don’t care about enforced foreign key releationships. Switching to InnoDB has significantly increased the performance of my app in a number of situations. InnoDB also supports row level locking, whereas MyISAM will lock the entire table when doing an update or an insert.

If you haven’t explicitly done anything, chances are that all of your grails-generated tables are MyISAM. You can change the default storage engine for mysql by specifying a different engine in your my.cnf:

default-storage-engine=innodb

To see the default is for your database execute:

show engines;

To see the engine for a particular table, use “show create table”:

mysql> show create table book\G
*************************** 1. row ***************************
       Table: book
Create Table: CREATE TABLE `book` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `version` bigint(20) NOT NULL,
  `edition` int(11) NOT NULL,
  `isbn` varchar(255) NOT NULL,
  `title` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=84192 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

If you need to alter the storage engine for an existing table, you can execute the alter table SQL statement:

ALTER TABLE book ENGINE = INNODB;

Grails Performance Tweaks

Adding an index and modifying the storage engine of our example book table to InnoDB make our sample app 25-30% faster. A nice boost from where we were originally, but things are still way too slow:

Adding indexes to the database drops the database CPU utilization to almost nothing, but grails is still pegged and it still gets much slower over time.

Something is amiss.

At this point, some people might be inclined to believe the myth that “grails is slow” and go through a painful rewrite in another “faster” language/technology.

Don’t go there yet! It’s only slow if you don’t understand what’s actually going on under the covers.

After reading through a couple of very helpful posts by Burt Beckwith, and reading through the Hibernate chapter on batch processing, we have all of the information we need to speed things up significantly.

Use a Monitoring Tool to Watch Memory/CPU usage Over Time

Because the import gets significantly slower over time, even with the index applied, we know there is some sort of leak.

If you don’t have your code instrumented with timing logic already in it, it’s easy to pop open jconsole or jvisualvm (both already installed and likely in your classpath if you have Java 1.6), connect to your grails instance and watch performance/cpu utilization over time.

The Culprits: Hibernate First-Level Cache and Grails Validation Cache

I found on Burt Beckwith’s blog, that there are 2 separate leaks, one of them is in the hibernate first-level cache, the other is a map that Grails uses for domain object validation errors.

Normally, a grails hibernate session executes something quickly and returns. During importing, we do a ton of processing, all with the same hibernate session. All of these objects that would normally be garbage collected when the session closed are piling up.

The easiest way to deal with this is to create a simple method to clear out these collections periodically.

We can modify our BookService to clean up GORM after every 100 books we insert:

class BookService {

    def sessionFactory
    def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP

    def importBooksInLibrary(library) {
        library.eachWithIndex { Map bookValueMap, index ->
            updateOrInsertBook(bookValueMap)
            if (index % 100 == 0) cleanUpGorm()
        }
    }    

    def cleanUpGorm() {
        def session = sessionFactory.currentSession
        session.flush()
        session.clear()
        propertyInstanceMap.get().clear()
    }

    def updateOrInsertBook(Map bookValueMap) {
        // ... same as above
    }
}	

This change makes everything much better. All 100,000 books get inserted in 3 minutes, the same time that it took the naive example to insert the last 1,000 books.

Also, because MySQL is now properly using indexes, the load on it is very light. If we’re looking for even more speed improvement, this would allow us to split our work up into batches and to make multiple BookService calls (potentially on different web servers). Because we’ve switched to using InnoDB, we’re now working with row-level locking, which isn’t hostile to this approach unlike MyISAM and it’s full table locking on Insert/Update.

Conclusion

This post is just some of what can be done to improve batch performance with Grails and MySQL. Using these tips, I was able to speed up my own imports to a level that met the speed requirments for our problem space, but kept our code much more maintainable. Our domain object remain the canonical code representation of our database model.

There are a number of other things that could be looked at to further speed up batch performance and that could be important to your particular application/infrastructure.

• connectivity between app server and DB
• giving enough memory to your app and DB servers
• other load into database and app servers
• anemic hardware
• poor database design

Be sure to weigh the cost of making changes in each of these areas, and run benchmark tests before committing to them. You’ll often find that what you think is the bottleneck, isn’t. Speed is a feature, but you need to balance it with the costs to achieve it.