Note
TL;DR: In decode oriented use-case with big payloads JSON decoders often use disproportionate amounts of memory. I gave up on JSON and switched to Msgpack.
You should draw your own conclusions by running the test code yourself.
⸻
Based on various feedback [*] I've did the benchmarks again, using ru_maxrss instead of Valgrind and with few more implementations.
Updated results: *
- my data (240Mb)
- citylots.json (189MB)
- canada.json (2.2Mb)
I have a peculiar use-case where I need to move around big chunks of data (some client connects to some HTTP API and gets some data). For whatever reason [1], JSON was chosen as the transport format. And one day that big chunk became very big - around few hundred megabytes. And it turned out that processes doing the JSON decoding were using lots of RAM - 4.4GB for a mere 240MB JSON payload? Insane. [2]
I was using the builtin json library, and the first thing I thought - "there must be a better JSON parser". So I've started measuring ...
Now measuring memory usage is a tricky thing, you can look at ps or look around in /proc/<pid> but you'd get very coarse snapshots and would be very hard to find out the real peak usage. Luckily enough Valgrind can instrument any program to track allocations (as opposed to recompiling everything to use a custom memory allocator) and it has a really nice tool called massif.
So I've started building a little benchmark using Valgrind. My input looks like this:
{
"foo": [{
"bar": [
'A"\\ :,;\n1' * 20000000,
],
"b": [
1, 0.333, True,
],
"c": None,
}]
}
That generates a 240MB JSON with a structure pretty close to my app's problematic data.
Running valgrind --tool=massif --pages-as-heap=yes --heap=yes --threshold=0 --peak-inaccuracy=0 --max-snapshots=1000 ... for each parser gets me something like this on Python 2.7 (scroll down for results on Python 3.5):
Peak memory usage (Python 2.7):
cjson: 485.4 Mb
rapidjson: 670.5 Mb
yajl: 1,199.2 Mb
ujson: 1,862.0 Mb
jsonlib2: 2,882.7 Mb
jsonlib: 2,884.2 Mb
simplejson: 2,953.6 Mb
json: 4,397.9 Mb
Would you look at that. Now you can argue that my sample data is crazy but sadly, but that's just how my data looks sometimes. Few of the strings blow up to horrid proportions once in a while.
json has a severe weakness here, it needs a dozen more times memory than the input. WAT.
cjson is right there in my face, begging me to use it. There are some rumours that it has VeryBadBugs™ [6] but I think the lack of a bug tracker is what makes that project ultimately unappealing.
rapidjson seems to be a new player [3], however the Python 2 binding seems to have some gaps in essential parts. Still, it's interesting to at least get an idea of how it performs. The Python 3-only binding looks more mature, but sadly this app only run on Python 2 right now.
yajl and ujson appear to be mature enough but they simply still use lots of memory. There must be a better way ...
It looks like whatever I choose it's bad. There's a very good proverb [†] that applies here:
Best solution to problem is not having the problem in the first place.
Remember that time a customer asked for a thing but in fact he only needed something simpler and less costly. Talking through requirements and refining them solves lots of problems right there. This is that kind of situation. I wish I had realized I don't really need JSON at all sooner ...
I have to do more changes to switch the format of the HTTP API but that can't be worse than maintaining/fixing the cjson or rapidjson bindings myself.
If we try msgpack (and some old friends [‡], just for kicks) we get this:
Peak memory usage (Python 2):
pickle: 368.9 Mb
marshal: 368.9 Mb
msgpack: 373.2 Mb
cjson: 485.4 Mb
rapidjson: 670.4 Mb
yajl: 1,199.2 Mb
ujson: 1,862.0 Mb
jsonlib2: 2,882.7 Mb
jsonlib: 2,884.2 Mb
simplejson: 2,953.6 Mb
json: 4,397.9 Mb
If you look at the test code you'll notice that I use msgpack with very specific options. Because the initial version of Msgpack wasn't very smart about strings (it had a single string type [5]) some specific options are needed:
msgpack.dumps(obj, use_bin_type=True) - use a different type for byte-strings. By default Msgpack will lump all kinds of strings into same type and you can't tell what the original type was.
On Python 2:
- str goes into the bin type
- unicode goes into the string type
On Python 3:
- bytes goes into the bin type
- str goes into the string type
msgpack.loads(payload, encoding='utf8') - decode the strings (so you get unicode back).
What about speed? *
Using pytest-benchmark we get this [5]:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 59.2630 (1.0)
test_speed[pickle] 59.4530 (1.00)
test_speed[msgpack] 59.7100 (1.01)
test_speed[rapidjson] 443.0561 (7.48)
test_speed[cjson] 676.6071 (11.42)
test_speed[ujson] 681.8101 (11.50)
test_speed[yajl] 1,590.4601 (26.84)
test_speed[jsonlib] 1,873.3799 (31.61)
test_speed[jsonlib2] 2,006.7949 (33.86)
test_speed[simplejson] 3,592.2401 (60.62)
test_speed[json] 5,193.2762 (87.63)
-----------------------------------------------
Only the minimum time is shown. This is intentional - run the test code on your own hardware if you care about anything else.
Python 3 *
This app where I had the issue runs only on Python 2 for a very good (and also sad) reason. But no reason to dig myself further into a hole - gotta see how this performs on the latest and greatest. It will get ported one day ...
Peak memory usage (Python 3.5):
marshal: 372.1 Mb
pickle: 372.9 Mb
msgpack: 376.6 Mb
rapidjson: 668.6 Mb
yajl: 687.3 Mb
ujson: 1,578.9 Mb
json: 3,422.3 Mb
simplejson: 6,681.4 Mb
Speed (Python 3.5)
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[msgpack] 69.0613 (1.0)
test_speed[pickle] 69.9465 (1.01)
test_speed[marshal] 74.9914 (1.09)
test_speed[rapidjson] 337.5243 (4.89)
test_speed[ujson] 902.8647 (13.07)
test_speed[yajl] 1,195.4298 (17.31)
test_speed[json] 4,404.9523 (63.78)
test_speed[simplejson] 6,524.9919 (94.48)
-----------------------------------------------
No cjson or jsonlib on Python 3. I don't even know what's the story behind jsonlib2. Looks like Msgpack is a safe bet here.
Different kind of data *
Now this is highly skewed towards some might call a completely atypical data shape. So I advise you take the test code and run the benchmarks with your own data.
But if you're lazy here are some results with different kinds of data, just to get an idea of how much the input can change memory use and speed.
Lots off small objects *
The 189MB citylots.json gets us wildly different results.
It appears simplejson works way better on small objects, and json is quite improved on Python 3:
Peak memory usage (Python 2.7):
simplejson: 1,171.7 Mb
cjson: 1,304.2 Mb
msgpack: 1,357.2 Mb
marshal: 1,385.2 Mb
yajl: 1,457.1 Mb
json: 1,468.0 Mb
rapidjson: 1,561.6 Mb
pickle: 1,854.1 Mb
jsonlib2: 2,134.9 Mb
jsonlib: 2,137.0 Mb
ujson: 2,149.9 Mb
Peak memory usage (Python 3.5):
marshal: 951.0 Mb
json: 1,059.8 Mb
simplejson: 1,063.6 Mb
pickle: 1,098.4 Mb
msgpack: 1,115.9 Mb
yajl: 1,226.6 Mb
rapidjson: 1,404.9 Mb
ujson: 2,077.6 Mb
Speed:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 3.9999 (1.0)
test_speed[ujson] 4.2569 (1.06)
test_speed[simplejson] 5.1105 (1.28)
test_speed[cjson] 5.2355 (1.31)
test_speed[msgpack] 5.9742 (1.49)
test_speed[yajl] 6.1059 (1.53)
test_speed[json] 6.3822 (1.60)
test_speed[jsonlib2] 6.7880 (1.70)
test_speed[jsonlib] 6.9587 (1.74)
test_speed[rapidjson] 7.4734 (1.87)
test_speed[pickle] 18.8649 (4.72)
-----------------------------------------------
Speed (Python 3.5):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 1.1784 (1.0)
test_speed[ujson] 3.6378 (3.09)
test_speed[msgpack] 3.7226 (3.16)
test_speed[pickle] 3.7739 (3.20)
test_speed[rapidjson] 4.1379 (3.51)
test_speed[json] 5.1150 (4.34)
test_speed[simplejson] 5.1530 (4.37)
test_speed[yajl] 5.9426 (5.04)
-----------------------------------------------
Smaller data *
The tiny 2.2MB canada.json, again, gives us very different results. Memory use becomes irrelevant:
Peak memory usage (Python 2.7):
marshal: 35.2 Mb
cjson: 38.9 Mb
yajl: 39.0 Mb
json: 39.3 Mb
msgpack: 39.5 Mb
simplejson: 40.5 Mb
pickle: 42.1 Mb
jsonlib2: 47.4 Mb
rapidjson: 48.5 Mb
jsonlib: 48.8 Mb
ujson: 50.9 Mb
Peak memory usage (Python 3.5):
marshal: 38.3 Mb
pickle: 40.4 Mb
yajl: 42.1 Mb
json: 42.2 Mb
msgpack: 42.7 Mb
simplejson: 45.3 Mb
rapidjson: 52.3 Mb
ujson: 55.5 Mb
And speed is again different:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[msgpack] 12.3210 (1.0)
test_speed[marshal] 15.1060 (1.23)
test_speed[ujson] 19.8410 (1.61)
test_speed[json] 48.0320 (3.90)
test_speed[cjson] 48.6560 (3.95)
test_speed[simplejson] 52.0709 (4.23)
test_speed[yajl] 62.1090 (5.04)
test_speed[jsonlib2] 81.6209 (6.62)
test_speed[jsonlib] 83.2670 (6.76)
test_speed[rapidjson] 102.3500 (8.31)
test_speed[pickle] 258.6429 (20.99)
-----------------------------------------------
Speed (Python 3.5):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 10.0271 (1.0)
test_speed[msgpack] 10.2731 (1.02)
test_speed[pickle] 17.2853 (1.72)
test_speed[ujson] 17.7634 (1.77)
test_speed[rapidjson] 25.6136 (2.55)
test_speed[json] 54.8634 (5.47)
test_speed[yajl] 58.3519 (5.82)
test_speed[simplejson] 65.0913 (6.49)
-----------------------------------------------
I suppose better use of freelists happens here?
Bottom line *
Both speed and memory are affected by data shape. Speed is not always proportional to memory use.
Again, don't trust the numbers, run the benchmarks yourself, with your own data. Even if your data is identical in shape your hardware might behave differently than mine. Even the memory use can be different on your machine (example: different architecture, different shared libraries). And what's the chance your data has the exact shape as whatever was used in the benchmark?
Test setup:
- Ubuntu 14.04 (a VM on an unoccupied host, at least in theory)
- Sandy Bridge i7 (TurboBoost off, but frequency scaling was on)
- Python 2.7.6
- Python 3.5.0
- Valgrind 3.11.0
As you can see the setup is less then perfect. If you really care you'd run the test code yourself.
| [1] | I believe I've fallen victim to cargo-culting ... |
| [2] | But not more insane than Linux doing it's memory overcommit thing and swapping everything. You can dismiss this as an ignorant remark about the kernel but bottom line is the same: trashing the swap is very bad. |
| [3] | I've noticed rapidjson in this C++ JSON library benchmark. It seems it was the only library with a working Python binding. |
| [4] | The fact that there are lots of languages you can use Msgpack on and they don't have an Unicode type doesn't help either. |
| [5] | (1, 2) Generated with something like script -c tox | ansi2html. |
| [6] | It seems that no one can really point out what's wrong with cjson. The author of jsonlib even did this FUD post and it seems people have been buying that bullshit. StackOverflow is practically plastered with transcriptions of that. He may be right, and there are actual issues with encoding and conformance in cjson, but that ain't the right way to explain it. Also, kinda ironic that jsonlib was abandoned one year later after that post ... |
| [*] | Feedback from commenters here, Reddit, HackerNews and Google+. |
| [†] | Where did this idea originate from? If you got a source please comment. |
| [‡] | Not something that you would use in a HTTP API at all. Put security concerns aside, the problem is mixing different versions of python. But they are still interesting to look at, for ideas of what's possible performance-wise. |