cj

compress-json

Store JSON data in space efficient manner

Showing:

Popularity

Downloads/wk

1.1K

GitHub Stars

12

Maintenance

Last Commit

7d ago

Contributors

0

Package

Dependencies

0

License

BSD-2-Clause

Type Definitions

Built-In

Tree-Shakeable

No?

Categories

Readme

compress-json

Store JSON data in space efficient manner.

npm Package Version

Inspired by compressed-json and jsonpack.

This library is optimized to reduce represent json object in compact format, which can save network bandwidth and disk space. It is not optimized for writing nor querying throughput. Although the reduced IO may speed up usage of lmdb on frequently redundant data, that is not the design goal.

Features

  • Object key order is preserved
  • Repeated values are stored only once
  • Numbers are encoded in base62 format (0-9A-Za-z)
  • Support multiple storage backend
    • in-memory object / array / Map
    • localStorage
    • lmdb
    • leveldb (sync mode)
    • custom adapter

Installation

npm i -S compress-json

Usage

import { compress, decompress } from 'compress-json'

let data = { 
  user: 'Alice',
  // more fields of any json values (string, number, array, object, e.t.c.)
}

let compressed = compress(data) // the result is an array
fetch('/submit', {
  method: 'post',
  body: JSON.stringify(compressed) // convert into string if needed
})

let reversed = decompress(compressed)
data === reversed // will be false
JSON.stringify(data) === JSON.stringify(reversed) // will be true

Format

Sample data:

let longStr = 'A very very long string, that is repeated'
let data = {
  int: 42,
  float: 12.34,
  str: 'Alice',
  longStr,
  longNum: 9876543210.123455,
  bool: true,
  bool2: false,
  arr: [42, longStr],
  arr2: [42, longStr], // identical values will be deduplidated, including array and object
  obj: { // nested values are supported
    id: 123,
    name: 'Alice',
    role: [ 'Admin', 'User', 'Guest' ],
    longStr: 'A very very long string, that is repeated',
    longNum: 9876543210.123455
  },
  escape: [ 's|str', 'n|123', 'o|1', 'a|1', 'b|T', 'b|F' ]
}

Compressed data:

// [ encoded value array, root value index ]
let compressed = [
  [  // encoded value array
    'int', // string
    'float',
    'str',
    'longStr',
    'longNum',
    'bool',
    'bool2',
    'arr',
    'arr2',
    'obj',
    'escape',
    'a|0|1|2|3|4|5|6|7|8|9|A',
    'n|g', // number (integer) (base62-encoded)
    'n|C.h', // number (float) (integer part and decimals are base62-encoded separately)
    'Alice',
    'A very very long string, that is repeated',
    'n|AmOy42.2KCf',
    'b|T', // boolean (true)
    'b|F', // boolean (false)
    'a|C|F', // array
    'id',
    'name',
    'role',
    'a|K|L|M|3|4',
    'n|1z',
    'Admin',
    'User',
    'Guest',
    'a|P|Q|R',
    'o|N|O|E|S|F|G', // object
    's|s|str', // escaped string
    's|n|123', // escaped number
    's|o|1',
    's|a|1',
    's|b|T', // escaped boolean
    's|b|F',
    'a|U|V|W|X|Y|Z',
    'o|B|C|D|E|F|G|H|I|J|J|T|a'
  ],
  'b' // root value index
]

Benchmark

Test file: compress-test.ts

Sample data in use: json data of 109,164 threads and 724,905 post crawled from a discuz forum. Truncated in varies of size for testing.

Algorithms in comparison:

  • JSON (JSON.stringify without indentation)
  • compressed-json
  • jsonpack
  • compress-json (this library)

(Binary compression algorithm is not considered)

Compressed Size

sampleJSONcompressed-jsonjsonpackcompress-json
all263M199M-176M
100,000235M178M-158M
50,00070M55M-50M
10,00034M26M-23M
2,0006.6M5.0M5.3M4.4M
1,0004.8M3.7M3.8M3.3M
100335K265K271K243K
104.0K3.3K3.0K3.2K

Compression Time

sampleJSONcompressed-jsonjsonpackcompress-json
all1,654ms12,674mstimeout*15,788ms
100,0001,500ms10,921mstimeout*12,715ms
50,000462ms3,047mstimeout*3,935ms
10,000146ms1,278mstimeout*1,733ms
2,00035ms328ms21,018ms456ms
1,00020ms270ms12,960ms390ms
1001ms18ms47ms37ms
100.3ms1.8ms1.6ms1.9ms

*timeout: excess 1 minute

Decompress Time

sampleJSONcompressed-jsonjsonpackcompress-json
all1,908ms4,611ms-9,225ms
100,0001,744ms3,740ms-7,576ms
50,000558ms1,066ms-2,452ms
10,000173ms460ms-1,055ms
2,00047ms108ms189ms317ms
1,00034ms90ms160ms263ms
1002ms11ms16ms19ms
102.9ms2.1ms1.1ms1.3ms

License

BSD 2-Clause License (Free Open Sourced Software)

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100
No reviews found
Be the first to rate

Alternatives

No alternatives found

Tutorials

No tutorials found
Add a tutorial