stm

save-to-mongo

Save an object stream directly to MongoDB. Bulk modes supported (ordered / unordered) with configurable ops group size.

Showing:

Popularity

Downloads/wk

0

GitHub Stars

5

Maintenance

Last Commit

5yrs ago

Contributors

2

Package

Dependencies

2

License

MIT

Type Definitions

Tree-Shakeable

No?

Categories

Readme

Save-To-Mongo

A Node.js Writable stream useful for saving (big) object streams directly to MongoDB.

Features:

Usage

See /examples folder. But basically:

var JSONStream = require('JSONStream');
var fs = require('fs');
var path = require('path');

// init stream

var SaveToMongo = require('save-to-mongo');

var saveToMongo = SaveToMongo({
  uri: 'mongodb://127.0.0.1:27017/test',
  collection: 'savetomongo',
  bulk: {
    mode: 'unordered'
  }
});

// go!!!

fs.createReadStream(path.join(__dirname, './accounts.json'));
  .pipe(JSONStream.parse('*'))
  .pipe(saveToMongo)
  .on('execute-error', function(err) {
    console.log(err);
  })
  .on('done', function() {
    console.log('All done!');
    process.exit(0);
  });

JSON random data was generated using json-generator.

Documentation

To obtain a Save-To-Mongo instance, you have to do:

  var SaveToMongo = require('save-to-mongo');
  var stream = SaveToMongo({ /* options */ });

stream available options are:

  • uri (String, required): MongoDB connection uri.
  • collection (String, required): MongoDB target collection.
  • connection (Object, optional): optional connection options. You can pass all options supported by node-mongodb-native.
  • streamOptions (Object, optional): underlying Writable stream options. Default: { objectMode: true }.
  • bulk (Object, optional): enables bulk mode. If you set this, you may want to set also:
    • mode: (String, required): can be ordered or unordered. See MongoDB docs if you want more info on them.
    • bufferSize (Number, required): By default all bulk operations are splitted in groups, each contains 1000 ops. Change this if you want a lower value.

Events

"write-error"

Emitted every time a MongoDB insert op fails (e.g: duplicate _id). Note that this will not break the stream.

"execute-error"

Emitted every time a MongoDB execute op fails. Note that this will not break the stream.

"done"

Emitted when all writes are executed (after stream's finish event).

Performance

Both bulk modes (expecially unordered mode) increases significantly insert speed.

On provided examples:


# for 100 records

time node ./examples/default.js # real: 0m0.450s
time node ./examples/unordered.js # real: 0m0.340s

UPDATE: for 100000 records (similar to the ones used in examples), using bulk unordered method drop execution time from 2:42m to 1:42m. Not bad :)

Debug

This module is built with debug, so you can inspect what's happening.

On examples, you can do:

DEBUG=save-to-mongo node ./exampes/default.js

Install

With npm:

npm install save-to-mongo

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100
No reviews found
Be the first to rate

Alternatives

No alternatives found

Tutorials

No tutorials found
Add a tutorial