brs

batch-request-stream

Make batched, rate-limited requests for data flowing through a node.js Stream

Showing:

Popularity

Downloads/wk

22

GitHub Stars

2

Maintenance

Last Commit

7yrs ago

Contributors

0

Package

Dependencies

1

Size (min+gzip)

5.4KB

License

MIT

Type Definitions

Tree-Shakeable

No?

Categories

Readme

Build Status

NPM

batch request stream

Make batched and rate-limited requests for data flowing through a node.js Stream; useful when you need to make slow, asynchronous I/O calls (which are optimally batched) for items coming down a fast read stream, and don't want to throttle the responding service.

api

The package exports a single function which will create and return a Writable stream.

createBatchRequestStream(options)

  • options is a configuration object which must contain the following mandatory keys and may contain the optional ones:
  • request(batch, requestCompleted): The function to execute on each buffered batch of data. Must accept two arguments: batch, an array of the objects written to the stream, and requestCompleted, a function that must be called by whatever callback is sent to the asynchronous I/O function called inside request(); see the example for clarification. This is used to keep track of the number of live requests and, thus, for rate-limiting.
  • batchSize (optional, default value: 100): The number of items to buffer before calling request(). If the stream has ended and a non-empty batch remains (because the total number of items written isn't evenly divisible by batchSize), it'll still be sent to request().
  • maxLiveRequests (optional, default value: 100): The maximum number of concurrent requests to keep open.
  • streamOptions (optional): Options to pass to new stream.Writable().

example

var createBatchRequestStream = require("batch-request-stream");

function request(batch, requestCompleted){
    asyncIORequest(batch, function callback(err){
        requestCompleted(); // MUST be called
    });
}

var batchRequestStream = createBatchRequestStream({
    request: request,
    batchSize: 4,
    maxLiveRequests: 3,
    streamOptions: {objectMode: true}
});

fastReadStream.pipe(batchRequestStream);

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100