sm

stream-mmmagic

Node module to sniff the start of a stream to detect the file type and encoding

Showing:

Popularity

Downloads/wk

1.2K

GitHub Stars

28

Maintenance

Last Commit

3mos ago

Contributors

7

Package

Dependencies

2

License

MIT

Type Definitions

Built-In

Tree-Shakeable

No?

Categories

Readme

node-stream-mmmagic

Build Status

Node module to sniff the start of a stream (non-destructively) to detect the file type and encoding when you don't have the luxury of being able to restart the stream again.

It does so by using buffer-peek-stream to get the first 16KB of the stream then send that to mmmagic (which uses libmagic). Before it's finished the peek stream will unshift the bytes it's received back onto the origin stream thereby making it appear as if the origin stream was new.

npm install stream-mmmagic

Use

const magic = require('stream-mmmagic');
const input = fs.createReadStream('somefile.csv');

const [mime, output] = await magic.promise(input);
console.log('TYPE:', mime.type);
console.log('ENCODING:', mime.encoding);
output.pipe(process.stdout);

//- TYPE: text/plain
//- ENCODING: us-ascii
//- <the file content>

Use (Callbacks)

var magic = require('stream-mmmagic');

var input = fs.createReadStream('somefile.csv');

magic(input, function (err, mime, output) {
  if (err) throw err;

  console.log('TYPE:', mime.type);
  console.log('ENCODING:', mime.encoding);

  // will print the *whole* file
  output.pipe(process.stdout);
});

//- TYPE: text/plain
//- ENCODING: us-ascii
//- <the file content>

options.magicFile Custom Magic File

A magic file is bundled with the mmmagic npm module but if you want to use your own then set the path to the file on the magicFile option.

const magicFile = '/usr/share/magic';
magic(input, {magicFile}, callback);

options.splitMime Original Mime String

Use {splitMime: false} option to get back the original mime string instead of a split object.

const [mime] = magic.promise(input, {splitMime: false});
console.log(mime);
//- text/plain; charset=us-ascii

options.peekBytes Control Bytes Used for Analysis

As the input stream starts to get data the first 16KB is buffered and sent to libmagic for analysis to get file type and encoding. 1KB is more than enough for detecting file type with a standard magicFile but the reliabilty of getting the correct encoding is increased the more bytes are buffered. The tradeoff is performance and memory use.

Set peekBytes to the number of bytes you want buffered and sent to libmagic. For best results do not set below 256 bytes.

// somefile.txt is a utf8 file where the first doublebyte char is after the first 1KB of the file
const input = fs.createReadStream('somefile.txt');

const [{encoding}, output] = magic.promise(input, {peekBytes: 1024});
console.log(encoding);
// not detected as utf8 because the first doublebyte char wasn't until later in the stream
//- us-ascii

const [{encoding}, output] = magic.promise(input, {peekBytes: 16384});
console.log(encoding);
// now we're peeking 16KB into the file libmagic gets that first doublebyte char and knows it's utf8
//- charset=utf8

LICENSE

MIT

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100
No reviews found
Be the first to rate

Alternatives

No alternatives found

Tutorials

No tutorials found
Add a tutorial