fas
fastchr
cargo install fastchr
fas

fastchr

A command-line tool for deduplicating entries in a file or stream.

by Chris McDonald

0.3.0 (see all)License:MIT OR Apache-2.0
cargo install fastchr
Readme

Build Status Build status

A better deduplicator written in Rust.

Basic usage: dedup <INPUT> [-o <OUTPUTFILE>]

Run dedup --help to see:

USAGE:
    dedup.exe [FLAGS] [OPTIONS] [INPUT]

FLAGS:
    -l, --count-lines        If flag is set only print the number of unique entries found.
        --mmap               Enables use of memory mapped files. This is enabled by default.
        --no-mmap            Prohibits usage of memory mapped files. This will slow down the deduplication process
                             significantly!
    -z, --zero-terminated    Specifies that entries should be intepreted as being separated by a null byte rather than a
                             newline.
    -h, --help               Prints help information
    -V, --version            Prints version information

OPTIONS:
    -o, --output <OUTPUT>
        --terminator <TERMINATOR>    Specifies the single-byte pattern to separate entries by. Default is newline.
                                     [default: \n]

ARGS:
    <INPUT>    Specifies the input file to read from. Omit or supply '-' to read from stdin.

To run the benchmark run python benchsuite/benchrunner. This will download a large (400MB+) text file to use as a benchmark case.

Feature requests and bug reports are always welcome! Please raise them as an issue in this Github repository.

GitHub Stars

11

LAST COMMIT

4yrs ago

MAINTAINERS

1

CONTRIBUTORS

1

OPEN ISSUES

2

OPEN PRs

0
VersionTagPublished
0.3.0
4yrs ago
0.2.0
4yrs ago
0.1.0
4yrs ago
No alternatives found
No tutorials found
Add a tutorial