rs

remote-sync

A cross-platform LFTP wrapper for automating file synchronization.

Showing:

Popularity

Downloads/wk

0

GitHub Stars

10

Maintenance

Last Commit

3yrs ago

Contributors

3

Package

Dependencies

0

License

MIT

Type Definitions

Tree-Shakeable

No?

Categories

Readme

Remote Sync

A cross-platform LFTP wrapper for automating file synchronization.

NPM

Documentation | Repository

Features

  • Leverage the power of LFTP to transfer files to/from remote host(s).
  • Logically organize LFTP operations into a workflow that is easily revisable.
  • For speed, open a persistent connection to a designated host and execute your commands in series.
  • Define separate settings (host|user|password) for independent operations that can be executed (a)synchronously.
  • Create a conditional chain of operations where you create the condition for further execution.
  • Use RemoteSync as a general purpose LFTP client.

Dependency

You'll need to have LFTP installed on your machine as its the child process RemoteSync wraps. I chose LFTP as my FTP(S) client because its easy to work with, comes preinstalled on most Unix-based OS distributions and has a nice port to Windows. If you don't already have LFTP installed (Windows) use a package management tool to get the latest version:

Windows (Chocolatey)

C:\> choco install lftp

OSX (Homebrew)

sudo brew install lftp

Linux

sudo apt-get install lftp
sudo yum install lftp

Configuration

Constructor Object

{
  operations : [{},{},..,{}],   // Array of operation objects. See below. 
  lftp_settings : {             // OPTIONAL: LFTP settings. See LFTP man page.
    'setting':'value'
  },
  user : 'user',                // User name for host. Default: ''
  pw : 'password',              // Password for host. Default: ''
  protocol : 'ftp',             // Protocol (Everything LFTP supports) Default: ftp
  host : 'some.host.com',       // Host name. Default: ''
  port : '21',                  // Port number. Default: ''
  persistent : false,           // Persistent connection. Default: false
  sync : true,                  // Synchronously execute operations. Default: true
  exit : false,                 // If persistent, close connection after operations finish. Default: false
  debug : true,                 // Pass debug flag to LFTP for verbose logging. Default: false
  stdio : stdio                 // OPTIONAL: Override stdio configuration of child process. This is overridden
                                // by operation object's stdio property.
}

Operation Object

{
    operation : 'download',     // Operation name. Something that relates to the command.
    command : 'lftp command',   // The LFTP command to run. Refer to man page.
    user : 'user',              // OPTIONAL: User name. If !persistent, overrides constructor value.
    pw : 'password',            // OPTIONAL: Password. If !persistent, overrides constructor value.
    protocol : 'ftp',           // OPTIONAL: Protocol. If !persistent, overrides constructor value.
    host : 'some.host.com',     // OPTIONAL: Host name. If !persistent, overrides constructor value.
    port : '21',                // OPTIONAL: Port number. If !persistent, overrides constructor value.
    settings : {                // OPTIONAL: Do stuff based on configuration.
        sync : fn(child),       // OPTIONAL: If !persistent and sync = true, call fn(child) on finish.
        stdio : stdio           // OPTIONAL: Override stdio configuration of child process. This overrides
                                // constructor object's stdio property.
    }
}

stdio Object

// https://nodejs.org/api/child_process.html#child_process_options_stdio
{
    stdio_config : {stdio:[]},  // See above link for custom configuration.
    stdout : data => fn(data),  // Custom behavior for child's stdout event.
    stderr : data => fn(data),  // Custom behavior for child's stderr event.
    close : code => fn(code),   // Custom behavior for child's close event.
    error : error => fn(error)  // Custom behavior for child's error event.
}

Examples

Fancy level 0: The following example will use a minimal configuration to demonstrate basic functionality. Open a non-persistent connection to ftp.host.com, execute operation 1 and exit.

// client.js
const RemoteSync = require('remote-sync');
const config = {
    operations : [
        {
            operation : 'List',
            command : 'nlist files'
        }
    ],
    user : 'kurt',
    pw : 'foobar',
    host : 'ftp.host.com'
};
const client = new RemoteSync(config);
client.perform(); // Returns a remote listing of files at ftp.host.com/files/

Fancy level 1: Add lftp_settings to the constructor object to customize the session. Open a non-persistent connection to ftp.host.com using FTPES (Explicit FTP over TLS), set parallel transfer count to 5, execute operation 1 and exit. (Read about the flags passed to mirror at LFTP.)

// client.js
const RemoteSync = require('remote-sync');
const command = 'mirror -c --only-missing <source> <dest>';
const config = {
    operations : [
        {
            operation : 'mirror directory',
            command : command
        }
    ],
    user : 'kurt',
    pw : 'foobar',
    host : 'ftp.host.com',
    lftp_settings : {
        'ftp:ssl-force':'true',
        'ftp:ssl-protect-data':'true',
        'ssl:verify-certificate':'false',
        'net:max-retries':'2',
        'net:timeout':'10',
        'net:connection-limit':'5',
        'net:reconnect-interval-base':'5',
        'net:reconnect-interval-multiplier':'1',
        'mirror:parallel-transfer-count':'5'
    }
};
const client = new RemoteSync(config);
client.perform(); // Mirror only missing files from remote source to local disk.

Fancy level 2: Create a conditional chain of operations where you create the condition for further execution. Open a non-persistent connection to ftp.host.com and execute operation 1. If operation 1's status is not 0 (success) exit the parent process, halting any further execution. If operation 1 is successful, operations 2 & 3 will be executed.

// client.js
const RemoteSync = require('remote-sync');
const mirror = 'mirror -c --only-missing <source> <dest>';
const remove = 'rm -r <source>';
const list = 'nlist files';
const config = {
    operations : [
        {
            operation : 'mirror directory',
            command : mirror,
            settings : {
                sync : child => {
                    if (child.status != 0) {
                        process.exit(1);
                    }
                }
            }
        },
        {  
            operation : 'delete directory',
            command : remove
        },
        {
            operation : 'list directory',
            command : list,
            user : 'username',
            pw : 'password',
            host : 'other.host.com'
        }
    ],
    user : 'kurt',
    pw : 'foobar',
    host : 'ftp.host.com',
    lftp_settings : settings_obj, // omitted for brevity
    sync : true
};
const client = new RemoteSync(config);
client.perform();   // Mirror only missing files from remote source to local disk.
                    // If successful, delete the remote source and get remote listing
                    // from other.host.com/files/

Fancy level 3: Open a persistent connection (persistent: true) to the designated host and perform operations in series. In this configuration a connection is established to ftp.host.com, each operation's command is executed in series followed by severing the connection (exit: true).

// client.js
const RemoteSync = require('remote-sync');
const mirror = 'mirror -c --only-missing <source> <dest>';
const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>'; // reverse mirror -R
const config = {
    operations : [
        {
            operation : 'mirror directory',
            command : mirror
        },
        {  
            operation : 'upload directory',
            command : upload
        }
    ],
    user : 'kurt',
    pw : 'foobar',
    host : 'ftp.host.com',
    lftp_settings : settings_obj, // omitted for brevity
    persistent : true,
    exit : true
};
const client = new RemoteSync(config);
client.perform();   // Connection stays open until `mirror` and `upload` complete

General Purpose LFTP Client

'use strict';
const RemoteSync = require('remote-sync');
const rs = new RemoteSync({
  user : 'kurt',
  pw : 'foobar',
  host : 'ftp.host.com'
});
// Define an iterator and execute commands in series
rs[Symbol.iterator] = function* () {
    yield rs.commands('nlist files').execute();
    yield rs.commands('nlist files/completed').execute();
    yield rs.commands('nlist files').execute();
};
[...rs];

Common Commands

Mirror directory from source to dest

const mirror = 'mirror -c --only-missing <source> <dest>';

Upload directory from local to remote

const upload = 'mirror -R -c --only-newer --overwrite --exclude .git/ <local> <remote>';

Delete remote source

const remove = 'rm -r <source>';

Get remote listing source

const list = 'nlist <source>';

Cronjob

# Run client.js every 15 minutes
0/15 * * * * path/to/node path/to/client.js

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100
No reviews found
Be the first to rate

Alternatives

No alternatives found

Tutorials

No tutorials found
Add a tutorial