ana
github.com/michaelbironneau/analyst
go get github.com/michaelbironneau/analyst
ana

github.com/michaelbironneau/analyst

A declarative, SQL-like DSL for data integration tasks.

by Michael Bironneau

v0.0.0-20180704152941-abd831de7a77 (see all)License:AGPL-3.0
go get github.com/michaelbironneau/analyst
Readme

Analyst

Go Report Card Build Status

Purpose

Analyst is a tool to validate and run Analyst Query Language (AQL) scripts. AQL is an ETL configuration language for developers that aims to be:

  • Declarative: the developer defines the components, how they depend on one another, and any additional synchronization (i.e. AFTER); the runtime figures out the DAG and executes it
  • Intuitive: similar syntax to SQL, but any options for external programs such as MS Excel use native conventions such as Excel Ranges
  • Maintainable: support large jobs and code reuse through language features like INCLUDE and EXTERN
  • Extensible: use stdin/stdout protocol and pipes to write ETL logic in any language. Native support for Python and Javascript.
  • Stateful: Components can persist state in an SQLite3 database unique to each job run (GLOBAL source/destination).

It has connectors to:

  • MS SQL Server (source/destination)
  • Postgres (source/destination)
  • SQLite3 (source/destination)
  • Mandrill transactional email API (destination)
  • Web APIs (source)
  • Slack (for logging only)
  • Flat file (source)
  • Console (destination)
  • Built-in in-memory SQLite3 database (source/destination)
  • JSON-RPC plugins (source/destination)

Getting Started

  1. Grab the latest binary from the releases tab and place it on your PATH.
  2. Create and save an AQL script.
  3. Run analyst run --script <path-to-your-script>.

For a "hello world" example, try

DATA 'MyMessage' (
    [
      ["Hello, World"]
    ]
) INTO CONSOLE WITH (COLUMNS = 'Message')

Documentation

Docs are on Github pages here.

Table of Contents

  1. Get Started
  2. Recipes
  3. Blocks

Contributing

All contributions are welcome:

  • To report any bugs or feature suggestion, please open an issue
  • If you wish to fix a minor bug or issue, please open a PR directly
  • For enhancements, refactoring, or major issues, please open an issue before opening a PR

License

All source code and artifacts are released under GNU Affero General Public License v3.0, as detailed in LICENSE.md.

If this not suitable for your use case please get in touch by opening an issue or Twitter @MikeBrno.

GitHub Stars

12

LAST COMMIT

4yrs ago

MAINTAINERS

0

CONTRIBUTORS

3

OPEN ISSUES

0

OPEN PRs

0
VersionTagPublished
v0.0.0-20180704152941-abd831de7a77
3yrs ago
No alternatives found
No tutorials found
Add a tutorial