ploomber

Write maintainable, production-ready pipelines using Jupyter or your favorite text editor. Develop locally, deploy to the cloud. ☁️

Showing:

Popularity

Downloads/wk

0

GitHub Stars

522

Maintenance

Last Commit

5d ago

Contributors

8

Package

Dependencies

51

License

Categories

Readme

CI Linux CI macOS CI Windows Documentation Status PyPI Conda (channel only) Conda Coverage Twitter Binder

Join our community | Newsletter | Contact us | Docs | Blog

Notebooks are hard to maintain. Teams often prototype projects in notebooks, but maintaining them is an error-prone process that slows progress down. Ploomber overcomes the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor.

Main Features

  1. Scripts as notebooks. Open .py files as notebooks, then execute them from the terminal and generate an output notebook to review results.
  2. Dependency resolution. Quickly build a DAG by referring to previous tasks in your code; Ploomber infers execution order and orchestrates execution.
  3. Incremental builds. Speed up iterations by skipping tasks whose source code hasn't changed since the last execution.
  4. Production-ready. Deploy to Kubernetes (via Argo Workflows), Airflow, and AWS Batch without code changes.
  5. Parallelization. Run independent tasks in parallel.
  6. Testing. Import pipelines in any testing frameworks and test them with any CI service (e.g. GitHub Actions).
  7. Flexible. Use Jupyter notebooks, Python scripts, R scripts, SQL scripts, Python functions, or a combination of them as pipeline tasks. Write pipelines using a pipeline.yaml file or with Python.

repo-lab-example

Community

Resources

Installation

Compatible with Python 3.6 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Use Binder to try out Ploomber without setting up an environment:

image

Or run an example locally:

# ML pipeline example
ploomber examples --name ml-basic
cd ml-basic

# if using pip
pip install -r requirements.txt

# if using conda
conda env create --file environment.yml
conda activate ml-basic

# run pipeline
ploomber build

Pipeline output saved in the output/ folder. Check out the pipeline definition in the pipeline.yaml file.

To get a list of examples, run ploomber examples.

Click here to go to our examples repository.

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100