Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.
You declare functions that you want to run and the data assets that those functions produce or update. Dagster then helps you run your functions at the right time and keep your assets up-to-date.
Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.
An asset graph defined in Python:
from dagster import asset from pandas import DataFrame, read_html, get_dummies from sklearn.linear_model import LinearRegression def country_populations() -> DataFrame: df = read_html("https://tinyurl.com/mry64ebh") df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"] df["change"] = df["change"].str.rstrip("%").str.replace("−", "-").astype("float") return df def continent_change_model(country_populations: DataFrame) -> LinearRegression: data = country_populations.dropna(subset=["change"]) return LinearRegression().fit( get_dummies(data[["continent"]]), data["change"] ) def continent_stats( country_populations: DataFrame, continent_change_model: LinearRegression ) -> DataFrame: result = country_populations.groupby("continent").sum() result["pop_change_factor"] = continent_change_model.coef_ return result
The graph loaded into Dagster's web UI:
Dagster is available on PyPI and officially supports Python 3.7+.
pip install dagster dagit
This installs two modules:
You can find the full Dagster documentation here.
Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.
Join our community here:
For details on contributing or running the project for development, check out our contributing guide.
Dagster is Apache 2.0 licensed.