dagster-twilio

dagster-twilio

pypi i dagster-twilio

383 Versions

0.16.10

6 days ago

0.15.10

12 days ago

0.16.9

13 days ago

0.16.8

19 days ago

0.16.7

1 month ago

0.15.9

1 month ago

0.16.6

1 month ago

0.16.4

1 month ago

0.16.3

2 months ago

0.16.2

2 months ago

0.16.1

2 months ago

0.16.0

2 months ago

0.16.0rc2

2 months ago

0.15.8

2 months ago
New
  • Software-defined asset config schemas are no longer restricted to dicts.
  • The OpDefinition constructor now accept ins and outs arguments, to make direct construction easier.
  • define_dagstermill_op accepts ins and outs in order to make direct construction easier.
Bugfixes
  • Fixed a bug where default configuration was not applied when assets were selected for materialization in Dagit.
  • Fixed a bug where RunRequests returned from run_status_sensors caused the sensor to error.
  • When supplying config to define_asset_job, an error would occur when selecting most asset subsets. This has been fixed.
  • Fixed an error introduced in 0.15.7 that would prevent viewing the execution plan for a job re-execution from 0.15.0 → 0.15.6
  • [dagit] The Dagit server now returns 500 http status codes for GraphQL requests that encountered an unexpected server error.
  • [dagit] Fixed a bug that made it impossible to kick off materializations of partitioned asset if the day_offset, hour_offset, or minute_offset parameters were set on the asset’s partitions definition.
  • [dagster-k8s] Fixed a bug where overriding the Kubernetes command to use to run a Dagster job by setting the dagster-k8s/config didn’t actually override the command.
  • [dagster-datahub] Pinned version of acryl-datahub to avoid build error.
Breaking Changes
  • The constructor of JobDefinition objects now accept a config argument, and the preset_defs argument has been removed.
Deprecations
  • DagsterPipelineRunMetadataValue has been renamed to DagsterRunMetadataValue. DagsterPipelineRunMetadataValue will be removed in 1.0.
Community Contributions
  • Thanks to @hassen-io for fixing a broken link in the docs!
Documentation
  • MetadataEntry static methods are now marked as deprecated in the docs.
  • PartitionMappings are now included in the API reference.
  • A dbt example and memoization example using legacy APIs have been removed from the docs site.

0.15.7

2 months ago
New
  • DagsterRun now has a job_name property, which should be used instead of pipeline_name.
  • TimeWindowPartitionsDefinition now has a get_partition_keys_in_range method which returns a sequence of all the partition keys between two partition keys.
  • OpExecutionContext now has asset_partitions_def_for_output and asset_partitions_def_for_input methods.
  • Dagster now errors immediately with an informative message when two AssetsDefinition objects with the same key are provided to the same repository.
  • build_output_context now accepts a partition_key argument that can be used when testing the handle_output method of an IO manager.
Bugfixes
  • Fixed a bug that made it impossible to load inputs using a DagsterTypeLoader if the InputDefinition had an asset_key set.
  • Ops created with the @asset and @multi_asset decorators no longer have a top-level “assets” entry in their config schema. This entry was unused.
  • In 0.15.6, a bug was introduced that made it impossible to load repositories if assets that had non-standard metadata attached to them were present. This has been fixed.
  • [dagster-dbt] In some cases, using load_assets_from_dbt_manifest with a select parameter that included sources would result in an error. This has been fixed.
  • [dagit] Fixed an error where a race condition of a sensor/schedule page load and the sensor/schedule removal caused a GraphQL exception to be raised.
  • [dagit] The “Materialize” button no longer changes to “Rematerialize” in some scenarios
  • [dagit] The live overlays on asset views, showing latest materialization and run info, now load faster
  • [dagit] Typing whitespace into the launchpad Yaml editor no longer causes execution to fail to start
  • [dagit] The explorer sidebar no longer displays “mode” label and description for jobs, since modes are deprecated.
Community Contributions
  • An error will now be raised if a @repository decorated function expects parameters. Thanks @roeij!
Documentation
  • The non-asset version of the Hacker News example, which lived inside examples/hacker_news/, has been removed, because it hadn’t received updates in a long time and had drifted from best practices. The asset version is still there and has an updated README. Check it out here
All Changes

https://github.com/dagster-io/dagster/compare/0.15.6...0.15.7

See All Contributors
  • 57acdd5 - Correct check for pickle s3 io manager (#8834) by @ripplekhera

  • 4114910 - [dagit] Always show “Materialize” instead of “Rematerialize” based on status (#8711) by @bengotow

  • 972274c - [dagster-dbt] make group configurable for load_assets_from_dbt (#8863) by @OwenKephart

  • 05fc596 - asset_partitions_def on InputContext and OutputContext (#8858) by @sryza

  • 2d86c76 - [dagster-dbt] refactor the dbt asset integration (#8793) by @OwenKephart

  • 150bc3e - PartitionMappings when non-partitioned assets depend on partitioned assets (#8866) by @sryza

  • 70a7dbf - [dagster-dbt] seeds and snapshots are assets when using dbt build (#8794) by @OwenKephart

  • e46b9a0 - Document valid names for asset keys (#8765) by @jamiedemaria

  • 2357a02 - [docs] Dagster + dbt guide (#8714) by @OwenKephart

  • dbaed58 - 0.15.6 changelog (#8876) by @yuhan

  • 5bb50c0 - provide description for MAP ConfigType (#8824) by @Jiafi

  • b1aa83a - Retrieve minimal set of asset records for assetsLatestInfo (#8835) by @bengotow

  • b8493f3 - error when duplicate asset keys on a repository (#8874) by @sryza

  • 084c66c - [docs] - Add Airflow Operator to Op Docs (#8875) by @clairelin135

  • 2f15fbf - dagster-datahub Rest and Kafka Emitter integration (#8764) by @Jiafi

  • 0988274 - Automation: versioned docs for 0.15.6 by @elementl-devtools

  • 0e83834 - [1.0] move solid to dagster.legacy (#8843) by @dpeng817

  • bc5e502 - Extract ECS task overrides (#8842) by @jmsanders

  • e3ea175 - [graphql] tolerate empty runConfigData (#8886) by @alangenfeld

  • 56c7023 - [dagit] Fix edge case where “ “ launchpad config is not coerced to an empty object (#8895) by @bengotow

  • ee2e977 - Fix ScheduleRootQuery typo (#8903) by @johannkm

  • 61c1c20 - unloadable shit (#8887) by @prha

  • 711b323 - Change base image for OSS release builds (#8902) by @gibsondan

  • c85e158 - change deprecation warnings to 1.0 (#8892) by @dpeng817

  • cd779b1 - update README for hacker news assets example (#8904) by @sryza

  • e657abd - [hacker news] add missing key prefix to activity analytics python assets (#8817) by @sryza

  • 4da2a9e - [buildkite] Specify internal branch used for compatibility pipeline (#8881) by @smackesey

  • 6c97c75 - [dagit] Remove “mode” label and description in explorer sidebar (#8907) by @bengotow

  • 4cefd84 - remove the non-asset version of the hacker news example (#8883) by @sryza

  • 23a9997 - Error when @repository-decorated function has arguments (#8913) by @roeij

  • f787d6d - [docs] - Correct snippets for dbt (#8923) by @erinkcochran87

  • 1961e51 - [bug] fix input loading regression (#8885) by @OwenKephart

  • ff87738 - [docs] - graph backed assets doc fix (#8927) by @jamiedemaria

  • 18f254d - silence system-originated experimental warning for PartitionMapping (#8931) by @sryza

  • a2df1de - Add partition key to build_output_context, add documentation for partition_key on build_op_context (#8774) by @dpeng817

  • 53287b9 - fix dimensions of screenshot on connecting ops tutorial page (#8908) by @sryza

  • c00de5b - ttv: remove undocumented/legacy example - user in loop (#8934) by @yuhan

  • 31f3283 - [docs] - Clean up graph-backed asset example, put under test (#8893) by @dpeng817

  • 7c60a46 - [docs] - Fix garbled sentence in ops.mdx (#8935) by @schrockn

  • c554461 - enable getting asset partitions def from op context (#8924) by @sryza

  • 7c13e28 - Increase test_docker_monitoring timeout (#8906) by @johannkm

  • 6365996 - PartitionsDefinition.get_partition_keys_in_range (#8933) by @sryza

  • b58d711 - Move pipeline to dagster.legacy (#8888) by @dpeng817

  • 7e11df2 - [dagit] Rename search open event (#8954) by @hellendag

  • f3caeae - [dagit] Adjust shift-selection behavior in asset graphs (#8950) by @bengotow

  • 65caf79 - [dagit] Clean up code around the graph sidebar (#8914) by @bengotow

  • 5bd5c8b - add a job_name property to PipelineRun (#8928) by @sryza

  • 9421f73 - remove partition entries from asset op config schema (#8951) by @sryza

  • 000d37a - avoid pipelines in run status sensor doc snippets (#8929) by @sryza

  • a9b25dd - [bug] Fix issue where 'invalid' asset metadata resulted in an error (#8947) by @OwenKephart

  • 4dadcd4 - [dagster-dbt] fix tagged source asset loading (#8943) by @OwenKephart

  • 065adbd - fix black in run status sensor docs example (#8974) by @sryza

  • 164c585 - [known state] fix build_for_reexecution bug (#8975) by @alangenfeld

  • cc70c88 - Document deprecation of MetadataEntry static constructors (#8984) by @smackesey

  • eed9277 - changelog (#8986) by @jamiedemaria

  • b283b8a - 0.15.7 by @elementl-devtools


0.15.6

2 months ago
New
  • When an exception is wrapped by another exception and raised within an op, Dagit will now display the full chain of exceptions, instead of stopping after a single exception level.
  • A default_logger_defs argument has been added to the @repository decorator. Check out the docs on specifying default loggers to learn more.
  • AssetsDefinition.from_graph and AssetsDefinition.from_op now both accept a partition_mappings argument.
  • AssetsDefinition.from_graph and AssetsDefinition.from_op now both accept a metadata_by_output_name argument.
  • define_asset_job now accepts an executor_def argument.
  • Removed package pin for gql in dagster-graphql.
  • You can now apply a group name to assets produced with the @multi_asset decorator, either by supplying a group_name argument (which will apply to all of the output assets), or by setting the group_name argument on individual AssetOuts.
  • InputContext and OutputContext now each have an asset_partitions_def property, which returns the PartitionsDefinition of the asset that’s being loaded or stored.
  • build_schedule_from_partitioned_job now raises a more informative error when provided a non-partitioned asset job
  • PartitionMapping, IdentityPartitionMapping, AllPartitionMapping, and LastPartitionMapping are exposed at the top-level dagster package. They're currently marked experimental.
  • When a non-partitioned asset depends on a partitioned asset, you can now control which partitions of the upstream asset are used by the downstream asset, by supplying a PartitionMapping.
  • You can now set PartitionMappings on AssetIn.
  • [dagit] Made performance improvements to the loading of the partitions and backfill pages.
  • [dagit] The Global Asset Graph is back by popular demand, and can be reached via a new “View global asset lineage ”link on asset group and asset catalog pages! The global graph keeps asset in the same group visually clustered together and the query bar allows you to visualize a custom slice of your asset graph.
  • [dagit] Simplified the Content Security Policy and removed frame-ancestors restriction.
  • [dagster-dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now support a node_info_to_group_name_fn parameter, allowing you to customize which group Dagster will assign each dbt asset to.
  • [dagster-dbt] When you supply a runtime_metadata_fn when loading dbt assets, this metadata is added to the default metadata that dagster-dbt generates, rather than replacing it entirely.
  • [dagster-dbt] When you load dbt assets with use_build_command=True, seeds and snapshots will now be represented as Dagster assets. Previously, only models would be loaded as assets.
Bugfixes
  • Fixed an issue where runs that were launched using the DockerRunLauncher would sometimes use Dagit’s Python environment as the entrypoint to launch the run, even if that environment did not exist in the container.
  • Dagster no longer raises a “Duplicate definition found” error when a schedule definition targets a partitioned asset job.
  • Silenced some erroneous warnings that arose when using software-defined assets.
  • When returning multiple outputs as a tuple, empty list values no longer cause unexpected exceptions.
  • [dagit] Fixed an issue with graph-backed assets causing a GraphQL error when graph inputs were type-annotated.
  • [dagit] Fixed an issue where attempting to materialize graph-backed assets caused a graphql error.
  • [dagit] Fixed an issue where partitions could not be selected when materializing partitioned assets with associated resources.
  • [dagit] Attempting to materialize assets with required resources now only presents the launchpad modal if at least one resource defines a config schema.
Breaking Changes
  • An op with a non-optional DynamicOutput will now error if no outputs are returned or yielded for that dynamic output.
  • If an Output object is used to type annotate the return of an op, an Output object must be returned or an error will result.
Community Contributions
  • Dagit now displays the path of the output handled by PickledObjectS3IOManager in run logs and Asset view. Thanks @danielgafni
Documentation
  • The Hacker News example now uses stable 0.15+ asset APIs, instead of the deprecated 0.14.x asset APIs.
  • Fixed the build command in the instructions for contributing docs changes.
  • [dagster-dbt] The dagster-dbt integration guide now contains information on using dbt with Software-Defined Assets.
All Changes

https://github.com/dagster-io/dagster/compare/0.15.5...0.15.6

See All Contributors
  • 583ce34 - Fold asset_defs submodule into main dagster structure (#8446) by @smackesey

  • b4fa57e - Op that runs a kubernetes job (#8161) by @gibsondan

  • a3a5ccb - add validation for graph backed asset graphs (#8754) by @OwenKephart

  • 5d5dd71 - chore: mark snapshots as generated (#8758) by @rexledesma

  • 4a9718d - [op] fix empty list output bug (#8763) by @alangenfeld

  • 397ad03 - [dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725) by @OwenKephart

  • b007a0c - docs: update config schema descriptions for default executors (#8757) by @rexledesma

  • 58518d9 - Restore sensor daemon yielding when evaluating sensors synchronously (#8756) by @prha

  • c449c0f - bypass bucketed queries for mysql versions that do not support it (#8753) by @prha

  • b1ac8d6 - [dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775) by @OwenKephart

  • 2a237a0 - Specifying executor docs examples (#8530) by @dpeng817

  • eccdac2 - prevent multiple sensor evaluations from multithreaded race conditions (#8720) by @prha

  • 519d77f - Fix config case for default executor (#8777) by @dpeng817

  • daadb5a - Ensure graph inputs/outputs are included in all_dagster_types (#8736) by @smackesey

  • a94210a - improve error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776) by @sryza

  • 0bab6fd - [dagit] Bring back the global asset graph as an “all asset groups” view (#8709) by @bengotow

  • f6987da - fix source asset regression (#8784) by @smackesey

  • 530c321 - fix issue with repos and partitioned scheduled asset jobs (#8779) by @sryza

  • b3ba40c - 0.15.5 Changelog (#8781) by @prha

  • 0e32054 - changelog (#8788) by @prha

  • 1452280 - Option to hide daemon heartbeat timestamp in Dagit (#8785) by @johannkm

  • 1e7691d - Fix bug with how resources are applied in materialize (#8790) by @dpeng817

  • ccd1893 - Add default_logger_defs arg to repository (#8512) by @dpeng817

  • e10b6f3 - update hackernews tests to use asset invocation and materialize_to_memory (#8592) by @dpeng817

  • 5bdd6cf - Add MetaDataEntry.path to PickledObjectS3IOManager (#8732) by @danielgafni

  • 6d19b1d - Automation: versioned docs for 0.15.5 by @elementl-devtools

  • f459da0 - add define_asset_job to __all__ (#8791) by @sryza

  • f1a4612 - eliminate incorrect SDA warnings (#8769) by @sryza

  • 738e7eb - update hacker news assets example for post-0.15.0 APIs (#7904) by @sryza

  • 5938830 - partition mappings on graph-backed assets (#8768) by @sryza

  • b721c70 - Snowflake IO Manager handles pandas timestamps (#8760) by @jamiedemaria

  • cb178b1 - Add Python 3.10 testing to BK and other image adjustments (#7700) by @smackesey

  • f4415aa - Option to skip daemon heartbeats with no errors (#8670) by @johannkm

  • 2d35b84 - Assorted type annotations (#8356) by @smackesey

  • 3e1d539 - Bump urllib3 (#8808) by @dependabot[bot]

  • de366db - Bump rsa (#8807) by @dependabot[bot]

  • 6771e78 - Bump pyyaml (#8806) by @dependabot[bot]

  • 44492db - Change default local grpc behavior to send back "dagster" as the entry point to use, rather than dagit's python environment (#8571) by @gibsondan

  • 776d701 - move Metadata and Tags concept page under jobs section (#8813) by @sryza

  • c142bd9 - updates to multi-assets docs page (#8814) by @sryza

  • 3eb0afd - Update asset ID to contain repository location and name (#8762) by @clairelin135

  • a05f9b1 - unpin gql (#8822) by @prha

  • c89d01e - remove unused old partitions ui (#8796) by @prha

  • 1539cfb - fix broken asset deps in hacker news example (#8809) by @sryza

  • 7f5cd8e - [dagit] Bump TS version (#8704) by @hellendag

  • 8da4644 - fix contributing docs (#8789) by @prha

  • 1714bc9 - Better support for nested causes in dagit (#8823) by @gibsondan

  • 641c707 - [easy] Fix docs link for RetryPolicy (#8830) by @gibsondan

  • 04fb5c1 - Remove the "cronjobs" permission from the helm chart (#8827) by @gibsondan

  • e7111f9 - fix gql resolver for graph-backed assets resources (#8825) by @smackesey

  • 418952d - refactor run storage to enable backfill status queries (#8695) by @prha

  • 3eae463 - refactor backfill / partition pages to stop run fetching (#8696) by @prha

  • 47238c2 - add multi_or_in_process_executor to __all__ (#8831) by @smackesey

  • a3ec60b - avoid apollo cache collision for partition/backfill status (#8841) by @prha

  • f741443 - distinguish between [] and None for asset queries (#8838) by @prha

  • 9906c4b - override batch loader to use asset records instead of legacy event materialization method (#8839) by @prha

  • e523dad - [dagit] Add analytics.group (#8832) by @hellendag

  • 52d4fdf - [dagster-io/ui] Fix disabled Button (#8844) by @hellendag

  • 633d6b4 - Fix issue where partitioned assets with resources fail materialization in dagit (#8837) by @smackesey

  • 652d12e - [dagit] Tweak analytics function sigs (#8851) by @hellendag

  • 7cd7de8 - [asset-defs] allow multi assets to have group names (#8847) by @OwenKephart

  • acf8c4d - Refactor op return checking code (#8755) by @dpeng817

  • df5833e - [dagit] Remove frame-ancestors restriction (#8850) by @hellendag

  • e5aca1b - adjust error messages (#8853) by @dpeng817

  • fb89b1f - [dagit] Update CRA, simplify CSP (#8854) by @hellendag

  • 50dbd6a - key_prefix for AssetsDefinition from_graph and from_op (#8859) by @jamiedemaria

  • 105ad88 - easy: fix dagster pandas link (#8862) by @yuhan

  • 1d33b3f - executor_definition on define_asset_job (#8856) by @sryza

  • c9cc22b - include airflow_operator_to_op in apidoc (#8860) by @sryza

  • 0fc21f5 - add metdata_by_output_name (#8861) by @OwenKephart

  • 3916741 - [dagster-dbt] make group configurable for load_assets_from_dbt (#8863) by @OwenKephart

  • 7eded6b - asset_partitions_def on InputContext and OutputContext (#8858) by @sryza

  • 2dfcff7 - [dagster-dbt] refactor the dbt asset integration (#8793) by @OwenKephart

  • 915948e - [dagster-dbt] seeds and snapshots are assets when using dbt build (#8794) by @OwenKephart

  • e7a82d0 - PartitionMappings when non-partitioned assets depend on partitioned assets (#8866) by @sryza

  • a555b22 - [docs] Dagster + dbt guide (#8714) by @OwenKephart

  • 6e4dfcb - 0.15.6 by @elementl-devtools


0.15.5

3 months ago
New
  • Added documentation and helm chart configuration for threaded sensor evaluations.
  • Added documentation and helm chart configuration for tick retention policies.
  • Added descriptions for default config schema. Fields like execution, loggers, ops, and resources are now documented.
  • UnresolvedAssetJob objects can now be passed to run status sensors.
  • [dagit] A new global asset lineage view, linked from the Asset Catalog and Asset Group pages, allows you to view a graph of assets in all loaded asset groups and filter by query selector and repo.
  • [dagit] A new option on Asset Lineage pages allows you to choose how many layers of the upstream / downstream graph to display.
  • [dagit] Dagit's DAG view now collapses large sets of edges between the same ops for improved readability and rendering performance.
Bugfixes
  • Fixed a bug with materialize that would cause required resources to not be applied correctly.
  • Fixed issue that caused repositories to fail to load when build_schedule_from_partitioned_job and define_asset_job were used together.
  • Fixed a bug that caused auto run retries to always use the FROM_FAILURE strategy
  • Previously, it was possible to construct Software-Defined Assets from graphs whose leaf ops were not mapped to assets. This is invalid, as these ops are not required for the production of any assets, and would cause confusing behavior or errors on execution. This will now result in an error at definition time, as intended.
  • Fixed issue where the run monitoring daemon could mark completed runs as failed if they transitioned quickly between STARTING and SUCCESS status.
  • Fixed stability issues with the sensor daemon introduced in 0.15.3 that caused the daemon to fail heartbeat checks if the sensor evaluation took too long.
  • Fixed issues with the thread pool implementation of the sensor daemon where race conditions caused the sensor to fire more frequently than the minimum interval.
  • Fixed an issue with storage implementations using MySQL server version 5.6 which caused SQL syntax exceptions to surface when rendering the Instance overview pages in Dagit.
  • Fixed a bug with the default_executor_def argument on repository where asset jobs that defined executor config would result in errors.
  • Fixed a bug where an erroneous exception would be raised if an empty list was returned for a list output of an op.
  • [dagit] Clicking the "Materialize" button for assets with configurable resources will now present the asset launchpad.
  • [dagit] If you have an asset group and no jobs, Dagit will display it by default rather than directing you to the asset catalog.
  • [dagit] DAG renderings of software-defined assets now display only the last component of the asset's key for improved readability.
  • [dagit] Fixes a regression where clicking on a source asset would trigger a GraphQL error.
  • [dagit] Fixed issue where the “Unloadable” section on the sensors / schedules pages in Dagit were populated erroneously with loadable sensors and schedules
  • [dagster-dbt] Fixed an issue where an exception would be raised when using the dbt build command with Software-Defined Assets if a test was defined on a source.
Deprecations
  • Removed the deprecated dagster-daemon health-check CLI command
Community Contributions
  • TimeWindow is now exported from the dagster package (Thanks @nvinhphuc!)
  • Added a fix to allow customization of slack messages (Thanks @solarisa21!)
  • [dagster-databricks] The databricks_pyspark_step_launcher now allows you to configure the following (Thanks @Phazure!):
    • the aws_attributes of the cluster that will be spun up for the step.
    • arbitrary environment variables to be copied over to databricks from the host machine, rather than requiring these variables to be stored as secrets.
    • job and cluster permissions, allowing users to view the completed runs through the databricks console, even if they’re kicked off by a service account.
Experimental
  • [dagster-k8s] Added k8s_job_op to launch a Kubernetes Job with an arbitrary image and CLI command. This is in contrast with the k8s_job_executor, which runs each Dagster op in a Dagster job in its own k8s job. This op may be useful when you need to orchestrate a command that isn't a Dagster op (or isn't written in Python). Usage:

    from dagster_k8s import k8s_job_op
    
    my_k8s_op = k8s_job_op.configured({
     "image": "busybox",
     "command": ["/bin/sh", "-c"],
     "args": ["echo HELLO"],
     },
     name="my_k8s_op",
    )
    
  • [dagster-dbt] The dbt asset-loading functions now support partitions_def and partition_key_to_vars_fn parameters, adding preliminary support for partitioned dbt assets. To learn more, check out the Github issue!

All Changes

https://github.com/dagster-io/dagster/compare/0.15.4...0.15.5

See All Contributors
  • 5192aa0 - Remove unused check_heartbeats arg (#8673) by @johannkm

  • 45e5019 - docs: use dagster brand colors for README (#8660) by @rexledesma

  • a09234a - fix: use absolute url for README images (#8676) by @rexledesma

  • 7a34e49 - Expose TimeWindow in dagster package (#8643) by @nvinhphuc

  • 9842ba8 - hold shift to force asset config modal (#8668) by @smackesey

  • d0ffb1e - add helm values for configuring instance sensor config (#8657) by @prha

  • b335d19 - [dagit] If you have no jobs, prefer routing to asset group over asset catalog (#8613) by @bengotow

  • 44bac29 - [dagit] Add control for graph depth on the Asset Lineage page, default to 5 (#8531) by @bengotow

  • 593cfc4 - [assets] Fix issue with graph backed assets + partitions (#8682) by @OwenKephart

  • eeb23e6 - [dagster-dbt] rework dagster dbt logging, cleanup (#8681) by @OwenKephart

  • b6396aa - Yield run requests from run status sensors (#8635) by @jamiedemaria

  • f53d5f4 - fix docs build (#8688) by @OwenKephart

  • 62801c8 - apidoc for AssetSelection and AssetsDefinition (#8618) by @sryza

  • 17b5233 - Materialize has resources arg, materialize_to_memory sets mem_io_manager for all io managers (#8659) by @dpeng817

  • 44d95b1 - Load asset launchpad for assets with configurable resources (#8677) by @smackesey

  • 01cfe05 - docs: update external links for github issues (#8661) by @rexledesma

  • c1cb62b - docs: convert github issue templates to forms (#8663) by @rexledesma

  • d338a0e - switch unloadable states to dedupe by selector id (#8656) by @prha

  • d3678bc - 0.15.3 changelog (#8690) by @dpeng817

  • e0d5dcb - Automation: versioned docs for 0.15.3 by @elementl-devtools

  • dc91a2d - docs: remove extra cruft from issue form (#8701) by @rexledesma

  • ca432f7 - docs: improve left nav for items with children DREL-359 (#8693) by @yuhan

  • b88b33e - docs: fix prev/next pagination (#8697) by @yuhan

  • 47118bb - docs: docs test should capture Next Image broken links and fix broken links (#8702) by @yuhan

  • 293db00 - [dagit] Display last asset key component on DAG rather than truncated full path (#8692) by @bengotow

  • 0d65304 - [dagit] Collapsed DAG rendering of multiple edges between the same ops (#8479) by @bengotow

  • 7d5bcc8 - [dagit] Add Analytics context (#8674) by @hellendag

  • 3e2fe7e - order the backfill partitions before creating (#8703) by @prha

  • 0533455 - skip threadpool sensor daemon tests (#8717) by @prha

  • 863ba80 - skip threaded sensor tests in py36 (#8726) by @prha

  • dc0fbe8 - Allow run status sensors to support unresolved asset jobs (#8689) by @smackesey

  • 979d644 - [dagster_databricks] - support configuration of job / cluster permissions (#8683) by @Phazure

  • a9ff7ca - [dagster_databricks] support aws_attributes (#8684) by @Phazure

  • 150a06b - [Job Log perf] Use rAF to call throttleSetNodes (#8735) by @salazarm

  • 33f059f - [dagster_databricks] support arbitrary env variables (#8685) by @Phazure

  • c72634c - minor changes to make dev_install (#8745) by @smackesey

  • bc227bb - Remove deprecated daemon health-check cli (#8751) by @johannkm

  • a57caf3 - Add reexecution strategy to auto run retries (#8718) by @johannkm

  • d90fc4a - Fix for blocks_fn option (#8448) by @clairelin135

  • 2d9a9e7 - Automation: versioned docs for 0.15.4 by @elementl-devtools

  • d4735dc - Improve race condition in run monitor (#8729) by @brad-alexander

  • 6f4b300 - Add get_daemon_statuses instance method (#8752) by @johannkm

  • b974184 - 0.15.4 changelog here (#8766) by @prha

  • bdca087 - add retention helm values (#8724) by @prha

  • 823a5d8 - Op that runs a kubernetes job (#8161) by @gibsondan

  • 62f41c4 - [op] fix empty list output bug (#8763) by @alangenfeld

  • 4d88332 - [dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725) by @OwenKephart

  • 874a368 - add validation for graph backed asset graphs (#8754) by @OwenKephart

  • 38ae84b - docs: update config schema descriptions for default executors (#8757) by @rexledesma

  • 0779ab7 - [dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775) by @OwenKephart

  • 5f92ca1 - Restore sensor daemon yielding when evaluating sensors synchronously (#8756) by @prha

  • 4176e42 - bypass bucketed queries for mysql versions that do not support it (#8753) by @prha

  • fd1a3d9 - prevent multiple sensor evaluations from multithreaded race conditions (#8720) by @prha

  • 4058bb5 - Fold asset_defs submodule into main dagster structure (#8446) by @smackesey

  • 87e291e - Fix config case for default executor (#8777) by @dpeng817

  • 1544643 - Ensure graph inputs/outputs are included in all_dagster_types (#8736) by @smackesey

  • eeeb82c - [dagit] Bring back the global asset graph as an “all asset groups” view (#8709) by @bengotow

  • e40122e - fix source asset regression (#8784) by @smackesey

  • 2fe13df - improve error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776) by @sryza

  • a42146f - fix issue with repos and partitioned scheduled asset jobs (#8779) by @sryza

  • 798e9c1 - 0.15.5 Changelog (#8781) by @prha

  • 39fecd4 - Fix bug with how resources are applied in materialize (#8790) by @dpeng817

  • 8f9f0d0 - 0.15.5 by @elementl-devtools


0.15.4

3 months ago

No new features in 0.15.4

All Contributions

Full Changelog: https://github.com/dagster-io/dagster/compare/0.15.3...0.15.4


0.15.3

3 months ago
New
  • When loading an upstream asset or op output as an input, you can now set custom loading behavior using the input_manager_key argument to AssetIn and In
  • The list of objects returned by a repository can now contain nested lists.
  • Added a data retention instance setting in dagster.yaml that enables the automatic removal of sensor/schedule ticks after a certain number of days.
  • Added a sensor daemon setting in dagster.yaml that enables sensor evaluations to happen in a thread pool to increase throughput.
  • materialize_to_memory and materialize now both have the partition_key argument.
  • Output and DynamicOutput objects now work with deep equality checks:
Output(value=5, name="foo") == Output(value=5, name="foo") # evaluates to True
  • RunRequests can now be returned from run status sensors
  • Added resource_defs argument to AssetsDefinition.from_graph. Allows for specifying resources required by constituent ops directly on the asset.
  • When adding a tag to the Run search filter in Dagit by clicking the hover menu on the tag, the tag will now be appended to the filter instead of replacing the entire filter state.
Bugfixes
  • [dagster-dbt] An exception is now emitted if you attempt to invoke the library without having dbt-core installed. dbt-core is now also added as a dependency to the library.
  • Asset group names can now contain reserved python keywords
  • Fixed a run config parsing bug that was introduced in 0.15.1 that caused Dagit to interpret datetime strings as datetime objects and octal strings as integers.
  • Runs that have failed to start are now represented in the Instance Timeline view on Dagit.
  • Fixed an issue where the partition status was missing for partitioned jobs that had no runs.
  • Fixed a bug where op/resource invocation would error when resources were required, no context was used in the body of the function, and no context was provided when invoking.
  • [dagster-databricks] Fixed an issue where an exception related to the deprecated prior_attempts_count field when using the databricks_pyspark_step_launcher.
  • [dagster-databricks] Polling information logged from the databricks_pyspark_step_launcher is now emitted at the DEBUG level instead of INFO.
  • In the yaml editor in Dagit, the typeahead feature now correctly shows suggestions for nullable schema types.
  • When editing asset configuration in Dagit, the “Scaffold config” button in the Dagit launchpad sometimes showed the scaffold dialog beneath the launchpad. This has been fixed.
  • A recent change added execution timezones to some human-readable cron strings on schedules in Dagit. This was added incorrectly in some cases, and has now been fixed.
  • In the Dagit launchpad, a config state containing only empty newlines could lead to an error that could break the editor. This has been fixed.
  • Fixed issue that could cause partitioned graph-backed assets to attempt to load upstream inputs from the incorrect path when using the fs_io_manager (or other similar io managers).
  • [dagster-dbt] Fixed issue where errors generated from issuing dbt cli commands would only show json-formatted output, rather than a parsed, human-readable output.
  • [dagster-dbt] By default, dagster will invoke the dbt cli with a --log-format json flag. In some cases, this may cause dbt to report incorrect or misleading error messages. As a workaround, it is now possible to disable this behavior by setting the json_log_format configuration option on the dbt_cli_resource to False.
  • materialize_to_memory erroneously allowed non-in-memory io managers to be used. Now, providing io managers to materialize_to_memory will result in an error, and mem_io_manager will be provided to all io manager keys.
All Changes

https://github.com/dagster-io/dagster/compare/0.15.2...0.15.3

See All Contributors

0.15.2

3 months ago
Bugfixes
  • Fixed an issue where asset dependency resolution would break when two assets in the same group had the same name
All Changes

https://github.com/dagster-io/dagster/compare/0.15.1...0.15.2

See All Contributors
  • f78b5ee - Automation: versioned docs for 0.15.1 by @elementl-devtools

  • d13e6b4 - fix asset dep resolution bug when two assets in group have same name (#8609) by @sryza

  • 552cb44 - 0.15.2 by @elementl-devtools


0.15.1

3 months ago
New
  • When Dagster loads an event from the event log of a type that it doesn’t recognize (for example, because it was created by a newer version of Dagster) it will now return a placeholder event rather than raising an exception.
  • AssetsDefinition.from_graph() now accepts a group_name parameter. All assets created by from_graph are assigned to this group.
  • You can define an asset from an op via a new utility method AssetsDefinition.from_op. Dagster will infer asset inputs and outputs from the ins/outs defined on the @op in the same way as @graphs.
  • A default executor definition can be defined on a repository using the default_executor_def argument. The default executor definition will be used for all op/asset jobs that don’t explicitly define their own executor.
  • JobDefinition.run_request_for_partition now accepts a tags argument (Thanks @jburnich!)
  • In Dagit, the graph canvas now has a dotted background to help it stand out from the reset of the UI.
  • @multi_asset now accepts a resource_defs argument. The provided resources can be either used on the context, or satisfy the io manager requirements of the outs on the asset.
  • In Dagit, show execution timezone on cron strings, and use 12-hour or 24-hour time format depending on the user’s locale.
  • In Dagit, when viewing a run and selecting a specific step in the Gantt chart, the compute log selection state will now update to that step as well.
  • define_asset_job and to_job now can now accept a partitions_def argument and a config argument at the same time, as long as the value for the config argument is a hardcoded config dictionary (not a PartitionedConfig or ConfigMapping)
Bugfixes
  • Fixed an issue where entering a string in the launchpad that is valid YAML but invalid JSON would render incorrectly in Dagit.
  • Fixed an issue where steps using the k8s_job_executor and docker_executor would sometimes return the same event lines twice in the command-line output for the step.
  • Fixed type annotations on the @op decorator (Thanks Milos Tomic!)
  • Fixed an issue where job backfills were not displayed correctly on the Partition view in Dagit.
  • UnresolvedAssetJobDefinition now supports the run_request_for_partition method.
  • Fixed an issue in Dagit where the Instance Overview page would briefly flash a loading state while loading fresh data.
Breaking Changes
  • Runs that were executed in newer versions of Dagster may produce errors when their event logs are loaded in older versions of Dagit, due to new event types that were recently added. Going forward, Dagit has been made more resilient to handling new events.
Deprecations
  • Updated deprecation warnings to clarify that the deprecated metadata APIs will be removed in 0.16.0, not 0.15.0.
Experimental
  • If two assets are in the same group and the upstream asset has a multi-segment asset key, the downstream asset doesn’t need to specify the full asset key when declaring its dependency on the upstream asset - just the last segment.
Documentation
  • Added dedicated sections for op, graph, and job Concept docs in the sidenav
  • Moved graph documentation from the jobs docs into its own page
  • Added documentation for assigning asset groups and viewing them in Dagit
  • Added apidoc for AssetOut and AssetIn
  • Fixed a typo on the Run Configuration concept page (Thanks Wenshuai Hou!)
  • Updated screenshots in the software-defined assets tutorial to match the new Dagit UI
  • Fixed a typo in the Defining an asset section of the software-defined assets tutorial (Thanks Daniel Kim!)

0.15.0

3 months ago
Major Changes
  • Software-defined assets are now marked fully stable and are ready for prime time - we recommend using them whenever your goal using Dagster is to build and maintain data assets.
  • You can now organize software defined assets into groups by providing a group_name on your asset definition. These assets will be grouped together in Dagit.
  • Software-defined assets now accept configuration, similar to ops. E.g.
    from dagster import asset
    @asset(config_schema={"iterations": int})
    def my_asset(context):
        for i in range(context.op_config["iterations"]):
            ...
    
  • Asset definitions can now be created from graphs via AssetsDefinition.from_graph:
    @graph(out={"asset_one": GraphOut(), "asset_two": GraphOut()})
    def my_graph(input_asset):
        ...
    graph_asset = AssetsDefinition.from_graph(my_graph)
    
  • execute_in_process and GraphDefinition.to_job now both accept an input_values argument, so you can pass arbitrary Python objects to the root inputs of your graphs and jobs.
  • Ops that return Outputs and DynamicOutputs now work well with Python type annotations. You no longer need to sacrifice static type checking just because you want to include metadata on an output. E.g.
    from dagster import Output, op
    @op
    def my_op() -> Output[int]:
        return Output(5, metadata={"a": "b"})
    
  • You can now automatically re-execute runs from failure. This is analogous to op-level retries, except at the job level.
  • You can now supply arbitrary structured metadata on jobs, which will be displayed in Dagit.
  • The partitions and backfills pages in Dagit have been redesigned to be faster and show the status of all partitions, instead of just the last 30 or so.
  • The left navigation pane in Dagit is now grouped by repository, which makes it easier to work with when you have large numbers of jobs, especially when jobs in different repositories have the same name.
  • The Asset Details page for a software-defined asset now includes a Lineage tab, which makes it easy to see all the assets that are upstream or downstream of an asset.
Breaking Changes and Deprecations Software-defined assets

This release marks the official transition of software-defined assets from experimental to stable. We made some final changes to incorporate feedback and make the APIs as consistent as possible:

  • Support for adding tags to asset materializations, which was previously marked as experimental, has been removed.
  • Some of the properties of the previously-experimental AssetsDefinition class have been renamed. group_names is now group_names_by_key, asset_keys_by_input_name is now keys_by_input_name, and asset_keys_by_output_name is now keys_by_output_name, asset_key is now key, and asset_keys is now keys.
  • Removes previously experimental IO manager fs_asset_io_manager in favor of merging its functionality with fs_io_manager. fs_io_manager is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key. Similarly, removed adls2_pickle_asset_io_manager, gcs_pickle_asset_io_manager , and s3_pickle_asset_io_manager. Instead, adls2_pickle_io_manager, gcs_pickle_io_manager, and s3_pickle_io_manager now support software-defined assets.
  • (deprecation) The namespace argument on the @asset decorator and AssetIn has been deprecated. Users should use key_prefix instead.
  • (deprecation) AssetGroup has been deprecated. Users should instead place assets directly on repositories, optionally attaching resources using with_resources. Asset jobs should be defined using define_assets_job (replacing AssetGroup.build_job), and arbitrary sets of assets can be materialized using the standalone function materialize (replacing AssetGroup.materialize).
  • (deprecation) The outs property of the previously-experimental @multi_asset decorator now prefers a dictionary whose values are AssetOut objects instead of a dictionary whose values are Out objects. The latter still works, but is deprecated.
  • The previously-experimental property on OpExecutionContext called output_asset_partition_key is now deprecated in favor of asset_partition_key_for_output
Event records
  • The get_event_records method on DagsterInstance now requires a non-None argument event_records_filter. Passing a None value for the event_records_filter argument will now raise an exception where previously it generated a deprecation warning.
  • Removed methods events_for_asset_key and get_asset_events, which have been deprecated since 0.12.0.
Extension libraries
  • [dagster-dbt] (breaks previously-experimental API) When using the load_assets_from_dbt_project or load_assets_from_dbt_manifest , the AssetKeys generated for dbt sources are now the union of the source name and the table name, and the AssetKeys generated for models are now the union of the configured schema name for a given model (if any), and the model name. To revert to the old behavior: dbt_assets = load_assets_from_dbt_project(..., node_info_to_asset_key=lambda node_info: AssetKey(node_info["name"]).
  • [dagster-k8s] In the Dagster Helm chart, user code deployment configuration (like secrets, configmaps, or volumes) is now automatically included in any runs launched from that code. Previously, this behavior was opt-in. In most cases, this will not be a breaking change, but in less common cases where a user code deployment was running in a different kubernetes namespace or using a different service account, this could result in missing secrets or configmaps in a launched run that previously worked. You can return to the previous behavior where config on the user code deployment was not applied to any runs by setting the includeConfigInLaunchedRuns.enabled field to false for the user code deployment. See the Kubernetes Deployment docs for more details.
  • [dagster-snowflake] dagster-snowflake has dropped support for python 3.6. The library it is currently built on, snowflake-connector-python, dropped 3.6 support in their recent 2.7.5 release.
Other
  • The prior_attempts_count parameter is now removed from step-launching APIs. This parameter was not being used, as the information it held was stored elsewhere in all cases. It can safely be removed from invocations without changing behavior.
  • The FileCache class has been removed.
  • Previously, when schedules/sensors targeted jobs with the same name as other jobs in the repo, the jobs on the sensor/schedule would silently overwrite the other jobs. Now, this will cause an error.
New since 0.14.20
  • A new define_asset_job function allows you to define a selection of assets that should be executed together. The selection can be a simple string, or an AssetSelection object. This selection will be resolved into a set of assets once placed on the repository.

    from dagster import repository, define_asset_job, AssetSelection
        
    string_selection_job = define_asset_job(
        name="foo_job", selection="*foo"
    )
    object_selection_job = define_asset_job(
        name="bar_job", selection=AssetSelection.groups("some_group")
    )
    @repository
    def my_repo():
        return [
            *my_list_of_assets,
            string_selection_job,
            object_selection_job,
        ]
    
  • [dagster-dbt] Assets loaded with load_assets_from_dbt_project and load_assets_from_dbt_manifest will now be sorted into groups based on the subdirectory of the project that each model resides in.

  • @asset and @multi_asset are no longer considered experimental.

  • Adds new utility methods load_assets_from_modules, assets_from_current_module, assets_from_package_module, and assets_from_package_name to fetch and return a list of assets from within the specified python modules.

  • Resources and io managers can now be provided directly on assets and source assets.

    from dagster import asset, SourceAsset, resource, io_manager
          
    @resource
    def foo_resource():
        pass
        
    @asset(resource_defs={"foo": foo_resource})
    def the_resource(context):
        foo = context.resources.foo
        
    @io_manager
    def the_manager():
        ...
        
    @asset(io_manager_def=the_manager)
    def the_asset():
        ...
    

    Note that assets provided to a job must not have conflicting resource for the same key. For a given job, all resource definitions must match by reference equality for a given key.

  • A materialize_to_memory method which will load the materializations of a provided list of assets into memory:

    from dagster import asset, materialize_to_memory
          
    @asset
    def the_asset():
        return 5
        
    result = materialize_to_memory([the_asset])
    output = result.output_for_node("the_asset")
    
  • A with_resources method, which allows resources to be added to multiple assets / source assets at once:

    from dagster import asset, with_resources, resource
        
    @asset(required_resource_keys={"foo"})
    def requires_foo(context):
        ...
        
    @asset(required_resource_keys={"foo"})
    def also_requires_foo(context):
        ...
        
    @resource
    def foo_resource():
        ...
        
    requires_foo, also_requires_foo = with_resources(
        [requires_foo, also_requires_foo], 
        {"foo": foo_resource},
    ) 
    
  • You can now include asset definitions directly on repositories. A default_executor_def property has been added to the repository, which will be used on any materializations of assets provided directly to the repository.

    from dagster import asset, repository, multiprocess_executor
    @asset
    def my_asset():
      ...
    @repository(default_executor_def=multiprocess_executor)
    def repo():
        return [my_asset]
    
  • The run_storage, event_log_storage, and schedule_storage configuration sections of the dagster.yaml can now be replaced by a unified storage configuration section. This should avoid duplicate configuration blocks with your dagster.yaml. For example, instead of:

    # dagster.yaml
    run_storage:
    module: dagster_postgres.run_storage
    class: PostgresRunStorage
    config:
        postgres_url: { PG_DB_CONN_STRING }
    event_log_storage:
    module: dagster_postgres.event_log
    class: PostgresEventLogStorage
    config:
        postgres_url: { PG_DB_CONN_STRING }
    schedule_storage:
    module: dagster_postgres.schedule_storage
    class: PostgresScheduleStorage
    config:
        postgres_url: { PG_DB_CONN_STRING }
    

    You can now write:

    storage:
      postgres:
        postgres_url: { PG_DB_CONN_STRING }
    
  • All assets where a group_name is not provided are now part of a group called default.

  • The group_name parameter value for @asset is now restricted to only allow letters, numbers and underscore.

  • You can now set policies to automatically retry Job runs. This is analogous to op-level retries, except at the job level. By default the retries pick up from failure, meaning only failed ops and their dependents are executed.

  • [dagit] The new repository-grouped left navigation is fully launched, and is no longer behind a feature flag.

  • [dagit] The left navigation can now be collapsed even when the viewport window is wide. Previously, the navigation was collapsible only for small viewports, but kept in a fixed, visible state for wide viewports. This visible/collapsed state for wide viewports is now tracked in localStorage, so your preference will persist across sessions.

  • [dagit] Queued runs can now be terminated from the Run page.

  • [dagit] The log filter on a Run page now shows counts for each filter type, and the filters have higher contrast and a switch to indicate when they are on or off.

  • [dagit] The partitions and backfill pages have been redesigned to focus on easily viewing the last run state by partition. These redesigned pages were previously gated behind a feature flag — they are now loaded by default.

  • [dagster-k8s] Overriding labels in the K8sRunLauncher will now apply to both the Kubernetes job and the Kubernetes pod created for each run, instead of just the Kubernetes pod.

Bugfixes
  • [dagster-dbt] In some cases, if Dagster attempted to rematerialize a dbt asset, but dbt failed to start execution, asset materialization events would still be emitted. This has been fixed.
  • [dagit] On the Instance Overview page, the popover showing details of overlapping batches of runs is now scrollable.
  • [dagit] When viewing Instance Overview, reloading a repository via controls in the left navigation could lead to an error that would crash the page due to a bug in client-side cache state. This has been fixed.
  • [dagit] When scrolling through a list of runs, scrolling would sometimes get stuck on certain tags, specifically those with content overflowing the width of the tag. This has been fixed.
  • [dagit] While viewing a job page, the left navigation item corresponding to that job will be highlighted, and the navigation pane will scroll to bring it into view.
  • [dagit] Fixed a bug where the “Scaffold config” button was always enabled.
Community Contributions
  • You can now provide dagster-mlflow configuration parameters as environment variables, thanks @chasleslr!
Documentation
  • Added a guide that helps users who are familiar with ops and graphs understand how and when to use software-defined assets.
  • Updated and reorganized docs to document software-defined assets changes since 0.14.0.
  • The Deploying in Docker example now includes an example of using the docker_executor to run each step of a job in a different Docker container.
  • Descriptions for the top-level fields of Dagit GraphQL queries, mutations, and subscriptions have been added.
All Changes

https://github.com/dagster-io/dagster/compare/0.14.0...0.15.0

See All Contributors
  • fcba344 - add fallback error state for snapshot types (#6671) by @prha

  • 43835eb - Getting Started and Basic Tutorial fixes (#6673) by @jmsanders

  • 5db89f5 - run timeline blank state (#6674) by @prha

  • 7b0b436 - [multiproc] default to spawn (#6676) by @alangenfeld

  • c71062f - Asset Observations Doc (#6630) by @clairelin135

  • d2117a4 - Restore missing repo.py file (#6677) by @gibsondan

  • 85aff95 - [dagster-aws] Fix bug when logging records from pyspark_step_launcher (#6680) by @OwenKephart

  • 360f7b2 - Automation: versioned docs for 0.14.0 by @elementl-devtools

  • a4fd446 - Changelog (#6667) by @clairelin135

  • 67b0426 - Fix formatting on schedules feature in changelog (#6686) by @gibsondan

  • fb38c93 - Pin grpcio-health-checking (#6685) by @johannkm

  • eb65568 - fix changelog page in docs (#6694) by @yuhan

  • 1f79613 - redirect ecs page (#6695) by @yuhan

  • 93b0bf9 - Remove crons from our example dockerfiles (#6645) by @gibsondan

  • 787c7ca - pin markupsafe (#6713) by @alangenfeld

  • 95333ec - add cursor label to tick timeline (#6712) by @prha

  • 3583858 - whitelist_for_serdes WorkerStatus enum (#6716) by @johannkm

  • e81374a - docstrings for AssetGroup and AssetGroup.build_job (#6696) by @dpeng817

  • b65900c - 0.14.1 changelog (#6720) by @alangenfeld

  • fe61ba5 - Remove bad invariant from back when sensor status was a fixed status, not a default status (#6711) by @gibsondan

  • be11e4a - Automation: versioned docs for 0.14.1 by @elementl-devtools

  • 9eba0d0 - Add a check in run execution that the run actually exists, just like in step execution (#6689) by @gibsondan

  • 19b106c - add back mutation tests for sensor/schedules with default status (#6721) by @prha

  • 47e8533 - Export build_reconstructable_target to match cragified docs (#6317) by @prha

  • 46c3e0b - add build_run_status_sensor_context (#6543) by @jamiedemaria

  • f55ef00 - Rename metadata (remove "Event") in graphql/dagit (#6656) by @smackesey

  • 8cba839 - Speed up get_assets_for_run_id when there are a lot of events (#6735) by @gibsondan

  • ef56abe - undefined (#6742) by @alangenfeld

  • fe88147 - Fix invariant check in databricks job/pipeline path (#6718) by @iswariyam

  • cded56d - make Content Gap an issue label instead of part of title (#6745) by @sryza

  • 229578c - 0.14.0 migration.md (#6715) by @clairelin135

  • 091702a - Move bollinger example to AssetGroup and add README (#6749) by @sryza

  • d7d1351 - move software-defined assets toy to asset collection (#6643) by @sryza

  • 640aa5d - rename conditional branching example to nyt and add README (#6757) by @sryza

  • b2b2672 - [docs infra] remove old versions from versioned_navigation (#6704) by @yuhan

  • 0c55d96 - fix typo in ops rst (#6758) by @yuhan

  • e83b11c - update error messages for ExternalAssetDependency/ExternalAssetDependedBy (#6748) by @OwenKephart

  • 38b8283 - [Docs] Upgrade Next.js to 12.1.0 (#6722) by @keyz

  • e602e45 - [docs infra] fix changelog's On This Page (#6706) by @yuhan

  • da26522 - fix(helm): remove registry field (#6744) by @rexledesma

  • 2b6ded6 - docs(helm): update chart metadata (#6778) by @rexledesma

  • e53dec4 - respect default_value in loadable input check (#6771) by @alangenfeld

  • f470ea9 - display non-sda assets in asset catalog (#6779) by @prha

  • b26ba51 - replace EventMetadata in docs (#6774) by @smackesey

  • a360bbd - Migrate user event docs to use log_event, various docs cleanups (#6772) by @dpeng817

  • 5cec1c8 - [dagster-dbt] fix issues with dbt asset integration (#6777) by @OwenKephart

  • 0ed9ad7 - Filter custom events from stats_for_run query (#6781) by @johannkm

  • 90faed0 - [Dagit] Fix missing op name in the asset sidebar (#6765) by @bengotow

  • 83643b2 - add validate_run_config to api docs (#6782) by @prha

  • 472c4dc - add api doc for op_def, graph_def (#6776) by @prha

  • 64095bd - [examples] dbt -> dbt-core in dbt_example (#6786) by @OwenKephart

  • 5f3966f - [dagit] Show full asset paths on details pages, tweak filter interactions (#6764) by @bengotow

  • 7495fea - Add small toggles to Asset Details graphs, consolidate localStorage-backed state (#6762) by @bengotow

  • f700471 - [dagster-shell] Fix deadlock with large output (#6435) (#6490) by @kbd

  • e574200 - [dagster-dbt] pin some stuff (#6791) by @OwenKephart

  • 50eda7a - deflake test_retry_policy (#6644) by @alangenfeld

  • f17a1fa - 0.14.2 changelog (#6789) by @sryza

  • d90d1dd - add run_request_for_partition for job definition (#6790) by @prha

  • 4d89075 - Add a "log framework error" hook to execute_step invocations, similar to the one in execute_run (#6690) by @gibsondan

  • 82c80c3 - fix up reexecution api example docs (#6787) by @prha

  • 7b3dde3 - reorganize dagster.check module (#6750) by @smackesey

  • 1a3f5c1 - Adjustments/condensing of dagster.check (#6751) by @smackesey

  • b4a1715 - Automation: versioned docs for 0.14.2 by @elementl-devtools

  • 308429b - fix: use run status to determine success for an in-process result (#6784) by @rexledesma

  • 47b9a68 - namedtuple to NamedTuple (1st batch) (#6752) by @smackesey

  • 44f4f2a - corrected export message from "~/dagster_home" to ~"/dagster_home" (#6327) by @Proteusiq

  • a1acd3f - docs: remove duplicate entries under integrations (#6724) by @kahnwong

  • 032f442 - namedtuple to NamedTuple (2nd batch) (#6753) by @smackesey

  • e0c66db - namedtuple to NamedTuple (3rd batch) (#6767) by @smackesey

  • 6327300 - namedtuple to NamedTuple (4th batch) (#6785) by @smackesey

  • 1f5a7ec - [docs infra] speed up vercel build with ISR (#6773) by @yuhan

  • ce7441c - [docs infra] no s3 when pulling versioned content (#6705) by @yuhan

  • ee07c9c - Still write cursors even when a sensor tick fails (#6807) by @gibsondan

  • ef3ad62 - Direct invocation of assets for unit testing (#6761) by @jamiedemaria

  • 6098bf0 - Partition docs updates (#6814) by @jamiedemaria

  • 779e27f - fix pandera link on integrations page (#6813) by @smackesey

  • 6ac6349 - [docs] regenerate yarn.lock (#6815) by @alangenfeld

  • ea19544 - [events] remove message field (#6769) by @alangenfeld

  • c157260 - [dagster-postgres] improve connection retry (#6831) by @alangenfeld

  • ca94d8d - set ErrorSource when retry request limit exceeded (#6806) by @alangenfeld

  • 4dbfa80 - Update black, isort (#6714) by @smackesey

  • 9dd063a - Exclude materializing run from materialization warning run count (#6692) by @clairelin135

  • 63556df - rename PipelineRunsFilter => RunsFilter (#6811) by @prha

  • c1a6018 - assorted function type annotations (batch 1) (#6800) by @smackesey

  • ab08410 - restore old metadata name deprecation warnings (#6688) by @smackesey

  • a3ecc86 - Also override cpu/memory in containerOverrides (#6836) by @jmsanders

  • f330e26 - move toys from build_assets_job to AssetGroup (#6801) by @sryza

  • 3408263 - Simplify IWorkspace API (#6821) by @gibsondan

  • 0158c07 - [dagster-aws] Add secretsmanager resources (#6802) by @benpankow

  • d98aa5c - chore(daemon): retrieve daemon heartbeats in batch (#6843) by @rexledesma

  • 5b10719 - Revert isort Makefile config to use git ls-files (#6845) by @smackesey

  • 6092f3e - fix: query heartbeats db once (#6854) by @rexledesma

  • 42d2d72 - [examples] Modern Data Stack + SDA Example (#6862) by @OwenKephart

  • f0323af - add table api doc page (#6840) by @smackesey

  • de1b5ef - [dagit] Avoid copying definition-time tags into launchpad state (#6851) by @bengotow

  • 7269df9 - [dagit] Replace black tag tooltips with as-needed, selectable ones (#6844) by @bengotow

  • 4d4d6a7 - default to fs_asset_io_manager in AssetGroup (#6859) by @sryza

  • e9834b4 - move slack apidoc example to op/job (#6855) by @sryza

  • 0ac9e1c - Remove @abstractproperty (#6867) by @johannkm

  • fe4a530 - gate bucketing queries on sqlalchemy version (#6841) by @prha

  • 8f29bf9 - Don't carry forward all container definitions (#6850) by @jmsanders

  • 8b7b27d - when multiple repos have the same asset, have graphql return the non-source one (#6856) by @sryza

  • 3067d32 - [dagit] Add a helper for building instance/runs paths, resolved #6847 (#6848) by @bengotow

  • e6fb91b - [dagster-fivetranAdd Fivetran Resync Op (#6868) by @dwallace0723

  • 9b08239 - [dagit] Use a custom hook to manage query polling (#6805) by @bengotow

  • cbc48e8 - enable cross-job asset partitions (#6865) by @sryza

  • c808d3b - [dagit] Add a gradient behind the Gantt chart step selector now by @souterjk

  • e33302a - remove gitignore setting from isort (#6872) by @smackesey

  • 7601eab - Random formatting fixes (#6888) by @dpeng817

  • 8467f5c - AssetGroup.from_modules (#6884) by @sryza

  • 6299284 - isort dagster-aws conftest (#6892) by @sryza

  • c14ac93 - fix expansion so its shell-neutral (#6894) by @smackesey

  • 6b57eb6 - build_assets_job -> AssetGroup in SDA guide (#6879) by @sryza

  • 4b27d8b - Changelog (#6908) by @dpeng817

  • c577ec5 - add entries to 0.14.3 changelog (#6909) by @sryza

  • d894501 - Rename build_run_status_sensor_context in changelog (#6904) by @gibsondan

  • e4fabe9 - make EventLogEntry.message change safe to downgrade (#6912) by @alangenfeld

  • f8319ea - For those whome detest wordwrap in vscode (#6887) by @dpeng817

  • e3e2143 - Solid error message migration (#6883) by @dpeng817

  • 919fb31 - Error when amultiple asset groups are used in one repo, when a job is passed in with the asset group reserved name (#6891) by @dpeng817

  • 498b261 - assorted sensor and schedule fixes (#6759) by @jamiedemaria

  • 7f35164 - [examples] For modern_data_stack_assets, more detailed README, helper script (#6914) by @OwenKephart

  • 28fe827 - fix bug with subset selection (#6921) by @OwenKephart

  • 7fa5308 - enable vixie-style cron strings (#6810) by @prha

  • 66bdcff - Automation: versioned docs for 0.14.3 by @elementl-devtools

  • 66f5ce6 - In run status sensors, snapshot the status in a PipelineRunReaction rather than pulling it from the PipelineRun object (#6931) by @gibsondan

  • 7ea0c61 - [dagster-airbyte][examples] fix bug with empty prefix (#6932) by @OwenKephart

  • c52b495 - RFC: resolve k8s job and ops tag entirely using k8s_model_from_dict (#6205) by @gibsondan

  • 096a17d - revert stray dagster[lint] for BK mypy (#6936) by @smackesey

  • bfcc59b - Migrate reference to solid in error message (#6935) by @dpeng817

  • 8b5a4ca - dagit-debug fix in-mem daemon heartbeats (#6943) by @alangenfeld

  • 9168e73 - [docs] details on mapping dynamic outputs (#6140) by @alangenfeld

  • 66ae1c0 - [docs] op retries concepts section (#6818) by @alangenfeld

  • 4145d10 - fix makefile issue with check_black/check_isort (#6944) by @smackesey

  • cf3f6d2 - change query to be compatible with sqlalchemy 1.3 (#6930) by @prha

  • c4fe9a6 - add a blank state message for the asset table (#6792) by @prha

  • 81fdc76 - fix black by @alangenfeld

  • ca7aedf - Remove update_doc_snapshot from root Makefile (#6837) by @smackesey

  • d1d6003 - namedtuple to NamedTuple (5th batch) (#6917) by @smackesey

  • 3c2a0b5 - function type annotations (batch 2) (#6933) by @smackesey

  • cddc63c - when building AssetGroups from modules, gather lists of assets (#6967) by @sryza

  • 5bfadb9 - remove legacy version of hacker news schedule (#6797) by @sryza

  • c193ccd - ungate run batching (#6966) by @prha

  • 803063e - Use AssetGroup in hacker news demo (#6965) by @sryza

  • 4262be1 - AssetGroup.from_current_module (#6959) by @sryza

  • 3f5aeb7 - [Dagit] Nits and polish based on user feedback (#6799) by @bengotow

  • 9dc310c - add support for filtering ticks by status (#6919) by @prha

  • 2910338 - [dagit] Only warn about the backfill daemon when backfills are present (#6846) by @bengotow

  • ed210ae - switch arg name for metadata value class (#6675) by @smackesey

  • 3df7e14 - [dagit] Use a virtualized list in <SuggestWIP /> to fix launchpad partition picker perf (#6852) by @bengotow

  • 63f6048 - Fix missing partition bars in large partition data-sets (#6877) by @souterjk

  • 07f2386 - Quote selection to avoid grabbing downstreams with the same name prefix when viewing upstream graph, fix react table unhappy log (#6907) by @souterjk

  • 37be307 - make all optional params explicit (#6934) by @smackesey

  • 2649808 - Adds pez popover as requested in #6470 (#6871) by @souterjk

  • e703ac6 - fix pylint for modern data stack example (#6975) by @smackesey

  • 73ec3a2 - dev_install tweaks (#6970) by @smackesey

  • 9e1fa57 - partitioned assets toy (#6960) by @sryza

  • a242d6b - Update create-new-project.mdx (#6898) by @frcode

  • e0dc13b - Add short-circuit in run start time if run failed to start (#6991) by @gibsondan

  • 02679a1 - Allow PathMetadataValue to accept os.PathLike input (#6920) by @abkfenris

  • 68b2746 - Raise run_task errors (#6993) by @jmsanders

  • fae347c - [dagit] React.FunctionComponent -> React.FC (#7002) by @hellendag

  • b86b0f1 - Fix partitioned asset jobs with double downstream non-partitioned (#6986) by @sryza

  • 7542130 - add ipython_genutils as dagit, dagstermill dep (#7016) by @smackesey

  • 64033df - Set up graphql mutation for setting a sensor cursor value (#6922) by @prha

  • 02524aa - [dagit] Fix warning icon tests, add one for present-but-stopped schedule (#7026) by @bengotow

  • 06f6fa8 - Stop building duplicate downstream asset dependencies (#7001) (#7012) by @bengotow

  • 5d5d2b0 - [dagit] Add error handling to Asset "Rematerialize Partitions" modal (#7011) by @bengotow

  • 07ce468 - [dagit] Fix tooltips in Gantt chart, use monospace font and middle truncation (#7025) by @bengotow

  • 16b7b4d - [dagit] Fix right Asset Graph panel missing if saved "explorer" state is absent (#7013) by @bengotow

  • 1dc6cf5 - Fix dagster k8s typo (#7028) by @johannkm

  • 1cb6d56 - Add sequence and mapping check functions (#7006) by @smackesey

  • b0a0dc2 - change backcompat pin to nbconvert (#7027) by @smackesey

  • 57f44c0 - [dagit] Hide the asset group job on the instance overview (#7019) by @bengotow

  • 3d22aef - [dagit] When materializing assets, link to run instead of showing it (#7014) by @bengotow

  • feb1c5e - Assorted typing additions and adjustments (#7007) by @smackesey

  • ce4b5ae - remove unneeded deps from dagit tox file (#7022) by @gibsondan

  • daa1188 - Switch sensor/schedule view to show tick list instead of run list (#6861) by @prha

  • 336ee6b - update AssetGraph dedup logic (#7021) by @smackesey

  • b2df1b0 - Make parse_run_config_input more defensive (#7003) by @gibsondan

  • 809665e - Expose metadata on AssetNode (and source assets) directly (#6900) by @smackesey

  • 94c8e7f - adds source asset metadata to dagit (#7015) by @smackesey

  • 1244427 - Adllow telemetry URL to be overridden with env var (#6946) by @dpeng817

  • 79827ef - Remove space from dbt example notebook path (#7038) by @smackesey

  • 52b1527 - add tick batching to schedule storage (#6995) by @prha

  • 6e7cd25 - add batch tick loader for repository-scoped schedule/sensor ticks (#6997) by @prha

  • b60c070 - K8s executor filter down to dagster events (#7000) by @johannkm

  • 8e6bccd - Allow the use of environment variables in k8s_job_executor (#7031) by @johannkm

  • c7bfb9c - [bug] hacky fix for step launcher behavior (#6866) by @OwenKephart

  • 1327a6d - make resource config available when loading from assets outside job (#7029) by @sryza

  • a88e658 - Rename step jobs 'dagster-step' in k8s and docker (#6982) by @johannkm

  • 756527a - minor improvements to check (#7036) by @smackesey

  • 28d74c3 - test raising exceptions in AssetGroup in deterministic order (#6945) by @jamiedemaria

  • 07c3887 - SDA docs fixes (#6905) by @jamiedemaria

  • d9b79ec - skip test_giant_external_repository_streaming_grpc (#7049) by @alangenfeld

  • 43cf866 - Fault tolerance docs (#5432) by @johannkm

  • 396da6c - Silence experimental warnings triggered by internal code (#6873) by @jamiedemaria

  • 8ad1195 - K8s docs ignore tag and show executor (#7053) by @johannkm

  • aa5712e - Assorted fixes to the hackernews assets demo (#7030) by @sryza

  • b2fe162 - Support the same Secrets config syntax as ECS (#6985) by @jmsanders

  • 8045d57 - fix the sqlite gating logic (#7058) by @prha

  • 8d0319a - [dagster-airbyte] [dagster-fivetran] cleanup + tests (#7056) by @OwenKephart

  • 07e3403 - Reuse ECS task definitions (#7009) by @jmsanders

  • 6bbfce2 - Update last_materialization_timestamp to also update on observation (#6885) by @clairelin135

  • d906c85 - Optimize Latest Runs Query (#7066) by @clairelin135

  • 8a611ae - [dagit] Remove default InstanceWarning mocks injecting randomness (#7054) by @bengotow

  • 7a78835 - fix tag batching with tag-based filters (#7055) by @prha

  • 75b48ea - Fix #6512 (#6969) by @petehunt

  • ee146ef - K8s check run health test (#7070) by @johannkm

  • 0f61c12 - hn local repo that uses duckdb as warehouse (#6968) by @sryza

  • b3e776b - fix bug in fetch_assets.get_asset_nodes_by_asset_key (#7063) by @sryza

  • 9941cbc - Clear out 'dependsOn' when creating a default ECS task definition (#6929) by @gibsondan

  • f6baac7 - [dagit] Render status page navigation when factory floor loading (#7079) by @benpankow

  • adcf93f - run mypy on dagster-graphql (#7075) by @prha

  • 2e2029d - fix logic for turning schedule ticks into hour boundaries when there's an offset (#7071) by @gibsondan

  • 3df4da6 - Fixes #7082 (#7086) by @smackesey

  • 61c43eb - Fix failing check.path test on windows (#7077) by @smackesey

  • 15b5ad9 - [dagit] refactor graphql execute methods (#7065) by @alangenfeld

  • 47f8fbc - dagster-snowflake drop 3.6 support (#7092) by @alangenfeld

  • 862fe7e - [dagit] Fix Gantt chart layout hang on certain graph structures #6942 (#7085) by @bengotow

  • 69f5e3b - Log failed compute log to event log rather than buried in a logging output (#7093) by @gibsondan

  • e282d44 - [dagit] Disallow use of gql from graphql.macro (#7042) by @bengotow

  • 72807a1 - [dagit] Add sensors/schedules to the asset details page (#7041) by @bengotow

  • 7bcdd05 - [dagit] Keep user-entered tags when changing presets / partitions (#7068) by @bengotow

  • 17313fe - Add gitpod.yml (#7088) by @petehunt

  • f32802e - [dagit] Lower factory floor timeline fidelity to seconds from ms (#7091) by @benpankow

  • d07a108 - [dagit] Show job failures on asset graph, fix in-progress runs on global graph (#7067) by @bengotow

  • 5765c38 - Optionally use sidecars (#7087) by @jmsanders

  • aa1dc4a - Fix kwarg case with invocation (#7095) by @dpeng817

  • 3f6458c - Return an empty secrets list for task definitions (#7096) by @jmsanders

  • eff26c4 - Fixup ECS secrets docs (#7097) by @jmsanders

  • 8c3cfd2 - 0.14.4 changelog (#7099) by @yuhan

  • ffd8318 - refetch the backfill before updating, to avoid clobbering cancels (#7094) by @prha

  • 0c72a16 - Automation: versioned docs for 0.14.4 by @elementl-devtools

  • 1c7f030 - fix perms in dagit (#7115) by @prha

  • 24290ad - Remove hard failure if missing inputs (#7116) by @dpeng817

  • 5c2d0bf - Fix internal code metadata warnings (#7112) by @smackesey

  • 40fdc1e - 0.14.5 Changelog (#7119) by @jmsanders

  • e1209fa - Automation: versioned docs for 0.14.5 by @elementl-devtools

  • b0a885f - batch run fetches in sensor daemon (#6950) by @prha

  • 8063405 - gcs asset io manager (#7081) by @jamiedemaria

  • 62883d7 - Don't give up if the terminate() call raises an exception during force-termination (#7110) by @gibsondan

  • 8b3ad9f - [dagit] Add optional close button to Alert component (#7107) by @benpankow

  • 5efccff - Add dagit test to make sure Permission flags stay in sync (#7117) by @prha

  • 8aaa2e6 - expose compute_kind tag value on gql AssetNode (#7120) by @smackesey

  • dfeff6c - replace internal uses of MetadataEntry static api (#7126) by @smackesey

  • 4812f93 - dynamic out bug fix (#6780) by @alangenfeld

  • 821d3e0 - Create oss-issues.yml (#7132) by @yuhan

  • cd90176 - fix docs search: js_render shouldnt be true (#7146) by @yuhan

  • 3509c98 - mypy version bump (#7149) by @smackesey

  • 98e99c8 - docs(sda): fix typos and edit wording (#7136) by @rexledesma

  • 3ee5790 - add permissions guard for schedule mutations (#7133) by @prha

  • b17183e - enable saving run key for ticks that skip runs because of run key idempotence (#7130) by @prha

  • 291fbf5 - add tick tooltip info for skipped ticks due to run key idempotence (#7131) by @prha

  • bfb8978 - Rename the InstanceMigratedError to be less pushy about migrating (#7141) by @gibsondan

  • f426230 - [postgres] share db engine with watcher_thread (#7143) by @alangenfeld

  • 01294b4 - Add Map to config type API docs (#7154) by @gibsondan

  • 6f10ab0 - Pass on PythonError handling that should be using PythonErrorFragment (#7152) by @gibsondan

  • 2cffa49 - [oss telemetry] Get rid of telemetry version (#6760) by @dpeng817

  • 35f5f7e - only show warning when force termination checkbox is checked (#7157) by @prha

  • ff8d594 - rename several internal submodules (#7134) by @smackesey

  • dc994c9 - Remove another cron from example dockerfile (#7155) by @gibsondan

  • 0f75e1a - Be more willing to return serialized errors in schedule/sensor/grpc partition methods (#7164) by @gibsondan

  • 4f91896 - docs: fix spelling (#7166) by @antquinonez

  • b986894 - [multiprocess] tweak preload modules logic (#7142) by @alangenfeld

  • 0c7f490 - feat(dagster-k8s): add run id label to run/step workers (#7167) by @rexledesma

  • bcc1fd7 - docs(logging): remove example about capturing from the root logger (#7168) by @rexledesma

  • 070610d - docs: fix typo in job/graph concepts page (#7172) by @antquinonez

  • a3b44e7 - [docs] fix dynamic graph examples (#7174) by @alangenfeld

  • e8f1745 - Fix inclusion of tests in our published packages (#7169) by @smackesey

  • 0fc9ba4 - RFC: add partition tags to partitioned config (#7111) by @prha

  • 2bb2da5 - add gcs pickle asset manager to api docs (#7177) by @jamiedemaria

  • 1fff21f - [dagster-databricks] better retry support for step launchers (#7171) by @OwenKephart

  • c1d674d - fix lint/mypy failures (#7183) by @prha

  • 0c1fea5 - Automation: versioned docs for 0.14.6 by @elementl-devtools

  • a892e0d - 0.14.6 changelog (#7184) by @jamiedemaria

  • efa9363 - update docs on Dict type (#7179) by @jamiedemaria

  • 7c69b8e - docs: use word "op" instead of "solid" (#7195) by @joe-hdai

  • e642465 - refactor: rename is_failure to is_failure_or_canceled (#7200) by @rexledesma

  • 8378c0c - Time window partitions with custom minute, hour, day offsets (#7125) by @jamiedemaria

  • d0188e4 - Fixup ECS docs (#7193) by @jmsanders

  • fd7189a - Update version placeholder "dev" to "0+dev" (#7175) by @smackesey

  • 7abf706 - fix: prevent runs filter from accepting an empty list of run ids (#7217) by @rexledesma

  • 3800687 - refactor: remove screenshots (#7225) by @rexledesma

  • a7989b7 - refactor: remove scala modules (#7226) by @rexledesma

  • 9444b2f - get MetadataValue.dagster_run to show up in apidoc (#7231) by @sryza

  • 1fc1bbc - Azure asset IO Manager (#7201) by @jamiedemaria

  • ff9596f - PartitionedConfig testing helpers (#7216) by @jamiedemaria

  • e66faf9 - [postgres] dont dispose engine in watch thread (#7241) by @alangenfeld

  • 404a7c1 - feat(helm): add configuration for Dagster telemetry (#7190) by @rexledesma

  • 847889d - add some type annotations for code loading (#7244) by @sryza

  • 7024d7c - feat: add alert failure event (#7238) by @rexledesma

  • 81360ce - [dagit] Remove the Asset Node context menu (#7151) by @bengotow

  • 6550409 - Only consider repos defining the asset with a corresponding op for multiple repo notice (#7186) by @bengotow

  • 6ecb496 - [dagit] Fix Gantt chart layout hang on certain graphs, round 2 #6942 (#7188) by @bengotow

  • 8f271ef - [dagit] If re-executing from failure fails, default to from failure again #5019 (#7189) by @bengotow

  • f064389 - [dagit] Ignore source assets for “Materialize All” button (#7196) by @bengotow

  • 4588b22 - [dagit] Load “Launch Asset partitions” modal when it’s opened + every time (#7207) by @bengotow

  • b9284ce - [dagit] Explicitly add the dagster/step_selection tag to asset runs (#7213) by @bengotow

  • dd6f57e - Remove redundant k8s launcher event (#7250) by @johannkm

  • 0831538 - Allow asset observations in IO Managers (#6653) by @clairelin135

  • 023adc4 - update docs for adls2 asset io manager (#7252) by @jamiedemaria

  • 42b65dd - 0.14.7 changelog (#7256) by @OwenKephart

  • ca369c4 - Update CHANGES.md by @prha

  • 14204d5 - Automation: versioned docs for 0.14.7 by @elementl-devtools

  • 99d40dd - fix typecheck for In.dagster_type (#7242) by @smackesey

  • b61db9d - [dagit] RunTimeline: Fix overlap with ongoing run (#7245) by @hellendag

  • c997aa3 - [dagster-io/ui] Remove WIP suffixes (#7229) by @hellendag

  • 5c7ec2e - [dagit] Modify "Open in Launchpad" from Run (#7263) by @hellendag

  • 60059f9 - RFC: Add register_run_asset event (#7098) by @clairelin135

  • 8503e6a - Add table to keep track of schedule data migrations 1/5 (#7182) by @prha

  • ae93b45 - add schema for instigators table, keyed by selector 2/5 (#7185) by @prha

  • 6262a51 - double-write selector_id to ticks, jobs, instigators 3/5 (#7191) by @prha

  • 8b8ce0d - add tick selector index migration (#7198) by @prha

  • 2e6eabd - Read instigator state / ticks off of selector_id instead of origin id 5/5 (#7268) by @prha

  • 4115505 - [dagster-io/ui] Declaration files for TS, Rollup for built JS (#7220) by @hellendag

  • 96d2a25 - Ensure telemetry logging dir is created before using logger (#7192) by @dpeng817

  • ec6402d - cross-repo assets toy (#7210) by @sryza

  • da2205e - update apidoc for non_argument_deps (#7271) by @clairelin135

  • 79235ab - enable loaded_input metadata entries to display in Dagit (#7255) by @clairelin135

  • 38cbd24 - [@dagster-io/ui] Cleanup for publish (#7270) by @hellendag

  • de5f244 - Remove empty k8s_job_task file (#7286) by @johannkm

  • 298d8b1 - Add MenuLink shim for internal compatibility (#7297) by @hellendag

  • ce69cc1 - Avoid mentioning pipelines in Dagit empty state (#7288) by @sryza

  • 052bf22 - fix invariant in batch tick loader (#7280) by @prha

  • e965190 - [dagster-airbyte] fix airbyte materializations without streamStats (#7281) by @OwenKephart

  • 34ed951 - point top-level workspace to latest toys (#6876) by @sryza

  • 36b189f - [dagster-io/ui] Eliminate RR dependency (#7285) by @hellendag

  • 21c8a12 - fix selector injection for k8s-celery test (#7307) by @prha

  • ef95e46 - remove experimental warning for mysql storage (#7290) by @prha

  • 4996767 - make some edits to changelog (#7264) by @prha

  • 0692f86 - [@dagster-io/ui] Clean up deps (#7302) by @hellendag

  • f7f2b0f - [dagit] Fix config refresh when reloading repo (#7325) by @hellendag

  • 84550f6 - [@dagster-io/ui] Prepare for 1.0.3 (#7326) by @hellendag

  • 504c65f - Add run_id as execute_in_process arg (#7317) by @dpeng817

  • 8dca877 - Fixes to root input manager memoization and no context provided (#7316) by @dpeng817

  • 2e900d5 - add apple m1 instructions to contributor guide (#7279) by @jamiedemaria

  • 1860b3a - Fix celery docker executor failing on None metadata (#7330) by @jamiedemaria

  • e073b6f - replace old celery docs site (#7331) by @jamiedemaria

  • 48f07e9 - [dagster-dbt] fix issue w/ conflicting dbt op names for assets (#7311) by @OwenKephart

  • e80494f - docs infra: remove github/slack bot from docs site (#7315) by @yuhan

  • 98c41b4 - [dagit] Fix CodeMirror show-hint error (#7334) by @hellendag

  • 7551d4f - 0.14.8 Release Notes (#7336) by @clairelin135

  • 8c85ad3 - add email icon (#7346) by @salazarm

  • 4ebba64 - [dagit] Don't show instigation switch for assets (#7347) by @hellendag

  • 4f2b613 - [dagit] Add "Copy config" button to "View tags and config" dialog (#7348) by @hellendag

  • f1441de - [dagit] Allow toggling repos in bulk (#7344) by @hellendag

  • 8e7d7d6 - [dagit] Store whitespace state in localStorage (#7350) by @hellendag

  • a7bf12e - [dagit] Adjust details in Run header (#7345) by @hellendag

  • 3039616 - [dagit] Add feature flag to collapse the left Jobs navigation by default (#7204) by @bengotow

  • 793aeda - Automation: versioned docs for 0.14.8 by @elementl-devtools

  • 4cb70cc - update README - include github discussion (#7339) by @yuhan

  • a515c58 - [@dagster-io/eslint-config] Create shared config (#7328) by @hellendag

  • ff2f632 - [@dagster-io/eslint-config] Fix dependencies (#7354) by @hellendag

  • d344eed - Add Slack icon (#7355) by @salazarm

  • ca8a616 - support container kwargs for celery-docker (#7335) by @jamiedemaria

  • c444420 - [dagster-dbt] Add /runs method to DbtCloudResourceV2 (#7258) by @kstennettlull

  • e0dc0a4 - [dagster-dbt] add "build" to DBT_RUN_RESULTS_COMMANDS (#7362) by @OwenKephart

  • 4a0adb3 - [dagit] Show timestamps for “failed to start” runs, other metadata tweaks (#7359) by @bengotow

  • 60840ac - Fix contention bug in windows (#7353) by @dpeng817

  • bdd4dd7 - fix dagster-ge library (#7369) by @prha

  • 3d82da9 - have fs io manager record path metadata entry (#7261) by @sryza

  • bdb1192 - AssetGroup.materialize_in_process (#7260) by @sryza

  • 7ecd4b3 - Add authenticator field to conn_args (#6983) by @swotai

  • d52463c - Make shell ops inherit env vars from outside environment (#6513) by @kbd

  • 4e9ee2e - Allow jobs to be passed in lazily to repository definitions (#7382) by @gibsondan

  • fac5a5f - [dagit] Make the “Latest Run” tag on the Job page auto-refresh (#7360) by @bengotow

  • 6d5eae5 - [dagit] Move data fetching for the asset graph into hooks (#7363) by @bengotow

  • 7472843 - autodiscover assets at module scope (#7247) by @sryza

  • 00bde2b - [hacker news] pin duckdb (#7390) by @alangenfeld

  • 38625ae - Let InProcessRepositoryLocationOrigin take in multiple repos (#7386) by @gibsondan

  • 7acd90e - [dagster-fivetran] Fix format strings (#6436) by @mgasner

  • 96a6762 - [dagster-airbyte] Handle 204 reponse in Airbyte Library (#7209) by @HAMZA310

  • 4f77c97 - run materializations (#7394) by @prha

  • 72e5b85 - [dagit] Fix default Dagit page in single repo, multiple jobs case (#7368) by @bengotow

  • 104e780 - Add grpc server timeout config to instance settings (#7387) by @gibsondan

  • 0ba3b4e - infer resource descriptions from docstrings (#7309) by @sryza

  • 3f5011b - [dagit] Async layout of large asset graphs (#7366) by @bengotow

  • bdb8126 - [dagit] Align terminology and data structures in asset / op graph layout (#7372) by @bengotow

  • 925ddbc - [dagit] Move asset-graph out of workspace dir, non-graph code to /assets (#7373) by @bengotow

  • f23a00b - Update AssetKeyTable.last_run_id upon materialization planned (#7319) by @clairelin135

  • 1600323 - asset tutorial (#7269) by @sryza

  • 9f1d04e - Add a field to the now-somewhat-poorly-named RepositoryPythonOrigin that can add 'deployment target' information within a repo location (#6925) by @gibsondan

  • 9f8e68f - Add EcsContainerContext (just with secrets for now) (#7176) by @gibsondan

  • 1a3a76e - Add K8sContainerContext for configuring k8s pods from the repository location (#7203) by @gibsondan

  • 71661a7 - Fixed typo in deploying-with-helm.mdx (#7416) by @LeoHuckvale

  • ec89e79 - [dagit] Use react-scripts@5.0.2 (#7393) by @hellendag

  • 94f7984 - [dagit] Add arrow key and double click interactions to asset graph #6407 (#7374) by @bengotow

  • 8120421 - [dagit] Allow colon in run tag value (#7409) by @hellendag

  • e2de0b7 - Re-execute run instance method (#7417) by @johannkm

  • afe0492 - skip adls test (#7420) by @jamiedemaria

  • 95aea0a - add open launch pad to run details page (#7304) by @salazarm

  • 0e46052 - Fix prerequisite typos in docs (#7424) by @johannkm

  • 9f9db4e - [@dagster-io/eslint-config] Enable object-shorthand (#7423) by @hellendag

  • 6d1cf77 - Filter out asset_materialization_planned event logs client-side (#7397) by @clairelin135

  • 8e75a98 - Make user repo filters per deployment (#7376) by @salazarm

  • cb642cb - [postgres] urlquote usernames as well (#7430) by @alangenfeld

  • e615623 - [toys] squelch warnings (#7419) by @alangenfeld

  • f38e1c3 - docs for gRPC sever timeout flag (#7400) by @gibsondan

  • 7a9e8c9 - Fix master and add DagsterEventType import (#7435) by @clairelin135

  • 6cf75eb - avoid extra experimental warning when asset has partitions def (#7431) by @sryza

  • 90ac1b5 - Add option to disable max_concurrent_runs (#7327) by @johannkm

  • a5c2854 - RFC: Fetch Cross-Repo Asset Deps (#7259) by @clairelin135

  • 7aef716 - make k8s container context volumes and volume mounts use snake case, not camel case (#7437) by @gibsondan

  • b09c8e1 - move expensive tests out of core_tests (#7436) by @gibsondan

  • 3b55c4e - allow AssetGroups to have assets with different partitions defs (#7388) by @sryza

  • be2469b - Update key for saving data in useExecutionSessionStorage (#7333) by @salazarm

  • bb27b2d - asset partitioned io manager (#7413) by @sryza

  • 0bc57e9 - Unbreak stopping unloadable schedules/sensors (#7446) by @gibsondan

  • e0b991b - [dagit] Check for asset group job prefix to support multiple groups per repo (#7418) by @bengotow

  • 5a27fcd - changelog for 0.14.9 (#7450) by @sryza

  • 7ad594b - update isort comment formatting (#7448) by @smackesey

  • a0a321d - bump black to 22.3.0 (#7449) by @smackesey

  • 1bdb802 - Automation: versioned docs for 0.14.9 by @elementl-devtools

  • 12b87c3 - Typo in the create a new project doc (#7441) by @ascrookes

  • 2e8136d - Fix issue with blank telemetry instance id file (#7433) by @dpeng817

  • 1ec9318 - Show repo selector in unloadable schedules/sensors instead of repo location metadata (#7444) by @gibsondan

  • a0cc868 - Split up screenshot specs (#7460) by @sryza

  • 9ad8f54 - resource dependencies for root input managers (#7459) by @sryza

  • 920c759 - avoid framework code triggering metadata_entries warning (#7412) by @sryza

  • a744f84 - allow combining memoization with step selection (#6431) by @sryza

  • 53b4f09 - refactor: remove local storage file that was committed (#7440) by @rexledesma

  • 74dafde - [dagster-graphql] remove gevent (#7402) by @alangenfeld

  • 2a9b06f - [dagit] Apply middle truncation to long asset nodes, hide run info (#7187) by @bengotow

  • 7f8f9d8 - [dagit] Use CodeMirror for all config displays (#7467) by @hellendag

  • f22619d - Add container_context to gRPC server and Helm chart (#7236) by @gibsondan

  • d1a1e44 - Add asset metadata to one of the dagster_test asset groups (#7462) by @bengotow

  • 2c54844 - Fix back-compat for new helm chart param for container context (#7476) by @gibsondan

  • 39c853a - [graph] use the correct input definition for type checks (#7453) by @alangenfeld

  • b94eeb0 - [dagit] Repair async graph layout for path-prefix (#7480) by @hellendag

  • e781ce6 - Add docs to k8s docs about includeConfigInLaunchedRuns (#7477) by @gibsondan

  • 98cdc50 - Acquire leases when using ADLS (#7410) by @jamiedemaria

  • 83bfccf - Update Materialization Warning (#7265) by @clairelin135

  • ad6c38a - [dagit] Fix Suggest (#7489) by @hellendag

  • cca4169 - Make the scheduler/sensor daemons key the schedules that it inspects by selector ID, not origin ID (#7492) by @gibsondan

  • a34a9e6 - Add right of search bar prop to top nav (#7500) by @salazarm

  • d57d83b - add flag argument to backfill all partitions in launch backfill graphql mutation (#7499) by @prha

  • 725e6d9 - [dagit] Fix job header “View assets” link, add missing graphQL types (#7493) by @bengotow

  • 4572857 - fix sqlite gating logic for batch tick queries (#7505) by @prha

  • 017bd49 - [dagster-census] Census Integration Library (#7249) by @dehume

  • 768c1b3 - [dagit] Show a better error on invalid run filters (#7494) by @hellendag

  • 13a420a - Add log message to grpc health check (#7506) by @johannkm

  • 8fc5c61 - refactor: remove duplicate solids in event log test (#7511) by @johannkm

  • a4618b4 - correct workspaces icon (#7504) by @salazarm

  • d7ff7bc - [dagit] More cleanup of query callsites performing background polling (#7508) by @bengotow

  • f5bdb5e - [dagit] Avoid resetting hidden repos on each page load #7519 (#7520) by @bengotow

  • 4181eba - [dagit] Allow disabling keyboard shortcuts (#7515) by @hellendag

  • eab1d7d - Update pylint and associated config (#6517) by @smackesey

  • b258c47 - 0.14.10 Changelog (#7521) by @johannkm

  • 36f51f4 - fix asset partitions runtime check (#7525) by @smackesey

  • fc57d23 - [dagit] Tiny fix for “asset graph search, dropdown stops showing when ">" character is used” (#7514) by @bengotow

  • 078d828 - [dagit] Move yarn analyze to packages/app (#7523) by @hellendag

  • 45d9c92 - [dagit] Clean up lodash imports (#7526) by @hellendag

  • 1f1e233 - [dagit] Display repositories in a stable order (#7518) by @bengotow

  • 81e7cba - Automation: versioned docs for 0.14.10 by @elementl-devtools

  • 460e482 - [hooks] fix resource reqs in nested graphs (#7529) by @alangenfeld

  • 610745b - [instigator] dont throw in start/stop if already in desired state (#7483) by @alangenfeld

  • 772afa6 - [dagit] Fix InstigationTick "Skipped" message if zero runs (#7527) by @hellendag

  • d9b523d - Get rid of mode in JobDefinition constructor (#7497) by @dpeng817

  • 45db8cd - Fix issue where we were calling the partition function n times, where n is the number of partitions returned by the function (#7539) by @gibsondan

  • 3f1f0fd - [bugfix] Check that oldData is defined before trying to migrate (#7537) by @salazarm

  • 697e701 - Assorted tox updates (#6971) by @smackesey

  • 1052405 - 0.14.11 changelog (#7543) by @gibsondan

  • f160f7d - Automation: versioned docs for 0.14.11 by @elementl-devtools

  • 19dff51 - Reexecute run mutation using just parent run (#7510) by @johannkm

  • 942c995 - Updated changelog for 0.14.12 (#7545) by @gibsondan

  • 5bf224a - Revert "[bugfix] Check that oldData is defined before trying to migrate (#7537)" and "Update key for saving data in useExecutionSessionStorage (#7333)" (#7551) by @gibsondan

  • ade2be8 - [graphql] subscription handle_graphql_errors (#7482) by @alangenfeld

  • 501d053 - Automation: versioned docs for 0.14.12 by @elementl-devtools

  • 812f0b9 - [dagit] Add bulk re-execution dialog to Runs page (#7451) by @hellendag

  • 3bb37d3 - Downgrade tox pip for Python 3.6 (#7547) by @smackesey

  • 7599886 - annotate dagster.builtins.Nothing with correct type (#7546) by @smackesey

  • 1756b97 - Pylint: Add encoding to open() (#7548) by @johannkm

  • e9f0ebc - asset op tags (#7472) by @sryza

  • 65b8cbd - [dagit] Show asset “compute kind” in the global asset graph (#7516) by @bengotow

  • 91ced9c - [dagit] Hide __asset_group partition sets from search bar (#7556) by @bengotow

  • 7452ec3 - Add typing-extensions >=3.10 as dependency. (#7555) by @smackesey

  • a141905 - validate tags in partition sets (to turn dicts into strings) (#7554) by @gibsondan

  • 5480b04 - [dagit] Feature flag for disabling WebSocket usage (#7557) by @hellendag

  • 2f7ccad - [dagit] Add opNames field to AssetNode (#7565) by @hellendag

  • c31de97 - add DEFAULT_PYTHON_VERSIONS env var for BK (#7553) by @smackesey

  • 922d786 - add DEFAULT_PYTHON_VERSIONS env var for BK (#7553) by @smackesey

  • fdb2274 - [1/n] Interop Stack: Use AssetJobInfo at runtime instead of OutputDefinition (#7473) by @OwenKephart

  • 4d44a79 - [2/n] Interop Stack: Use AssetJobInfo to construct ExternalAssetNodes (#7475) by @OwenKephart

  • 65f15cd - [dagit] Introduce sticky table headers (#7568) by @hellendag

  • 36e7e6a - [dagit] Truncate global search result list (#7569) by @hellendag

  • 38c650e - Test test_create_reexecuted_run_from_failure (#7566) by @johannkm

  • bb87a40 - Tag behavior option for reexecute method (#7574) by @johannkm

  • 6e12f83 - add setuptools as dep (#7576) by @smackesey

  • b87ef89 - fix errors (#7577) by @smackesey

  • b49aadc - Register a new task definition for IAM changes (#7564) by @jmsanders

  • 69546e9 - Improve typing of schedule/sensor and hooks (#7560) by @smackesey

  • 7ef7384 - All steps option for rexecute method (#7575) by @johannkm

  • e9af45b - [docs] fix multi asset sensor example (#7583) by @alangenfeld

  • 5926b2c - Create runs during event log tests (#7502) by @johannkm

  • 22e0267 - [dagit] Refactor ExecutionSessionStorage (#7559) by @hellendag

  • 4516740 - fix(helm): support numeric type image tags (#7579) by @jrouly

  • fc4358e - [dagit] Add bulk re-execution of all steps (#7591) by @hellendag

  • 0b25e62 - ResourceDefinition.none_resource instead of no_step_launcher (#7381) by @sryza

  • db0bf2b - Remove unused imports from event log tests (#7592) by @johannkm

  • 81a09ff - [dagit] Fix asset graph navigation edge cases (#6955, #7208) (#7532) by @bengotow

  • 2a3595b - fullbuild in branch name for full BK build (#7595) by @smackesey

  • 5a85d2c - document partitioned assets (#7466) by @sryza

  • 61a6861 - fix secretsmanager example doc (#7593) by @prha

  • f8f87f6 - Add ConfigSchema type alias (#7596) by @smackesey

  • 8ecaadf - lint dagster_tests (#7594) by @alangenfeld

  • 7ed2dbd - add dialect-specific update logic (#7572) by @prha

  • d0d4d5d - fix volumes example for docker run launcher (#7607) by @gibsondan

  • 0620355 - [tests] set platform on docker build (#7603) by @alangenfeld

  • ed8323c - Use explicitly set env vars rather than autoenvvar prefixes for the grpc command (#7611) by @gibsondan

  • 0a1616b - Fixes for running the user-code-deployment helm chart in a different namespace than the system namespace (#7597) by @gibsondan

  • e947de9 - standardize additional_message kwarg in check methods (#7605) by @smackesey

  • 0cbc38b - [3/n] Interop Stack: Remove asset key from inputs/outputs (#7550) by @OwenKephart

  • cc2060a - fix(helm): allow max concurrent runs to be 0 (#7618) by @rexledesma

  • 6d27f54 - [4/n] Interop Stack: node_def on AssetsDefinition can be graph (#7573) by @OwenKephart

  • 3bcb757 - Asset key to node handle mapping (#7599) by @clairelin135

  • 00fa234 - [dagster-aws] update emr pyspark step launcher (#7604) by @OwenKephart

  • 00d8e1c - Make dagster-user-deployments service account set up a role and role binding by default, like the main dagster chart service account (#7622) by @gibsondan

  • 4cb262c - Add resources to K8sContainerContext (#7619) by @gibsondan

  • d71cb6f - [dagit] Add "shared key path" outlines, experimental asset graph flag (#7608) by @bengotow

  • ee4cb0e - [@dagster-io/ui] Separate tsconfig for build (#7584) by @hellendag

  • 98fe635 - Fixup XXXSource docs (#7629) by @gibsondan

  • febd18f - Changelog 0.14.13 (#7631) by @dpeng817

  • ba9d867 - AssetGroup.prefixed (#7395) by @sryza

  • b8be100 - AssetsDefinition.from_graph (#7620) by @clairelin135

  • b9be0d4 - [Testing] Support for resolvers in ApolloTestProvider (#7612) by @salazarm

  • 9345ebc - [dagit] Show default value for config (#7598) by @hellendag

  • aa37ece - [docker] improve test time (#7589) by @alangenfeld

  • b9b9d97 - Update mypy config and bump version (#7625) by @smackesey

  • 67547d0 - [dagster-ge] drop 3.6 (#7638) by @alangenfeld

  • b12af8c - Automation: versioned docs for 0.14.13 by @elementl-devtools

  • 6e52818 - Unify all alembic scripts into single directory, creating single history (#7411) by @prha

  • 8ab7bfb - enable mypy checking on dagit (#7626) by @smackesey

  • 5870039 - [dagit] Add dummy graphName field to AssetNode (#7637) by @hellendag

  • 3630d97 - [RFC] lighter weight pull request template (#7538) by @alangenfeld

  • cc07c17 - [dagit] AssetView: Don't show historical view message if it's not (#7639) by @hellendag

  • a83f387 - allow flexible types for SourceAsset key arg (#7633) by @sryza

  • 1cfcac3 - add graphql fields to quickly query partition status (#7614) by @prha

  • b0da1bf - add option to GraphQueryInput to apply changes without hitting enter (#7635) by @prha

  • 8f80e59 - Refactors the PartitionGraph component to be divorced from the actual graphql query (#7616) by @prha

  • 6003fdb - Adds a new partition page behind a feature flag (#7617) by @prha

  • a953214 - enable adding asset groups together (#7634) by @sryza

  • 8fbd524 - Fix mypy config for dagster main package (#7644) by @smackesey

  • 596515d - allow multiple asset groups on a repository (#7649) by @sryza

  • f7e85ff - refactor: remove airline demo (#7653) by @rexledesma

  • 4c65d82 - to_source_assets (#7643) by @sryza

  • d543f82 - adding a Terminate Run method for in-progress execution to the Python GraphQL client (#7443) by @Javier162380

  • 87c0f27 - fix apidoc for fs io manager (#7671) by @sryza

  • 90b4739 - Fix celery docker jobs that involve writing directly to command-line output (#7665) by @gibsondan

  • f46fc50 - Fix docs lib links (#7680) by @johannkm

  • 9941f61 - Fix configmaps that need to be manually set when running the user deployments helm chart in a different namespace (#7660) by @gibsondan

  • 37d9629 - Add all repos to hacker_news __init__.py (#7677) by @gibsondan

  • dbbeadf - remove primitive types from apidoc (#7674) by @sryza

  • 7b54c30 - Add py.typed to dagster and all extension libs (#7561) by @smackesey

  • c6ff832 - [eslint-config] Bump dependencies (#7681) by @hellendag

  • faad5ea - [dagster-airbyte] terminate airbyte sync w python process (#7687) by @OwenKephart

  • c99184c - [dagster-dbt] remove annoying color formatting strings from log output (#7688) by @OwenKephart

  • d79bb7b - Pull out PostgresEventWatcher into its own module and parameterize it more (#7666) by @gibsondan

  • 0422087 - graph and job apidoc fixes (#7673) by @sryza

  • 7bdffdb - docs: fix example for multiprocess run config (#7693) by @rexledesma

  • 2db9bd1 - finish dagster-fivetran types (#7563) by @smackesey

  • 636decb - finish dagster-airbyte types (#7562) by @smackesey

  • e9dd228 - chore(graphql): add linter to ensure docstrings on graphene graphql objects (#7669) by @rexledesma

  • acf8f9f - docs(graphql): add docstrings to mutations.py (#7691) by @rexledesma

  • 1bb7112 - Move EventSpecificData out of TYPE_CHECKING (#7697) by @smackesey

  • 441f5ba - Add ability to set mapping key on op/solid invocation context (#7364) by @dpeng817

  • 671cea6 - Make Output generic (#7202) by @dpeng817

  • c55e3f4 - update fake adls resource to work with leases (#7587) by @jamiedemaria

  • de0eae9 - add core partition status storage query (#7662) by @prha

  • 439b4d5 - docs: specify the correct mysql event log module (#7703) by @rexledesma

  • 8566057 - Revert "Make Output generic (#7202)" (#7715) by @dpeng817

  • 11f9496 - Add bool metadata type (#7694) by @clairelin135

  • 52ff71f - Add platform arg and other improvements to dagster-docker CLI (#7698) by @smackesey

  • d4293b8 - Add a guard around dagit's JSON parsing of incoming requests (#7714) by @gibsondan

  • d32fcff - Switch new partition page to use more efficient partition status query (#7663) by @prha

  • bc2e62d - [docs] retry policy invocation example (#7716) by @alangenfeld

  • 756540e - docs: fix example with lazy loading repository (#7704) by @rexledesma

  • 2a579a6 - update slack api call in op hook docs (#7712) by @jamiedemaria

  • 2f828aa - Improve Buildkite test version specification API (#7699) by @smackesey

  • 64c6d85 - use ">" for multi-component selection in AssetGroup.build_job (#7661) by @sryza

  • d6af7e1 - finish dagster-pagerduty types (#7668) by @smackesey

  • 857079d - enable providing asset_key to build_output_context (#7696) by @sryza

  • 7504407 - [dagit] Support special cron strings (#7717) by @hellendag

  • 5fac185 - solid -> op in dagster-slack (#7684) by @sryza

  • 6e467c5 - Typing additions (#7039) by @smackesey

  • 84cc1a4 - [instance] rm get_addresses_for_step_output_versions (#7601) by @alangenfeld

  • e90d8d9 - Add run tags for repository/location names (#6893) by @prha

  • c81a825 - Surface retried error in Dagit (#7692) by @johannkm

  • 504aa08 - RFC: Keep firing hooks even if a framework exception is raised mid-execution (#7652) by @gibsondan

  • 948499e - combine validate_tags and check_tags to JSONify nested tags (#7720) by @gibsondan

  • ae55df0 - test dynamic_output output for node (#7718) by @clairelin135

  • 5e32caa - [postgres] dont use urlquote_plus (#7723) by @alangenfeld

  • d9cc191 - [docs] fix jobs from graphs example (#7722) by @alangenfeld

  • 4c57949 - Make output generic (#7719) by @dpeng817

  • 2155bf5 - Missed changes for Dagit surface retry error (#7731) by @johannkm

  • ed04dab - Change head_bucket to list_objects. (#7485) by @trevenrawr

  • fbb1abb - RFC: include schema in default dbt asset keys (#7645) by @sryza

  • 59a9915 - populate op names and graph name for ExternalAssetNode (#7721) by @OwenKephart

  • 69c57d3 - add extra sensor state fetch to minimize chance of clobber state (#7738) by @prha

  • 97ed20f - Workaround for mypy cache bug (#7732) by @smackesey

  • 8cc4bdc - fix run list for new partitions view (#7742) by @prha

  • 81a0421 - 0.14.14 changelog (#7743) by @prha

  • ee3081f - Make Output and DynamicOutput no longer be tuple types (#7740) by @dpeng817

  • de65762 - [dagit] Use a CSP header instead of meta tag (#7727) by @hellendag

  • 0260b0f - Add a test with various wait/act use cases (#7728) by @hellendag

  • da15d8d - Add job/op equivalents to execution context (#7734) by @johannkm

  • 30e7ced - Rename ReexecutionPolicy -> ReexecutionStrategy (#7746) by @johannkm

  • 9beeff4 - [dagit] Update tsconfig to es2022, update browserslist (#7749) by @hellendag

  • 2930507 - [dagit] Use Array flat (#7754) by @hellendag

  • a4ffde0 - remove usable_as_dagster_type from integrations (#7682) by @sryza

  • 41f705c - Automation: versioned docs for 0.14.14 by @elementl-devtools

  • 1d35e58 - enable returning lists of run requests from sensor/schedule evaluation functions (#7755) by @prha

  • db8f725 - remove usable_as_dagster_type from types concept page (#7758) by @sryza

  • f305b65 - [easy] fix missing space in error message for util function (#7761) by @gibsondan

  • f85b25d - [dagit] Add more security headers (#7764) by @hellendag

  • 89efb37 - Additional asset graph improvements (#7707) by @bengotow

  • 9f73f3f - [dagit] Add Reload All button to the asset catalog, remove repo filter (#7708) by @bengotow

  • 6e50207 - [dagit] Better indentation guide in our CodeMirrors (#7765) by @hellendag

  • afe23af - Allow python objects to be passed as top-level inputs to job (#7032) by @dpeng817

  • 9366a7e - Add a static method on DefaultRunLauncher that launches a run without a workspace (#7777) by @gibsondan

  • 68a99d6 - Revert "Allow python objects to be passed as top-level inputs to job (#7032)" (#7779) by @dpeng817

  • cba9144 - Allow repositories to contain asset definitions and source assets for the same asset key (#7781) by @sryza

  • 8e25adc - fix error message when can't find upstream asset for dep (#7648) by @sryza

  • 8e05a7d - Allow TextInput to have a type of number (#7778) by @salazarm

  • 1b08360 - fix error with inferring description from empty docstring (#7788) by @sryza

  • 8096b17 - [dagit] Add hover actions to run Tags (#7751) by @hellendag

  • 42e00e1 - Fixes to the apidocs page (#7729) by @sryza

  • 9cf7cde - [dagit] Show asset nodes on Asset Details (#7689) by @hellendag

  • 7055f90 - [dagit] Make log filter input clearable (#7805) by @hellendag

  • 10d9fc5 - [eslint-config] Add recommended jest lint (#7807) by @hellendag

  • 64cf6a2 - pin xmltodict to fix aws test failures, until moto 3.1.9 (#7806) by @prha

  • 5f1e9fb - update s3 sensor docs to use context.cursor instead of context.last_run_key (#7748) by @prha

  • 8cfa136 - Add namespaces / prefixes to the hacker news assets demo (#7782) by @sryza

  • 742ba2b - toposort frozenset workaround (#7793) by @smackesey

  • 4462e81 - Increase executor event log poll interval (#7803) by @johannkm

  • abd59f6 - Add a few __init__.py in test packages (#7809) by @smackesey

  • b4115ef - remove TableSchema experimental warnings (#7799) by @smackesey

  • 1f0d9fb - snowflake io manager (#7726) by @sryza

  • ccb2fd1 - Change the fake repo that simulates an import error while loading dagster code (#7816) by @gibsondan

  • e6c5bb0 - remove jinja / nbconvert pin (#7819) by @prha

  • 9ca7fdd - [dagit] Use /instead of> for displaying asset paths (#7818) by @bengotow

  • b0e3ca2 - fix snowflake io manager tests on python 3.7 (#7824) by @sryza

  • 35a198b - link to examples in concepts pages (#7753) by @jamiedemaria

  • 0540339 - [docs] job execution (#7776) by @alangenfeld

  • 14947c2 - [dagit] Clean up my todos (#7810) by @hellendag

  • 99259a7 - [dagit] Fix global search for repeated item keys (#7827) by @hellendag

  • cd7de4d - sys.meta_path import mapping layer (#7040) by @smackesey

  • 629a689 - Allow passing posargs to tox commands (#7822) by @smackesey

  • 2572d66 - Rename check > _check (#7808) by @smackesey

  • 62b2ffb - context.partition_time_window (#7795) by @sryza

  • 7abbdb6 - take out airline demo from make dev_install (#7839) by @sryza

  • 27ca0c0 - [dagit] Make schedule/sensor tags more prominent in RunTags (#7834) by @hellendag

  • 106dca2 - add BoolMetadataValue to __all__ in dagster/__init__.py (#7838) by @sryza

  • 42e35af - [dagit server] guard on async websocket send (#7833) by @alangenfeld

  • 03c8691 - [top-level inputs 1/2] Move InvalidSubsetError try-catch up a level (#7780) by @dpeng817

  • 8cb4101 - docs design update (#7531) by @yuhan

  • bb75f3b - Remove test-connection pod (#7842) by @gibsondan

  • d60c3c8 - remove validate_asset_key_string (#7811) by @sryza

  • 3125227 - partitioned assets from graphs (#7837) by @sryza

  • 15e9eb5 - fallback logic for resolvers (#7766) by @OwenKephart

  • d099e43 - migrate concept docs code snippet paths to graph/job/op (#6775) by @sryza

  • 0356320 - [rfc] generic + returnable dynamic outputs (#7744) by @dpeng817

  • 923342e - [dagster-dbt] dbt build + AssetObservations from tests (#7783) by @OwenKephart

  • 77dff37 - [dagit] New “folder grid” asset view behind experimental flag (#7767) by @bengotow

  • 7207a6e - Handle backcompat asset observation (#7831) by @clairelin135

  • 2238f8b - [dagit] Remove search item key (#7830) by @hellendag

  • 0ba6b2a - use generic Output type annotations in hacker news (#7600) by @sryza

  • 5e8f310 - changelog 0.14.15 (#7850) by @yuhan

  • 773cf30 - [dagit] add log-level flag (#7853) by @alangenfeld

  • b71f150 - [changelog][skip ci] dagit line and autofmt (#7857) by @alangenfeld

  • 61afd9a - [dagit] tweak websocket disable (#7860) by @alangenfeld

  • 21237cf - Limit pods per run with k8s_job_executor (#7846) by @johannkm

  • d917df5 - additions to the 0.14.15 changelog (#7856) by @sryza

  • c0f86a3 - fix asset key prefixes in MDS assets example (#7855) by @sryza

  • 042d3d2 - [helm] dagit.logLevel (#7862) by @alangenfeld

  • f06689b - warn for instance.get_event_records calls without an event type filter (#7848) by @prha

  • 8455843 - [top-level inputs 2/2] top level inputs on job and execute_in_process (#7786) by @dpeng817

  • e362fad - Automation: versioned docs for 0.14.15 by @elementl-devtools

  • 632cc79 - rm duplicate changelog entry from navigation (#7873) by @gibsondan

  • b5d52ea - Fix error message assertions in Helm tests (#7858) by @johannkm

  • 6732a87 - Toxfile adjustments (adding mypy) (#7867) by @smackesey

  • 993a3aa - [resources on repos 1/n] Reorganize job/pipeline/graph duplication error logic in repository (#7817) by @dpeng817

  • a4d79e3 - docs: fix vercel build (#7876) by @yuhan

  • bfbfa62 - enable @asset-decorated functions to accept kwargs (#7871) by @sryza

  • 21b6a5e - fix type error (#7887) by @smackesey

  • 78d4b57 - dagster-dbt types (#7878) by @smackesey

  • 272b250 - Add health check for ECS tasks (#7695) by @jamiedemaria

  • 91b6c0f - [cloud dagit] Add optional nav tabs parameter for settings root page (#7892) by @benpankow

  • ea27cdc - feat(helm): support custom configmap for workspace (#7882) by @johannkm

  • 9e786a1 - add data migration for run repo tags (#7815) by @prha

  • c71b117 - make docker_compose_cm a contextmanager rather than a fixture (#7905) by @gibsondan

  • 727c059 - [@dagster-io/eslint-config] v1.0.4 (#7913) by @hellendag

  • 5897837 - docs: fix flash white box (#7877) by @yuhan

  • fe59b68 - InputContext.upstream_output is missing key when source asset (#7919) by @sryza

  • 80153e0 - remove handle_schema_errors that wrap db errors with schema outdated exceptions (#7886) by @prha

  • c95cfd1 - Enable asset partitioning support for @multi_asset defined assets (#7908) by @aroig

  • 5e22b7b - update multiple dynamic outs example (#7917) by @jamiedemaria

  • 6e15dfb - dagster-databricks mypy (#7879) by @smackesey

  • 252f285 - [1/n] Subsetting Stack: use subset_selector for AssetGoup.build_job (#7796) by @OwenKephart

  • 06149b6 - Don't call it a 'framework error' when a run worker unexpectedly restarts (#7885) by @gibsondan

  • 17fa439 - chore(helm): remove local from compute log manager options (#7924) by @rexledesma

  • f6d63d1 - Executor and dagster-docker typing (#7881) by @smackesey

  • a7e0b69 - remove markupsafe pin (#7898) by @bollwyvl

  • 1564c9f - [2/n] Subsetting Stack: AssetsDefinition subsetting (#7797) by @OwenKephart

  • 3b5c135 - fix docstring (#7948) by @OwenKephart

  • d0524c2 - RFC: Better error handling when an ExternalRepository grpc call fails (#7929) by @gibsondan

  • f27ee8b - Use correct configured link in resources (#7949) by @gibsondan

  • 445194b - add CLI command to print the storage schema version (#7910) by @prha

  • 2d219f4 - [dagit] Present location reload errors after “Workspace > Reload All” (#7907) by @bengotow

  • 943ab6c - [dagit] Asset details: Graph/op links (#7655) by @hellendag

  • 448ca76 - Fix detection of parent process death (#7914) by @aroig

  • 15887bc - Merge identical get_location and get_repository_location methods (#7952) by @gibsondan

  • 0d90a3c - add missing check to AssetsDefinition.from_graph (#7960) by @sryza

  • f7b14c4 - Fix issue where azure IO manager was sometimes failing on recursive deletes (#7956) by @gibsondan

  • 780f771 - make input defs deterministic (#7957) by @clairelin135

  • b8f3816 - [events] can_load mitigation (#7955) by @alangenfeld

  • 2cbf3cd - fix slack image link on Getting Started (#7964) by @yuhan

  • 1bd83e4 - Typing for automation package (#7812) by @smackesey

  • 205d811 - make StepLauncher and friends public (#7945) by @smackesey

  • fb025dc - feat(helm): bump minimum supported kubernetes version to 0.19 (#7925) by @rexledesma

  • 422fa71 - Asset Subselection (#7835) by @clairelin135

  • d6b4eaf - various fixes for new partition page (#7950) by @prha

  • 0424134 - [dagster-airbyte] don't cancel completed syncs (#7888) by @OwenKephart

  • 2a56fa2 - [dagster-dbt] small tweaks (#7967) by @OwenKephart

  • e168466 - Error rather than warn when an event log query comes in that tries to use an int cursor on an event log DB that needs a RunShardedEventsCursor (#7970) by @gibsondan

  • c7b2387 - Still make cursor editable if it is not set (#7969) by @gibsondan

  • 738799d - [daemon] tweak per-thread instance/workspace setup (#7965) by @alangenfeld

  • b66a915 - Revert "feat(helm): bump minimum supported kubernetes version to 0.19 (#7925)" (#7968) by @rexledesma

  • 50ac89f - 0.14.16 Changelog (#7976) by @johannkm

  • abc35c0 - fix reexecution (#7980) by @clairelin135

  • 129a8d6 - Automation: versioned docs for 0.14.16 by @elementl-devtools

  • b49cfe4 - [dagit] Fix SearchBootstrapQuery to be lazy (#7981) by @hellendag

  • 7a72d66 - split graphql query for the backfill page (#7986) by @prha

  • a8d3526 - Remove can_terminate from run launchers and run coordinators, use a status check instead in graphql (#7983) by @gibsondan

  • 6afdaf9 - style(helm): remove unnecessary pylint suppressions (#7923) by @rexledesma

  • 8f375c4 - provide job_timeout_in_seconds in dataproc resource config (#7941) by @3cham

  • 605d274 - [dagit] Replace opName with opNames everywhere (#7977) by @hellendag

  • d910d23 - Set default hour of day and minute of hour in build_schedule_from_partitioned_job based on the offsets of the parittioned job (#7954) by @gibsondan

  • 4293e7c - WIP: Don't fetch every single run in the InstanceBackfillsQuery (#7985) by @gibsondan

  • cbd7da9 - [dagit] Show assets on run pages in more scenarios (#7874) by @bengotow

  • 5ae7d32 - [dagit] Do not display the partition_set, step_selection tags on asset group runs (#7953) by @bengotow

  • f51d7e6 - [dagit] WebSocketProvider: Time out and fall back to disabled (#7978) by @hellendag

  • 515e194 - Coerce asset_key on AssetIn (#8009) by @aroig

  • d19ec31 - Fix issue where we added the run ID repeatedly to the runs for the tick during scheduler failure recovery (#8003) by @gibsondan

  • 58230bf - Load fewer backfills on each page (#7998) by @gibsondan

  • f625b6d - add loading indicator for job backfills (#8004) by @prha

  • abac74a - Remove duplicate count_resume_run_attempts method (#7915) by @johannkm

  • d9133bb - Make asset related checks more robust (#8008) by @aroig

  • c4d844d - Fix relative path for pytest fixture in backcompat tests (#7999) by @dpeng817

  • 60e8774 - feat(helm): allow name override for Dagit deployment (#8005) by @rexledesma

  • be20db5 - Add a __main__.py to dagster/daemon (#8012) by @gibsondan

  • 07210b9 - dagster-buildkite refactor (#7813) by @smackesey

  • d5d7068 - Fix unsightly scrollbar in custom alert (#8018) by @gibsondan

  • 1bda005 - create get_records_for_run for event log storage, using an opaque string cursor (#7973) by @prha

  • 7cc7681 - Fix error when dagit passes in a non-dict string to run config valiation (#8020) by @gibsondan

  • bf78a28 - [dagster-io/ui] Fix Dialog header/footer alignment (#7975) by @hellendag

  • 8c6a8fe - [dagit] Disable unloadables if no permissions (#8021) by @hellendag

  • 71802bd - use mkdir -p in migration guide by @johannkm

  • aebfe43 - [dagit] Sectioned left nav (#8017) by @hellendag

  • e385dec - Handle InvalidSubsetError in launch run results (#8014) by @gibsondan

  • 08ef634 - Use dagster-postgres in alembic migration guide (#7992) by @johannkm

  • 0b50cc2 - Generate buildkite headers inside dagster-buildkite (#7987) by @smackesey

  • 3ac0dd2 - bug fix: should allow re-executing all dynamic steps generated from mapper step (#7979) by @yuhan

  • 654a748 - refactor: has_index for migrations (#8034) by @johannkm

  • 08e37e3 - In-progress indicator for graph-backed assets (#8015) by @clairelin135

  • b1ab74b - resource defs on asset def (#7918) by @dpeng817

  • 9d98be1 - Load all code artifacts on grpc server startup, not just pipelines/jobs (#8040) by @gibsondan

  • 4062bca - Accept string values for non_argument_deps (#8023) by @shalabhc

  • 49aa991 - add example Python incompatibilities on Py 3.6 (#8036) by @smackesey

  • 31e1f3d - Add missing deps for docs-snippets and skip 36 in BK (#8048) by @smackesey

  • e7fca20 - docs: update readme with social links (#8046) by @rexledesma

  • 4e1a53f - OpExecutionContext.output_asset_key (#7961) by @sryza

  • 10b9225 - Add missing method to SnowflakePandasTypeHandler (#8051) by @gibsondan

  • 10d8e65 - [dagit] Flip section arrow in left nav (#8056) by @hellendag

  • b5604ce - [dagster-io/ui] Make disabled+checked Switch state more obvious (#8033) by @hellendag

  • 9122968 - [dagit] Enable sectioned left nav by default (#8039) by @hellendag

  • 407e698 - [3/n] Subsetting Stack: dbt assets can be subset (#7798) by @OwenKephart

  • 84ccc32 - Raise error upon incomplete graph-backed asset subset (#8041) by @clairelin135

  • ac6007c - throw error for empty asset key (#8069) by @smackesey

  • 71a85ba - [dagit] Use dialog for multiple schedules/sensors in left nav (#8065) by @hellendag

  • f8ff6f6 - remove dynamic mapping and collect section from jobs concepts page (#7844) by @sryza

  • 307b363 - grammar (#7984) by @dwinston

  • 71872c3 - Pass in engine rather than connection in SnowflakePandasTypeHandler (#8070) by @gibsondan

  • b6caeaf - [dagit] Launchpad: Show disabled "Scaffold" and "Remove" buttons instead of hiding them (#8066) by @hellendag

  • 220e74d - chore(buildkite): mention user in private channel (#8071) by @rexledesma

  • d7129a5 - [dagster-dbt] allow for static manifest.json-based selection (#8047) by @OwenKephart

  • 03b24cd - [asset-resources 2/n][rfc] io manager defs directly on asset defs (#7920) by @dpeng817

  • 1e411ad - adding a FromSourceAsset StepInputSource (#7942) by @OwenKephart

  • 284e912 - Add protobuf pin to dagster (#8078) by @gibsondan

  • 0e0bf21 - Skip flaky snowflake+pandas tests (#8081) by @gibsondan

  • e557f0e - docs: revamp the README (#8052) by @rexledesma

  • 28f9930 - 0.14.17 changelog (#8083) by @smackesey

  • 8e8ee85 - Fix upstream context handling in fs_asset_io_manager (#8007) by @aroig

  • 3e68731 - KnownState.ready_outputs (#8016) by @alangenfeld

  • d072b41 - Automation: versioned docs for 0.14.17 by @elementl-devtools

  • aad66c0 - chore: mark FileCache for deprecation in 0.15.0 (#7922) by @rexledesma

  • d980112 - remove step stats query (#8089) by @clairelin135

  • f733499 - Revert "throw error for empty asset key (#8069)" (#8093) by @gibsondan

  • c7427a0 - migrate from deprecated sqlalchemy methods (#7864) by @alangenfeld

  • 67ca747 - fix typos (#8090) by @OwenKephart

  • e9771fd - Add skip for flaky grpc server test (#8097) by @gibsondan

  • 1e9ac3d - chore(buildkite): ignore notifications on canceled builds (#8099) by @rexledesma

  • 46722df - [dagit] Break apart LaunchpadSessionContainer (#8101) by @hellendag

  • 003c161 - [dagit] Small changes to the asset catalog (#7993) by @bengotow

  • bea40a4 - [dagit] Track partition set sort order for Launchpad (#8104) by @hellendag

  • 08fad28 - Lint rule to make sure queries requiring QueryVariables have them (#8102) by @salazarm

  • aa6d4ba - Make dagster-images build work when it is called outside of a git repo (#8088) by @gibsondan

  • 18624a9 - Fix configuration schema for k8s executor (#8107) by @fahadkh

  • f921f04 - [dagit] Clean up some yarn peer deps (#8108) by @hellendag

  • 8551940 - Add latestRun resolver to AssetsLatestInfo (#8072) by @clairelin135

  • 6b8f907 - Handle op outputs in default asset IO manager (#8074) by @clairelin135

  • 35e8a66 - [dagit] Add last materialization, latest run columns to the asset table (#7996) by @bengotow

  • 8f1d403 - fix(mypy): refine types (#8129) by @rexledesma

  • cc7360a - [dagit] Remove the global asset graph in preparation for asset group graphs (#8125) by @bengotow

  • dbae025 - [dagit] Rename isAssetGroup => isHiddenAssetGroupJob for clarity (#8124) by @bengotow

  • 72c4e50 - fix(helm): allow numeric quoted strings as image tags (#8120) by @rexledesma

  • 1429ac2 - docs: add a button to copy/paste code snippets (#8106) by @yuhan

  • 9af83ae - [asset-resources 4/n][rfc] Refactor resource requirement checking code (#7947) by @dpeng817

  • 0d4dd5a - fix(helm): use templated comment (#8137) by @rexledesma

  • cd3f69e - docs(helm): add doc hint for dagster-user-deployments.imagePullSecrets (#8112) by @ceefour

  • dc922f9 - Added group_name to asset (#8110) by @shalabhc

  • a3cf6c9 - Fix invocation on ops that use generic dynamic outputs (#8133) by @dpeng817

  • 48507e9 - [dagit] Show more information for last run on Schedules/Sensors (#8130) by @hellendag

  • bc3fde4 - [dagit] Remove asset graph bundling based on path prefixes, experimental flag (#8127) by @bengotow

  • ac59da6 - KnownState parent run info (#8030) by @alangenfeld

  • cfe0766 - add metadata to jobs (#7849) by @jamiedemaria

  • 0e6f9a9 - fix dagit-debug (#8148) by @alangenfeld

  • 64bd480 - add kwargs to pipeline snapshot from storage (#8149) by @jamiedemaria

  • 8fc3838 - Fix label sanitization for strings that end in a period (#8151) by @gibsondan

  • 1d216c5 - [dagit] Show counts next to log filter tags (#8141) by @hellendag

  • 5112ead - increase test_ping timeout (#8150) by @alangenfeld

  • 0c1dc58 - [dagit] Fix duplicate styled-components (#8144) by @hellendag

  • 992e271 - Basic asset config (#7590) by @smackesey

  • f2beff3 - Avoid launching ECS runs with large overrides (#8152) by @jmsanders

  • 217f636 - Optionally tag images as latest (#8132) by @jmsanders

  • bc038fe - chore(helm): christen the code server service port name as grpc (#8142) by @calebfornari

  • b4b4e33 - feat(helm): allow the postgresql scheme to be configurable (#8126) by @peay

  • 1936f23 - [dagit] Remove clear-site-data header (#8134) by @hellendag

  • 35b5241 - [dagit] Allow run termination on queued run (#8157) by @hellendag

  • 6fddda8 - [dagit] Add active state to left nav items (#8147) by @hellendag

  • 6324cc4 - [asset-resources 5/n] io manager defs on source assets (#8105) by @dpeng817

  • 59ae56b - [dagster-io/eslint-config] v1.0.5 (#8121) by @hellendag

  • a0aeb78 - fix(helm): use pre-2022 bitnami repository (#8166) by @rexledesma

  • 04f9c4e - Asset config gql resolver (#8163) by @smackesey

  • 67e4b01 - Allow environment variables in dagster-mlflow schema (#7997) by @chasleslr

  • 09fddd9 - changes (#8169) by @OwenKephart

  • 5c35d39 - [dagit] asset Config (#8154) by @smackesey

  • 974af9f - clean up create_pg_connection (#8165) by @alangenfeld

  • ccab3c0 - [dagit] Add a Lineage tab to the Asset Details page (#8143) by @bengotow

  • 11f5f1b - [easy] 0.14.18 => 0.14.19 in changelog (#8180) by @gibsondan

  • c6f7fba - revert "Scaffold Config" always enabled (#8181) by @smackesey

  • 6e684e1 - Add asset groups to graphql (#8140) by @shalabhc

  • cca4561 - [0.15.0] remove attempt count from step launching APIs (#8068) by @alangenfeld

  • c8e5fa4 - [dagit] Make the left nav stay open/closed (#8173) by @hellendag

  • 6fee2dc - step launcher fix up (#8186) by @alangenfeld

  • b47f6ee - Explicit asset key args to assetsLatestInfo (#8178) by @clairelin135

  • 5f7d322 - Improve documentation around Output objects and op output annotations. (#8139) by @dpeng817

  • 102e9e5 - Automation: versioned docs for 0.14.19 by @elementl-devtools

  • a217002 - Revert "Automation: versioned docs for 0.14.19" (#8190) by @gibsondan

  • ac4a1d8 - Automation: versioned docs for 0.14.19 by @elementl-devtools

  • efae88a - [docs] - RFC: add a Learning More section to the asset tutorial [CON-17] (#8031) by @sryza

  • 585eeaf - Merge Asset IO functionality with Library IO managers (#8189) by @clairelin135

  • d2e058a - let tag inherit pointer style (#8208) by @salazarm

  • bd32fae - extract event connection to top-level graphql query (#8077) by @prha

  • ad8a0a2 - Fixes for asset graphql tests to make them easier to generalize and call with other graphql context fixtures (#8199) by @gibsondan

  • b89f4d0 - [instance] rm events_for_asset_key and get_asset_events (#7602) by @alangenfeld

  • 62cbc10 - Restrict group names to VALID_NAME_REGEX_STR (#8214) by @shalabhc

  • 18af5e7 - [asset-resources 6/n] with_resources method (#8019) by @dpeng817

  • 75e295d - fix encoding for windows test (#8227) by @prha

  • 1b20e2e - Migration: add columns action_type and selector_id to bulk_actions (#7995) by @johannkm

  • a5a1c0b - Create unified storage configuration for dagster.yaml (#7283) by @prha

  • 076fb86 - Add image pull secrets to k8s container context (#8221) by @gibsondan

  • 73d459b - [dagit] Update MenuLink to support a disabled state (#8200) by @bengotow

  • ef928a4 - [docs] - Document partitioned IO managers [CON-37] (#8191) by @smackesey

  • a34abb7 - ungate new partitions backfill (#8224) by @prha

  • 7fa711e - [dagit] Add top level Asset Group pages, Asset Groups in left nav (#8203) by @bengotow

  • b1aa959 - Fix upgrade story for new scheme field in postgres (#8220) by @gibsondan

  • 148e3e5 - fix workspace.yaml formatting (#8182) by @smackesey

  • 8cc09fe - docs: improve search 1/ (#8229) by @yuhan

  • cc31c77 - [dagit] Add asset group filter to the Asset Catalog (#8204) by @bengotow

  • be6d055 - `assets_from_module methods [1/2] (#8225)` by @clairelin135

  • b2c3741 - Graph-backed asset IO Fix (#8171) by @clairelin135

  • b434a3e - bring back source asset metadata (#8195) by @sryza

  • 7dfdfe1 - [dagit] Asset graph GraphQL query audit + cleanup (#8205) by @bengotow

  • 5733b39 - [dagit] Remove flat left nav (#8231) by @hellendag

  • b34f04d - [assets] cycle resolution (#8222) by @OwenKephart

  • 85e9c3a - [docs] add multi_assets docs page [CON-33] (#8192) by @OwenKephart

  • 660cd52 - Assets now have a default group name (#8226) by @shalabhc

  • 6498f34 - Move config editor to core so we can reuse in admin portal (#8237) by @salazarm

  • 20f5889 - Renamed the prefix for auto created jobs from '__ASSET_GROUP' to '__ASSET_JOB'. (#8235) by @shalabhc

  • 38b71a2 - Derive origin from pipeline run instead of the arg to ExecuteRunArgs (#8156) by @gibsondan

  • c48efb2 - unique name for assets in adls2 tests (#8232) by @jamiedemaria

  • 4b691da - docs: improve search 2/ (#8244) by @yuhan

  • f7718fa - AssetSelection (#8202) by @smackesey

  • 4ee0b99 - [for 0.15.0] default includeConfigInLaunchedRuns in helm chart to true (#7488) by @gibsondan

  • 9894016 - [dagit] Don't block scroll when mouse hits custom tooltip (#8242) by @hellendag

  • 5f833e8 - [dagit] Switch asset group pages to /list and /lineage instead of ?tab= (#8250) by @bengotow

  • 0215e82 - [docs] Asset Job Schedules/Sensors [CON-28, CON-29] (#8155) by @clairelin135

  • 3c8d1bf - [docs] - Document asset metadata [CON-31] (#8084) by @erinkcochran87

  • 872c2a8 - [docs] - Upstream changes for assets [CON-36] (#8175) by @erinkcochran87

  • d759550 - add data migration to bulk actions table for backfill jobs (#8153) by @prha

  • 2e236b2 - [docs] - Asset config [CON-89] (#8119) by @erinkcochran87

  • 1804af5 - [docs] - Update Ops & Asset pages for release [CON-21] (#8158) by @erinkcochran87

  • e9ebf8f - Override resource defs when invoking assets (#8217) by @dpeng817

  • a1a31d4 - Add resource defs to source asset, handle transitive dependencies (#8223) by @dpeng817

  • eea9a77 - asset defs directly on repository (#8197) by @sryza

  • 003bfc0 - [docs] - Graph-backed assets [CON-34] (#8174) by @erinkcochran87

  • 833c7a1 - `assets_from_module methods [2/2] (#8236)` by @clairelin135

  • 198053a - [assets] UnresolvedAssetJobDefinition (#8207) by @OwenKephart

  • 8e6b70f - [dagster-dbt] update dbt keys (#8228) by @OwenKephart

  • ca009fc - with_resources top level export (#8264) by @dpeng817

  • 90fd674 - chore(buildkite): separate docs only changes (#8261) by @rexledesma

  • db7fb76 - [docs] Asset storage description in filesystem IO Manager docs (#8240) by @clairelin135

  • 08b4fcf - [dagit] Add Materialize button to the Asset Catalog (#8206) by @bengotow

  • df10793 - Added the default group name for assets defined using Out(...) (#8259) by @shalabhc

  • 455096b - [assets] Allow graph_name to be None when not specified (#8247) by @bengotow

  • f6ab196 - [dagit] Asset group styling adjustments, polish (#8246) by @bengotow

  • c6aa327 - add backfill blank state (#8267) by @prha

  • 9567e1e - Fix descriptor that only references solid (#8159) by @dpeng817

  • ff3392a - add an "environment" key to EcsRunLauncher / EcsContainerContext that sets env vars (#8243) by @gibsondan

  • ec409c2 - [dagit] Add a Materialize All button to Asset Details > Lineage (#8248) by @bengotow

  • 6552e43 - [assets] Update modern-data-stack assets (#8271) by @OwenKephart

  • b54ed41 - [assets] fix tests (#8275) by @OwenKephart

  • 43e5d1a - docs: improve search 3/ handle GH discussion entries in search results (#8252) by @yuhan

  • 44c3075 - [docs] - Cross-repository assets [CON-32] (#8075) by @erinkcochran87

  • e8f256e - Fix hackernews after change to AssetGroup behavior (#8269) by @dpeng817

  • a2d7861 - [docs] - Document non-argument deps for assets [CON-16] (#7962) by @erinkcochran87

  • d99815d - assets_from_package_module -> load_assets_from_package_module (#8280) by @dpeng817

  • 40d8268 - validate resources for assets passed directly to a repository (#8270) by @dpeng817

  • 66ea7c9 - Standalone materialize method (#8268) by @dpeng817

  • fc5f07e - [0.15.0] move from warnings to errors for sensor/schedule target duplication (#7861) by @dpeng817

  • c3f4d36 - refactor: remove FileCache (#7701) by @rexledesma

  • d684848 - [assets] add partition_def and config to define_asset_job (#8282) by @OwenKephart

  • b86fe4d - Add docs for top level job inputs (#8212) by @dpeng817

  • f0b7346 - default executor on repo (#8272) by @dpeng817

  • 947a9f6 - Deprecate asset namespace (#8274) by @smackesey

  • d14ae4c - Fix black (#8288) by @johannkm

  • e34d9ca - Add kvs table to oss (#8213) by @johannkm

  • a03ccb8 - [dagit] Hold the lineage graph zoom level constant as you navigate (#8251) by @bengotow

  • b9de066 - Add max_retries to user editable tags (#8285) by @johannkm

  • 19a4d6e - fix is_subclass bug where issubclass(list[str], DagsterType) throws a surprising exception (#8287) by @gibsondan

  • 079bdd9 - take experimental decorators off of asset APIs (#8260) by @sryza

  • c52c203 - fix group_name in to_source_assets (#8279) by @sryza

  • 9fc1b11 - KVS storage methods (#8249) by @johannkm

  • 3b8c1e6 - [dagster-dbt] Fix bug where a dbt invocation that did not successfully start could emit materialization events. (#8293) by @OwenKephart

  • 04e2a7a - Fix typo in multi_asset docstring (#8292) by @johannkm

  • 028eb42 - Make dagster/reexecution_strategy tag editable (#8286) by @johannkm

  • 33d6a38 - asset selection tweaks (#8290) by @smackesey

  • 406b501 - Revert "KVS storage methods (#8249)" by @johannkm

  • a896660 - only return config field for solids (#8278) by @clairelin135

  • 09194c3 - docs: fix white box flashing when navigating between pages (#8281) by @yuhan

  • 79ea5d9 - [dagit] Fix JS error on repo reload (#8297) by @hellendag

  • 02e9d20 - [docs] - Move assets into intro tutorial + up in sidenav [CON-18] (#8241) by @erinkcochran87

  • 0e1fc50 - un-asset-group-ify SDA guide (#8283) by @dpeng817

  • 3954423 - asset concepts page without AssetGroups (#7901) by @sryza

  • b6a8f2f - 0.14.20 Changelog (#8301) by @johannkm

  • a6107b7 - solid -> op in message for omitted outputs (#7903) by @sryza

  • e0879bf - [docs] - Resources in assets [CON-27] (#8168) by @erinkcochran87

  • 82d510b - [docs] - SDA guide for existing Dagster users [CON-30] (#8188) by @sryza

  • 3f55bd2 - fix subclass test on py36 (#8302) by @gibsondan

  • e61fc61 - [dagit] Repair issue where last ten runs are missing for job (#8305) by @hellendag

  • a8bc2a2 - Deprecate AssetGroup (#8276) by @smackesey

  • 6ec3af6 - remove experimental materialization tags (#6650) by @prha

  • f6acfdc - Automation: versioned docs for 0.14.20 by @elementl-devtools

  • e2f3342 - [docs] - Remove advanced tutorials [CON-290] (#8298) by @erinkcochran87

  • 2fbac77 - RFC: asset tutorial without AssetGroups (#7909) by @sryza

  • 8ea1a15 - [dagit] Don’t let description overflow on asset details page (#8309) by @bengotow

  • c881f12 - [dagit] Small left-nav design adjustments (#8307) by @bengotow

  • b3d063e - [dagit] Make the warning icon optional in AppTopNav (#8294) by @hellendag

  • 2623403 - Add docker executor example to deploy_docker (#8219) by @gibsondan

  • e6fad5b - AssetSelection.assets (#8316) by @sryza

  • 8b0d51c - upload backcompat artifacts to bk (#8184) by @jamiedemaria

  • 81de3d9 - [dagster-io/ui] Expose some text input styles (#8318) by @hellendag

  • 70558e1 - output_asset_partition_key -> asset_partition_key_for_output (#8327) by @sryza

  • e4a00e3 - [dagit] Fix text overflow behavior in the left nav, keep open if one repo (#8324) by @bengotow

  • bca8b6a - [dagit] Run timeline: make popover scrollable (#8319) by @hellendag

  • 3a1d95b - docs: make sidenav sticky [DREL-329] (#8310) by @yuhan

  • ab62a16 - use DisplayableEvent in dagit log viewer (#8323) by @gibsondan

  • 5ba7a9f - uniform prefix parsing (#8332) by @OwenKephart

  • 1fbce53 - docs: fix link from Kubernetes example (#8335) by @rexledesma

  • e97cc73 - [dagster-dbt] group names on dbt assets (#8303) by @OwenKephart

  • d46e5d8 - Attempt to speed up query for existing run keys for a given sensor (#8329) by @gibsondan

  • ffb3ddc - [docs] - Getting Started/quick start updates [CON-39] (#8187) by @erinkcochran87

  • 7a86d60 - in load_assets_from_... functions, make key_prefix docstring consistent with group (#8333) by @sryza

  • 2dc2d9c - reorder concept super-sections in left nav (#8315) by @sryza

  • 510bf07 - docs: improve search 4/ house cleaning - remove unused config/cmd (#8273) by @yuhan

  • 632ea3d - [dagit] Repair font-sizes on Safari (#8351) by @hellendag

  • 9d11ca4 - [assets] fix issues with job selection (#8340) by @OwenKephart

  • 7bdf0ce - AssetsDefinition.asset_keys -> keys and similar (#8325) by @sryza

  • b9ac2a3 - require get_event_records to have filter arg (#8284) by @prha

  • 5db2182 - Revert "Revert "KVS storage methods (#8249)"" (#8296) by @johannkm

  • 8f43d56 - Allow number type for id in ButtonGroup (#8352) by @salazarm

  • de10b7f - [dagit] Resolve redundant tag buttons on Launchpad (#8353) by @hellendag

  • c32caa0 - test(graphql): ensure type names do not leak graphene (#8345) by @rexledesma

  • 1ed9846 - docs(graphql): ensure descriptions for all mutation fields (#8346) by @rexledesma

  • 426a17a - docs(graphql): ensure descriptions for all query fields (#8347) by @rexledesma

  • 2e50477 - docs(graphql): ensure descriptions for all subscription fields (#8348) by @rexledesma

  • d8992f3 - docs(graphql): ensure descriptions for all types with opt-in enforcement (#8349) by @rexledesma

  • 2e46b28 - fix 0.14.20 versioned_navigation entry (#8360) by @yuhan

  • 233575a - Event log methods for event log consumer (#8253) by @johannkm

  • ff22edd - [docs] - Clarify explanation of graph-backed assets (#8328) by @sryza

  • 227a336 - AssetOut (#8359) by @sryza

  • 8b3770c - [docs] - Update IO Manager docs to contain Asset IO management (#8337) by @clairelin135

  • 511e5ec - Auto run reexecution daemon for oss (#8254) by @johannkm

  • 13f7f16 - Toggle to enable auto run reexecution daemon (#8277) by @johannkm

  • 8d5806e - [with_resources changes 1/n] with_resources docstring, config argument (#8322) by @dpeng817

  • fc6fdeb - Update k8s docs to reflect new includeConfigInLaunchedRuns default (#8361) by @gibsondan

  • d954334 - Simplify execute_in_process result (#8365) by @gibsondan

  • 142cd80 - [docs] - Graph-backed asset examples (#8339) by @clairelin135

  • 56fa984 - Run retries helm config (#8369) by @johannkm

  • c212e85 - [with-resources changes 2/n] If resources collide when using with_resources, error. (#8330) by @dpeng817

  • 8856374 - AssetsDefinition property name changes (#8317) by @sryza

  • d679440 - pass on partitions page for assets (#8355) by @sryza

  • 17ba710 - Type annotations for IOManager (#8308) by @smackesey

  • 3c618ed - materialize_in_process method (#8364) by @dpeng817

  • a661b5f - [assets] assorted bugfixes (#8372) by @OwenKephart

  • 42d4903 - [docs] - Update repository concept doc with asset info (#8373) by @smackesey

  • a20b02b - asset apiref improvements (#8374) by @sryza

  • c97ddaa - add custom k8s labels to dagster k8s jobs, not just dagster k8s pods (#8381) by @gibsondan

  • a7e75eb - asset apiref improvements (#8374) by @sryza

  • b5106af - Merge branch 'master' of https://github.com/dagster-io/dagster into release-0.15.0 by @clairelin135

  • 662f47a - [dagit] Update the linking between asset groups + jobs (#8377) by @bengotow

  • 0889686 - [docs] - Asset grouping [CON-295] (#8375) by @erinkcochran87

  • 13b8735 - [docs] - New Jobs & Graphs structure [CON-289] (#8035) by @erinkcochran87

  • fd3eae4 - fix docs lint (#8387) by @gibsondan

  • 2a2653e - Patch op definition resolution for assets in GQL (#8384) by @smackesey

  • 14696ae - Run retries docs (#8367) by @johannkm

  • e03eefe - Make asset invocation error if resources conflict (#8390) by @dpeng817

  • a60e889 - materialize_in_process -> materialize_to_memory (#8391) by @dpeng817

  • db08e07 - remove old asset lineage from docs (#8382) by @sryza

  • 531e5da - Fix problems from "remove old asset lineage from docs (#8382)" (#8389) by @sryza

  • bc1fa3e - apidoc for AssetOut and AssetIn (#8388) by @sryza

  • 3f051e4 - 0.15.0 by @elementl-devtools


0.14.21rc0

4 months ago

0.14.20

4 months ago
New
  • [dagster-aws] Added an env_vars field to the EcsRunLauncher that allows you to configure environment variables in the ECS task for launched runs.
  • [dagster-k8s] The env_vars field on K8sRunLauncher and k8s_job_executor can now except input of the form ENV_VAR_NAME=ENV_VAR_VALUE, and will set the value of ENV_VAR_NAME to ENV_VAR_VALUE. Previously, it only accepted input of the form ENV_VAR_NAME, and the environment variable had to be available in the pod launching the job.
  • [dagster-k8s] setting ‘includeConfigInLaunchedRuns’ on a user code deployment will now also include any image pull secrets from the user code deployment in the pod for the launched runs.
Bugfixes
  • A recent change had made it so that, when IOManager.load_input was called to load an asset that was not being materialized as part of the run, the provided context would not include the metadata for that asset. context.upstream_output.metadata now correctly returns the metadata on the upstream asset.
  • Fixed an issue where using generic type aliases introduced in Python 3.9 (like list[str]) as the type of an input would raise an exception.
  • [dagster-k8s] Fixed an issue where upgrading the Helm chart version without upgrading your user code deployment version would result in an “Received unexpected config entry "scheme" at path root:postgres_db" error.
All Changes

https://github.com/dagster-io/dagster/compare/0.14.19...0.14.20

See All Contributors
  • b6cf5d2 - 0.14.20rc0 by @elementl-devtools

  • d421a1b - Fixes for asset graphql tests to make them easier to generalize and call with other graphql context fixtures (#8199) by @gibsondan

  • 58fa8ca - bring back source asset metadata (#8195) by @sryza

  • 561b81a - fix is_subclass bug where issubclass(list[str], DagsterType) throws a surprising exception (#8287) by @gibsondan

  • de28811 - Add max_retries to user editable tags (#8285) by @johannkm

  • 37d0062 - Derive origin from pipeline run instead of the arg to ExecuteRunArgs (#8156) by @gibsondan

  • cb800cc - add an "environment" key to EcsRunLauncher / EcsContainerContext that sets env vars (#8243) by @gibsondan

  • ed1b9b0 - Fix upgrade story for new scheme field in postgres (#8220) by @gibsondan

  • 455fb97 - Add image pull secrets to k8s container context (#8221) by @gibsondan

  • d1319e2 - extract event connection to top-level graphql query (#8077) by @prha

  • 5e2badf - Create unified storage configuration for dagster.yaml (#7283) by @prha

  • 601a9cb - [dagster-dbt] Fix bug where a dbt invocation that did not successfully start could emit materialization events. (#8293) by @OwenKephart

  • aa5d8ca - Move config editor to core so we can reuse in admin portal (#8237) by @salazarm

  • 8f9294a - Make dagster/reexecution_strategy tag editable (#8286) by @johannkm

  • 2a8d755 - only return config field for solids (#8278) by @clairelin135

  • 5f13fcd - clean up create_pg_connection (#8165) by @alangenfeld

  • b6c65e4 - 0.14.20 Changelog (#8301) by @johannkm

  • a44bb67 - 0.14.20 by @elementl-devtools


0.14.20rc0

4 months ago

0.14.19

4 months ago
New
  • Metadata can now be added to jobs (via the metadata parameter) and viewed in dagit. You can use it to track code owners, link to docs, or add other useful information.
  • In the Dagit launchpad, the panel below the config editor now shows more detailed information about the state of the config, including error state and whether the config requires further scaffolding or the removal of extra config.
  • FileCache is now marked for deprecation in 0.15.0.
  • In Dagit, the asset catalog now shows the last materialization for each asset and links to the latest run.
  • Assets can now have a config_schema. If you attempt to materialize an asset with a config schema in Dagit, you'll be able to enter the required config via a modal.
Bugfixes
  • [helm] Fixed an issue where string floats and integers were not properly templated as image tags.
  • [dagster-k8s] Fixed an issue when using the k8s_job_executor where ops with long names sometimes failed to create a pod due to a validation error with the label names automatically generated by Dagster.
  • [dagster-aws] Fixed an issue where ECS tasks with large container contexts would sometimes fail to launch because their request to the ECS RunTask API was too large.
Breaking Changes
  • fs_asset_io_manager has been removed in favor of merging its functionality with fs_io_manager. fs_io_manager is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key.
Community Contributions
  • Fixed a bug that broke the k8s_job_executor’s max_conccurent configuration. Thanks @fahadkh!
  • Fixed a bug that caused the fs_io_manager to incorrectly handle assets associated with upstream assets. Thanks @aroig!
Documentation
  • [helm] Add documentation for code server image pull secrets in the main chart.
  • The Dagster README has been revamped with documentation and community links.
All Changes

https://github.com/dagster-io/dagster/compare/0.14.17...0.14.19


0.14.18rc6

4 months ago

0.14.18rc5

4 months ago

0.14.18rc4

4 months ago

0.14.18rc3

4 months ago

0.14.18rc2

4 months ago

0.14.17

4 months ago
New
  • Added a pin to protobuf version 3 due to a backwards incompatible change in the probobuf version 4 release.
  • [helm] The name of the Dagit deployment can now be overridden in the Dagster Helm chart.
  • [dagit] The left navigation now shows jobs as expandable lists grouped by repository. You can opt out of this change using the feature flag in User Settings.
  • [dagit] In the left navigation, when a job has more than one schedule or sensor, clicking the schedule/sensor icon will now display a dialog containing the full list of schedules and sensors for that job.
  • [dagit] Assets on the runs page are now shown in more scenarios.
  • [dagster-dbt] dbt assets now support subsetting! In dagit, you can launch off a dbt command which will only refresh the selected models, and when you’re building jobs using AssetGroup.build_job(), you can define selections which select subsets of the loaded dbt project.
  • [dagster-dbt][experimental] The load_assets_from_dbt_manifest function now supports an experimental select parameter. This allows you to use dbt selection syntax to select from an existing manifest.json file, rather than having Dagster re-compile the project on demand.
  • For software-defined assets, OpExecutionContext now exposes an asset_key_for_output method, which returns the asset key that one of the op’s outputs corresponds too.
  • The Backfills tab in Dagit loads much faster when there have been backfills that produced large numbers of runs.
  • Added the ability to run the Dagster Daemon as a Python module, by running python -m dagster.daemon.
  • The non_argument_deps parameter for the asset and multi_asset decorators can now be a set of strings in addition to a set of AssetKey.
Bugfixes
  • [dagit] In cases where Dagit is unable to make successful WebSocket connections, run logs could become stuck in a loading state. Dagit will now time out on the WebSocket connection attempt after a brief period of time. This allows run logs to fall back to http requests and move past the loading state.
  • In version 0.14.16, launching an asset materialization run with source assets would error with an InvalidSubsetError. This is now fixed.
  • Empty strings are no longer allowed as AssetKeys.
  • Fixed an issue where schedules built from partitioned job config always ran at midnight, ignoring any hour or minute offset that was specified on the config.
  • Fixed an issue where if the scheduler was interrupted and resumed in the middle of running a schedule tick that produced multiple RunRequests, it would show the same run ID multiple times on the list of runs for the schedule tick.
  • Fixed an issue where Dagit would raise a GraphQL error when a non-dictionary YAML string was entered into the Launchpad.
  • Fixed an issue where Dagster gRPC servers would sometimes raise an exception when loading repositories with many partition sets.
  • Fixed an issue where the snowflake_io_manager would sometimes raise an error with pandas 1.4 or later installed.
  • Fixed an issue where re-executing an entire set of dynamic steps together with their upstream step resulted in DagsterExecutionStepNotFoundError. This is now fixed.
  • [dagit] Added loading indicator for job-scoped partition backfills.
  • Fixed an issue that made it impossible to have graph-backed assets with upstream SourceAssets.
Community Contributions
  • AssetIn can now accept a string that will be coerced to an AssetKey. Thanks @aroig!
  • Runtime type checks improved for some asset-related functions. Thanks @aroig!
  • Docs grammar fixes. Thanks @dwinston!
  • Dataproc ops for dagster-gcp now have user-configurable timeout length. Thanks @3cham!
All Changes

https://github.com/dagster-io/dagster/compare/0.14.16...0.14.17

See All Contributors

0.14.17rc10

4 months ago

0.14.17rc8

4 months ago

0.14.17rc7

4 months ago

0.14.17rc6

4 months ago

0.14.17rc5

4 months ago

0.14.17rc4

4 months ago

0.14.17rc3

4 months ago

0.14.16

4 months ago
New
  • AssetsDefinition.from_graph now accepts a partitions_def argument.
  • @asset-decorated functions can now accept variable keyword arguments.
  • Jobs executed in ECS tasks now report the health status of the ECS task
  • The CLI command dagster instance info now prints the current schema migration state for the configured instance storage.
  • [dagster-dbt] You can now configure a docs_url on the dbt_cli_resource. If this value is set, AssetMaterializations associated with each dbt model will contain a link to the dbt docs for that model.
  • [dagster-dbt] You can now configure a dbt_cloud_host on the dbt_cloud_resource, in the case that your dbt cloud instance is under a custom domain.
Bugfixes
  • Fixed a bug where InputContext.upstream_output was missing the asset_key when it referred to an asset outside the run.
  • When specifying a selection parameter in AssetGroup.build_job(), the generated job would include an incorrect set of assets in certain situations. This has been fixed.
  • Previously, a set of database operational exceptions were masked with a DagsterInstanceSchemaOutdated exception if the instance storage was not up to date with the latest schema. We no longer wrap these exceptions, allowing the underlying exceptions to bubble up.
  • [dagster-airbyte] Fixed issue where successfully completed Airbyte syncs would send a cancellation request on completion. While this did not impact the sync itself, if alerts were set up on that connection, they would get triggered regardless of if the sync was successful or not.
  • [dagster-azure] Fixed an issue where the Azure Data Lake Storage adls2_pickle_io_manager would sometimes fail to recursively delete a folder when cleaning up an output.
  • Previously, if two different jobs with the same name were provided to the same repo, and one was targeted by a sensor/schedule, the job provided by the sensor/schedule would silently overwrite the other job instead of failing. In this release, a warning is fired when this case is hit, which will turn into an error in 0.15.0.
  • Dagit will now display workspace errors after reloading all repositories.
Breaking Changes
  • Calls to instance.get_event_records without an event type filter is now deprecated and will generate a warning. These calls will raise an exception starting in 0.15.0.
Community Contributions
  • @multi_asset now supports partitioning. Thanks @aroig!
  • Orphaned process detection now works correctly across a broader set of platforms. Thanks @aroig!
  • [K8s] Added a new max_concurrent field to the k8s_job_executor that limits the number of concurrent Ops that will execute per run. Since this executor launches a Kubernetes Job per Op, this also limits the number of concurrent Kuberenetes Jobs. Note that this limit is per run, not global. Thanks @kervel!
  • [Helm] Added a new externalConfigmap field as an alternative to dagit.workspace.servers when running the user deployments chart in a separate release. This allows the workspace to be managed outside of the main Helm chart. Thanks @peay!
  • Removed the pin on markupsafe<=2.0.1. Thanks @bollwyvl!

0.14.16rc4

4 months ago

0.14.16rc3

4 months ago

0.14.16rc2

4 months ago

0.14.15

5 months ago
New
  • Sensors / schedules can now return a list of RunRequest objects instead of yielding them.
  • Repositories can now contain asset definitions and source assets for the same asset key.
  • OpExecutionContext (provided as the context argument to Ops) now has fields for, run, job_def, job_name, op_def, and op_config. These replace pipeline_run, pipeline_def, etc. (though they are still available).
  • When a job is partitioned using an hourly, daily, weekly, or monthly partitions definition, OpExecutionContext now offers a partition_time_window attribute, which returns a tuple of datetime objects that mark the bounds of the partition’s time window.
  • AssetsDefinition.from_graph now accepts a partitions_def argument.
  • [dagster-k8s] Removed an unnecessary dagster-test-connection pod from the Dagster Helm chart.
  • [dagster-k8s] The k8s_job_executor now polls the event log on a ~1 second interval (previously 0.1). Performance testing showed that this reduced DB load while not significantly impacting run time.
  • [dagit] Removed package pins for Jinja2 and nbconvert.
  • [dagit] When viewing a list of Runs, tags with information about schedules, sensors, and backfills are now more visually prominent and are sorted to the front of the list.
  • [dagit] The log view on Run pages now includes a button to clear the filter input.
  • [dagit] When viewing a list of Runs, you can now hover over a tag to see a menu with an option to copy the tag, and in filtered Run views, an option to add the tag to the filter.
  • [dagit] Configuration editors throughout Dagit now display clear indentation guides, and our previous whitespace indicators have been removed.
  • [dagit] The Dagit Content-Security-Policy has been moved from a <meta> tag to a response header, and several more security and privacy related headers have been added as well.
  • [dagit] Assets with multi-component key paths are always shown as foo/bar in dagit, rather than appearing as foo > bar in some contexts.
  • [dagit] The Asset graph now includes a “Reload definitions” button which reloads your repositories.
  • [dagit] On all DAGs, you can hold shift on the keyboard to switch from mouse wheel / touch pad zooming to panning. This makes it much easier to scroll horizontally at high speed without click-drag-click-drag-click-drag.
  • [dagit] a --log-level flag is now available in the dagit cli for controlling the uvicorn log level.
  • [dagster-dbt] The load_assets_from_dbt_project() and load_assets_from_dbt_manifest() utilities now have a use_build_command parameter. If this flag is set, when materializing your dbt assets, Dagster will use the dbt build command instead of dbt run. Any tests run during this process will be represented with AssetObservation events attached to the relevant assets. For more information on dbt build, see the dbt docs.
  • [dagster-dbt] If a dbt project successfully runs some models and then fails, AssetMaterializations will now be generated for the successful models.
  • [dagster-snowflake] The new Snowflake IO manager, which you can create using build_snowflake_io_manager offers a way to store assets and op outputs in Snowflake. The PandasSnowflakeTypeHandler stores Pandas DataFrames in Snowflake.
  • [helm] dagit.logLevel has been added to values.yaml to access the newly added dagit --log-level cli option.
Bugfixes
  • Fixed incorrect text in the error message that’s triggered when building a job and an asset can’t be found that corresponds to one of the asset dependencies.
  • An error is no longer raised when an op/job/graph/other definition has an empty docstring.
  • Fixed a bug where pipelines could not be executed if toposort<=1.6 was installed.
  • [dagit] Fixed an issue in global search where rendering and navigation broke when results included objects of different types but with identical names.
  • [dagit] server errors regarding websocket send after close no longer occur.
  • [dagit] Fixed an issue where software-defined assets could be rendered improperly when the dagster and dagit versions were out of sync.
Community Contributions
  • [dagster-aws] PickledObjectS3IOManager now uses list_objects to check the access permission. Thanks @trevenrawr!
Breaking Changes
  • [dagster-dbt] The asset definitions produced by the experimental load_assets_from_dbt_project and load_assets_from_dbt_manifest functions now include the schemas of the dbt models in their asset keys. To revert to the old behavior: dbt_assets = load_assets_from_dbt_project(..., node_info_to_asset_key=lambda node_info: AssetKey(node_info["name"]).
Experimental
  • The TableSchema API is no longer experimental.
Documentation
  • Docs site now has a new design!
  • Concepts pages now have links to code snippets in our examples that use those concepts.

0.14.14

5 months ago
New
  • When viewing a config schema in the Dagit launchpad, default values are now shown. Hover over an underlined key in the schema view to see the default value for that key.

  • dagster, dagit, and all extension libraries (dagster-*) now contain py.typed files. This exposes them as typed libraries to static type checking tools like mypy. If your project is using mypy or another type checker, this may surface new type errors. For mypy, to restore the previous state and treat dagster or an extension library as untyped (i.e. ignore Dagster’s type annotations), add the following to your configuration file:

    [mypy-dagster]  (or e.g. mypy-dagster-dbt)
    follow_imports = "skip"
    
  • Op retries now surface the underlying exception in Dagit.

  • Made some internal changes to how we store schema migrations across our different storage implementations.

  • build_output_context now accepts an asset_key argument.

  • They key argument to the SourceAsset constructor now accepts values that are strings or sequences of strings and coerces them to AssetKeys.

  • You can now use the + operator to add two AssetGroups together, which forms an AssetGroup that contains a union of the assets in the operands.

  • AssetGroup.from_package_module, from_modules, from_package_name, and from_current_module now accept an extra_source_assets argument that includes a set of source assets into the group in addition to the source assets scraped from modules.

  • AssetsDefinition and AssetGroup now both expose a to_source_assets method that return SourceAsset versions of their assets, which can be used as source assets for downstream AssetGroups.

  • Repositories can now include multiple AssetGroups.

  • The new prefixed method on AssetGroup returns a new AssetGroup where a given prefix is prepended to the asset key of every asset in the group.

  • Dagster now has a BoolMetadataValue representing boolean-type metadata. Specifying True or False values in metadata will automatically be casted to the boolean type.

  • Tags on schedules can now be expressed as nested JSON dictionaries, instead of requiring that all tag values are strings.

  • If an exception is raised during an op, Dagster will now always run the failure hooks for that op. Before, certain system exceptions would prevent failure hooks from being run.

  • mapping_key can now be provided as an argument to build_op_context/build_solid_context. Doing so will allow the use of OpExecutionContext.get_mapping_key().

Bugfixes
  • [dagit] Previously, when viewing a list of an asset’s materializations from a specified date/time, a banner would always indicate that it was a historical view. This banner is no longer shown when viewing the most recent materialization.
  • [dagit] Special cron strings like @daily were treated as invalid when converting to human-readable strings. These are now handled correctly.
  • The selection argument to AssetGroup.build_job now uses > instead of . for delimiting the components within asset keys, which is consistent with how selection works in Dagit.
  • [postgres] passwords and usernames are now correctly url quoted when forming a connection string. Previously spaces were replaced with +.
  • Fixed an issue where the celery_docker_executor would sometimes fail to execute with a JSON deserialization error when using Dagster resources that write to stdout.
  • [dagster-k8s] Fixed an issue where the Helm chart failed to work when the user code deployment subchart was used in a different namespace than the main dagster Helm chart, due to missing configmaps.
  • [dagster-airbyte] When a Dagster run is terminated while executing an Airbyte sync operation, the corresponding Airbyte sync will also be terminated.
  • [dagster-dbt] Log output from dbt cli commands will no longer have distracting color-formatting characters.
  • [dagit] Fixed issue where multi_assets would not show correct asset dependency information.
  • Fixed an issue with the sensor daemon, where the sensor would sometimes enter a race condition and overwrite the sensor status.
Community Contributions
  • [dagster-graphql] The Python DagsterGraphQLClient now supports terminating in-progress runs using client.terminate_run(run_id). Thanks @Javier162380!
Experimental
  • Added an experimental view of the Partitions page / Backfill page, gated behind a feature flag in Dagit.

0.14.13

5 months ago
New
  • [dagster-k8s] You can now specify resource requests and limits to the K8sRunLauncher when using the Dagster helm chart, that will apply to all runs. Before, you could only set resource configuration by tagging individual jobs. For example, you can set this config in your values.yaml file:
runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      resources:
        limits:
          cpu: 100m
          memory: 128Mi
        requests:
          cpu: 100m
          memory: 128Mi
  • [dagster-k8s] Specifying includeConfigInLaunchedRuns: true in a user code deployment will now launch runs using the same namespace and service account as the user code deployment.
  • The @asset decorator now accepts an op_tags argument, which allows e.g. providing k8s resource requirements on the op that computes the asset.
  • Added CLI output to dagster api grpc-health-check (previously it just returned via exit codes)
  • [dagster-aws] The emr_pyspark_step_launcher now supports dynamic orchestration, RetryPolicys defined on ops, and re-execution from failure. For failed steps, the stack trace of the root error will now be available in the event logs, as will logs generated with context.log.info.
  • Partition sets and can now return a nested dictionary in the tags_fn_for_partition function, instead of requiring that the dictionary have string keys and values.
  • [dagit] It is now possible to perform bulk re-execution of runs from the Runs page. Failed runs can be re-executed from failure.
  • [dagit] Table headers are now sticky on Runs and Assets lists.
  • [dagit] Keyboard shortcuts may now be disabled from User Settings. This allows users with certain keyboard layouts (e.g. QWERTZ) to inadvertently avoid triggering unwanted shortcuts.
  • [dagit] Dagit no longer continues making some queries in the background, improving performance when many browser tabs are open.
  • [dagit] On the asset graph, you can now filter for multi-component asset keys in the search bar and see the “kind” tags displayed on assets with a specified compute_kind.
  • [dagit] Repositories are now displayed in a stable order each time you launch Dagster.
Bugfixes
  • [dagster-k8s] Fixed an issue where the Dagster helm chart sometimes failed to parse container images with numeric tags. Thanks @jrouly!
  • [dagster-aws] The EcsRunLauncher now registers new task definitions if the task’s execution role or task role changes.
  • Dagster now correctly includes setuptools as a runtime dependency.
  • In can now accept asset_partitions without crashing.
  • [dagit] Fixed a bug in the Launchpad, where default configuration failed to load.
  • [dagit] Global search now truncates the displayed list of results, which should improve rendering performance.
  • [dagit] When entering an invalid search filter on Runs, the user will now see an appropriate error message instead of a spinner and an alert about a GraphQL error.
Documentation
  • Added documentation for partitioned assets
  • [dagster-aws] Fixed example code of a job using secretsmanager_resource.

0.14.12

5 months ago
Bugfixes
  • Fixed an issue where the Launchpad in Dagit sometimes incorrectly launched in an empty state.

0.14.11

5 months ago
Bugfixes
  • Fixed an issue where schedules created from partition sets that launched runs for multiple partitions in a single schedule tick would sometimes time out while generating runs in the scheduler.
  • Fixed an issue where nested graphs would sometimes incorrectly determine the set of required resources for a hook.

0.14.10

5 months ago
New
  • [dagster-k8s] Added an includeConfigInLaunchedRuns flag to the Helm chart that can be used to automatically include configmaps, secrets, and volumes in any runs launched from code in a user code deployment. See https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment for more information.
  • [dagit] Improved display of configuration yaml throughout Dagit, including better syntax highlighting and the addition of line numbers.
  • The GraphQL input argument type BackfillParams (used for launching backfills), now has an allPartitions boolean flag, which can be used instead of specifying all the individual partition names.
  • Removed gevent and gevent-websocket dependencies from dagster-graphql
  • Memoization is now supported while using step selection
  • Cleaned up various warnings across the project
  • The default IO Managers now support asset partitions
Bugfixes
  • Fixed sqlite3.OperationalError error when viewing schedules/sensors pages in Dagit. This was affecting dagit instances using the default SQLite schedule storage with a SQLite version < 3.25.0.
  • Fixed a bug with type-checking of the asset_partitions arg passed to In
  • Fixed an issues where schedules and sensors would sometimes fail to run when the daemon and dagit were running in different Python environments.
  • Fixed an exception when the telemetry file is empty
  • fixed a bug with @graph composition which would cause the wrong input definition to be used for type checks
  • [dagit] For users running Dagit with --path-prefix, large DAGs failed to render due to a WebWorker error, and the user would see an endless spinner instead. This has been fixed.
  • [dagit] Fixed a rendering bug in partition set selector dropdown on Launchpad.
  • [dagit] Fixed the ‘View Assets’ link in Job headers
  • Fixed an issue where root input managers with resource dependencies would not work with software defined assets
Community Contributions
  • dagster-census is a new library that includes a census_resource for interacting the Census REST API, census_trigger_sync_op for triggering a sync and registering an asset once it has finished, and a CensusOutput type. Thanks @dehume!
  • Docs fix. Thanks @ascrookes!

0.14.9

5 months ago
New
  • Added a parameter in dagster.yaml that can be used to increase the time that Dagster waits when spinning up a gRPC server before timing out. For more information, see https://docs.dagster.io/deployment/dagster-instance#code-servers.
  • Added a new graphQL field assetMaterializations that can be queried off of a DagsterRun field. You can use this field to fetch the set of asset materialization events generated in a given run within a GraphQL query.
  • Docstrings on functions decorated with the @resource decorator will now be used as resource descriptions, if no description is explicitly provided.
  • You can now point dagit -m or dagit -f at a module or file that has asset definitions but no jobs or asset groups, and all the asset definitions will be loaded into Dagit.
  • AssetGroup now has a materialize method which executes an in-process run to materialize all the assets in the group.
  • AssetGroups can now contain assets with different partition_defs.
  • Asset materializations produced by the default asset IO manager, fs_asset_io_manager, now include the path of the file where the values were saved.
  • You can now disable the max_concurrent_runs limit on the QueuedRunCoordinator by setting it to -1. Use this if you only want to limit runs using tag_concurrency_limits.
  • [dagit] Asset graphs are now rendered asynchronously, which means that Dagit will no longer freeze when rendering a large asset graph.
  • [dagit] When viewing an asset graph, you can now double-click on an asset to zoom in, and you can use arrow keys to navigate between selected assets.
  • [dagit] The “show whitespace” setting in the Launchpad is now persistent.
  • [dagit] A bulk selection checkbox has been added to the repository filter in navigation or Instance Overview.
  • [dagit] A “Copy config” button has been added to the run configuration dialog on Run pages.
  • [dagit] An “Open in Launchpad” button has been added to the run details page.
  • [dagit] The Run page now surfaces more information about start time and elapsed time in the header.
  • [dagster-dbt] The dbt_cloud_resource has a new get_runs() function to get a list of runs matching certain paramters from the dbt Cloud API (thanks @kstennettlull!)
  • [dagster-snowflake] Added an authenticator field to the connection arguments for the snowflake_resource (thanks @swotai!).
  • [celery-docker] The celery docker executor has a new configuration entry container_kwargs that allows you to specify additional arguments to pass to your docker containers when they are run.
Bugfixes
  • Fixed an issue where loading a Dagster repository would fail if it included a function to lazily load a job, instead of a JobDefinition.
  • Fixed an issue where trying to stop an unloadable schedule or sensor within Dagit would fail with an error.
  • Fixed telemetry contention bug on windows when running the daemon.
  • [dagit] Fixed a bug where the Dagit homepage would claim that no jobs or pipelines had been loaded, even though jobs appeared in the sidebar.
  • [dagit] When filtering runs by tag, tag values that contained the : character would fail to parse correctly, and filtering would therefore fail. This has been fixed.
  • [dagster-dbt] When running the “build” command using the dbt_cli_resource, the run_results.json file will no longer be ignored, allowing asset materializations to be produced from the resulting output.
  • [dagster-airbyte] Responses from the Airbyte API with a 204 status code (like you would get from /connections/delete) will no longer produce raise an error (thanks @HAMZA310!)
  • [dagster-shell] Fixed a bug where shell ops would not inherit environment variables if any environment variables were added for ops (thanks @kbd!)
  • [dagster-postgres] usernames are now urlqouted in addition to passwords
Documentation

0.14.9rc0

5 months ago

0.14.8

6 months ago
New
  • The MySQL storage implementations for Dagster storage is no longer marked as experimental.
  • run_id can now be provided as an argument to execute_in_process.
  • The text on dagit’s empty state no longer mentions the legacy concept “Pipelines”.
  • Now, within the IOManager.load_input method, you can add input metadata via InputContext.add_input_metadata. These metadata entries will appear on the LOADED_INPUT event and if the input is an asset, be attached to an AssetObservation. This metadata is viewable in dagit.
Bugfixes
  • Fixed a set of bugs where schedules and sensors would get out of sync between dagit and dagster-daemon processes. This would manifest in schedules / sensors getting marked as “Unloadable” in dagit, and ticks not being registered correctly. The fix involves changing how Dagster stores schedule/sensor state and requires a schema change using the CLI command dagster instance migrate. Users who are not running into this class of bugs may consider the migration optional.
  • root_input_manager can now be specified without a context argument.
  • Fixed a bug that prevented root_input_manager from being used with VersionStrategy.
  • Fixed a race condition between daemon and dagit writing to the same telemetry logs.
  • [dagit] In dagit, using the “Open in Launchpad” feature for a run could cause server errors if the run configuration yaml was too long. Runs can now be opened from this feature regardless of config length.
  • [dagit] On the Instance Overview page in dagit, runs in the timeline view sometimes showed incorrect end times, especially batches that included in-progress runs. This has been fixed.
  • [dagit] In the dagit launchpad, reloading a repository should present the user with an option to refresh config that may have become stale. This feature was broken for jobs without partition sets, and has now been fixed.
  • Fixed issue where passing a stdlib typing type as dagster_type to input and output definition was incorrectly being rejected.
  • [dagster-airbyte] Fixed issue where AssetMaterialization events would not be generated for streams that had no updated records for a given sync.
  • [dagster-dbt] Fixed issue where including multiple sets of dbt assets in a single repository could cause a conflict with the names of the underlying ops.

0.14.8rc1

6 months ago

0.14.7

6 months ago
New
  • [helm] Added configuration to explicitly enable or disable telemetry.
  • Added a new IO manager for materializing assets to Azure ADLS. You can specify this IO manager for your AssetGroups by using the following config:
from dagster import AssetGroup
from dagster_azure import adls2_pickle_asset_io_manager, adls2_resource
asset_group = AssetGroup(
    [upstream_asset, downstream_asset],
    resource_defs={"io_manager": adls2_pickle_asset_io_manager, "adls2": adls2_resource}
)
  • Added ability to set a custom start time for partitions when using @hourly_partitioned_config , @daily_partitioned_config, @weekly_partitioned_config, and @monthly_partitioned_config
  • Run configs generated from partitions can be retrieved using the PartitionedConfig,get_partition_for_partition_key function. This will allow the use of the validate_run_config function in unit tests.
  • [dagit] If a run is re-executed from failure, and the run fails again, the default action will be to re-execute from the point of failure, rather than to re-execute the entire job.
Bugfixes
  • Fixed a bug in the message for reporting Kubernetes run worker failures
  • [dagit] Fixed issue where re-executing a run that materialized a single asset could end up re-executing all steps in the job.
  • [dagit] Fixed issue where the health of an asset’s partitions would not always be up to date in certain views.
  • [dagit] Fixed issue where the “Materialize All” button would be greyed out if a job had SourceAssets defined.
Documentation
  • Updated resource docs to reference “ops” instead of “solids” (thanks @joe-hdai!)
  • Fixed formatting issues in the ECS docs

0.14.7rc0

6 months ago

0.14.6

6 months ago
New
  • Added IO manager for materializing assets to GCS. You can specify the GCS asset IO manager by using the following config for resource_defs in AssetGroup:
from dagster import AssetGroup, gcs_pickle_asset_io_manager, gcs_resource
asset_group = AssetGroup(
    [upstream_asset, downstream_asset],
    resource_defs={"io_manager": gcs_pickle_asset_io_manager, "gcs": gcs_resource}
)
  • Improved the performance of storage queries run by the sensor daemon to enforce the idempotency of run keys. This should reduce the database CPU when evaluating sensors with a large volume of run requests with run keys that repeat across evaluations.
  • [dagit] Added information on sensor ticks to show when a sensor has requested runs that did not result in the creation of a new run due to the enforcement of idempotency using run keys.
  • [k8s] Run and step workers are now labeled with the Dagster run id that they are currently handling.
  • If a step launched with a StepLauncher encounters an exception, that exception / stack trace will now appear in the event log.
Bugfixes
  • Fixed a race condition where canceled backfills would resume under certain conditions.
  • Fixed an issue where exceptions that were raised during sensor and schedule execution didn’t always show a stack trace in Dagit.
  • During execution, dependencies will now resolve correctly for certain dynamic graph structures that were previously resolving incorrectly.
  • When using the forkserver start_method on the multiprocess executor, preload_modules have been adjusted to prevent libraries that change namedtuple serialization from causing unexpected exceptions.
  • Fixed a naming collision between dagster decorators and submodules that sometimes interfered with static type checkers (e.g. pyright).
  • [dagit] postgres database connection management has improved when watching actively executing runs
  • [dagster-databricks] The databricks_pyspark_step_launcher now supports steps with RetryPolicies defined, as well as RetryRequested exceptions.
Community Contributions
  • Docs spelling fixes - thanks @antquinonez!

0.14.6rc0

6 months ago

0.14.5

6 months ago
0.14.5 Bugfixes
  • [dagit] Fixed issue where sensors could not be turned on/off in dagit.
  • Fixed a bug with direct op invocation when used with funcsigs.partial that would cause incorrect InvalidInvocationErrors to be thrown.
  • Internal code no longer triggers deprecation warnings for all runs.

0.14.5rc0

6 months ago

0.14.4

6 months ago
New
  • Dagster now supports non-standard vixie-style cron strings, like @hourly, @daily, @weekly, and @monthly in addition to the standard 5-field cron strings (e.g. * * * * *).
  • value is now an alias argument of entry_data (deprecated) for the MetadataEntry constructor.
  • Typed metadata can now be attached to SourceAssets and is rendered in dagit.
  • When a step fails to upload its compute log to Dagster, it will now add an event to the event log with the stack trace of the error instead of only logging the error to the process output.
  • [dagit] Made a number of improvements to the Schedule/Sensor pages in Dagit, including showing a paginated table of tick information, showing historical cursor state, and adding the ability to set a cursor from Dagit. Previously, we only showed tick information on the timeline view and cursors could only be set using the dagster CLI.
  • [dagit] When materializing assets, Dagit presents a link to the run rather than jumping to it, and the status of the materialization (pending, running, failed) is shown on nodes in the asset graph.
  • [dagit] Dagit now shows sensor and schedule information at the top of asset pages based on the jobs in which the asset appears.
  • [dagit] Dagit now performs "middle truncation" on gantt chart steps and graph nodes, making it much easier to differentiate long assets and ops.
  • [dagit] Dagit no longer refreshes data when tabs are in the background, lowering browser CPU usage.
  • dagster-k8s, dagster-celery-k8s, and dagster-docker now name step workers dagster-step-... rather than dagster-job-....
  • [dagit] The launchpad is significantly more responsive when you're working with very large partition sets.
  • [dagit] We now show an informative message on the Asset catalog table when there are no matching assets to display. Previously, we would show a blank white space.
  • [dagit] Running Dagit without a backfill daemon no longer generates a warning unless queued backfills are present. Similarly, a missing sensor or schedule daemon only yields a warning if sensors or schedules are turned on.
  • [dagit] On the instance summary page, hovering over a recent run’s status dot shows a more helpful tooltip.
  • [dagster-k8s] Improved performance of the k8s_job_executor for runs with many user logs
  • [dagster-k8s] When using the dagster-k8s/config tag to configure Dagster Kubernetes pods, the tags can now accept any valid Kubernetes config, and can be written in either snake case (node_selector_terms) or camel case (nodeSelectorTerms). See the docs for more information.
  • [dagster-aws] You can now set secrets on the EcsRunLauncher using the same syntax that you use to set secrets in the ECS API.
  • [dagster-aws] The EcsRunLauncher now attempts to reuse task definitions instead of registering a new task definition for every run.
  • [dagster-aws] The EcsRunLauncher now raises the underlying ECS API failure if it cannot successfully start a task.
Software-Defined Assets
  • When loading assets from modules using AssetGroup.from_package_name and similar methods, lists of assets at module scope are now loaded.
  • Added the static methods AssetGroup.from_modules and AssetGroup.from_current_module, which automatically load assets at module scope from particular modules.
  • Software-defined assets jobs can now load partitioned assets that are defined outside the job.
  • AssetGraph.from_modules now correctly raises an error if multiple assets with the same key are detected.
  • The InputContext object provided to IOManager.load_input previously did not include resource config. Now it does.
  • Previously, if an assets job had a partitioned asset as well as a non-partitioned asset that depended on another non-partitioned asset, it would fail to run. Now it runs without issue.
  • [dagit] The asset "View Upstream Graph" links no longer select the current asset, making it easier to click "Materialize All".
  • [dagit] The asset page's "partition health bar" highlights missing partitions better in large partition sets.
  • [dagit] The asset "Materialize Partitions" modal now presents an error when partition config or tags cannot be generated.
  • [dagit] The right sidebar of the global asset graph no longer defaults to 0% wide in fresh / incognito browser windows, which made it difficult to click nodes in the global graph.
  • [dagit] In the asset catalog, the search bar now matches substrings so it's easier to find assets with long path prefixes.
  • [dagit] Dagit no longer displays duplicate downstream dependencies on the Asset Details page in some scenarios.
  • [dagster-fivetran] Assets created using build_fivetran_assets will now be properly tagged with a fivetran pill in Dagit.
Bugfixes
  • Fixed issue causing step launchers to fail in many scenarios involving re-execution or dynamic execution.
  • Previously, incorrect selections (generally, step selections) could be generated for strings of the form ++item. This has been fixed.
  • Fixed an issue where run status sensors sometimes logged the wrong status to the event log if the run moved into a different status while the sensor was running.
  • Fixed an issue where daily schedules sometimes produced an incorrect partition name on spring Daylight Savings time boundaries.
  • [dagit] Certain workspace or repo-scoped pages relied on versions of the SQLAlchemy package to be 1.4 or greater to be installed. We are now using queries supported by SQLAlchemy>=1.3. Previously we would raise an error including the message: 'Select' object has no attribute 'filter'.
  • [dagit] Certain workspace or repo-scoped pages relied on versions of sqlite to be 3.25.0 or greater to be installed. This has been relaxed to support older versions of sqlite. This was previously marked as fixed in our 0.14.0 notes, but a handful of cases that were still broken have now been fixed. Previously we would raise an error (sqlite3.OperationalError).
  • [dagit] When changing presets / partitions in the launchpad, Dagit preserves user-entered tags and replaces only the tags inherited from the previous base.
  • [dagit] Dagit no longer hangs when rendering the run gantt chart for certain graph structures.
  • [dagster-airbyte] Fixed issues that could cause failures when generating asset materializations from an Airbyte API response.
  • [dagster-aws] 0.14.3 removed the ability for the EcsRunLauncher to use sidecars without you providing your own custom task definition. Now, you can continue to inherit sidecars from the launching task’s task definition by setting include_sidecars: True in your run launcher config.
Breaking Changes
  • dagster-snowflake has dropped support for python 3.6. The library it is currently built on, snowflake-connector-python, dropped 3.6 support in their recent 2.7.5 release.
Community Contributions
  • MetadataValue.path() and PathMetadataValue now accept os.PathLike objects in addition to strings. Thanks@abkfenris!
  • [dagster-k8s] Fixed configuration of env_vars on the k8s_job_executor. Thanks @kervel!
  • Typo fix on the Create a New Project page. Thanks @frcode!
Documentation
  • Concepts sections added for Op Retries and Dynamic Graphs
  • The Hacker News Assets demo now uses AssetGroup instead of build_assets_job, and it can now be run entirely from a local machine with no additional infrastructure (storing data inside DuckDB).
  • The Software-Defined Assets guide in the docs now uses AssetGroup instead of build_assets_job.

0.14.4rc0

6 months ago

0.14.3

7 months ago
New
  • When using an executor that runs each op in its own process, exceptions in the Dagster system code that result in the op process failing will now be surfaced in the event log.
  • Introduced new SecretsManager resources to the dagster-aws package to enable loading secrets into Jobs more easily. For more information, see the documentation.
  • Daemon heartbeats are now processed in a batch request to the database.
  • Job definitions now contain a method called run_request_for_partition, which returns a RunRequest that can be returned in a sensor or schedule evaluation function to launch a run for a particular partition for that job. See our documentation for more information.
  • Renamed the filter class from PipelineRunsFilter => RunsFilter.
  • Assets can now be directly invoked for unit testing.
  • [dagster-dbt] load_assets_from_dbt_project will now attach schema information to the generated assets if it is available in the dbt project (schema.yml).
  • [examples] Added an example that demonstrates using Software Defined Assets with Airbyte, dbt, and custom Python.
  • The default io manager used in the AssetGroup api is now the fs_asset_io_manager.
  • It's now possible to build a job where partitioned assets depend on partitioned assets that are maintained outside the job, and for those upstream partitions to show up on the context in the op and IOManager load_input function.
  • SourceAssets can now be partitioned, by setting the partitions_def argument.
Bugfixes
  • Fixed an issue where run status sensors would sometimes fire multiple times for the same run if the sensor function raised an error.
  • [ECS] Previously, setting cpu/memory tags on a job would override the ECS task’s cpu/memory, but not individual containers. If you were using a custom task definition that explicitly sets a container’s cpu/memory, the container would not resize even if you resized the task. Now, setting cpu/memory tags on a job overrides both the ECS task’s cpu/memory and the container's cpu/memory.
  • [ECS] Previously, if the EcsRunLauncher launched a run from a task with multiple containers - for example if both dagit and daemon were running in the same task - then the run would be launched with too many containers. Now, the EcsRunLauncher only launches tasks with a single container.
  • Fixed an issue where the run status of job invoked through execute_in_process was not updated properly.
  • Fixed some storage queries that were incompatible with versions of SQLAlchemy<1.4.0.
  • [dagster-dbt] Fixed issue where load_assets_from_dbt_project would fail if models were organized into subdirectories.
  • [dagster-dbt] Fixed issue where load_assets_from_dbt_project would fail if seeds or snapshots were present in the project.
Community Contributions
  • [dagster-fivetran] A new fivetran_resync_op (along with a corresponding resync_and_poll method on the fivetran_resource) allows you to kick off Fivetran resyncs using Dagster (thanks @dwallace0723!)

  • [dagster-shell] Fixed an issue where large log output could cause operations to hang (thanks @kbd!)

  • [documentation] Fixed export message with dagster home path (thanks @proteusiq)!

  • [documentation] Remove duplicate entries under integrations (thanks @kahnwong)!

UI
  • Added a small toggle to the right of each graph on the asset details page, allowing them to be toggled on and off.
  • Full asset paths are now displayed on the asset details page.
Documentation
  • Added API doc entries for validate_run_config.
  • Fixed the example code for the reexecute_pipeline API.
  • TableRecord, TableSchema and its constituents are now documented in the API docs.
  • Docs now correctly use new metadata names MetadataEntry and MetadataValue instead of old ones.

0.14.3rc0

7 months ago

0.14.2

7 months ago
New
  • Run status sensors can now be invoked in unit tests. Added build_run_sensor_status_context to help build context objects for run status sensors
Bugfixes
  • An issue preventing the use of default_value on inputs has been resolved. Previously, a defensive error that did not take default_value in to account was thrown.
  • [dagster-aws] Fixed issue where re-emitting log records from the pyspark_step_launcher would occasionally cause a failure.
  • [dagit] The asset catalog now displays entries for materialized assets when only a subset of repositories were selected. Previously, it only showed the software-defined assets unless all repositories were selected in Dagit.
Community Contributions
  • Fixed an invariant check in the databricks step launcher that was causing failures when setting the local_dagster_job_package_path config option (Thanks Iswariya Manivannan!)
Documentation
  • Fixed the example code in the reconstructable API docs.

0.14.2rc0

7 months ago

0.14.1

7 months ago
New
  • [dagit] The sensor tick timeline now shows cursor values in the tick tooltip if they exist.
Bugfixes
  • Pinned dependency on markupsafe to function with existing Jinja2 pin.
  • Sensors that have a default status can now be manually started. Previously, this would fail with an invariant exception.

0.14.1rc0

7 months ago

0.14.0

7 months ago
“Never Felt Like This Before” Major Changes
  • Software-defined assets, which offer a declarative approach to data orchestration on top of Dagster’s core job/op/graph APIs, have matured significantly. Improvements include partitioned assets, a revamped asset details page in Dagit, a cross-repository asset graph view in Dagit, Dagster types on assets, structured metadata on assets, and the ability to materialize ad-hoc selections of assets without defining jobs. Users can expect the APIs to only undergo minor changes before being declared fully stable in Dagster’s next major release. For more information, view the software-defined assets concepts page here.

  • We’ve made it easier to define a set of software-defined assets where each Dagster asset maps to a dbt model. All of the dependency information between the dbt models will be reflected in the Dagster asset graph, while still running your dbt project in a single step.

  • Dagit has a new homepage, dubbed the “factory floor” view, that provides an overview of recent runs of all the jobs. From it, you can monitor the status of each job’s latest run or quickly re-execute a job. The new timeline view reports the status of all recent runs in a convenient gantt chart.

  • You can now write schedules and sensors that default to running as soon as they are loaded in your workspace, without needing to be started manually in Dagit. For example, you can create a sensor like this:

    from dagster import sensor, DefaultSensorStatus
    
    @sensor(job=my_job, default_status=DefaultSensorStatus.RUNNING)
    def my_running_sensor():
        ...
    

    or a schedule like this:

    from dagster import schedule, DefaultScheduleStatus, ScheduleEvaluationContext
    
    @schedule(job=my_job, cron_schedule="0 0 * * *", default_status=DefaultScheduleStatus.RUNNING)
    def my_running_schedule(context: ScheduleEvaluationContext):
        ...
    

    As soon as schedules or sensors with the default_status field set to RUNNING are included in the workspace loaded by your Dagster Daemon, they will begin creating ticks and submitting runs.

  • Op selection now supports selecting ops inside subgraphs. For example, to select an op my_op inside a subgraph my_graph, you can now specify the query as my_graph.my_op. This is supported in both Dagit and Python APIs.

  • Dagster Types can now have attached metadata. This allows TableSchema objects to be attached to Dagster Types via TableSchemaMetadata. A Dagster Type with a TableSchema will have the schema rendered in Dagit.

  • A new Pandera integration (dagster-pandera) allows you to use Pandera’s dataframe validation library to wrap dataframe schemas in Dagster types. This provides two main benefits: (1) Pandera’s rich schema validation can be used for runtime data validation of Pandas dataframes in Dagster ops/assets; (2) Pandera schema information is displayed in Dagit using a new TableSchema API for representing arbitrary table schemas.

  • The new AssetObservation event enables recording metadata about an asset without indicating that the asset has been updated.

  • AssetMaterializations, ExpectationResults, and AssetObservations can be logged via the context of an op using the OpExecutionContext.log_event method. Output metadata can also be logged using the OpExecutionContext.add_output_metadata method. Previously, Dagster expected these events to be yielded within the body of an op, which caused lint errors for many users, made it difficult to add mypy types to ops, and also forced usage of the verbose Output API. Here’s an example of the new invocations:

    from dagster import op, AssetMaterialization
    @op
    def the_op(context):
        context.log_event(AssetMaterialization(...))
        context.add_output_metadata({"foo": "bar"})
        ...
    
  • A new Airbyte integration (dagster-airbyte) allows you to kick off and monitor Airbyte syncs from within Dagster. The original contribution from @airbytehq’s own @marcosmarxm includes a resource implementation as well as a pre-built op for this purpose, and we’ve extended this library to support software-defined asset use cases as well. Regardless of which interface you use, Dagster will automatically capture the Airbyte log output (in the compute logs for the relevant steps) and track the created tables over time (via AssetMaterializations).

  • The ECSRunLauncher (introduced in Dagster 0.11.15) is no longer considered experimental. You can bootstrap your own Dagster deployment on ECS using our docker compose example or you can use it in conjunction with a managed Dagster Cloud deployment. Since its introduction, we’ve added the ability to customize Fargate container memory and CPU, mount secrets from AWS SecretsManager, and run with a variety of AWS networking configurations. Join us in #dagster-ecs in Slack!

  • [Helm] The default liveness and startup probes for Dagit and user deployments have been replaced with readiness probes. The liveness and startup probe for the Daemon has been removed. We observed and heard from users that under load, Dagit could fail the liveness probe which would result in the pod restarting. With the new readiness probe, the pod will not restart but will stop serving new traffic until it recovers. If you experience issues with any of the probe changes, you can revert to the old behavior by specifying liveness and startup probes in your Helm values (and reach out via an issue or Slack).

Breaking Changes and Deprecations
  • The Dagster Daemon now uses the same workspace.yaml file as Dagit to locate your Dagster code. You should ensure that if you make any changes to your workspace.yaml file, they are included in both Dagit’s copy and the Dagster Daemon’s copy. When you make changes to the workspace.yaml file, you don’t need to restart either Dagit or the Dagster Daemon - in Dagit, you can reload the workspace from the Workspace tab, and the Dagster Daemon will periodically check the workspace.yaml file for changes every 60 seconds. If you are using the Dagster Helm chart, no changes are required to include the workspace in the Dagster Daemon.
  • Dagster’s metadata API has undergone a signficant overhaul. Changes include:
    • To reflect the fact that metadata can be specified on definitions in addition to events, the following names are changing. The old names are deprecated, and will function as aliases for the new names until 0.15.0:
      • EventMetadata > MetadataValue
      • EventMetadataEntry > MetadataEntry
      • XMetadataEntryData > XMetadataValue (e.g. TextMetadataEntryData > TextMetadataValue)
    • The metadata_entries keyword argument to events and Dagster types is deprecated. Instead, users should use the metadata keyword argument, which takes a dictionary mapping string labels to MetadataValues.
    • Arbitrary metadata on In/InputDefinition and Out/OutputDefinition is deprecated. In 0.15.0, metadata passed for these classes will need to be resolvable to MetadataValue (i.e. function like metadata everywhere else in Dagster).
    • The description attribute of EventMetadataEntry is deprecated.
    • The static API of EventMetadataEntry (e.g. EventMetadataEntry.text) is deprecated. In 0.15.0, users should avoid constructing EventMetadataEntry objects directly, instead utilizing the metadata dictionary keyword argument, which maps string labels to MetadataValues.
  • In previous releases, it was possible to supply either an AssetKey, or a function that produced an AssetKey from an OutputContext as the asset_key argument to an Out/OutputDefinition. The latter behavior makes it impossible to gain information about these relationships without running a job, and has been deprecated. However, we still support supplying a static AssetKey as an argument.
  • We have renamed many of the core APIs that interact with ScheduleStorage, which keeps track of sensor/schedule state and ticks. The old term for the generic schedule/sensor “job” has been replaced by the term “instigator” in order to avoid confusion with the execution API introduced in 0.12.0. If you have implemented your own schedule storage, you may need to change your method signatures appropriately.
  • Dagit is now powered by Starlette instead of Flask. If you have implemented a custom run coordinator, you may need to make the following change:
    from flask import has_request_context, request
    def submit_run(self, context: SubmitRunContext) -> PipelineRun:
        jwt_claims_header = (
            request.headers.get("X-Amzn-Oidc-Data", None) if has_request_context() else None
        )
    
    Should be replaced by:
    def submit_run(self, context: SubmitRunContext) -> PipelineRun:
        jwt_claims_header = context.get_request_header("X-Amzn-Oidc-Data")
    
  • Dagit
    • Dagit no longer allows non-software-defined asset materializations to be be graphed or grouped by partition. This feature could render in incorrect / incomplete ways because no partition space was defined for the asset.
    • Dagit’s “Jobs” sidebar now collapses by default on Instance, Job, and Asset pages. To show the left sidebar, click the “hamburger” icon in the upper left.
    • “Step Execution Time” is no longer graphed on the asset details page in Dagit, which significantly improves page load time. To view this graph, go to the asset graph for the job, uncheck “View as Asset Graph” and click the step to view its details.
    • The “experimental asset UI” feature flag has been removed from Dagit, this feature is shipped in 0.14.0!
  • The Dagster Daemon now requires a workspace.yaml file, much like Dagit.
  • Ellipsis (“...”) is now an invalid substring of a partition key. This is because Dagit accepts an ellipsis to specify partition ranges.
  • [Helm] The Dagster Helm chart now only supported Kubernetes clusters above version 1.18.
New since 0.13.19
  • Software Defined Assets:
    • In Dagit, the Asset Catalog now offers a third display mode - a global graph of your software-defined assets.
    • The Asset Catalog now allows you to filter by repository to see a subset of your assets, and offers a “View in Asset Graph” button for quickly seeing software-defined assets in context.
    • The Asset page in Dagit has been split into two tabs, “Activity” and “Definition”.
    • Dagit now displays a warning on the Asset page if the most recent run including the asset’s step key failed without yielding a materialization, making it easier to jump to error logs.
    • Dagit now gives you the option to view jobs with software-defined assets as an Asset Graph (default) or as an Op Graph, and displays asset<>op relationships more prominently when a single op yields multiple assets.
    • You can now include your assets in a repository with the use of an AssetGroup. Each repository can only have one AssetGroup, and it can provide a jumping off point for creating the jobs you plan on using from your assets.
      from dagster import AssetGroup, repository, asset
      @asset(required_resource_keys={"foo"})
      def asset1():
          ...
      @asset
      def asset2():
          ...
      @repository
      def the_repo():
          asset_group = AssetGroup(assets=[asset1, asset2], resource_defs={"foo": ...})
          return [asset_group, asset_group.build_job(selection="asset1-")]
      
    • AssetGroup.build_job supports a selection syntax similar to that found in op selection.
  • Asset Observations:
    • You can now yield AssetObservations to log metadata about a particular asset from beyond its materialization site. AssetObservations appear on the asset details page alongside materializations and numerical metadata is graphed. For assets with software-defined partitions, materialized and observed metadata about each partition is rolled up and presented together. For more information, view the docs page here.
    • Added an asset_observations_for_node method to ExecuteInProcessResult for fetching the AssetObservations from an in-process execution.
  • Dagster Types with an attached TableSchemaMetadataValue now render the schema in Dagit UI.
  • [dagster-pandera] New integration library dagster-pandera provides runtime validation from the Pandera dataframe validation library and renders table schema information in Dagit.
  • OpExecutionContext.log_event provides a way to log AssetMaterializations, ExpectationResults, and AssetObservations from the body of an op without having to yield anything. Likewise, you can use OpExecutionContext.add_output_metadata to attach metadata to an output without having to explicitly use the Output object.
  • OutputContext.log_event provides a way to log AssetMaterializations from within the handle_output method of an IO manager without yielding. Likewise, output metadata can be added using OutputContext.add_output_metadata.
  • [dagster-dbt] The load_assets_from_dbt_project function now returns a set of assets that map to a single dbt run command (rather than compiling each dbt model into a separate step). It also supports a new node_info_to_asset_key argument which allows you to customize the asset key that will be used for each dbt node.
  • [dagster-airbyte] The dagster-airbyte integration now collects the Airbyte log output for each run as compute logs, and generates AssetMaterializations for each table that Airbyte updates or creates.
  • [dagster-airbyte] The dagster-airbyte integration now supports the creation of software-defined assets, with the build_airbyte_assets function.
  • [dagster-fivetran] The dagster-fivetran integration now supports the creation of software-defined assets with the build_fivetran_assets function.
  • The multiprocess executor now supports choosing between spawn or forkserver for how its subprocesses are created. When using forkserver we attempt to intelligently preload modules to reduce the per-op overhead.
  • [Helm] Labels can now be set on the Dagit and daemon deployments.
  • [Helm] The default liveness and startup probes for Dagit and user deployments have been replaced with readiness probes. The liveness and startup probe for the Daemon has been removed. We observed and heard from users that under load, Dagit could fail the liveness probe which would result in the pod restarting. With the new readiness probe, the pod will not restart but will stop serving new traffic until it recovers. If you experience issues with any of the probe changes, you can revert to the old behavior by specifying liveness and startup probes in your Helm values (and reach out via an issue or Slack).
  • [Helm] The Ingress v1 is now supported.
Community Contributions
  • Typo fix from @jiafi, thank you!
Bugfixes
  • Fixed an issue where long job names were truncated prematurely in the Jobs page in Dagit.
  • Fixed an issue where loading the sensor timeline would sometimes load slowly or fail with a timeout error.
  • Fixed an issue where the first time a run_status_sensor executed, it would sometimes run very slowly or time out.
  • Fixed an issue where Launchpad mistakenly defaulted with invalid subset error in Dagit.
  • Multi-component asset keys can now be used in the asset graph filter bar.
  • Increased the storage query statement timeout to better handle more complex batch queries.
  • Added fallback support for older versions of sqlite to service top-level repository views in Dagit (e.g. the top-level jobs, schedule, and sensor pages).
Documentation
  • Images in the documentation now enlarge when clicked.
  • New example in examples/bollinger demonstrates dagster-pandera and TableSchema , and software-defined assets in the context of analyzing stock price data.

0.14.0rc0

7 months ago

0.13.19

8 months ago
New
  • [dagit] Various performance improvements for asset graph views.
  • [dagster-aws] The EcsRunLauncher can now override the secrets_tag parameter to None, which will cause it to not look for any secrets to be included in the tasks for the run. This can be useful in situations where the run launcher does not have permissions to query AWS Secretsmanager.
Bugfixes
  • [dagster-mysql] For instances using MySQL for their run storage, runs created using dagster versions 0.13.17 / 0.13.18 might display an incorrect timestamp for its start time on the Runs page. Running the dagster instance migrate CLI command should resolve the issue.

0.13.19rc2

8 months ago

0.13.18

8 months ago
New
  • Op selection now supports selecting ops inside subgraphs. For example, to select an op my_op inside a subgraph my_graph, you can now specify the query as "my_graph.my_op".
  • The error message raised on failed Dagster type check on an output now includes the description provided on the TypeCheck object.
  • The dagster asset wipe CLI command now takes a --noprompt option.
  • Added the new Map config type, used to represent mappings between arbitrary scalar keys and typed values. For more information, see the Map ConfigType docs.
  • build_resources has been added to the top level API. It provides a way to initialize resources outside of execution. This provides a way to use resources within the body of a sensor or schedule: https://github.com/dagster-io/dagster/issues/3794
  • The dagster-daemon process now creates fewer log entries when no actions are taken (for example, if the run queue is empty)
  • [dagster-k8s] When upgrading the Dagster helm chart, the old dagster-daemon pod will now spin down completely before the new dagster-daemon pod is started.
  • [dagster-k8s] A flag can now be set in the Dagster helm chart to control whether the Kubernetes Jobs and Pods created by the K8sRunLauncher should fail if the Dagster run fails. To enable this flag, set the `failPodOnRunFailure key to true in the run launcher portion of the Helm chart.
  • [dagster-dbt] Fixed compatibility issues with dbt 1.0. The schema and data arguments on the DbtCliResource.test function no longer need to be set to False to avoid errors, and the dbt output will be no longer be displayed in json format in the event logs.
  • Dagster Types can now have metadata entries attached to them.
  • DagsterGraphQLClient now supports submitting runs with op/solid sub-selections.
  • [dagit] The Asset Catalog view will now include information from both AssetMaterializations and AssetObservation events for each asset.
  • [dagit][software-defined-assets] A warning will now be displayed if you attempt to backfill partitions of an asset whose upstream dependencies are missing.
Bugfixes
  • When Dagit fails to load a list of ops, the error message used the legacy term “solids”. Now it uses “ops”.
  • Runs created using dagster versions 0.13.15 / 0.13.16 / 0.13.17 might display an incorrect timestamp for its start time on the Runs page. This would only happen if you had run a schema migration (using one of those versions) with the dagster instance migrate CLI command. Running the dagster instance reindex command should run a data migration that resolves this issue.
  • When attempting to invoke run status sensors or run failure sensors, it will now incur an error. Run status/failure sensor invocation is not yet supported.
  • [dagster-k8s] Fixed a bug in the sanitization of K8s label values with uppercase characters and underscores
Community Contributions
  • [software-defined-assets] Language in dagit has been updated from “refreshing” to “rematerializing” assets (thanks @Sync271!)
  • [docs] The changelog page is now mobile friendly (thanks @keyz!)
  • [docs] The loading shimmer for text on docs pages now has correct padding (also @keyz!)
Experimental
  • [software-defined-assets] The namespace argument of the @asset decorator now accepts a list of strings in addition to a single string.
  • [memoization] Added a missing space to the error thrown when trying to use memoization without a persistent Dagster instance.
  • [metadata] Two new metadata types, TableSchemaMetadataEntryData and TableMetadataEntryData allow you to emit metadata representing the schema / contents of a table, to be displayed in Dagit.

0.13.18rc0

8 months ago

0.13.17

8 months ago
New
  • When a user-generated context.log call fails while writing to the event log, it will now log a system error in the event log instead of failing the run.
  • [dagit] Made performance improvements to the Runs page, which can be realized after running an optional storage schema migration using dagster instance migrate.
  • When a job is created from a graph, it will now use the graph’s description if a description is not explicitly provided to override it. (Thanks @AndreaGiardini!)
  • [dagit] Log job names are now truncated in Dagit.
  • [dagit] The execution timezone is shown beside schedule cron strings, since their timezone may be UTC or a custom value.
  • [dagit] Graph filter inputs now default to using quoted strings, and this syntax matches ops, steps, or assets via an exact string match. "build_table"+ will select that asset and it's downstream children without selecting another containing that string, such as build_table_result. Removing the quotes provides the old string matching behavior
  • [dagster-aws] When using the emr_pyspark_step_launcher to run Dagster ops in an Amazon EMR cluster, the raw stdout output of the Spark driver is now written to stdout and will appear in the compute logs for the op in dagit, rather than being written to the Dagster event log.
  • [dagit] Improved performance loading the Asset entry page in Dagit.
Bugfixes
  • [dagster-mysql] Added a schema migration script that was mistakenly omitted from 0.13.16. Migrating instance storage using dagster instance migrate should now complete without error.
  • [dagster-airbyte] Fixed a packaging dependency issue with dagster-airbyte. (Thanks bollwyvl!)
  • Fixed a bug where config provided to the config arg on to_job required environment variables to exist at definition time.
  • [dagit] The asset graph view now supports ops that yield multiple assets and renders long asset key paths correctly.
  • [dagit] The asset graph’s filter input now allows you to filter on assets with multi-component key paths.
  • [dagit] The asset graph properly displays downstream asset links to other asset jobs in your workspace.
Experimental
  • [dagster-celery-k8s] Experimental run monitoring is now supported with the CeleryK8sRunLauncher. This will detect when a run worker K8s Job has failed (due to an OOM, a Node shutting down, etc.) and mark the run as failed so that it doesn’t hang in STARTED. To enable this feature, set dagsterDaemon.runMonitoring.enabled to true in your Helm values.
Documentation
  • [dagster-snowflake] Fixed some example code in the API doc for snowflake_resource, which incorrectly constructed a Dagster job using the snowflake resource.

0.13.17rc0

8 months ago

0.13.16

8 months ago
New
  • Added an integration with Airbyte, under the dagster-airbyte package (thanks Marcos Marx).
  • An op that has a config schema is no longer required to have a context argument.
Bugfixes
  • Fixed an issue introduced in 0.13.13 where jobs with DynamicOutputs would fail when using the k8s_job_executor due to a label validation error when creating the step pod.
  • In Dagit, when searching for asset keys on the Assets page, string matches beyond a certain character threshold on deeply nested key paths were ignored. This has been fixed, and all keys in the asset path are now searchable.
  • In Dagit, links to Partitions views were broken in several places due to recent URL querystring changes, resulting in page crashes due to JS errors. These links have been fixed.
  • The “Download Debug File” menu link is fixed on the Runs page in Dagit.
  • In the “Launch Backfill” dialog on the Partitions page in Dagit, the range input sometimes discarded user input due to page updates. This has been fixed. Additionally, pressing the return key now commits changes to the input.
  • When using a mouse wheel or touchpad gestures to zoom on a DAG view for a job or graph in Dagit, the zoom behavior sometimes was applied to the entire browser instead of just the DAG. This has been fixed.
  • Dagit fonts now load correctly when using the --path-prefix option.
  • Date strings in tool tips and tick labels on time-based charts no longer duplicate the meridiem indicator.
Experimental
  • Software-defined assets can now be partitioned. The @asset decorator has a partitions_def argument, which accepts a PartitionsDefinition value. The asset details page in Dagit now represents which partitions are filled in.
Documentation
  • Fixed the documented return type for the sync_and_poll method of the dagster-fivetran resource (thanks Marcos Marx).
  • Fixed a typo in the Ops concepts page (thanks Oluwashina Aladejubelo).

0.13.16rc0

8 months ago

0.13.15

8 months ago

0.13.15rc0

8 months ago

0.13.14

8 months ago
New
  • When you produce a PartitionedConfig object using a decorator like daily_partitioned_config or static_partitioned_config, you can now directly invoke that object to invoke the decorated function.
  • The end_offset argument to PartitionedConfig can now be negative. This allows you to define a schedule that fills in partitions further in the past than the current partition (for example, you could define a daily schedule that fills in the partition from two days ago by setting end_offset to -1.
  • The runConfigData argument to the launchRun GraphQL mutation can now be either a JSON-serialized string or a JSON object , instead of being required to be passed in as a JSON object. This makes it easier to use the mutation in typed languages where passing in unserialized JSON objects as arguments can be cumbersome.
  • Dagster now always uses the local working directory when resolving local imports in job code, in all workspaces. In the case where you want to use a different base folder to resolve local imports in your code, the working_directory argument can now always be specified (before, it was only available when using the python_file key in your workspace). See the Workspace docs (https://docs.dagster.io/concepts/repositories-workspaces/workspaces#loading-relative-imports) for more information.
Bugfixes
  • In Dagit, when viewing an in-progress run, the logic used to render the “Terminate” button was backward: it would appear for a completed run, but not for an in-progress run. This bug was introduced in 0.13.13, and is now fixed.
  • Previously, errors in the instance’s configured compute log manager would cause runs to fail. Now, these errors are logged but do not affect job execution.
  • The full set of DynamicOutputs returned by a op are no longer retained in memory if there is no hook to receive the values. This allows for DynamicOutput to be used for breaking up a large data set that can not fit in memory.
Breaking Changes
  • When running your own gRPC server to serve Dagster code, jobs that launch in a container using code from that server will now default to using dagster as the entry point. Previously, the jobs would run using PYTHON_EXECUTABLE -m dagster, where PYTHON_EXECUTABLE was the value of sys.executable on the gRPC server. For the vast majority of Dagster jobs, these entry points will be equivalent. To keep the old behavior (for example, if you have multiple Python virtualenvs in your image and want to ensure that runs also launch in a certain virtualenv), you can launch the gRPC server using the new ----use-python-environment-entry-point command-line arg.
Community Contributions
  • [bugfix] Fixed an issue where log levels on handlers defined in dagster.yaml would be ignored (thanks @lambdaTW!)
Documentation
  • Typo fix in the jobs page (thanks @kmiku7!)
  • Added docs on how to modify k8s job TTL
UI
  • When re-launching a run, the log/step filters are now preserved in the new run’s page.
  • Step execution times/recent runs now appear in the job/graph sidebar.

0.13.14rc0

8 months ago

0.13.13

9 months ago
New
  • [dagster-dbt] dbt rpc resources now surface dbt log messages in the Dagster event log.
  • [dagster-databricks] The databricks_pyspark_step_launcher now streams Dagster logs back from Databricks rather than waiting for the step to completely finish before exporting all events. Fixed an issue where all events from the external step would share the same timestamp. Immediately after execution, stdout and stderr logs captured from the Databricks worker will be automatically surfaced to the event log, removing the need to set the wait_for_logs option in most scenarios.
  • [dagster-databricks] The databricks_pyspark_step_launcher now supports dynamically mapped steps.
  • If the scheduler is unable to reach a code server when executing a schedule tick, it will now wait until the code server is reachable again before continuing, instead of marking the schedule tick as failed.
  • The scheduler will now check every 5 seconds for new schedules to run, instead of every 30 seconds.
  • The run viewer and workspace pages of Dagit are significantly more performant.
  • Dagit loads large (100+ node) asset graphs faster and retrieves information about the assets being rendered only.
  • When viewing an asset graph in Dagit, you can now rematerialize the entire graph by clicking a single “Refresh” button, or select assets to rematerialize them individually. You can also launch a job to rebuild an asset directly from the asset details page.
  • When viewing a software-defined asset, Dagit displays its upstream and downstream assets in two lists instead of a mini-graph for easier scrolling and navigation. The statuses of these assets are updated in real-time. This new UI also resolves a bug where only one downstream asset would appear.
Bugfixes
  • Fixed bug where execute_in_process would not work for graphs with nothing inputs.
  • In the Launchpad in Dagit, the Ctrl+A command did not correctly allow select-all behavior in the editor for non-Mac users, this has now been fixed.
  • When viewing a DAG in Dagit and hovering on a specific input or output for an op, the connections between the highlighted inputs and outputs were too subtle to see. These are now a bright blue color.
  • In Dagit, when viewing an in-progress run, a caching bug prevented the page from updating in real time in some cases. For instance, runs might appear to be stuck in a queued state long after being dequeued. This has been fixed.
  • Fixed a bug in the k8s_job_executor where the same step could start twice in rare cases.
  • Enabled faster queries for the asset catalog by migrating asset database entries to store extra materialization data.
  • [dagster-aws] Viewing the compute logs for in-progress ops for instances configured with the S3ComputeLogManager would cause errors in Dagit. This is now fixed.
  • [dagster-pandas] Fixed bug where Pandas categorical dtype did not work by default with dagster-pandas categorical_column constraint.
  • Fixed an issue where schedules that yielded a SkipReason from the schedule function did not display the skip reason in the tick timeline in Dagit, or output the skip message in the dagster-daemon log output.
  • Fixed an issue where the snapshot link of a finished run in Dagit would sometimes fail to load with a GraphQL error.
  • Dagit now supports software-defined assets that are defined in multiple jobs within a repo, and displays a warning when assets in two repos share the same name.
Breaking Changes
  • We previously allowed schedules to be defined with cron strings like @daily rather than 0 0 * * *. However, these schedules would fail to actually run successfully in the daemon and would also cause errors when viewing certain pages in Dagit. We now raise an DagsterInvalidDefinitionError for schedules that do not have a cron expression consisting of a 5 space-separated fields.
Community Contributions
  • In dagster-dask, a schema can now be conditionally specified for ops materializing outputs to parquet files, thank you @kudryk!
  • Dagster-gcp change from @AndreaGiardini that replaces get_bucket() calls with bucket(), to avoid unnecessary bucket metadata fetches, thanks!
  • Typo fix from @sebastianbertoli, thank you!
  • [dagster-k8s] Kubernetes jobs and pods created by Dagster now have labels identifying the name of the Dagster job or op they are running. Thanks @skirino!
Experimental
  • [dagit] Made performance improvements for loading the asset graph.
  • [dagit] The debug console logging output now tracks calls to fetch data from the database, to help track inefficient queries.

0.13.13rc0

9 months ago

0.13.12

9 months ago
0.13.12 New
  • The dagit and dagster-daemon processes now use a structured Python logger for command-line output.
  • Dagster command-line logs now include the system timezone in the logging timestamp.
  • When running your own Dagster gRPC code server, the server process will now log a message to stdout when it starts up and when it shuts down.
  • [dagit] The sensor details page and sensor list page now display links to the assets tracked by @asset_sensors.
  • [dagit] Improved instance warning in Dagit. Previously, Dagit showed an instance warning for daemon not running when no repos have schedulers or sensors.
  • [dagster-celery-k8s] You can now specify volumes and volume mounts to runs using the CeleryK8sRunLauncher that will be included in all launched jobs.
  • [dagster-databricks] You are no longer required to specify storage configuration when using the databricks_pyspark_step_launcher.
  • [dagster-databricks] The databricks_pyspark_step_launcher can now be used with dynamic mapping and collect steps.
  • [dagster-mlflow] The end_mlflow_on_run_finished hook is now a top-level export of the dagster mlflow library. The API reference also now includes an entry for it.
Bugfixes
  • Better backwards-compatibility for fetching asset keys materialized from older versions of dagster.
  • Fixed an issue where jobs running with op subsets required some resource configuration as part of the run config, even when they weren’t required by the selected ops.
  • RetryPolicy is now respected when execution is interrupted.
  • [dagit] Fixed "Open in Playground" link on the scheduled ticks.
  • [dagit] Fixed the run ID links on the Asset list view.
  • [dagit] When viewing an in-progress run, the run status sometimes failed to update as new logs arrived, resulting in a Gantt chart that either never updated from a “queued” state or did so only after a long delay. The run status and Gantt chart now accurately match incoming logs.
Community Contributions
  • [dagster-k8s] Fixed an issue where specifying job_metadata in tags did not correctly propagate to Kubernetes jobs created by Dagster. Thanks @ibelikov!
Experimental
  • [dagit] Made performance improvements for loading the asset graph.
Documentation
  • The Versioning and Memoization guide has been updated to reflect a new set of core memoization APIs.
  • [dagster-dbt] Updated the dagster-dbt integration guide to mention the new dbt Cloud integration.
  • [dagster-dbt] Added documentation for the default_flags property of DbtCliResource.

0.13.12rc2

9 months ago

0.13.11

10 months ago
New
  • [dagit] Made performance improvements to the Run page.
  • [dagit] Highlighting a specific sensor / schedule ticks is now reflected in a shareable URL.
Bugfixes
  • [dagit] On the Runs page, when filtering runs with a tag containing a comma, the filter input would incorrectly break the tag apart. This has been fixed.
  • [dagit] For sensors that do not target a specific job (e.g. un_status_sensor, we are now hiding potentially confusing Job details
  • [dagit] Fixed an issue where some graph explorer views generated multiple scrollbars.
  • [dagit] Fixed an issue with the Run view where the Gantt view incorrectly showed in-progress steps when the run had exited.
  • [dagster-celery-k8s] Fixed an issue where setting a custom Celery broker URL but not a custom Celery backend URL in the helm chart would produce an incorrect Celery configuration.
  • [dagster-k8s] Fixed an issue where Kubernetes volumes using list or dict types could not be set in the Helm chart.
Community Contributions
  • [dagster-k8s] Added the ability to set a custom location name when configuring a workspace in the Helm chart. Thanks @pcherednichenko!
Experimental
  • [dagit] Asset jobs now display with spinners on assets that are currently in progress.
  • [dagit] Assets jobs that are in progress will now display a dot icon on all assets that are not yet running but will be re-materialized in the run.
  • [dagit] Fixed broken links to the asset catalog entries from the explorer view of asset jobs.
  • The AssetIn input object now accepts an asset key so upstream assets can be explicitly specified (e.g. AssetIn(asset_key=AssetKey("asset1")))
  • The @asset decorator now has an optional non_argument_deps parameter that accepts AssetKeys of assets that do not pass data but are upstream dependencies.
  • ForeignAsset objects now have an optional description attribute.
Documentation
  • “Validating Data with Dagster Type Factories” guide added.

0.13.11rc0

10 months ago

0.13.10

10 months ago
New
  • run_id, job_name, and op_exception have been added as parameters to build_hook_context.
  • You can now define inputs on the top-level job / graph. Those inputs can be can configured as an inputs key on the top level of your run config. For example, consider the following job:
from dagster import job, op

@op
def add_one(x):
    return x + 1

@job
def my_job(x):
    add_one(x)

You can now add config for x at the top level of my run_config like so:

run_config = {
  "inputs": {
    "x": {
      "value": 2
    }
  }
}
  • You can now create partitioned jobs and reference a run’s partition from inside an op body or IOManager load_input or handle_output method, without threading partition values through config. For example, where previously you might have written:
@op(config_schema={"partition_key": str})
def my_op(context):
    print("partition_key: " + context.op_config["partition_key"])

@static_partitioned_config(partition_keys=["a", "b"])
def my_static_partitioned_config(partition_key: str):
    return {"ops": {"my_op": {"config": {"partition_key": partition_key}}}}

@job(config=my_static_partitioned_config)
def my_partitioned_job():
    my_op()

You can now write:

@op
def my_op(context):
    print("partition_key: " + context.partition_key)

@job(partitions_def=StaticPartitionsDefinition(["a", "b"]))
def my_partitioned_job():
    my_op()
  • Added op_retry_policy to @job. You can also specify op_retry_policy when invoking to_job on graphs.
  • [dagster-fivetran] The fivetran_sync_op will now be rendered with a fivetran tag in Dagit.
  • [dagster-fivetran] The fivetran_sync_op now supports producing AssetMaterializations for each table updated during the sync. To this end, it now outputs a structured FivetranOutput containing this schema information, instead of an unstructured dictionary.
  • [dagster-dbt] AssetMaterializations produced from the dbt_cloud_run_op now include a link to the dbt Cloud docs for each asset (if docs were generated for that run).
  • You can now use the @schedule decorator with RunRequest - based evaluation functions. For example, you can now write:
@schedule(cron_schedule="* * * * *", job=my_job)
def my_schedule(context):
    yield RunRequest(run_key="a", ...)
    yield RunRequest(run_key="b", ...)
  • [dagster-k8s] You may now configure instance-level python_logs settings using the Dagster Helm chart.
  • [dagster-k8s] You can now manage a secret that contains the Celery broker and backend URLs, rather than the Helm chart
  • [Dagster-slack] Improved the default messages in make_slack_on_run_failure_sensor to use Slack layout blocks and include clickable link to Dagit. Previously, it sent a plain text message.
Dagit
  • Made performance improvements to the Run page.
  • The Run page now has a pane control that splits the Gantt view and log table evenly on the screen.
  • The Run page now includes a list of succeeded steps in the status panel next to the Gantt chart.
  • In the Schedules list, execution timezone is now shown alongside tick timestamps.
  • If no repositories are successfully loaded when viewing Dagit, we now redirect to /workspace to quickly surface errors to the user.
  • Increased the size of the reload repository button
  • Repositories that had been hidden from the left nav became inaccessible when loaded in a workspace containing only that repository. Now, when loading a workspace containing a single repository, jobs for that repository will always appear in the left nav.
  • In the Launchpad, selected ops were incorrectly hidden in the lower right panel.
  • Repaired asset search input keyboard interaction.
  • In the Run page, the list of previous runs was incorrectly ordered based on run ID, and is now ordered by start time.
  • Using keyboard commands with the / key (e.g. toggling commented code) in the config editor
Bugfixes
  • Previously, if an asset in software-defined assets job depended on a ForeignAsset, the repository containing that job would fail to load.
  • Incorrectly triggered global search. This has been fixed.
  • Fix type on tags of EMR cluster config (thanks Chris)!
  • Fixes to the tests in dagster new-project , which were previously using an outdated result API (thanks Vašek)!
Experimental
  • You can now mount AWS Secrets Manager secrets as environment variables in runs launched by the EcsRunLauncher.
  • You can now specify the CPU and Memory for runs launched by the EcsRunLauncher.
  • The EcsRunLauncher now dynamically chooses between assigning a public IP address or not based on whether it’s running in a public or private subnet.
  • The @asset and @multi_asset decorator now return AssetsDefinition objects instead of OpDefinitions
Documentation
  • The tutorial now uses get_dagster_logger instead of context.log.
  • In the API docs, most configurable objects (such as ops and resources) now have their configuration schema documented in-line.
  • Removed typo from CLI readme (thanks Kan (https://github.com/zkan))!

0.13.10rc3

10 months ago

0.13.9

10 months ago
New
  • Memoization can now be used with the multiprocess, k8s, celery-k8s, and dask executors.

0.13.9rc0

10 months ago

0.13.8

10 months ago
New
  • Improved the error message for situations where you try a, b = my_op(), inside @graph or @job, but my_op only has a single Out.
  • [dagster-dbt] A new integration with dbt Cloud allows you to launch dbt Cloud jobs as part of your Dagster jobs. This comes complete with rich error messages, links back to the dbt Cloud UI, and automatically generated Asset Materializations to help keep track of your dbt models in Dagit. It provides a pre-built dbt_cloud_run_op, as well as a more flexible dbt_cloud_resource for more customized use cases. Check out the api docs to learn more!
  • [dagster-gcp] Pinned the google-cloud-bigquery dependency to <3, because the new 3.0.0b1 version was causing some problems in tests.
  • [dagit] Verbiage update to make it clear that wiping an asset means deleting the materialization events for that asset.
Bugfixes
  • Fixed a bug with the pipeline launch / job launch CLIs that would spin up an ephemeral dagster instance for the launch, then tear it down before the run actually executed. Now, the CLI will enforce that your instance is non-ephemeral.
  • Fixed a bug with re-execution when upstream step skips some outputs. Previously, it mistakenly tried to load inputs from parent runs. Now, if an upstream step doesn’t yield outputs, the downstream step would skip.
  • [dagit] Fixed a bug where configs for unsatisfied input wasn’t properly resolved when op selection is specified in Launchpad.
  • [dagit] Restored local font files for Inter and Inconsolata instead of using the Google Fonts API. This allows correct font rendering for offline use.
  • [dagit] Improved initial workspace loading screen to indicate loading state instead of showing an empty repository message.
Breaking Changes
  • The pipeline argument of the InitExecutorContext constructor has been changed to job.
Experimental
  • The @asset decorator now accepts a dagster_type argument, which determines the DagsterType for the output of the asset op.
  • build_assets_job accepts an executor_def argument, which determines the executor for the job.
Documentation
  • A docs section on context manager resources has been added. Check it out here.
  • Removed the versions of the Hacker News example jobs that used the legacy solid & pipeline APIs.

0.13.8rc0

10 months ago

0.13.7

10 months ago
New
  • The Runs page in Dagit now loads much more quickly.
Bugfixes
  • Fixed an issue where Dagit would sometimes display a red "Invalid JSON" error message.
Dependencies
  • google-cloud-bigquery is temporarily pinned to be prior to version 3 due to a breaking change in that version.

0.13.7rc0

10 months ago

0.13.6

10 months ago
Bugfixes
  • Previously, the EcsRunLauncher tagged each ECS task with its corresponding Dagster Run ID. ECS tagging isn't supported for AWS accounts that have not yet migrated to using the long ARN format. Now, the EcsRunLauncher only adds this tag if your AWS account has the long ARN format enabled.
  • Fixed a bug in the k8s_job_executor and docker_executor that could result in jobs exiting as SUCCESS before all ops have run.
  • Fixed a bug in the k8s_job_executor and docker_executor that could result in jobs failing when an op is skipped.
  • [dagit] Improved performance of the Runs page.
Dependencies
  • graphene is temporarily pinned to be prior to version 3 to unbreak Dagit dependencies.

0.13.6rc0

10 months ago

0.13.5

1 year ago
New
  • [dagster-fivetran] A new dagster-fivetran integration allows you to launch Fivetran syncs and monitor their progress from within Dagster. It provides a pre-built fivetran_sync_op, as well as a more flexible fivetran_resource for more customized use cases. Check out the api docs to learn more!
  • When inferring a graph/job/op/solid/pipeline description from the docstring of the decorated function, we now dedent the docstring even if the first line isn’t indented. This allows descriptions to be formatted nicely even when the first line is on the same line as the triple-quotes.
  • The SourceHashVersionStrategy class has been added, which versions op and resource code. It can be provided to a job like so:
from dagster import job, SourceHashVersionStrategy

@job(version_strategy=SourceHashVersionStrategy())
def my_job():
     ...
  • [dagit] Improved performance on the initial page load of the Run page, as well as the partitions UI / launch backfill modal
  • [dagit] Fixed a bug where top-level graphs in the repo could not be viewed in the Workspace > Graph view.
Bugfixes
  • Fixed an issue where turning a partitioned schedule off and on again would sometimes result in unexpected past runs being created. (#5604)
  • Fixed an issue where partition sets that didn’t return a new copy of run configuration on each function call would sometimes apply the wrong config to partitions during backfills.
  • Fixed rare issue where using dynamic outputs in combination with optional outputs would cause errors when using certain executors.
  • [dagster-celery-k8s] Fixed bug where CeleryK8s executor would not respect job run config
  • [dagit] Fixed bug where graphs would sometimes appear off-center.
Breaking Changes
  • In 0.13.0, job CLI commands executed via dagster job selected both pipelines and jobs. This release changes the dagster job command to select only jobs and not pipelines.
Community Contributions
  • [dagster-dask] Updated DaskClusterTypes to have the correct import paths for certain cluster managers (thanks @kudryk!)
  • [dagster-azure] Updated version requirements for Azure to be more recent and more permissive (thanks @roeap !)
  • [dagster-shell] Ops will now copy the host environment variables at runtime, rather than copying them from the environment that their job is launched from (thanks @alexismanuel !)
Documentation
  • The job, op, graph migration guide was erroneously marked experimental. This has been fixed.

0.13.5rc0

1 year ago

0.13.4

1 year ago
New
  • [dagster-k8s] The k8s_job_executor is no longer experimental, and is recommended for production workloads. This executor runs each op in a separate Kubernetes job. We recommend this executor for Dagster jobs that require greater isolation than the multiprocess executor can provide within a single Kubernetes pod. The celery_k8s_job_executor will still be supported, but is recommended only for use cases where Celery is required (The most common example is to offer step concurrency limits using multiple Celery queues). Otherwise, the k8s_job_executor is the best way to get Kubernetes job isolation.
  • [dagster-airflow] Updated dagster-airflow to better support job/op/graph changes by adding a make_dagster_job_from_airflow_dag factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.
  • Removed a version pin of the chardet library that was required due to an incompatibility with an old version of the aiohttp library, which has since been fixed.
  • We now raise a more informative error if the wrong type is passed to the ins argument of the op decorator.
  • In the Dagit Launchpad, the button for launching a run now says “Launch Run” instead of “Launch Execution”
Bugfixes
  • Fixed an issue where job entries from Dagit search navigation were not linking to the correct job pages.
  • Fixed an issue where jobs / pipelines were showing up instead of the underlying graph in the list of repository graph definitions.
  • Fixed a bug with using custom loggers with default config on a job.
  • [dagster-slack] The slack_on_run_failure_sensor now says “Job” instead of “Pipeline” in its default message.
Community Contributions
  • Fixed a bug that was incorrectly causing a DagsterTypeCheckDidNotPass error when a Dagster Type contained a List inside a Tuple (thanks @jan-eat!)
  • Added information for setting DAGSTER_HOME in Powershell and batch for windows users. (thanks @slamer59!)
Experimental
  • Changed the job explorer view in Dagit to show asset-based graphs when the experimental Asset API flag is turned on for any job that has at least one software-defined asset.
Documentation
  • Updated API docs and integration guides to reference job/op/graph for various libraries (dagstermill, dagster-pandas, dagster-airflow, etc)
  • Improved documentation when attempting to retrieve output value from execute_in_process, when job does not have a top-level output.

383 Versions

TagPublished
0.16.106d ago
0.15.1012d ago
0.16.913d ago
0.16.819d ago
0.16.71mo ago
0.15.91mo ago
0.16.61mo ago
0.16.41mo ago
0.16.32mos ago
0.16.22mos ago
0.16.12mos ago
0.16.02mos ago
0.16.0rc22mos ago
0.15.82mos ago
0.15.72mos ago
0.15.62mos ago
0.15.53mos ago
0.15.43mos ago
0.15.33mos ago
0.15.23mos ago
0.15.13mos ago
0.15.03mos ago
0.14.21rc04mos ago
0.14.204mos ago
0.14.20rc04mos ago
0.14.194mos ago
0.14.18rc64mos ago
0.14.18rc54mos ago
0.14.18rc44mos ago
0.14.18rc34mos ago
0.14.18rc24mos ago
0.14.174mos ago
0.14.17rc104mos ago
0.14.17rc84mos ago
0.14.17rc74mos ago
0.14.17rc64mos ago
0.14.17rc54mos ago
0.14.17rc44mos ago
0.14.17rc34mos ago
0.14.164mos ago
0.14.16rc44mos ago
0.14.16rc34mos ago
0.14.16rc24mos ago
0.14.155mos ago
0.14.145mos ago
0.14.135mos ago
0.14.125mos ago
0.14.115mos ago
0.14.105mos ago
0.14.95mos ago
0.14.9rc05mos ago
0.14.86mos ago
0.14.8rc16mos ago
0.14.76mos ago
0.14.7rc06mos ago
0.14.66mos ago
0.14.6rc06mos ago
0.14.56mos ago
0.14.5rc06mos ago
0.14.46mos ago
0.14.4rc06mos ago
0.14.37mos ago
0.14.3rc07mos ago
0.14.27mos ago
0.14.2rc07mos ago
0.14.17mos ago
0.14.1rc07mos ago
0.14.07mos ago
0.14.0rc07mos ago
0.13.198mos ago
0.13.19rc28mos ago
0.13.188mos ago
0.13.18rc08mos ago
0.13.178mos ago
0.13.17rc08mos ago
0.13.168mos ago
0.13.16rc08mos ago
0.13.158mos ago
0.13.15rc08mos ago
0.13.148mos ago
0.13.14rc08mos ago
0.13.139mos ago
0.13.13rc09mos ago
0.13.129mos ago
0.13.12rc29mos ago
0.13.1110mos ago
0.13.11rc010mos ago
0.13.1010mos ago
0.13.10rc310mos ago
0.13.910mos ago
0.13.9rc010mos ago
0.13.810mos ago
0.13.8rc010mos ago
0.13.710mos ago
0.13.7rc010mos ago
0.13.610mos ago
0.13.6rc010mos ago
0.13.51yr ago
0.13.5rc01yr ago
0.13.41yr ago