Changelog#

0.6.6#

Breaking

The selector argument to PipelineDefinition has been removed. This API made it possible to construct a PipelineDefinition in an invalid state. Use PipelineDefinition.build_sub_pipeline instead.

New

Added the dagster_prometheus library, which exposes a basic Prometheus resource.
Dagster Airflow DAGs may now use GCS instead of S3 for storage.
Expanded interface for schedule management in Dagit.

Dagit

Performance improvements when loading, displaying, and editing config for large pipelines.
Smooth scrolling zoom in the explore tab replaces the previous two-step zoom.
No longer depends on internet fonts to run, allowing fully offline dev.
Typeahead behavior in search has improved.
Invocations of composite solids remain visible in the sidebar when the solid is expanded.
The config schema panel now appears when the config editor is first opened.
Interface now includes hints for autocompletion in the config editor.
Improved display of solid inputs and output in the explore tab.
Provides visual feedback while filter results are loading.
Better handling of pipelines that aren't present in the currently loaded repo.

Bugfix

Dagster Airflow DAGs previously could crash while handling Python errors in DAG logic.
Step failures when running Dagster Airflow DAGs were previously not being surfaced as task failures in Airflow.
Dagit could previously get into an invalid state when switching pipelines in the context of a solid subselection.
frozenlist and frozendict now pass Dagster's parameter type checks for list and dict.
The GraphQL playground in Dagit is now working again.

Nits

Dagit now prints its pid when it loads.
Third-party dependencies have been relaxed to reduce the risk of version conflicts.
Improvements to docs and example code.

Breaking

The interface for type checks has changed. Previously the type_check_fn on a custom type was required to return None (=passed) or else raise Failure (=failed). Now, a type_check_fn may return True/False to indicate success/failure in the ordinary case, or else return a TypeCheck. The newsuccess field on TypeCheck now indicates success/failure. This obviates the need for the typecheck_metadata_fn, which has been removed.
Executions of individual composite solids (e.g. in test) now produce a CompositeSolidExecutionResult rather than a SolidExecutionResult.
dagster.core.storage.sqlite_run_storage.SqliteRunStorage has moved to dagster.core.storage.runs.SqliteRunStorage. Any persisted dagster.yaml files should be updated with the new classpath.
is_secret has been removed from Field. It was not being used to any effect.
The environmentType and configTypes fields have been removed from the dagster-graphql Pipeline type. The configDefinition field on SolidDefinition has been renamed to configField.

Bugfix

PresetDefinition.from_files is now guaranteed to give identical results across all Python minor versions.
Nested composite solids with no config, but with config mapping functions, now behave as expected.
The dagster-airflow DagsterKubernetesPodOperator has been fixed.
Dagit is more robust to changes in repositories.
Improvements to Dagit interface.

New

dagster_pyspark now supports remote execution on EMR with the @pyspark_solid decorator.

Nits

Documentation has been improved.
The top level config field features in the dagster.yaml will no longer have any effect.
Third-party dependencies have been relaxed to reduce the risk of version conflicts.

Scheduler errors are now visible in Dagit
Run termination button no longer persists past execution completion
Fixes run termination for multiprocess execution
Fixes run termination on Windows
dagit no longer prematurely returns control to terminal on Windows
raise_on_error is now available on the execute_solid test utility
check_dagster_type added as a utility to help test type checks on custom types
Improved support in the type system for Set and Tuple types
Allow composite solids with config mapping to expose an empty config schema
Simplified graphql API arguments to single-step re-execution to use retryRunId, stepKeys execution parameters instead of a reexecutionConfig input object
Fixes missing step-level stdout/stderr from dagster CLI

Adds a type_check parameter to PythonObjectType, as_dagster_type, and @as_dagster_type to enable custom type checks in place of default isinstance checks. See documentation here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/types.html#custom-type-checks
Improved the type inference experience by automatically wrapping bare python types as dagster types.
Reworked our tutorial (now with more compelling/scary breakfast cereal examples) and public API documentation. See the new tutorial here: https://dagster.readthedocs.io/en/latest/sections/learn/tutorial/index.html
New solids explorer in Dagit allows you to browse and search for solids used across the repository.
Enabled solid dependency selection in the Dagit search filter.
- To select a solid and its upstream dependencies, search +{solid_name}.
- To select a solid and its downstream dependents, search {solid_name}+.
- For both search +{solid_name}+.
Added a terminate button in Dagit to terminate an active run.
Added an --output flag to dagster-graphql CLI.
Added confirmation step for dagster run wipe and dagster schedule wipe commands (Thanks @shahvineet98).
Fixed a wrong title in the dagster-snowflake library README (Thanks @Step2Web).

Changed composition functions @pipeline and @composite_solid to automatically give solids aliases with an incrementing integer suffix when there are conflicts. This removes to the need to manually alias solid definitions that are used multiple times.
Add dagster schedule wipe command to delete all schedules and remove all schedule cron jobs
execute_solid test util now works on composite solids.
Docs and example improvements: https://dagster.readthedocs.io/
Added --remote flag to dagster-graphql for querying remote Dagit servers.
Fixed issue with duplicate run tag autocomplete suggestions in Dagit (#1839)
Fixed Windows 10 / py3.6+ bug causing pipeline execution failures