Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
define_asset_job now accepts a hooks argument.
Added support for sqlalchemy==2.x
[ui] The Runs page has been revamped with better filtering support
[ui] The automaterialize policy page for SDA’s using the experimental AutomaterializePolicy feature now indicates time periods where no materializations happened due to no materialization conditions being met
[dagster-k8s] The Dagster Helm chart now includes an additionalInstanceConfig key that allows you to supply additional configuration to the Dagster instance.
[dagster-aws] The EcsRunLauncher now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.
[dagster-aws] The EcsRunLauncher now includes a run_ecs_tags config key that lets you configure tags on the launched ECS task for each run.
When a sensor had yield statement and also returned a SkipReason, the SkipReason would be ignored. This has been fixed.
[dagster-cloud] Fixed a bug in the docker user code launcher that was preventing code location containers from being properly cleaned up.
Fixed an issue where the Dagster UI would sometimes raise a `RuntimeError: dictionary changed size during iteration exception while code servers were being reloaded.
Fixed an issue where the Dagster daemon reloaded your code server every 60 seconds when using the new experimental dagster code-server start CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.
Fixed a GraphQL error which would occur when loading the default config for jobs without config.
[dagster-dbt] Fixed an error which would arise when trying to load assets from a DBT Cloud instance using the Pythonic-style resource.
@observable_source_asset-decorated functions can now return a DataVersionsByPartition to record versions for partitions.
@dbt_assets
DbtCliTask's created by invoking DbtCli.cli(...) now have a method .is_successful(), which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.
Descriptions of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_description.
Added support for streaming upload of compute logs to Dagster Cloud
The ECS agent now supports setting server_ecs_tags and run_ecs_tags that apply to each service or task created by the agent. See the docs for more information.
Fixed run filtering for calls to instance.get_run_partition_data in Dagster Cloud.
Adding a .env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment variables referenced in your dagster.yaml file using an env: key. Previously, setting a .env file only worked for environment variables referenced in your Dagster code.
When using the GraphQL Python client, submit_job_execution can now take in a RunConfig object. Previously, it could only take a Python dictionary with the run configuration.
Asset backfills can now be canceled via a button in the UI. This will terminate unfinished runs and prevent new runs from being queued.
Introduced a new user setting which allows automatically expanding default config for jobs in the launchpad.
[dagit] Dagit now supports displaying a Polars tag on the asset graph.
Fixed an issue where setting a resource in an op didn’t work if the Dagster job was only referenced within a schedule or sensor and wasn’t included in the jobs argument to Definitions.
[dagster-slack][dagster-pagerduty][dagster-msteams][dagster-airflow] Fixed issue where pre-built sensors and hooks which created urls to the runs page in the UI would use the old /instance/runs path instead of the new /runs.
Added an experimental dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc, however, dagster code-server start runs the code in a subprocess, so it can reload code from the Dagster UI without needing to restart the command. This can be useful for jobs that load code from some external source and may want to reload job definitions without restarting the process.
Added a new sensors.num_submit_workers key to dagster.yaml that can be used to decrease latency when a sensor emits multiple run requests within a single tick. See the docs for more information.
[dagster-k8s] The k8s_job_executor can now be used to launch each step of a job in its own Kubernetes, pod, even if the Dagster deployment is not using the K8sRunLauncher to launch each run in its own Kubernetes pod.
[ui] When viewing a list of schedules or sensors in Overview or on individual code locations, it is now possible to filter by running state.
[ui] A new experimental asset overview page is available via user settings.
Fixed issue where asset jobs featuring a subset of a larger multi_asset could be rendered as an op-based job.
Fixed an issue where Pythonic IO managers could not be passed to the io_manager_def param on an asset.
Fixed an issue where late-binding an executor to a job, such as providing one to Definitions, would not correctly update the config schema.
[dagster-k8s] Fixed an issue where setting maxResumeRunAttempts to null in the helm chart would cause it to be set to a default value of 3 instead of disabling run retries.
[dagster-k8s] Fixed an issue where the k8s_job_executor would sometimes fail with a 409 Conflict error after retrying the creation of a Kubernetes pod for a step, due to the job having already been created during a previous attempt despite raising an error.
[dagster-dbt] Fixed an issue where dbt logs were not being captured when invoking dbt commands using the resource.
[dagster-dbt] Fixed an issue where if op_name was passed to load_assets_from_dbt_manifest, and a select parameter was specified, a suffix would be appended to the desired op name.
[ui] Fixed an issue where using a path prefix for dagit would lead to JavaScript bundle loading errors.
[ui] Resizing the viewport while viewing the Run timeline now correctly resizes the timeline and its contents.
[ui] Ctrl-scroll to zoom the DAG view, and shift-scroll to pan horizontally now work on all platforms and an instructional tooltip explains the zoom interaction.
[dagster-dbt] Added a new implementation of the dbt Resource, DbtCli, in dagster_dbt.cli. This new resource only support dbt-core>=1.4.0.
[dagster-dbt] Added a new decorator @dbt_assets in dagster_dbt.asset_decorator that allows you to specify a compute function for a selected set of dbt assets that loaded as an AssetsDefinition.
[dagster-duckdb] New guide and API reference page for the DuckDB I/O managers and resource
[rbac] - Added documentation for the new Dagster Cloud Teams feature for role-based access control (RBAC). We’ve also revamped the documentation for Dagster Cloud user roles and permissions - check it out here.
Fixed a typo in the "Using Environment Variables and Secrets" guide (thanks snikch!)
A new max_materializations_per_minute parameter (with a default of 1) to AutoMaterializationPolicy.eager() and AutoMaterializationPolicy.lazy() allows you to set bounds on the volume of work that may be automatically kicked off for each asset. To restore the previous behavior, you can explicitly set this limit to None.
DailyPartitionsDefinition, HourlyPartitionsDefinition, WeeklyPartitionsDefinition, and MonthlyPartitionsDefinition now support and end_date attribute.
[ui] When GraphQL requests time out with 504 errors, a toaster message is now shown indicating the error, instead of failing silently.
[dagster-snowflake] The Snowflake I/O managers now support authentication via unencrypted private key.
When using AutoMaterializePolicys or build_asset_reconciliation_sensor, a single new data version from an observable source asset could trigger multiple runs of the downstream assets. This has been fixed.
Fixed a bug with pythonic resources where raw run config provided to a resource would be ignored.
We previously erroneously allowed the use of EnvVar and IntEnvVar within raw run config - although they just returned the name of the env var rather than retrieve its value. This has been fixed to error directly.
[ui] Fixed an issue in the left navigation where code locations with names with URI-encodable characters (e.g. whitespace) could not be expanded.
[ui] Fixed an issue where the time shown on the Runs page when a run was starting was shown in an incorrect timezone.
[dagster-dbt] Fixed an issue where selecting models by * was being interpreted as glob pattern, rather than as a dbt selector argument. We now explicitly set the default selection pattern as fqn:*.
[dagster-cloud cli] Fixed and issue where dagster-cloud serverless deploy did not create a unique image tag if the --image tag was not specified.
Run monitoring will now detect runs that are stuck in a CANCELING state due to an error during termination and move them into CANCELED. See the docs for more information.
TimeWindowPartitionMapping objects are now current-time aware. Subsequently, only upstream/downstream partitions existent at the current time are returned.
ExecuteJobResult was renamed to JobExecutionResult (ExecuteJobResult remains a deprecated alias)
New AssetSelection.key_prefixes method allows matching asset keys starting with a provided prefix.
[dagster-airflow] persistent database URI can now be passed via environment variable
[dagster-azure] New ConfigurablePickledObjectADLS2IOManager that uses pythonic config
[dagster-fivetran] Fivetran connectors that are broken or incomplete are now ignored
[dagster-gcp] New DataProcResource follows the Pythonic resource system. The existing dataproc_resource remains supported.
[dagster-k8s] The K8sRunLauncher and k8s_job_executor will now retry the api call to create a Kubernetes Job when it gets a transient error code (500, 503, 504, or 401).
[dagster-snowflake] The SnowflakeIOManager now supports private_keys that have been base64 encoded to avoid issues with newlines in the private key. Non-base64 encoded keys are still supported. See the SnowflakeIOManager documentation for more information on base64 encoded private keys.
[ui] Unpartitioned assets show up on the backfill page
[ui] On the experimental runs page you can open the “view all tags” dialog of a row by pressing the hotkey ‘t’ while hovering that row.
[ui] The “scroll-to-pan” feature flag has been removed, and scroll-to-pan is now default functionality.
The server side polling for events during a live run has had its rate adjusted and no longer uses a fixed interval.
[dagster-postgres] Fixed an issue where primary key constraints were not being created for the kvs, instance_info, and daemon_hearbeats table for existing Postgres storage instances that were migrating from before 1.2.2. This should unblock users relying on the existence of a primary key constraint for replication.
Fixed a bug that could cause incorrect counts to be shown for missing asset partitions when partitions are in progress
Fixed an issue within SensorResult evaluation where multipartitioned run requests containing a dynamic partition added in a dynamic partitions request object would raise an invalid partition key error.
[ui] When trying to terminate a queued or in-progress run from a Run page, forcing termination was incorrectly given as the only option. This has been fixed, and these runs can now be terminated normally.
[ui] Fixed an issue on the asset job partitions page where an infinite recursion error would be thrown when using TimeWindowPartitionMapping.
[dagster-databricks] Polling for the status of skipped Databricks runs now properly terminates.
[dagster-airbyte] When supplying an airbyte_resource to load_assets_from_connections , you may now provide an instance of the AirbyteResource class, rather than just airbyte_resource.configured(...) (thanks @joel-olazagasti!)
[dagster-airbyte] Fixed an issue connecting to destinations that support normalization (thanks @nina-j!)
Fix an error in the docs code snippets for IO managers (thanks out-running-27!)
Added an example to show how to build the Dagster's Software-Defined Assets for an analytics workflow with different deployments for a local and prod environment. (thanks @PedramNavid!)
[dagster-celery] Fixed an issue where the dagster-celery CLI accepted an inconsistent configuration format - it now matches the same format as the celery_executor. Thanks @boenshao!
The ECS agent will now display task logs and other debug information when a code location fails to start up.
You can now set ecs_timeout in your ECS user code launcher config to extend how long the ECS agent polls for new code servers to start. Extending this timeout is useful if your code server takes an unusually long time to start up - for example, because it uses a very large image.
Added support for running the Dagster Cloud Kubernetes agent in a cluster using istio.