We'll explore how this works in the next couple sections. With Airflow, it’s difficult and frustrating to do any of these. ![]() If you can run it in a unit test, you can write a suite that expresses your expectations about how it works and continually defends against future breakages. If you can run it on your laptop, you can iterate on it quickly. If you can run your data pipeline before merging it to production, you can catch problems before they break production. Your choice of orchestrator has a huge impact on how fast you can develop your data pipelines and how much of your time you spend fixing them. While you're here, we’d love for you to join Dagster’s community by starring it on GitHub and joining our Slack. ![]() We’ll also discuss Dagster’s Airflow integration, which allows you to build pipelines in Dagster even when you’re already using Airflow heavily. In this post, we’ll dig into each of these areas in greater detail, as well as differences in data-passing, event-driven execution, and backfills. Airflow makes it awkward to isolate dependencies and provision infrastructure.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |