Transform Data into Insights with Dagster and Deepnote
Data Engineering for Empowered Business Decisions: ETL, Exploration & Visualization
Transform Data into Insights with Dagster and Deepnote udemy course free download
Data Engineering for Empowered Business Decisions: ETL, Exploration & Visualization
Do you struggle with making data-driven decisions for your business due to scattered, inconsistent, and inaccessible data? This course is the solution! Learn to build a streamlined and efficient ETL pipeline that will allow you to turn data into actionable insights.
This course teaches you how to build a system that collects data from multiple sources, normalizes it, and stores it in a consistent and accessible format. You will learn how to extract data, explore and preprocess it, and ultimately visualize it to support better decision-making and optimize business processes.
Forget about big data and cluster management headaches, this course is designed to get you up and running quickly with a real-time ETL pipeline. With infrastructure costs under $50 a month, you can start seeing immediate results and return on investment for your clients or company.
In the first part of the course, I will walk you through the architecture and introduce you to the tools we will be using:
Deepnote, as a setup-free development environment
Dagster, as the pipeline orchestrator
Metabase, as a low-code data visualization platform
While the course will introduce you to the relevant features of Deepnote and Metabase, it is mostly focused on Dagster.
In the next part, we will get started by generating dummy sales data of a hypothetical company using Deepnote. The code will be provided for this. Once we have the data, the course will dive into data exploration and preprocessing techniques using Python and SQL in Deepnote, including cleaning and normalizing data from various sources such as relational and JSON data, Excel sheets, and more. We will implement the processing logic in Deepnote, then commit it to a Git repository that will be shared with Dagster.
In the following section, we will wrap the business logic with Dagster operations and jobs, then deploy them to Dagster Cloud (self-hosted option also available), which will allow you to manage everything from a single, unified view. In this section, you will also learn a few tricks to speed up and optimize processing, such as parallelization or streamed processing.
In the final section of this course, you'll bring your preprocessed data to life with Metabase. With a few simple clicks, even non-technical individuals will be able to create stunning, powerful visualizations that unlock the full potential of your data.
By the end of this course, you'll have a comprehensive understanding of the tools used and how they work together, empowering you to provide tangible benefits to your clients or company from day one, measured in thousands or tens of thousands of dollars.
The choice is yours - will you seize this opportunity to deliver massive benefits to your company or clients, and claim your fair share of the rewards?
