Building Pipelines with Dynamic Tables

With Dynamic Tables, Snowflake simplifies the creation and maintenance of data pipelines. Gone are the days when users had to manually consolidate their data. But how do you use Dynamic Tables exactly? What are the design decisions? How do you deal with errors and performance regressions?

In this video, you'll learn how to set up a full data pipeline. We'll navigate common issues using the many metric sources that come with Dynamic Tables. Where necessary, we'll change our setup on the fly to ensure our business requirements are met and explain the effect of the target lag parameter, the difference between full and incremental refreshes, and the reasons for skipped refresh.

❄ Join our YouTube community ❄

Learn how to build your application on Snowflake:

Join the Snowflake Community:

[mauricioz]

DT are very useful to simplify pipelines and object management, support, monitoring, documentation and support.

[vikihome]

Can I create the DT on top of external table ! looks it won't allow, is there any work around if in case I need to use external table ?

[radhekishansoni]

I could not find anywhere in documentation, but is there any optimisation in create-replace of a dynamic table. Does it have any way to know that may be some partial data is already there.
For example when we introduce a new column, it is highly likely that no other data was changed but does dynamic table consider this when it refreshes on create-replace.

[sharma_ji_in_uk]

we can use non dynamic tables in the dynamic sql?

[EdchelStephenNini]

Hi, can we write dynamic table definition using python instead of SQL?

[ur]

where you define call start_data_gen(); and call send_alert_email(); function ?

[TanuE-lxzw]

dynamic tables can be monitor through snowsight->activity->dynamic table
I need to know which snowflake account usage table it is using to display dynamic tables details in snowsight?
I need to get all the refresh history of all the tables, so how to get this details?

[VigyaanJyoti]

The demo example is pretty complex and the presentor should have chosen a simple example to explain things.

[luisfernandocardiel]

where you define call start_data_gen(); and call send_alert_email(); function ?