Snowflake DSA-C02 - SnowPro Advanced: Data Scientist Certification Exam

Explanation

Data processing and analysis cannot happen without data profiling—reviewing source data for con-tent and quality. As data gets bigger and infrastructure moves to the cloud, data profiling is increasingly important.

What is data profiling?

Data profiling is the process of reviewing source data, understanding structure, content and interrelationships, and identifying potential for data projects.

Data profiling is a crucial part of:

· Data warehouse and business intelligence (DW/BI) projects—dataprofiling can uncover data quality issues in data sources, and what needs to be corrected in ETL.

· Data conversion and migration projects—data profiling can identify data quality issues, which you can handle in scripts and data integration tools copying data from source to target. It can also un-cover new requirements for the target system.

· Source system data quality projects—data profiling can highlight data which suffers from serious or numerous quality issues, and the source of the issues (e.g. user inputs, errors in interfaces, data corruption).

Data profiling involves:

· Collecting descriptive statistics like min, max, count and sum.

· Collecting data types, length and recurring patterns.

· Tagging data with keywords, descriptions or categories.

· Performing data quality assessment, risk of performing joins on the data.

· Discovering metadata and assessing its accuracy.

· Identifying distributions, key candidates, foreign-key candidates, functional dependencies, embedded value dependencies, and performing inter-table analysis.

Explanation

Window Functions

A window function operates on a group (“window”) of related rows.

Each time a window function is called, it is passed a row (the current row in the window) and the window of rows that contain the current row. The window function returns one output row for each input row. The output depends on the individual row passed to the function and the values of the other rows in the window passed to the function.

Some window functions are order-sensitive. There are two main types of order-sensitive window functions:

Rank-related functions.

Window frame functions.

Rank-related functions list information based on the “rank” of a row. For example, if you rank stores in descending order by profit per year, the store with the most profit will be ranked 1; the second-most profitable store will be ranked 2, etc.

Window frame functions allow you to perform rolling operations, such as calculating a running total or a moving average, on a subset of the rows in the window.

Explanation

Supervised learning methodology applies conditional probability of all the variables with respective the dependent variable and generally conditional probability of variables is nothing but a basic method of estimating the statistics for few random experiments.

Conditional probability is thus the likelihood of an event or outcome occurring based on the occurrence of some other event or prior outcome. Two events are said tobe independent if one event occurring does not affect the probability that the other event will occur.

Explanation

To return the contents of a DataFrame as a Pandas DataFrame, use the to_pandas method.

For example:

1.>>> python_df = session.create_dataframe(["a", "b", "c"])

2.>>> pandas_df = python_df.to_pandas()

Explanation

As a consumer, you can do the following:

· Discover and test third-party data sources.

· Receive frictionless access to raw data products from vendors.

· Combine new datasets with your existing data in Snowflake to derive new business insights.

· Have datasets available instantly and updated continually for users.

· Eliminate the costs of building and maintaining various APIs and data pipelines to load and up-date data.

· Use the business intelligence (BI) tools of your choice.

Explanation

Moving Data from Spark to Snowflake

The steps for saving the contents of a DataFrame to a Snowflake table are similar to writing from Snowflake to Spark:

1. Use the write() method of the DataFrame to construct a DataFrameWriter.

2. Specify SNOWFLAKE_SOURCE_NAME using the format() method.

3. Specify the connector options using either the option() or options() method.

4. Use the dbtable option to specify the table to which data is written.

5. Use the mode() method to specify the save mode for the content.

Examples

1.df.write

2..format(SNOWFLAKE_SOURCE_NAME)

3..options(sfOptions)

4..option("dbtable", "t2")

5..mode(SaveMode.Overwrite)

6..save()

Explanation

To track changing data trends, create a data drift monitor that uses the training data as a baseline and the new data as a target.

Model drift and decay are concepts that describe the process during which the performance of a model deployed to production degrades on new, unseen data or the underlying assumptions about the data change.

These are important metrics to track once models are deployed toproduction. Models must be regularly re-trained on new data. This is referred to as refitting the model. This can be done either on a periodic basis, or, in an ideal scenario, retraining can be triggered when the performance of the model degrades below a certain pre-defined threshold.