Databricks Databricks-Machine-Learning-Professional - Databricks Certified Machine Learning Professional

Jensen-Shannon (JS) distance is a symmetric measure of the distance between two probability distributions. It is based on the Kullback-Leibler (KL) divergence, but unlike KL, it does not depend on which distribution is considered the reference and which is considered the comparison. JS distance is also bounded between 0 and 1, which makes it easier to interpret and compare across different features.

Kolmogorov-Smirnov (KS) test is a non-parametric test that compares the cumulative distribution functions (CDFs) of two samples. It calculates the maximum difference between the CDFs and compares it to a critical value to determine if the samples are from the same distribution. KS test is sensitive to the shape, location, and scale of the distributions, but it can also be too sensitive for large datasets, where small changes can result in significant differences.

JS distance is more robust than KS test for numeric feature drift detection because it can handle large datasets without being too sensitive to minor fluctuations. It can also capture changes in the shape and spread of the distributions, not just the location. JS distance does not require any manual threshold or cutoff determinations, unlike KS test, which requires choosing a significance level and a critical value. JS distance can also be generalized to multivariate cases, while KS test is limited to univariate cases. References:

Which test is the best? We compared 5 methods to detect data drift on large datasets

The Jensen-Shannon Divergence: A Measure of Distance Between Probability Distributions

Kullback–Leibler vs Kolmogorov-Smirnov distance

Batch deployment is the most common paradigm for machine learning projects, as it accounts for 80-90% of deployments1. Batch deployment leverages databases and object storage to process a large batch of input data at once, rather than processing each input data point individually in real time. Batch deployment is well suited for large data volume scenarios that need efficient processing but response time isn’t critical. For example, you might use batch deployment to generate recommendations for customers based on their purchase history, or to detect anomalies in sensor data collected over a period of time. Batch deployment is less expensive and more efficient than real-time deployment, as the underlying compute runs only during the batch job and can be scaled up or down as needed. Batch deployment can be implemented using Databricks Machine Learning endpoints, which provide a unified interface to invoke and manage model deployments across compute types2. References:

Databricks Machine Learning Professional Exam Guide, Section 3: Model Deployment, p. 4

Databricks Machine Learning Documentation, Deploying ML Models, Batch Inference, p. 1

Machine Learning in Production - Databricks, The Four Deployment Paradigms, p. 9

Deploy machine learning models in production environments, Deployment methods, p. 1

Concept drift is a change in the relationship between input and output data over time, which can affect the performance and accuracy of the model. To monitor concept drift, one needs to compare the predicted values from the model with the observed (actual) label values from the data source. A common way to do this is to compute an evaluation metric, such as accuracy, precision, recall, F1-score, etc., using the observed and predicted values. This metric can indicate how well the model is performing on the current data, and how it differs from the baseline performance on the training data. If the metric shows a significant decline over time, it may suggest that concept drift has occurred and the model needs to be updated123 References:

Monitoring Concept Drift - Databricks

Data drift, concept drift, and how to monitor for them - Mona Labs

8 Concept Drift Detection Methods To Use With ML Models - Aporia

To compute predictions for spark_df when the missing feature values can be found in the Feature Store by searching for features by customer_id, you can use the following code block:

Python

# Get the missing features from the Feature Store using the model URI and the customer_id column

df = fs.get_missing_features(spark_df, model_uri, lookup_key="customer_id")

# Score the DataFrame using the model URI and the Feature Store Client

fs.score_batch(model_uri, df)

AI-generated code. Review and use carefully. More info on FAQ.

The fs.get_missing_features method takes a Spark DataFrame, a model URI, and a lookup key as arguments. It returns a new Spark DataFrame that contains the originalcolumns plus the missing features that are required by the model. The missing features are retrieved from the Feature Store by joining the DataFrame with the feature tables using the lookup key. The lookup key must match the primary key of the feature tables. The model URI must point to a registered model that was trained using features from the Feature Store1.

The fs.score_batch method takes a model URI and a Spark DataFrame as arguments. It applies the model to the DataFrame and returns a new Spark DataFrame that contains the original columns plus a prediction column. The model URI must point to a registered model that was trained using features from the Feature Store2.

The other options are incorrect because:

Option A: fs.score_model is not a valid method name, as it is missing an underscore. The correct method name is fs.score_batch2.

Option B: fs.score_model without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.

Option D: fs.score_batch without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.

Option E: fs.score_batch without specifying the lookup key will not work, as the fs.get_missing_features method requires a lookup key to join the DataFrame with the feature tables. The correct way is to use fs.get_missing_features with the lookup key “customer_id” before fs.score_batch12. References: Get missing features, Score batch

MLflow provides a module called mlflow.shap that can be used to automatically calculate and log a Shapley feature importance plot using the SHAP library. The mlflow.shap.log_explanation function takes a predict function and a sample of input data, and computes the SHAP values and base values for each feature. It then logs the following artifacts to the current run:

Base values

SHAP values (computed using shap.KernelExplainer)

Summary bar plot (shows the average impact of each feature on model output) The mlflow.shap.log_explanation function can be used with any model that has a predict function, such as scikit-learn, xgboost, etc1234 References:

mlflow.shap — MLflow 2.9.1 documentation

Extract feature importance from a mlflow 1.9 PyFuncModel model

9.6 SHAP (SHapley Additive exPlanations) | Interpretable Machine Learning

load MLFlow model and plot feature importance with feature names

To view the data visualizations that are logged with each run, you can go to the Artifacts section of the MLflow Run page in Databricks. The Artifacts section shows the files and directories that are logged as artifacts for a run. You can browse the artifact hierarchy and preview the files, such as images, text, or HTML1. You can also download the artifacts or copy their URIs for further use2. The other options are incorrect because:

Option A: The MLflow Model Registry Model page shows the information and metadata of a registered model, such as its name, description, versions, stages, and lineage. It does not show the data visualizations that are logged with each run3.

Option B: The Artifacts section of the MLflow Experiment page shows the artifacts that are logged for an experiment, not for a specific run. It does not allow you to preview the files or browse the artifact hierarchy4.

Option C: Logged data visualizations can be viewed in Databricks using the Artifacts section of the MLflow Run page1.

Option E: There is no Figures section of the MLflow Run page in Databricks. The Figures section is only available in the open source MLflow UI, which shows the plots that are logged as figures for a run5. References: View run artifacts, Log, list, and download artifacts, Manage models, View experiment artifacts, Logging Visualizations with MLflow

The mlflow.register_model operation can be used to add a new version of an MLflow Model to an existing MLflow Model Registry model. This operation takes the URI of the model source (such as an MLflow run or a local path) and the name of the registered model as arguments, and returns a ModelVersion object that contains metadata about the registered model version. The mlflow.register_model operation can also create a new registered model if the name does not exist in the Model Registry.

The other options are incorrect for the following reasons:

B. MlflowClient.update_registered_model is used to update the metadata of an existing registered model, such as the name or description, not to add a new version of the model.

C. mlflow.add_model_version does not exist as a valid MLflow operation.

D. MlflowClient.get_model_version is used to fetch the metadata of a specific version of a registered model, not to add a new version of the model.

E. The machine learning engineer does not need to create an entirely new MLflow Model Registry model, as they can use the mlflow.register_model operation to add a new version of the model to the existing “project” model.

References: Adding an MLflow Model to the Model Registry, mlflow.register_model

To move a model version from the Staging stage to the Production stage using the MLflow Client, you can use the client.transition_model_version_stage method. This method takes the name of the registered model, the version of the model, and the target stage as arguments. It transitions the model version to the specified stage and returns the updated model version object. The syntax of the method is as follows:

Python

client.transition_model_version_stage (name, version, stage)

AI-generated code. Review and use carefully. More info on FAQ.

The other options are incorrect because:

Option A: client.transition_model_version_stage with the stage argument set to “Staging” will not move the model version from the Staging stage to the Production stage, but rather keep it in the same stage.

Option B: client.transition_model_stage is not a valid method name, as it is missing an underscore. The correct method name is client.transition_model_version_stage.

Option D: client.transition_model_stage is not a valid method name, as it is missing an underscore. The correct method name is client.transition_model_version_stage. Also, the method does not take from and to arguments, but only a stage argument.

Option E: client.transition_model_version_stage does not take from and to arguments, but only a stage argument. The from argument is redundant, as the method will automatically transition the model version from its current stage tothe target stage. References: MLflow Model Registry, MlflowClient.transition_model_version_stage

Categorical feature drift is a change in the distribution of the input data over time for categorical features, which can affect the performance and accuracy of the model. Monitoring categorical feature drift is important to ensure that the model is still valid and reliable for the current data. One simple statistic to monitor for categorical feature drift is the combination of mode, number of unique values, and percentage of missing values for each categorical feature. These statistics can provide a quick overview of the changes in the data distribution, such as the most frequent category, the diversity of categories, and the quality of data. If these statistics deviate significantly from the baseline values, it may indicate a categorical feature drift. However, these statistics may not capture all the nuances of the data distribution, such as the relative frequencies of different categories, the similarity of categories, etc. Therefore, other methods, such as statistical tests or information-theoretic measures, may be needed to complement the simple statistics and provide a more comprehensive analysis of the categorical feature drift123 References:

Monitoring Feature Drift - Databricks

Drift Metrics: How to Select the Right Metric to Analyze Drift

Detect data drift on datasets (preview) - Azure Machine Learning