Databricks Databricks-Machine-Learning-Professional - Databricks Certified Machine Learning Professional

MLflow provides a module called mlflow.pyfunc that can be used to deploy any Python model as a PySpark UDF. The mlflow.pyfunc.spark_udf function takes a Spark session and a model URI, and returns a Spark UDF that can be applied to a Spark DataFrame. The model URI can be a local path, a remote path, or a run ID with a relative path within the MLflow tracking server. The mlflow.pyfunc.spark_udf function can be used with any model that has a predict function, such as scikit-learn, xgboost, etc12

The other code blocks are incorrect because:

A. The mlflow.pyfunc.spark_udf function requires a Spark session as the first argument, not a Spark DataFrame.

B. It is possible to deploy a scikit-learn model on a Spark DataFrame using mlflow.pyfunc.spark_udf or other methods, such as broadcasting the model or using Fugue.

C. There is no module called sklearn.spark_udf. To use a scikit-learn model as a Spark UDF, mlflow.pyfunc.spark_udf or other methods should be used.

D. There is no method called spark.spark_udf. To create a Spark UDF, the spark.udf.register or pyspark.sql.functions.udf methods should be used.

References:

mlflow.pyfunc — MLflow 2.9.1 documentation

Deploy models for batch inference and prediction - Databricks

The Feature Store Client fs provides a method called create_table that can be used to create and populate a Feature Store table from a Spark DataFrame. The create_table method takes the following parameters:

name: The name of the feature table to create. It must include the database name, such as ‘recommender_system.new_table’.

primary_keys: The name or list of names of the columns that uniquely identify each row in the feature table. For example, ‘customer_id’ or [‘customer_id’, ‘date’].

df: The Spark DataFrame that contains the data to populate the feature table. It must have the same schema as the feature table, and include the primary key columns.

description: An optional string that describes the feature table.

partition_columns: An optional list of column names to partition the feature table by. For example, [‘date’, ‘region’].

online_store: An optional boolean flag that indicates whether to publish the feature table to an online store. The default value is False.

online_store_options: An optional dictionary that specifies the configuration options for the online store, such as table name, database name, and write mode. For example, {‘table_name’: ‘new_table_online’, ‘database_name’: ‘recommender_system’, ‘write_mode’: ‘overwrite’}.

The create_table method creates a Delta table with the specified name, primary keys, partition columns, and description, and writes the data from the DataFrame to the table. If online_store is True, it also publishes the feature table to an online store with the specified options. The create_table method returns a FeatureTable object that represents the feature table12

The other code blocks are incorrect because:

B. The create_table method requires the df parameter to populate the feature table with data.

C. The write method of the DataFrame does not have a mode called ‘fs’ or a path parameter. To write a DataFrame to a Delta table, the mode should be one of ‘append’, ‘overwrite’, ‘ignore’, or ‘error’, and the format should be ‘delta’. To write a DataFrame to a Feature Store table, the create_table or write_table method of the Feature Store Client should be used.

D. The create_table method does not have a function parameter. To use a Python function to compute the features, the function should be called first and the output DataFrame should be passed to the df parameter.

E. The write method of the DataFrame does not have a mode called ‘feature’ or a path parameter. To write a DataFrame to a Feature Store table, the create_table or write_table method of the Feature Store Client should be used.

References:

Feature Store Python API reference - Databricks

Work with features in Workspace Feature Store - Databricks

The Databricks REST APIs allow users to programmatically access Databricks features and services, such as clusters, jobs, workspaces, libraries, secrets, and more. The Jobs API enables users to create, edit, delete, run, and list jobs, as well as manage job runs. Users can use the Jobs API to programmatically create a new Databricks job whose schedule depends on the result of some automated tests in a machine learning pipeline. For example, users can use the create endpoint to create a new job with a cron_schedule parameter that specifies the cron expression for the job schedule. Users can also use the run_now endpoint to run an existing job with a run_name parameter that specifies the name of the run. Users can use the runs/get endpoint to get the run result state of a job run, and use conditional logic to trigger another job run based on the result. Users can also use the runs/list endpoint to list all the runs of a job, and use the runs/cancel endpoint to cancel a job run if needed. References:

Databricks REST API — Databricks Documentation, p. 1

Jobs API — Databricks Documentation, p. 1-2

Create and run Databricks Jobs | Databricks on AWS, Create and run jobs using the CLI, API, or notebooks, p. 1

To log single values like the number of trees in a random forest, you can use the mlflow.log_param function. This function allows you to log a parameter (e.g. model hyperparameter) under the current run. Parameters can be of any type, and can be logged using the following syntax:

Python

mlflow.log_param("num_trees", 100)

AI-generated code. Review and use carefully. More info on FAQ.

MLflow also offers a convenient way to log multiple parameters by indicating all of them using a dictionary1. The other options are incorrect because:

Option A: mlflow.log_artifact is used to log output files in any format, not single values2.

Option B: mlflow.log_model is used to log an MLflow Model along with its artifacts, not single values3.

Option C: mlflow.log_metric is used to log numeric metrics, not single values4.

Option E: There is a way to store values like this using the mlflow.log_param function1. References: mlflow.log_param, mlflow.log_artifact, mlflow.log_model, mlflow.log_metric

The correct line of code is “MODEL_VERSION_TRANSITIONED_STAGE”, which is the name of the event that occurs when a model version transitions to any stage in the MLflow Model Registry. This event can be used to trigger a webhook that will execute a Databricks Job based on the model version and the stage it transitions to. The other options are either invalid event names or too specific for the task.

References: MLflow Model Registry Webhooks, MLflow Model Registry Events

MLflow provides a utility function mlflow.models.signature.infer_signature that can infer the input schema and the output schema of a model from a sample dataset. The input schema and the output schema are represented as mlflow.types.schema.Schema objects, which define the names, data types, and shapes of the model inputs and outputs. The infer_signature function can be used for any model that implements a predict method that takes a pandas DataFrame or a numpy array as input and returns a pandas DataFrame, a pandas Series, a numpy array, or a list as output. The infer_signature function can also be used for models that implement a transform method that takes a pandas DataFrame as input and returns a pandas DataFrame as output. The infer_signature function can be used before logging the model with MLflow, as it does not require the model to be in MLflow format or registered with MLflow Model Registry. References:

MLflow Models Documentation, Model Signature and Input Example, p. 6-7

mlflow.types.schema Documentation, p. 1-2

Machine Learning Engineering with MLflow, Managing Model Signatures and Schemas, p. 1

The mlflow.sklearn module provides an API for logging and loading scikit-learn models. To load a model as an sklearn object for batch deployment, the mlflow.sklearn.load_model function can be used. This function takes a model_uri argument that specifies the location of the model to load. The model_uri can be a local file path, a remote file path, or a Databricks or MLflow Model Registry URI. The function returns an sklearn model object that can be used for inference or further training. The function also preserves the model signature and the input example, if they were logged with the model. References:

mlflow.sklearn Documentation, p. 1-2

MLflow Models Documentation, Model Signature and Input Example, p. 6-7

Containers are a type of software packaging that can assist in real-time deployments by packaging software with its own application, tools, and libraries. Containers are isolated and portable environments that run on a shared operating system, but have their ownfile system, dependencies, configuration, and resources. Containers can help developers and operators to build, test, and deploy software faster, easier, and more reliably, as they ensure that the software runs the same way in any environment, whether it is a developer’s laptop, a test server, or a production cluster. Containers can also help to achieve scalability, high availability, and load balancing, as they can be easily created, destroyed, replicated, and orchestrated using tools such as Kubernetes, Docker Swarm, or Amazon ECS. Containers can also integrate with other tools and services, such as MLflow, Databricks, AWS CodeDeploy, and Google Cloud Deployment Manager, to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance1234 References:

How to Package Your Application’s Infrastructure with Docker

21 open source DevOps tools (and what they do)

Microservices [What is it & Top Benefits]

9 Best Software Deployment Tools for 2023 (Paid & Free)