Amazon Web Services Data-Engineer-Associate - AWS Certified Data Engineer - Associate (DEA-C01)

Option A is the correct design for single-digit millisecond latency with unpredictable spikes and variable attributes. The study material describes Amazon DynamoDB as a NoSQL database “designed for highly dynamic datasets with frequent read and write operations,” providing low-latency performance at any scale—which directly matches the latency and traffic-spike requirements.

DynamoDB’s key-value and document model fits “product data that has variable attributes” because items can contain different attributes without needing schema migrations typical of relational databases. The requirement to retrieve items by Product ID maps naturally to DynamoDB’s primary key access pattern. The requirement for flexible queries on Category and Brand is met by creating global secondary indexes (GSIs) on those attributes so queries can be served efficiently without scanning the whole table.

Option B (Aurora) can scale reads, but it is not typically the best fit for sustained single-digit millisecond performance during sudden spikes without careful capacity planning. Option C is optimized for search and text/query relevance rather than primary-key transactional access patterns. Option D uses Athena (interactive SQL over S3) which is not designed for millisecond-latency, high-QPS query workloads.

Option B is correct because CloudWatch Logs subscription filters are the native way to send only matching log events to a downstream destination such as Amazon Data Firehose. AWS states that a subscription filter defines the filter pattern for which log events get delivered and where they are sent. AWS also documents that CloudWatch Logs events can be sent to Firehose by using subscription filters. This directly satisfies the requirement to deliver only relevant logs. Once the logs are in Firehose, AWS documents that Firehose can invoke an AWS Lambda function to transform incoming source data before delivering it to the destination, which satisfies the requirement to deliver the logs in a compatible format for analysis.

Option A is incorrect because S3 bucket rules do not filter CloudWatch log events by timestamp for ingestion control, and using S3 triggers adds unnecessary post-delivery processing. Option C is invalid because the subscription filter applies at the CloudWatch Logs log group, not by “monitoring the Firehose stream.” Option D is incorrect because Firehose does not natively ingest only CloudWatch log groups based on tags or automatically reformat arbitrary logs without a transformation step. The most reliable and AWS-native design is subscription filtering + Firehose + Lambda transformation.

Option D is the best solution to meet the requirements with the least operational overhead because AWS Lake Formation is a fully managed service that simplifies the process of building, securing, and managing data lakes. AWS Lake Formation allows you to define granular data access policies at the row and column level for different users and groups. AWS Lake Formation also integrates with Amazon Athena, Amazon Redshift Spectrum, and Apache Hive on Amazon EMR, enabling these services to access the data in the data lake through AWS Lake Formation.

Option A is not a good solution because S3 access policies cannot restrict data access by rows and columns. S3 access policies are based on the identity and permissions of the requester, the bucket and object ownership, and the object prefix and tags. S3 access policies cannot enforce fine-grained data access control at the row and column level.

Option B is not a good solution because it involves using Apache Ranger and Apache Pig, which are not fully managed services and require additional configuration and maintenance. Apache Ranger is a framework that provides centralized security administration for data stored in Hadoop clusters, such as Amazon EMR. Apache Ranger can enforce row-level and column-level access policies for Apache Hive tables. However, Apache Ranger is not a native AWS service and requires manual installation and configuration on Amazon EMR clusters. Apache Pig is a platform that allows you to analyze large data sets using a high-level scripting language called Pig Latin. Apache Pig can access data stored in Amazon S3 and process it using Apache Hive. However, Apache Pig is not a native AWS service and requires manual installation and configuration on Amazon EMR clusters.

Option C is not a good solution because Amazon Redshift is not a suitable service for data lake storage. Amazon Redshift is a fully managed data warehouse service that allows you to run complex analytical queries using standard SQL. Amazon Redshift can enforce row-level and column-level access policies for different users and groups. However, Amazon Redshift is not designed to store and process large volumes of unstructured or semi-structured data, which are typical characteristics of data lakes. Amazon Redshift is also more expensive and less scalable than Amazon S3 for data lake storage.

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide

What Is AWS Lake Formation? - AWS Lake Formation

Using AWS Lake Formation with Amazon Athena - AWS Lake Formation

Using AWS Lake Formation with Amazon Redshift Spectrum - AWS Lake Formation

Using AWS Lake Formation with Apache Hive on Amazon EMR - AWS Lake Formation

Using Bucket Policies and User Policies - Amazon Simple Storage Service

Apache Ranger

Apache Pig

What Is Amazon Redshift? - Amazon Redshift

In Amazon Managed Workflows for Apache Airflow (MWAA), the type of log that is most useful for diagnosing workflow (DAG) failures is the Task logs. These logs provide detailed information on the execution of each task within the DAG, including error messages, exceptions, and other critical details necessary for diagnosing failures.

Option D: YourEnvironmentName-TaskTask logs capture the output from the execution of each task within a workflow (DAG), which is crucial for understanding what went wrong when a DAG fails. These logs contain detailed execution information, including errors and stack traces, making them the best source for debugging.

Other options (WebServer, Scheduler, and DAGProcessing logs) provide general environment-level logs or logs related to scheduling and DAG parsing, but they do not provide the granular task-level execution details needed for diagnosing workflow failures.

Option C is correct because SSE-KMS encrypts Amazon S3 objects by using AWS KMS keys, and access to those keys can be controlled through IAM policies and KMS key policies so that only specific employees can use them. This directly satisfies both requirements: encryption of the S3 objects and restriction of key usage to a limited group of authorized users. AWS-managed S3 encryption with KMS also requires far less effort than building or operating a dedicated HSM-based solution. This is the standard AWS answer when the question asks for encrypted S3 data with fine-grained control over who can use the encryption keys.

Option A is far more operationally complex because CloudHSM requires managing dedicated HSM infrastructure. Option B is less desirable operationally because SSE-C requires the customer to provide and manage the keys for every request. Option D is incorrect because SSE-S3 uses Amazon S3 managed keys, and customers do not get the same granular control over key usage by specific employees. Therefore, the least-effort and best-governed solution is SSE-KMS with restricted KMS key access. This aligns with the study guide’s security-and-governance focus on using AWS KMS when controlled access to encryption keys is required.

Problem Analysis:

The company runs a daily AWS Glue ETL pipeline to clean and transform files received in an S3 bucket.

If a file is incomplete or empty, the previous day’s file should be retained.

Need a solution to validate files before overwriting the existing file.

Key Considerations:

Automate data validation with minimal human intervention.

Use built-in AWS Glue capabilities for ease of integration.

Ensure robust validation for missing or incomplete data.

Solution Analysis:

Option A: Lambda Function for Validation

Lambda can validate files, but it would require custom code.

Does not leverage AWS Glue’s built-in features, adding operational complexity.

Option B: AWS Glue Data Quality Rules

AWS Glue Data Quality allows defining Data Quality Definition Language (DQDL) rules.

Rules can validate if required fields are missing or if the file is empty.

Automatically integrates into the existing ETL pipeline.

If validation fails, retain the previous day’s file.

Option C: AWS Glue Studio with Filling Missing Values

Modifying ETL code to fill missing values with most common values risks introducing inaccuracies.

Does not handle empty files effectively.

Option D: Athena Query for Validation

Athena can drop rows with missing values, but this is a post-hoc solution.

Requires manual intervention to copy the corrected file to S3, increasing complexity.

Final Recommendation:

Use AWS Glue Data Quality to define validation rules in DQDL for identifying missing or incomplete data.

This solution integrates seamlessly with the ETL pipeline and minimizes manual effort.

Implementation Steps:

Enable AWS Glue Data Quality in the existing ETL pipeline.

Define DQDL Rules, such as:

Check if a file is empty.

Verify required fields are present and non-null.

Configure the pipeline to proceed with overwriting only if the file passes validation.

In case of failure, retain the previous day’s file.

AWS Glue Data Quality Overview

Defining DQDL Rules

AWS Glue Studio Documentation