Amazon Web Services Data-Engineer-Associate - AWS Certified Data Engineer - Associate (DEA-C01)

 QuickSight does not have access to the S3 bucket and QuickSight does not have access to decrypt S3 data are two possible factors that could cause the permissions-related errors. Amazon QuickSight is a business intelligence service that allows you to create and share interactive dashboards based on various data sources, including Amazon Athena. Amazon Athena is a serverless query service that allows you to analyze data stored in Amazon S3 using standard SQL. To use an Amazon QuickSight dashboard that is based on Amazon Athena queries on data that is stored in an Amazon S3 bucket, you need to grant QuickSight access to both Athena and S3, as well as any encryption keys that are used to encrypt the S3 data. If QuickSight does not have access to the S3 bucket or the encryption keys, it will not be able to read the data from Athena and display it on the dashboard, resulting in an error message that indicates insufficient permissions.

The other options are not factors that could cause the permissions-related errors. Option A, there is no connection between QuickSight and Athena, is not a factor, as QuickSight supports Athena as a native data source, and you can easily create a connection between them using the QuickSight console or the API. Option B, the Athena tables are not cataloged, is not a factor, as QuickSight can automatically discover the Athena tables that are cataloged in the AWS Glue Data Catalog, and you can also manually specify the Athena tables that are not cataloged. Option E, there is no IAM role assigned to QuickSight, is not a factor, as QuickSight requires an IAM role to access any AWS data sources, including Athena and S3, and you can create and assign an IAM role to QuickSight using the QuickSight console or the API. References:

Using Amazon Athena as a Data Source

Granting Amazon QuickSight Access to AWS Resources

Encrypting Data at Rest in Amazon S3

Options B and D are correct. AWS Prescriptive Guidance states that shuffle is a major cause of Spark performance issues and can exhaust local disk space on executors, leading to “No space left on device” failures. AWS specifically recommends assessing shuffle performance in CloudWatch metrics and in the Spark UI, and for data skew, it says to examine metrics in the Spark UI, including task duration and spill behavior across executors. That makes B the correct low-cost way to identify the source of the problem.

For remediation, AWS Glue documentation states that Spark can throw “No space left on device” when there is insufficient local disk on the executor, and that you can use Amazon S3 to store Spark shuffle data by enabling the AWS Glue Spark shuffle plugin. The S3 shuffle approach is specifically presented as a way to run shuffle-intensive jobs more reliably when they are bound by local disk capacity. In skew scenarios, using a skew-mitigation technique such as salting is also a standard way to distribute hot keys more evenly. That makes D the most cost-effective corrective step.

Options A and C increase cost by adding capacity before diagnosing or directly fixing the shuffle bottleneck. Option E is less precise than the Spark UI and Glue metrics for identifying skew.

AWS Glue Data Catalog stores metadata about data stored in Amazon S3, including table schemas and partitions. If new partitions are added to S3 and the Glue Data Catalog is not updated—either through crawlers or explicit API calls—query engines such as Amazon Athena and AWS Glue will not be aware of the new data.

This leads to analysis results that appear incomplete or outdated, even though the underlying data exists in S3. This is a common issue in time-sensitive analytics workloads where data arrives frequently and partitions change often.

IAM permission issues would typically cause access failures rather than missing records. S3 versioning affects object history, not query visibility. Overlapping Glue job schedules could cause processing delays but would not directly cause missing data during query time.

Therefore, the most likely cause is that the Glue Data Catalog metadata is not synchronized with the latest S3 partitions.

The streaming ingestion feature of Amazon Redshift enables you to ingest data from streaming sources, such as Amazon Kinesis Data Streams, into Amazon Redshift tables in near real-time. You can use the streaming ingestion feature to process the streaming data from the wearable devices, hospital equipment, and patient records. The streaming ingestion feature also supports incremental updates, which means you can append new data or update existing data in the Amazon Redshift tables. This way, you can store the data in an Amazon Redshift Serverless warehouse and support near real-time analytics of the streaming data and the previous day’s data. This solution meets the requirements with the least operational overhead, as it does not require any additional services or components to ingest and process the streaming data. The other options are either not feasible or not optimal. Loading data into Amazon Kinesis Data Firehose and then into Amazon Redshift (option A) would introduce additional latency and cost, as well as require additional configuration and management. Loading data into Amazon S3 and then using the COPY command to load the data into Amazon Redshift (option C) would also introduce additional latency and cost, as well as require additional storage space and ETL logic. Using the Amazon Aurora zero-ETL integration with Amazon Redshift (option D) would not work, as it requires the data to be stored in Amazon Aurora first, which is not the case for the streaming data from the healthcare company. References:

Using streaming ingestion with Amazon Redshift

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 3: Data Ingestion and Transformation, Section 3.5: Amazon Redshift Streaming Ingestion

A data mesh approach in AWS typically uses Lake Formation for domain-level access control and Athena for cross-domain querying through federated views. CloudTrail ensures auditing.

“For data mesh architectures, use AWS Lake Formation for fine-grained access control and Athena views for cross-domain analysis. Enable CloudTrail to audit access.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

This ensures scalability, security, and compliance across domains.

In AWS Glue, table partitions are managed as metadata objects within the AWS Glue Data Catalog. To add a new partition to an existing table, the correct and supported approach is to use the AWS Glue Data Catalog API, such as the CreatePartition operation, or equivalent console or SDK actions.

Updating the table schema alone does not create partitions or inform Glue about new partition values. Editing metadata files directly in Amazon S3 is unsupported and can corrupt the Data Catalog. Manually modifying the S3 bucket structure without registering partitions in Glue will result in Athena and other query engines being unable to recognize the partitions.

By adding partitions through the Glue Data Catalog API, query engines such as Amazon Athena and Amazon Redshift Spectrum can perform partition pruning, which significantly improves query performance by scanning only relevant data.

This method aligns with AWS best practices, ensures metadata consistency, and avoids unnecessary operational risk. Therefore, Option C is the correct solution.

Option D is correct because the problem is lock contention between ETL writes and analytical reads on the same Redshift tables. AWS documents that table locks can block both reads and writes, and that Redshift data sharing lets you share live data across Redshift clusters and workgroups without manually moving or copying the data. Because the shared data is live, consumer users see the most up-to-date information as soon as it is updated. Separating ETL and analytics onto different Redshift clusters while using data sharing is therefore the cleanest way to isolate workloads and still provide consistent data views.

Option A reduces contention only by changing schedules and does not fully solve concurrent peak-hour conflicts. Option B is not correct because Redshift federated queries are read-only against external sources and do not provide a Redshift “read replica” design for this case. Option C does not address the locking problem on the dimension tables that ETL is updating. The official Redshift guidance and service capabilities point to workload isolation plus data sharing as the best fix. This also matches the exam guide’s emphasis on choosing architectures that maintain performance and consistency without unnecessary copying.

To make the S3 data accessible daily in the AWS Glue Data Catalog, the data engineer needs to create a crawler that can crawl the S3 data and write the metadata to the Data Catalog. The crawler also needs to run on a daily schedule to keep the Data Catalog updated with the latest data. Therefore, the solution must include the following steps:

Create an IAM role that has the necessary permissions to access the S3 data and the Data Catalog. The AWSGlueServiceRole policy is a managed policy that grants these permissions1.

Associate the role with the crawler.

Specify the S3 bucket path of the source data as the crawler’s data store. The crawler will scan the data and infer the schema and format2.

Create a daily schedule to run the crawler. The crawler will run at the specified time every day and update the Data Catalog with any changes in the data3.

Specify a database name for the output. The crawler will create or update a table in the Data Catalog under the specified database. The table will contain the metadata about the data in the S3 bucket, such as the location, schema, and classification.

Option B is the only solution that includes all these steps. Therefore, option B is the correct answer.

Option A is incorrect because it configures the output destination to a new path in the existing S3 bucket. This is unnecessary and may cause confusion, as the crawler does not write any data to the S3 bucket, only metadata to the Data Catalog.

Option C is incorrect because it allocates data processing units (DPUs) to run the crawler every day. This is also unnecessary, as DPUs are only used for AWS Glue ETL jobs, not crawlers.

Option D is incorrect because it combines the errors of option A and C. It configures the output destination to a new path in the existing S3 bucket and allocates DPUs to run the crawler every day, both of which are irrelevant for the crawler.

1: AWS managed (predefined) policies for AWS Glue - AWS Glue

2: Data Catalog and crawlers in AWS Glue - AWS Glue

3: Scheduling an AWS Glue crawler - AWS Glue

[4]: Parameters set on Data Catalog tables by crawler - AWS Glue

[5]: AWS Glue pricing - Amazon Web Services (AWS)

The requirement to retain immutable object versions using a WORM model mandates the use of S3 Object Lock, not just S3 Versioning. Object Lock enforces immutability by preventing deletion or modification of objects for a specified retention period, which is required for compliance scenarios such as regulatory data retention.

To meet the access requirement of retrieving data within 5 hours, the appropriate archival storage class is S3 Glacier Flexible Retrieval, which supports expedited and standard retrieval within minutes to hours. S3 Glacier Deep Archive does not meet this requirement because retrieval times are typically 12 hours or longer.

Using S3 Intelligent-Tiering for active data optimizes storage costs automatically before transitioning older objects. A lifecycle policy can then transition objects older than 1 year to Glacier Flexible Retrieval and delete them after the 7-year retention period expires.

Options that rely only on S3 Versioning do not provide immutability guarantees. Options using Glacier Deep Archive fail the access-time requirement. Therefore, Option C is the only solution that satisfies WORM compliance, retrieval-time constraints, and cost optimization.

The problem likely arises from Athena not being able to read from the correct S3 location or missing partitions. The two most relevant troubleshooting steps involve checking the S3 location and repairing the table metadata.

A. Confirm that Athena is pointing to the correct Amazon S3 location:

One of the most common issues with missing data in Athena queries is that the query is pointed to an incorrect or outdated S3 location. Checking the S3 path ensures Athena is querying the correct data.