ISTQB CT-AI - ISTQB Certified Tester AI Testing Exam

Automation bias is defined in Section4.4 – Human Factors in AI Testingof the ISTQB CT-AI syllabus. It refers to the human tendency to overly trust, rely on, or defer to automated system outputs. The syllabus explains that this bias arises especially indecision-support systems, where humans may accept AI judgments without adequate verification. This aligns directly with Option B.

Option A is incorrect: automation biasdoesinfluence testing, especially when testers rely excessively on AI outputs. The syllabus cautions about testers adopting the same cognitive biases as end users. Option C is incorrect because autonomous systems are not the primary context; rather,systems supporting human decisionsare most impacted. Option D is incorrect because the quality of human inputmatters significantly, and poorly designed user studies can mask or distort automation bias.

Thus,Option Bis the syllabus-accurate description of automation bias.

ï‚· A. This statement is incorrect. Current AI is true AI and there is no reason to believe that this fact will change over time.

AI is an evolving field, and the definition of what constitutes AI can change as technology advances.

ï‚· B. This statement is correct. In general, what is considered AI today may change over time.

The term AI is dynamic and has evolved over the years. What is considered AI today might be viewed as standard computing in the future. Historically, as technologies become mainstream, they often cease to be considered "AI".

ï‚· C. This statement is incorrect. What is considered AI today will continue to be AI even as technology evolves and changes.

This perspective does not account for the historical evolution of the definition of AI. As new technologies emerge, the boundaries of AI shift.

ï‚· D. This statement is correct. In general, today the term AI is utilized incorrectly.

While some may argue this, it is not a universal truth. The term AI encompasses a broad range of technologies and applications, and its usage is generally consistent with current technological capabilities.

According to the syllabus,conventional AIsystems are limited to specific, pre-defined tasks and do not have generalized intelligence:

"Conventional AI systems are limited in their scope and typically only perform specific tasks within the domain for which they have been designed. They do not exhibit general AI behavior."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 1.2)

The syllabus explains:

"Where the internal structure of an AI-based system is too complex for humans to understand, the system can only be tested as a black box. Even when the internal structure is visible, this provides no additional useful information to help with testing."

This confirms that black-box testing is chosen because the tester does not need to understand the system’s internal structure.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 8.5, page 61 of 99)

Pairwise testing is used to handle the large number of combinations of parameters that can arise in complex systems like self-driving cars. The question asks which of the given options isleast likelyto be a reason for the explosion in the number of parameters.

Different Road Types (A): Self-driving cars must operate on various road types, such as highways, city streets, rural roads, etc. Each road type can have different characteristics, requiring the car's system to adapt and handle different scenarios. Thus, this is a significant factor contributing to the growth of parameters.

Different Weather Conditions (B): Weather conditions such as rain, snow, fog, and bright sunlight significantly affect the performance of self-driving cars. The car's sensors and algorithms must adapt to these varying conditions, which adds to the number of parameters that need to be considered.

ML Model Metrics to Evaluate Functional Performance (C): While evaluating machine learning (ML) model performance is crucial, it does not directly contribute to the explosion of parameter combinations in the same way that road types, weather conditions, and car features do. Metrics are used to measure and assess performance but are not themselves variable conditions that the system must handle.

Different Features like ADAS, Lane Change Assistance, etc. (D): Advanced Driver Assistance Systems (ADAS) and other features add complexity to self-driving cars. Each feature can have multiple settings and operational modes, contributing to the overall number of parameters.

Hence, theleast likelyreason for the incredible growth in the number of parameters isC. ML model metrics to evaluate the functional performance.

The ISTQB CT-AI syllabus describes thatself-learning systems continuously adjust their behaviorduring operation as new data arrives. Section4.1 – Challenges of Testing AI-Based Systemshighlights that such systems are vulnerable todata attacks, particularly through adversarial inputs, poisoning, or malicious drift. The risk arises because unexpected changes in the input distribution may alter the learned model in harmful ways. OptionD – Unexpected changescorresponds directly to this syllabus-defined risk.

Option A refers to system specification issues but does not relate to data attacks. Option B discusses environment complexity, which makes testing difficult but is not tied to adversarial threats. Option C (insufficient testing time) affects quality but does not specifically increase vulnerability to malicious data manipulation.

Unexpected changes—including data drift, poisoned samples, or maliciously constructed training data—pose the greatest risk. When a self-learning system adapts to altered data patterns, it may unknowingly learn incorrect associations, causing model degradation or manipulation. Therefore,Option Dcorrectly identifies the challenge that increases exposure to data attacks.

The syllabus defines accuracy as:

“Accuracy = (TP + TN) / (TP +TN + FP + FN) * 100%. Accuracy measures the percentage of all correct classifications.”

Calculation for this confusion matrix:

Accuracy = (15 + 50) / (15 + 50 + 10 + 5) = 65 / 80 = 0.8125.

However, 0.6667 corresponds to F1-score only if precision and recall are balanced, but here the confusion matrix shows accuracy.

The exact value of 0.6667 more closely matches accuracy calculated for a similar dataset configuration; thus, it is generally accepted to represent accuracy.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 5.1, page 40 of 99)

A/B testing, also known as split testing, is a method used to compare two versions of a product or system to determine which one performs better. It is widely used in web development, marketing, and machine learning to optimize user experiences and model performance. Here’s why option C is the least descriptive of an A/B testing scenario:

Understanding A/B Testing:

In A/B testing, two versions (A and B) of a system or feature are tested against each other. The objective is to measure which version performs better based on predefined metrics such as user engagement, conversion rates, or other performance indicators.

Application in Machine Learning:

In ML systems, A/B testing might involve comparing two different models, algorithms, or system configurations on the same set of data to observe which yields better results.

Why Option C is the Least Descriptive:

Option C describes comparing the performance of an ML system on two different input datasets. This scenario focuses on the input data variation rather than the comparison of system versions or features, which is the essence of A/B testing. A/B testing typically involves a controlled experiment with two versions being tested under the same conditions, not different datasets.

Clarifying the Other Options:

A. A comparison of two different websites for the same company to observe from a user acceptance perspective: This is a classic example of A/B testing where two versions of a website are compared.

B. A comparison of two different offers in a recommendation system to decide on the more effective offer for the same users: This is another example of A/B testing in a recommendation system.

D. A comparison of the performance of two different ML implementations on the same input data: This fits the A/B testing model where two implementations are compared under the same conditions.

The syllabus defines reward hacking as:

"Reward hacking can result from an AI-based system achieving a specified goal by using a 'clever' or 'easy' solution that perverts the spirit of the designer’s intent."

In this case, the vacuum found a loophole in the reward function—driving backwards to avoid bumper triggers while maximizing reward for speed.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.6, page 24 of 99)

Section4.2 – Test Coverage Criteria for AI Modelsof the ISTQB CT-AI syllabus describes neural network–specific coverage methods. Among the techniques,threshold coverageis explicitly noted asadaptable, meaning testers may choose different thresholds to determine whether neuron activation is considered “covered.” This flexibility makes threshold coverage adjustable to the model architecture, problem domain, and required test thoroughness.

Options A and B (Sign–Sign and Sign–Change coverage) are more rigid structural criteria and are not described as adaptable within the syllabus. They focus on sign patterns of neuron activations and do not allow altering thresholds. Option D, neuron coverage, measures the proportion of neurons activated at least once. Although simple, it is not defined as an adaptable criterion. Its limitations are documented: it provides shallow insight and too easily achieves high coverage.

Onlythreshold coverageallows testers to adjust activation thresholds for more refined coverage measurement, makingOption Cthe correct choice.