ISTQB CT-AI - ISTQB Certified Tester AI Testing Exam

The question asks which deficiency is most likely to be discovered by the test expert given the scenario of poor real-world performance despite good isolated accuracy.

A lack of similarity between the training and testing data (A): This is a common issue in ML where the model performs well on training data but poorly on real-world data due to a lack of representativeness in the training data. This leads to poor generalization to new, unseen data.

The input data has not been tested for quality prior to use for testing (B): While data quality is important, this option is less likely to be the primary reason for the described issue compared to the representativeness of training data.

A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial, but the issue described seems more related to the data mismatch rather than metric selection.

A lack of focus on non-functional requirements testing (D): Non-functional requirements are important, but the scenario specifically mentions issues with detecting real cancer cases, pointing more towards data issues.

In the syllabus, theevolutioncharacteristic for AI-based systems means the ability of the system to evolve and adapt its behavior in response to changes in the environment or in its own performance:

"Evolution is the system’s ability to change itself to adapt to new situations, different hardware, or a changing operational environment."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.3)

Prevalence Rate and Model Performance:

The cancer detection system being developed by "Splendid Healthcare" needs to account for the fact that the type of cancer has a 2% prevalence rate in the population. This indicates that the dataset is highly imbalanced with far fewer positive (cancer) cases compared to negative (normal) cases.

Importance of Recall:

Recall, also known as sensitivity or true positive rate, measures the proportion of actual positive cases that are correctly identified by the model. In medical diagnosis, especially cancer detection, recall is critical because missing a positive case (false negative) could have severe consequences for the patient. Therefore, maximizing recall ensures that most, if not all, cancer cases are detected.

Importance of Precision:

Precision measures the proportion of predicted positive cases that are actually positive. High precision reduces the number of false positives, meaning fewer people will be incorrectly diagnosed with cancer. This is also important to avoid unnecessary anxiety and further invasive testing for those who do not have the disease.

Balancing Recall and Precision:

In scenarios where both false negatives and false positives have significant consequences, it is crucial to balance recall and precision. This balance ensures that the model is not only good at detecting positive cases but also accurate in its predictions, reducing both types of errors.

Accuracy and Specificity:

While accuracy (the proportion of total correct predictions) is important, it can be misleading in imbalanced datasets. In this case, high accuracy could simply result from the model predicting the majority class (normal) correctly. Specificity (true negative rate) is also important, but for a cancer detection system, recall and precision take precedence to ensure positive cases are correctly and accurately identified.

Conclusion:

Therefore, for a cancer detection system with a low prevalence rate, maximizing both recall and precision is crucial to ensure effective and accurate detection of cancer cases.

The syllabus describes that data pre-processing includes cleaning (e.g., fixing or removing invalid data) and transforming data (e.g., changing data types such as numbers stored as strings).

"Transformation: The format of the given data is changed… converting categorical data into numerical data, changing image formats…"

(Reference: ISTQB CT-AI Syllabus v1.0, Section 4.1.1, Page 34 of 99)

The syllabus explains that supervised learning requires correctly labeled data so the algorithm can learn the relationship between input features and output labels:

"In supervised learning, the algorithm creates the ML model from labeled data during the training phase. The labeled data is used to infer the relationship between the input data and output labels."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.1.1)

The question asks which characteristic is least likely to cause safety-related issues for an AI system. Let's evaluate each option:

Non-determinism (A): Non-deterministic systems can produce different outcomes even with the same inputs, which can lead to unpredictable behavior and potential safety issues.

Robustness (B): Robustness refers to the ability of the system to handle errors, anomalies, and unexpected inputs gracefully. A robust system is less likely to cause safety issues because it can maintain functionality under varied conditions.

High complexity (C): High complexity in AI systems can lead to difficulties in understanding, predicting, and managing the system's behavior, which can cause safety-related issues.

Self-learning (D): Self-learning systems adapt based on new data, which can lead to unexpected changes in behavior. If not properly monitored and controlled, this can result in safety issues.

Section1.7 – Neural Networksof the ISTQB CT-AI syllabus explains that neural networks consist of neurons connected by weighted links. During training,learning occurs by adjusting the weights on these connections. This is the essence of gradient descent and backpropagation. Option B correctly states this behavior: only theweightsare modified, not the activation functions, neuron counts, or architectural structure.

Option A is incorrect because a neuron’sbiasis not determined by previous activations; it is an independent trainable parameter added to the weighted input sum. Option C is incorrect because the syllabus states that asingle-layer perceptron is a valid type of neural network, although limited to linearly separable problems. Option D is incorrect because no rule requires the number of input neurons to exceed or equal the number of output neurons. Instead, input neurons correspond to thenumber of features, while output neurons correspond totasks or classes.

Therefore,Option Bprecisely reflects the syllabus definition of what changes during neural network training.

The syllabus describes metamorphic testing as:

“Testing involves defining metamorphic relations and then applying those relations to check that the transformations result in expected outcomes, even when the expected output of the system is unknown or not well-defined.”

In this scenario, applying the metamorphic relation (speed differences) and checking the transformed outcome (arrival times) fits the definition of metamorphic testing.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.5, page 69 of 99)

Bias detection at thedata levelis performed by reviewingdata acquisition and preprocessing steps, as explained in Section2.3 – Data Quality and Biasof the ISTQB CT-AI syllabus. Sample bias arises when data is distorted or when preprocessing introduces unintended shifts—for example, by filtering, normalization, or labeling steps that disproportionately affect subsets of the data. OptionBcorrectly reflects this: reviewers can identify whether preprocessing steps have altered the dataset in a way that introducessample distortions. This aligns perfectly with syllabus guidance on reviewing data pipelines for bias sources.

Option A is incorrect because algorithmic bias originates from themodel, not data collection procedures. Option C is incorrect because LIME is anexplainabilitymethod applied post-model, not in data reviews. Option D incorrectly states “algorithmic bias,” but preprocessing affectssample bias, not algorithmic bias.

Thus, OptionBcorrectly matches the syllabus’ definition of how bias can be detected during data-related reviews.

The syllabus mentions that AI can help optimize regression test suites:

"An AI-based tool can perform optimization of the regression test suite by analyzing… the information from previous test results, associated defects, and the latest changes that have been made, such as features which are broken more frequently and which tests exercise code impacted by recent changes."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 11.4, page 79 of 99)