Machine Learning-Based Cognitive Instrumentation for Runtime AI Assurance

Machine learning (ML) techniques are outperforming traditional methods in many areas, including building performance evaluation (Ali et al., 2024), financial market forecasting (Vancsura et al., 2025), and mortality prediction in surgical risk models (Zaka et al., 2025). ML comes to the fore when data contains many variables with complex interactions among them. ML is also not beholden to relational assumptions or heuristics such as linearity, as compared to traditional methods. The advantage grows further as the volume of data available to train models increases, allowing ML to unlock performance that classical statistics cannot harness. One challenge that accompanies the improved performance of ML systems is greater difficulty to understand how the system functions, which becomes especially important as ML is incorporated into critical systems with the highest standards for assurance. This paper presents a conceptual framework and proof-of-concept implementation of a machine-learning-based monitor as an additional assurance layer for AI-enabled systems.

Assurance challenges for ML can be highlighted by adopting the taxonomy of Snowden & Boone (2007) from domains of operative contexts, to describe systems as simple, complicated, or complex. Simple systems display straightforward cause and effect relationships, complicated systems have more obscure cause and effect relationships, and complex systems exhibit apparent unpredictability which can often only be understood in retrospect. Complexity reflects that we cannot fully predict system behaviour under operational variability. Despite this limitation, complex systems still allow potential for insight via analysis of the instructive patterns they exhibit in their operation. Although deterministic under fixed conditions, ML-enabled systems may exhibit highly sensitive output with input variations, characterised as uncertainty. This uncertainty can exist whether we have access to the internal information of the component or not. Approaching systems that incorporate ML components as complex in the first instance serves to improve the overall assurance of the system.

We consider assurance as “grounds for justified confidence that a claim has been or will be achieved” (ISO/IEC/IEEE, 2025). Freeman (2020) has called for necessary innovation in assurance to certify that “ML and/or AI algorithms function as intended and are vulnerability free, either intentionally or unintentionally designed or inserted as data/algorithm parts”. Test and Evaluation (T&E) is offered as a process to provide the grounds for this assurance, but given the characteristics of complex systems defined above, it is impractical (if not impossible) for T&E of such a system to be comprehensive. System monitors can be used to provide additional assurance for these systems (Bloomfield & Rushby, 2021), taking many forms and implemented in many ways (Naveed et al., 2025). However, many existing monitoring approaches remain primarily rule-based and reactive. This study explores using an ML-based system monitor to leverage the instructive patterns exhibited by a system during its operation. We can consider a system monitor of this nature to be cognitive instrumentation (Haugh et al., 2018)—the ability to observe and interpret behavioural patterns within system decision processes.

We asked the following research question: Can an ML-based monitor serve as cognitive instrumentation to provide assurance for a complex system? To address this question, we propose a conceptual framework in which an ML-based monitor provides an additional layer of assurance for AI-enabled systems. By monitoring behavioural patterns during operation, the approach has the potential to support adaptive assurance and to detect behavioural deviations before they manifest as observable performance degradation. Although such a monitor offers great promise, we do not believe it would replace the need for system testing. Instead, overall assurance activities may become more extensive, as the monitor itself must also be validated using test inputs designed to induce system ‘misbehaviour’ (Pullum et al., 2007). The experimental study in this paper is intended as a proof-of-concept demonstration of the proposed approach rather than an investigation of monitor performance optimisation.

The remainder of this paper reviews relevant literature, presents the proposed conceptual framework and proof-of-concept implementation, discusses implications for AI assurance research, and concludes with recommendations for future work.

Liu et al. (2004) presented an approach to validating an adaptive neural network flight control system through an online operational monitor methodology. This methodology consisted of a novelty detection technique and online stability monitoring tools to check for Lyapunov stability via error values to give an accurate state of the system. Borg et al. (2018) presented a safety cage architecture to continuously monitor input data from the operational environment to shift focus from off-line verification to instead performing on-line testing at run-time. Peterson et al. (2020) focused on evaluation of Run Time Assurance (RTA) as an alternate concept for assuring safety of airborne systems applied to a novel, airborne system architecture. At the centre of the RTA approach is a monitor that decides which output should be active at any given time. This monitor required a formal definition of its behaviour to ensure correct output was produced under all anticipated conditions. These approaches demonstrate the growing importance of runtime monitoring as an operational assurance mechanism for AI-enabled systems, alongside their dependence on predefined rules, assumptions, or formally encoded constraints.

Beland et al. (2020) and Torens et al. (2024) take the position that runtime monitors are among the key assurance technologies to address the challenges posed by probabilistic and uncertain nature of learning-enabled autonomous systems. Against that approach, Baumeister et al. (2023) note that runtime monitoring is generally considered a light-weight alternative to formal verification; notwithstanding, they concede that in safety-critical systems, the monitor itself is a critical component. In the automotive domain, Burton et al. (2017) identified the need to monitor assumptions during runtime, if those assumptions were critical to system operation. They also proposed the use of secondary channels to minimise the impact on the primary machine learning function. Naveed et al. (2025) identify that monitoring can be viewed as an extension of testing to overcome the limitation of not being able to assess all possible scenarios. In a military context, Ryan (2019) argued that military institutions must build ‘checkpoints’ to ensure that the use of AI is aligned with institutional values. These checkpoints imply the need for monitoring.

NASA provided early works on assurance of neural-network-enabled systems (Gupta & Schumann, 2004; Jacklin et al., 2005; Nguyen & Jacklin, 2010; Schumann et al., 2003; Schumann & Nelson, 2002). Collectively, they identified that neural networks must have a certification process before being used in safety critical applications. A key concern was achieving stable and rapid convergent learning in neural networks for adaptive flight control systems. They discussed a layered approach to verification and validation, as well as methods for measuring and monitoring artificial neural networks. Focusing on testing as part of the verification and validation process, Clark et al. (2014) discuss autonomous systems in critical applications and note at that time the burden of providing assurance rests on the test and evaluation of such systems. The goal of their activity centred on the certification of safety critical autonomous systems through verification and validation. Given this burden, Helle et al. (2016) discuss the challenges to test autonomous systems in safety-critical applications in an assurance context. Collectively, these studies demonstrated that assuring learning-enabled systems presents challenges that extend beyond conventional verification and validation approaches.

Freeman (2020) made one of the earliest direct calls for AI assurance and offered T&E as a process for achieving this, including ten themes for enhancing T&E to address AI challenges. This was followed by Batarseh et al. (2021), who provided a systematic review of research relevant to AI assurance published between 1985 and 2021. Freeman et al. (2022) subsequently formalised the key components, including six goals for AI assurance. Importantly, they argued that assurance should not be treated as an afterthought. Schnitzer et al. (2024) proposed a novel methodology for safety assurance cases for AI-based systems by systematically demonstrating the absence of AI safety concerns. Bloomfield & Rushby (2025) examined AI assurance from a dependability perspective. Their approach prioritises fault avoidance before fault tolerance, with resilience as a last line of defence. Accordingly, they argue that testing alone is insufficient when high levels of assurance are required. This limitation motivates the investigation of complementary assurance mechanisms capable of operating during system runtime.

Haugh et al. (2018) brought the concept of cognitive instrumentation back into contemporary usage. They described cognitive instrumentation as “the ability to see into the inner working of the decision engines” and referred to it as “any measurement of internal system states of the software modules that provide the autonomous capabilities of a system”. The term ‘cognitive instrumentation’ was first used in an engineering context to describe a system’s ability to ‘feel’ whether a process is functioning properly (Kercel, 2000; Kercel et al., 2000). Bellman (2005) described cognitive instrumentation as a capability to “create feedback that is meaningful for decisions in a system that can reflectively reason about its own computations, resource use, and goals and behaviour within its environment”. Porter et al. (2020) extended the discussion by addressing the application of cognitive instrumentation – what to do with the ‘inner workings’ once made available. Wojton & Porter (2020) indicate that cognitive instrumentation could be used to support runtime monitoring through providing information for supervisory control, ensuring safe operating bounds, and for detecting adversarial attacks on a system. These concepts provide a theoretical foundation for the use of ML-based cognitive instrumentation as an adaptive assurance mechanism.

A test method leveraging monitoring was the use of test oracles to evaluate system behaviour (Chowdhary, 2020). The test oracle provides a practical application for ML-based cognitive instrumentation (Wojton et al., 2020). This methodology depends on comparing observed system behaviour against a standard of correct behaviour. Accordingly, an understanding of correct system behaviour in terms of expected outcome or ground truth is required, which is a challenge when desired outcomes are unknown beforehand (Chandrasekaran et al., 2024). Complex systems exacerbate this challenge, where it is possible for more than one correct answer to exist (Riccio et al., 2020). The use of oracles can be extended from testing to deployment with normative oracles sometimes adapted to directly support operational dependability via run-time monitoring (Tate et al., 2016). Oracles establish that any practical application of ML-based cognitive instrumentation needs to consider a comparative standard or truth in operation.

Explainability is a key component of assurance (Laplante & Kuhn, 2022), and eXplainable Artificial Intelligence (XAI) (Gunning, 2016) is a research area which seeks to address that need. One aim of XAI is transparency: in this case, transparency to facilitate monitoring and testing for safety standards (Weller, 2019). XAI techniques may therefore be useful for assurance purposes. However, while XAI might allow a user to develop trust in a system (Van Lent et al., 2004) assurance may require different evidence than that provided by XAI. Notwithstanding the different needs, Haugh et al. (2018) offer that Cognitive Instrumentation is a necessary condition for Explainable AI, and that any valid explanation of AI behaviour must be based on accurate measurements of its internal states. This highlights the broader movement toward improving observability and interpretability within AI assurance architectures. While this experimental study does not add explainability directly, this could be a future direction enabled by its inclusion.

Etzioni & Etzioni (2016) suggest that to proceed we need a new kind of AI program to monitor, audit, and hold operational AI programs accountable. While we do not offer an ‘Ethics bot’ as proposed by Etzioni & Etzioni (2016), we explore the concept of ‘AI assuring AI’ that aligns closely with the vision of oversight-oriented AI architectures. More recently, Johannssen et al. (2025) discussed embedding XAI into control charts which use AI techniques as they are black box by nature. The intent of XAI here was to enhance interpretability while maintaining high predictive accuracy. Yampolskiy (2025) provides a definition and taxonomy for AI monitoring and offers some considerations on how advanced monitoring can be done, including implementation of AI monitoring AI. The context was with regards to identifying emergent unwanted characteristics and so represents an extreme of considerations, but the approaches are still valid for monitoring in general.

This paper proposes a framework for ML-based cognitive instrumentation operating as an adaptive monitoring layer for AI-enabled systems. To demonstrate proof-of-concept, we conducted an experimental evaluation with an ML-based monitor. We refer to the monitor as the Machine Learning Monitor (MLM). A general schematic of the MLM setup, and how it relates to a baseline system, is shown in Figure 1.

Figure 1: MLM Schematic

The baseline system in this study was developed previously by McCutcheon et al. (2025) for assurance experimentation. It is used here as a proof-of-concept platform to explore the feasibility of the proposed assurance framework. The baseline system predicted a surface vessel track from an Autonomous Underwater Vehicle (AUV). The system used a Long Short Term Memory neural network model to predict the track based on a noisy range and bearing signal input; in essence, a neural Kalman filter. The baseline system used synthetic data generated from a Mission Oriented Operating Suite – Interval Programming (MOOS-IvP) AUV simulation, with Pareto distributed noise added to match error metrics from field data.

The baseline system transformed the range and bearing measurements into cartesian (x,y) coordinates using a classical conversion. The time interval between measurements was a constant 15 seconds. The data was then windowed into 3D sequences of shape (sample, 9, 2), with each sequence consisting of the 8 preceding and the current cartesian coordinates. Each sequence was then individually scaled, using a separate standard scaler for x and y coordinates. The MLM input then consisted of the baseline system’s converted and scaled inputs as well as corresponding output (i.e. current predicted x and y coordinates of the surface vessel). The MLM input vector therefore consisted of 20 elements, as shown in Equation 1,  where  were the converted x and y coordinates for the eight previous time steps and current time step, and X_p, Y_p were the predicted x and y coordinates at the current time step.

Three MLM multilayer perceptron architectures were used for the experiment, as illustrated in Figure 2. These architectures were selected for simplicity in evaluating the conceptual framework. Two are classifiers and one is an autoencoder.

The two classifier architectures each consisted of an input layer with 20 neurons, a fully connected hidden layer containing 40 neurons with sigmoid activation, and an output layer using softmax activation. For the output layers, the 2-category classifier had two neurons, and the 10-category classifier had ten neurons. Both used the Keras categorical_crossentropy loss function. Both classifier architectures were compiled using the Adam optimiser with default learning rate of 0.001. During classifier training, the target labels were determined from the Euclidean distance between the baseline system prediction and the truth position of the surface vessel. This distance was then categorised into one of 2 or 10 categories, depending on the classifier, as shown in Table 1 and Table 2 respectively. Note in Table 2, most spans are in the order of 10m, excepting the last two categories spanning ~30m and ~500m respectively, reflecting the pareto noise distribution of the baseline system. The category boundaries were selected such that each category contained an equal number of samples across the full training dataset, ensuring balanced class distributions during training. A strategy of equal class representation was selected for this conceptual model – this may not be appropriate for a deployable model. The classifier MLMs were therefore trained to predict the categorised accuracy of the baseline system outputs.

Figure 2: MLP architectures

Table 1: 2 Category training data distribution

Table 2: 10 Category training data distribution

The autoencoder architecture consisted of an input layer with 20 neurons, an encoded layer containing 6 neurons with ReLU activation, followed by a reconstruction layer of 20 neurons with sigmoid activation. The autoencoder used the same training inputs as the classifiers.

The MLMs were trained using a pre-trained baseline model. Baseline model training datasets were not used to train the MLM. The MLM models were trained with 24 training datasets, comprising 27 306 data points. The MLM training datasets were prepared in the same way as the baseline training datasets. This process ensured that the same Pareto statistical distribution of measurement noise was maintained across all training datasets. For the classifiers, the training curriculum was ordered according to decreasing baseline predicted mean radial error across the training dataset. Each dataset was split into training and validation sets using the train_test_split function from scikit-learn (version 1.9.0), with train_size = 0.75 and random_state = 1. Both classifier MLM were trained with each training set for 200 epochs, with a batch size of 16 for the 2-category classifier and autoencoder, and a batch size of 32 for the 10 category classifier. Model weights from the best epoch based on accuracy for each training dataset were used as the starting weights for the next training dataset run. No dropout or other regularisation techniques were used. These hyperparameters were used as proof-of-concept settings rather than because of any specific tuning. Only one run was performed.

Following training, the MLM models were evaluated using the same dataset employed for baseline model testing. The test dataset was reused from McCutcheon et al. (2025), consisting of a manoeuvring profile illustrative of the AUV homing to a surface vessel. Neither the baseline model nor the MLM had been exposed to the test dataset during training. While errors were representative of the field reported error in range and bearing, the test dataset was biased to smaller errors. This bias was due to the test dataset simulating an AUV homing to a surface vessel, where separation ranges were smaller overall relative to the training datasets. At shorter ranges, bearing errors produce smaller cartesian coordinate error compared with equivalent bearing errors at larger ranges. For the classifier architectures, prediction performance was evaluated using Scikit-Learn’s classification_report function (Pedregosa et al., 2011). The autoencoder used Root Mean Square Error (RMSE) between the input and output layer as its output error function.

Artificial variations were manually inserted into the measured data to evaluate the sensitivity of the MLM models to anomalous behaviour. Variations were introduced at datapoint 159 of the MLM input, corresponding to a region where both the baseline model prediction had low prediction errors and corresponding MLM output predicted low category error. The anomaly scenarios in Table 3 were introduced as conceptual disturbances or attack conditions. The variations were inserted into the measured data after conversion to cartesian coordinates, and before they were windowed and scaled. The single variations therefore influenced predicted output points 151 to 159, while the 10-variation sequence had an effect from point 151 to 168.

Table 3: Anomaly scenarios

Results for the two classifier MLM methods using the nominal test dataset, without inserted variations, are presented in Table 4. Accuracy was calculated as the proportion of correctly classified samples across all categories relative to the total number of test samples. The proportional chance criterion (PCC) provides a comparison to a random chance prediction given the dataset class distributions. With class prior distributions of pi, the PCC is calculated as the sum of the squared class probabilities    (Morrison, 1969). Balanced accuracy was used to evaluate classifier performance independent of class imbalance across the test dataset.

Table 4: MLM classifier results

The 2-category classifier achieved only marginal performance over the PCC and poor balanced accuracy, indicating performance worse than chance. The 2-category classifier produced a low false positive rate, but this was mainly due to the predominant low error in the testing dataset. For comparison, a ZeroR prediction strategy (all category ‘0’ prediction) would have produced an accuracy of 0.905. This highlights the importance of investigating multiple metrics, especially here, with a class imbalance in the test dataset. For this case, while the accuracy may appear high, comparison to the PCC serves to contextualise the result. Figure 3a shows the confusion matrix for the 2-category classifier.

The 10-category classifier also achieved performance marginally above the PCC with a balanced accuracy indicating performance only slightly above chance. When the classification tolerance was relaxed such that predictions within one or two adjacent categories were considered acceptable, the reported accuracy increased accordingly. Interpretation of this performance metric needs to be considered against the category distributions in Table 2, as the categories were distributed for even training samples, and not error distance. The confusion matrix for the 10-category classifier is shown in Figure 3b.

Figure 3: Confusion Matrices for classifier MLM

Overall, the two classifier methods demonstrated poor classification capability; however, their responses to inserted variations remained of interest for detecting out of distribution samples. Quantified results are provided in Table 5, showing a minimal effect on predictions. Figure 4 illustrates the response of the 2-category classifier (left) and the 10-category classifier (right) to the inserted variations. The top plots are without variations, and these are followed by inserted variations as labelled. The specific region of points has been expanded to show the effect of the inserted variation on the predicted categories. In the context of the whole dataset however, these changes are lost in the noise.

Table 5: Variation effects

Figure 4: Effect of variations

Figure 5 shows the responses to the inserted variations for the autoencoder MLM. The autoencoder error as presented is the RMSE of all features. The response to the single zero is small, reflective of the same for the classifier MLMs. However, the autoencoder response provides a clear signal against the truth for each of the 10-point signal dropout and the single outlier variations. For this conceptual model, if a threshold to flag an outlier were set to the 80^th percentile of the training data error (140m), then both variations would have been flagged. Setting an appropriate threshold to flag the signal (for both above and below the reference signal) is key to having this be of practical use in the field. This setting would be determined based on overall system performance, anticipated anomaly characterisation, and risk thresholds. The autoencoder appeared to show the most discernible response to the inserted variations overall.

Figure 5: Autoencoder response to variations

The MLM methods presented in this study used simple multi-layer perceptron architectures to demonstrate the proof of concept. Likewise, the MLM methods used the same input features as the baseline system. Alternative architectures, feature sets, or feature engineering approaches could support multiple complementary monitors operating together to provide more specialised monitoring functions. More refined architectures with improved classification accuracy may provide earlier or more reliable indications of degrading system behaviour before threshold exceedance.  However, improved performance is not guaranteed with the sophistication of the ML architecture, reflective of the complex nature of ML based systems. Additional runs would also be required to provide a statistically rigorous understanding of system performance, to include random seed changes during training as well as temporal shifts for the inserted artificial variations during evaluation. One limitation of the autoencoder presented in this paper was that the output error function is the RMSE across all features. Per feature reconstruction error could allow further insight on which feature or features were potentially anomalous.

A design limitation for the system was that the MLM used the same features as the baseline system. This means the MLM would not be able to detect failures, anomalies, or attacks originating from the inputs that the baseline also receives. This could be mitigated by using different or additional features as input to the MLM, including parameters that are internal to the baseline system. A further limitation of this study is the exclusive use of synthetic data. Although McCutcheon et al. (2025) attempted to ensure that the simulated errors were representative of field conditions, validation using field data would provide greater confidence in the proposed concept. Field data would not only focus on validating the synthetic errors, but also the variations included to test the MLM. The class imbalance in the test dataset may also confound the results, though further work would be needed to analyse this effect, given the operational use of the baseline system for homing.

Despite the limitations, the proposed autoencoder method demonstrated observable responses to the inserted variations without extensive model tuning or feature engineering. This highlights the power of ML approaches for application areas where pattern recognition or matching are of particular interest, which is the case for complex systems. Although optimal performance would likely require problem-specific model design and feature engineering, the present concept demonstration indicates that a general-purpose ML monitor can still show sensitivity to changes. Such a general monitor might not have the same performance as a specific (or tuned) method but could provide an acceptable level of performance for a given system. The acceptable level of performance would, in turn, be dictated by the given system and application.

Monitoring systems of this nature inherently require an operational response following anomaly detection. This could be invoking alternative (perhaps certified) methods or informing a decision to remove a monitored system from the field for retraining. As such, this method could be used in conjunction with other monitors or guards to achieve that aim. In practice, this type of assurance framework would likely involve multiple complementary monitoring mechanisms. Given the large number of potential things to monitor for, it is not unreasonable to follow that multiple monitors would be required. The power of using a monitor such as the MLM is the potential insight gained from the instructive patterns present, whether in the input features, outputs, and/or relationships between them. Depending on the assurance requirements of the application, selecting an appropriate assurance framework, such as those discussed by Bloomfield & Rushby (2025) will be critical.

These MLM concepts could equally be applied to a conventional system, with suitable features and targets (for the case of classifiers) identified. The core concept here is that the system, no matter whether simple, complicated, or complex, can display instructive patterns if allowed. This does not bypass the need to test the system, nor should this method be used as a sole means of assurance, but it can contribute to a layered assurance approach to achieve the level of assurance required for the application.

One practical limitation of an ML-based monitoring system is the computational resources required for deployment at the operational edge. Practical implementation therefore requires careful consideration of the trade-off between monitoring performance and computational resource consumption, noting that increased model complexity does not necessarily guarantee improved performance. This may be in the form of considering simpler models or simplifying more complex models with existing techniques.

Future work could extend this methodology to alternative datasets and application domains. Starting close to the problem, a different target for the classifier methods, or a different feature set for the autoencoder could be explored. Many open-source datasets and methods are available for anomaly-detection research. The proposed method could therefore be directly compared against another method or another dataset. Alternatively, applying another methodology to the present baseline system dataset may help identify limitations associated with the synthetic data generation process. Further research could also explore alternative MLM architectures or feature engineering approaches to improve the performance of the MLM. Finally, eXplainable AI (XAI) techniques could be integrated alongside the MLM framework to provide greater interpretability of system behaviour and anomaly detection outcomes. At present, it remains unclear whether samples producing high autoencoder reconstruction errors are genuinely anomalous relative to the dataset distribution or whether the observed behaviour reflects limitations of the MLM methodology itself. Future incorporation of alternative MLM architectures or XAI techniques must also consider the computational and resource constraints associated with edge deployment.

This paper investigates whether an ML-based monitor could be used as cognitive instrumentation to provide assurance for a complex system. While simple and complicated systems depend on T&E for providing grounds for assurance that they function as intended and are vulnerability free, complex systems require more. One characteristic of complexity is that a system can display instructive patterns if allowed, and ML techniques are known for their pattern matching capability. Training an ML component as a system monitor may allow deviations from normal system operation to be recognised and flagged for follow on action. Such a monitor contributes to the assurance of a system alongside T&E, rather than serving as an alternative.

In this paper we presented three MLM methods to demonstrate the concept of an ML based monitor as an additional assurance layer. The three MLM were comprised of two classifiers and one autoencoder. The two classifier MLMs did not show good accuracy when predicting the baseline system performance. The autoencoder showed the best result of the three presented architectures for anomaly detection. The autoencoder provided a discernible signal in the presence of a variation, whereas the responses from the classifier methods were not consistently distinguishable from the noise.

The three MLM methods presented were simple multilayer perceptron architectures, which could be further developed or refined for the specific application, likely with improved results. Further, development with real world data and real-world input variations is essential to considering such a system for fielding. Alongside refinement, future work could extend this methodology to different datasets and methods to provide a direct comparison of performance for assurance purposes. This study suggests the feasibility of using an ML based monitor as cognitive instrumentation in a proof-of-concept setting. The monitor can give an indication of performance degradation and flag potential vulnerabilities or threats. This study also reiterated the need to test the overall system and employ multiple metrics and methods to ensure a correct interpretation of assurance is achieved.

Ali, A., Jayaraman, R., Azar, E., & Maalouf, M. (2024). A comparative analysis of machine learning and statistical methods for evaluating building performance: A systematic review and future benchmarking framework. Building and Environment, 252, 111268. https://doi.org/10.1016/j.buildenv.2024.111268

Batarseh, F. A., Freeman, L., & Huang, C. H. (2021). A survey on artificial intelligence assurance. Journal of Big Data, 8. https://doi.org/10.1186/s40537-021-00445-7

Baumeister, J., Dauer, J. C., Finkbeiner, B., & Schirmer, S. (2023). Monitoring with verified guarantees. International Journal on Software Tools for Technology Transfer, 25, 593–616. https://doi.org/10.1007/s10009-023-00712-3

Beland, S., Chang, I., Chen, A., Moser, M., Paunicka, J., Stuart, D., Vian, J., Westover, C., & Yu, H. (2020). Towards assurance evaluation of autonomous systems. Proceedings of the 39th International Conference on Computer-Aided Design, 1–6. https://doi.org/10.1145/3400302.3415785

Bellman, K. L. (2005). Self-Conscious Modeling (Modellierung bewusster Systeme). It-Information Technology, 47(4), 188–194. https://doi.org/10.1524/itit.2005.47.4.188

Bloomfield, R., & Rushby, J. (2021). Assurance 2.0: A Manifesto. https://doi.org/https://doi.org/10.48550/arXiv.2004.10474

Bloomfield, R., & Rushby, J. (2025). Assurance of AI Systems From a Dependability Perspective. https://doi.org/https://doi.org/10.48550/arXiv.2407.13948

Borg, M., Englund, C., Wnuk, K., Duran, B., Levandowski, C., Gao, S., Tan, Y., Kaijser, H., Lönn, H., & Törnqvist, J. (2018). Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry. https://doi.org/10.48550/arXiv.1812.05389

Burton, S., Gauerhof, L., & Heinzemann, C. (2017). Making the Case for Safety of Machine Learning in Highly Automated Driving. In S. Tonetta, E. Schoitsch, & F. Bitsch (Eds.), SAFECOMP 2017 Workshops (pp. 5–16). Springer International Publishing. https://doi.org/10.1007/978-3-319-66284-8

Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., Freeman, L., & Alexander, K. (2024). Testing Machine Learning: Best Practices for the Life Cycle. Naval Engineers Journal, 136, 249–263.

Chowdhary, K. R. (2020). Fundamentals of Artificial Intelligence. Springer India. https://doi.org/10.1007/978-81-322-3972-7

Clark, M., Kearns, K., Overholt, J., Gross, K., Barthelemy, B., & Reed, C. (2014). Air Force Research Laboratory Test and Evaluation, Verification and Validation of Autonomous Systems Challenge Exploration Final Report. https://apps.dtic.mil/sti/pdfs/ADA614199.pdf

Etzioni, A., & Etzioni, O. (2016). AI assisted ethics. Ethics and Information Technology, 18(2), 149–156. https://doi.org/10.1007/s10676-016-9400-6

Freeman, L. (2020). Test and Evaluation for Artificial Intelligence. Insight (International Council on Systems Engineering), 23(1), 27–30. https://doi.org/https://doi.org/10.1002/inst.12281

Freeman, L., Batarseh, F. A., Kuhn, D. R., Raunak, M. S., Kacker, R. N., & Lin, H. Y. (2022). The Path to a Consensus on Artificial Intelligence Assurance. Computer, 55(3), 82–86. https://doi.org/10.1109/MC.2021.3129027

Gunning, D. (2016). Explainable Artificial Intelligence (XAI) [DARPA-BAA-16-53]. Defense Advanced Research Projects Agency.

Gupta, P., & Schumann, J. (2004). A tool for verification and validation of neural network based adaptive controllers for high assurance systems. Eighth IEEE International Symposium on High Assurance Systems Engineering. https://doi.org/10.1109/HASE.2004.1281757

Haugh, B., Sparrow, D., & Tate, D. (2018). The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems [IDA Paper P-9292]. Institute for Defense Analyses.

Helle, P., Schamai, W., & Strobel, C. (2016). Testing of Autonomous Systems – Challenges and Current State-of-the-Art. 26th Annual INCOSE International Symposium (IS 2016), 26(1), 571–584. https://doi.org/10.1002/j.2334-5837.2016.00179.x

ISO/IEC/IEEE. (2025). ISO/IEC/IEEE 15026-1 Systems and software engineering – Systems and software assurance – Part 1: Vocabulary and concepts.

Jacklin, S., Schumann, J., Gupta, P., Richard, M., Guenther, K., & Soares, F. (2005, September 26). Development of advanced verification and validation procedures and tools for the certification of learning systems in aerospace applications. Infotech@Aerospace. https://doi.org/https://doi.org/10.2514/6.2005-6912

Johannssen, A., Qiu, P., Yeganeh, A., & Chukhrova, N. (2025). Explainable AI for trustworthy intelligent process monitoring. Computers & Industrial Engineering, 209, 111407. https://doi.org/10.1016/j.cie.2025.111407

Kercel, S. W. (2000). Why should engineers be interested in bizarre systems? Smc 2000 Conference Proceedings. 2000 Ieee International Conference on Systems, Man and Cybernetics.’cybernetics Evolving to Systems, Humans, Organizations, and Their Complex Interactions’(Cat. No. 0, 3, 2210–2215. https://doi.org/10.1109/ICSMC.2000.886444

Kercel, S. W., Brown-VanHoozer, S., & VanHoozer, W. R. (2000). The model-based mind. Smc 2000 Conference Proceedings. 2000 Ieee International Conference on Systems, Man and Cybernetics.’cybernetics Evolving to Systems, Humans, Organizations, and Their Complex Interactions’(Cat. No. 0, 3, 2204–2209. https://doi.org/10.1109/ICSMC.2000.886443

Laplante, P., & Kuhn, R. (2022). AI Assurance for the Public – Trust but Verify, Continuously. 2022 IEEE 29th Annual Software Technology Conference (STC), 174–180. https://doi.org/10.1109/STC55697.2022.00032

Liu, Y., Yerramalla, S., Fuller, E., Cukic, B., & Gururajan, S. (2004). Adaptive control software: Can we guarantee safety? Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC’04), 2. https://doi.org/10.1002/0471648272.ch7

McCutcheon, R., Joiner, K., Qiao, L., Keane, J., Olsen, A., El-Fiqi, H., Garratt, M., & Arulampalam, S. (2025). Establishing a baseline for Assurance of Complex and Critical Artificially Intelligent Systems. Unpublished Manuscript.

Morrison, D. G. (1969). On the Interpretation of Discriminant Analysis. Journal of Marketing Research, 6(2), 156. https://doi.org/10.2307/3149666

Naveed, H., Barnett, S., Grundy, J., Arora, C., Khalajzadeh, H., & Haggag, O. (2025). Monitoring Machine Learning Systems: A Multivocal Literature Review. In ACM Transactions on Software Engineering and Methodology: XX. https://doi.org/https://doi.org/10.48550/arXiv.2509.14294

Nguyen, N., & Jacklin, S. (2010). Stability, Convergence, and Verification and Validation Challenges of Neural Net Adaptive Flight Control. In J. Schumann & Y. Liu (Eds.), Applications of Neural Networks in High Assurance Systems (Vol. 268, Number August 2007, pp. 77–110). Springer. https://doi.org/https://doi.org/10.1007/978-3-642-10690-3_5

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12.

Peterson, E. M., DeVore, M., Cooper, J., & Carr, G. (2020). Run Time Assurance as an Alternate Concept to Contemporary Development Assurance Processes. https://doi.org/NASA/CR–2020-220586

Porter, D. J., McAnally, M. O., Bieber, C. M., Wojton, H. M., & Medlin, R. (2020). Trustworthy Autonomy: A Roadmap to Assurance, Part I: System Effectiveness [IDA Document P-10768-NS]. Institute for Defense Analyses.

Pullum, L. L., Taylor, B. J., & Darrah, M. A. (2007). Verification and Validation of Neural Networks – Guidance. In Guidance for the Verification and Validation of Neural Networks (pp. 39–110). IEEE Computer Society. https://doi.org/10.1002/9781119134671.ch3

Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020). Testing machine learning based systems: a systematic mapping. Empirical Software Engineering, 25(6). https://doi.org/10.1007/s10664-020-09881-0

Ryan, M. (2019). Extending the intellectual edge with artificial intelligence. Australian Journal of Defence and Strategic Studies, 1(1), 23–40.

Schnitzer, R., Kilian, L., Roessner, S., Theodorou, K., & Zillner, S. (2024). Landscape of AI safety concerns — A methodology to support safety assurance for AI-based autonomous systems. 8th International Conference on System Reliability and Safety (ICSRS), 501–510. https://doi.org/doi:10.1109/ICSRS63046.2024.10927556

Schumann, J., Gupta, P., & Nelson, S. (2003). On verification & validation of neural network based controllers. https://api.semanticscholar.org/CorpusID:2265523

Schumann, J., & Nelson, S. (2002). Toward V&V of Neural Network Based Controllers. Proceedings of the First ACM SIGSOFT Workshop on Self-Healing Systems (WOSS’02). https://doi.org/10.1145/582128.582141

Snowden, D. J., & Boone, M. E. (2007, November). A Leader’s Framework for Decision Making. Harvard Business Review, 85(11)(11), 68–76.

Tate, D., Grier, R., Martin, C., Moses, F., & Sparrow, D. (2016). A framework for evidence-based licensure of adaptive autonomous systems [IDA Paper P-5325]. Institute for Defense Analyses.

Torens, C., Nagarajan, P., Schirmer, S., Dauer, J., Baumeister, J. E., Kohn, F., Finkbeiner, B., Manfredi, G., & Löhr, F. (2024, January 8). Certification Aspects of Runtime Assurance for Urban Air Mobility. AIAA SCITECH 2024 Forum. https://doi.org/10.2514/6.2024-1464

Van Lent, M., Fisher, W., & Mancuso, M. (2004). An explainable artificial intelligence system for small-unit tactical behavior. Proceedings of the 16th Conference on Innovative Applications of Artificial Intelligence, 900–907. https://api.semanticscholar.org/CorpusID:7286175

Vancsura, L., Tatay, T., & Bareith, T. (2025). Navigating AI-Driven Financial Forecasting: A Systematic Review of Current Status and Critical Research Gaps. Forecasting, 7(3), 36. https://doi.org/10.3390/forecast7030036

Weller, A. (2019). Transparency: Motivations and Challenges (pp. 23–40). https://doi.org/10.1007/978-3-030-28954-6_2

Wojton, H. M., & Porter, D. J. (2020). Briefing to the Air Force Scientific Advisory Board: T&E Contributions to Avoiding Unintended Behaviors in Autonomous Systems (Number NS D-12078). JSTOR. http://www.jstor.org/stable/resrep22757

Wojton, H. M., Porter, D. J., & Dennis, J. W. (2020). Test & Evaluation of AI-enabled and Autonomous Systems: A Literature Review [IDA Document NS-D-14331]. Institute for Defense Analyses.

Yampolskiy, R. V. (2025). On monitorability of AI. AI and Ethics, 5(1), 689–707. https://doi.org/10.1007/s43681-024-00420-x

Zaka, A., Mustafiz, C., Mutahar, D., Sinhal, S., Gorcilov, J., Muston, B., Evans, S., Gupta, A., Stretton, B., Kovoor, J., Mridha, N., Sivagangabalan, G., Thiagalingam, A., Ramponi, F., Chan, J., Bennetts, J., Murdoch, D. J., Zaman, S., Chow, C. K., … Bacchi, S. (2025). Machine-learning versus traditional methods for prediction of all-cause mortality after transcatheter aortic valve implantation: a systematic review and meta-analysis. Open Heart, 12(1). https://doi.org/10.1136/openhrt-2024-002779