SECC: An AI-Powered Assurance Agent for Complex Government System Integration

Government acquisition programs of the 21st century are defined by complexity at a scale for which traditional systems engineering processes were not designed. Multiple vendors develop interdependent subsystems in parallel. Requirements evolve across years of concurrent development. Thousands of engineering artifacts, concepts of operations (ConOps), requirements specifications, interface control documents (ICDs), and verification and validation (V&V) plans, must remain mutually consistent, traceable, and compliant throughout a program lifecycle that grows more compressed with each successive modernization initiative.

The test and evaluation (T&E) community has long recognized that the cost of discovering defects increases substantially with proximity to program completion. A requirements inconsistency found during document review costs a fraction of what that same inconsistency costs when discovered during system integration testing or, worse, after delivery. Yet current review practice is structurally constrained. Engineering teams conducting artifact review cover an estimated 10 to 15 percent of the total documentation corpus; the remainder goes unreviewed. Given that a major government program can generate tens of thousands of pages of technical documentation, this gap is not a reflection of inadequate effort, it is the inevitable outcome of applying human-bandwidth-dependent processes to a problem that has outscaled those processes.

The FAA’s ongoing ATC modernization effort illustrates the stakes. As Hayman and Keegan (2025) have described, this initiative requires the deployment of 618 radars, 800 digital voice switches, 25,000 ATC radios, 15 new towers, and two common automation platforms, all developed by different vendors, subject to different specifications, and required to interoperate within an airspace environment where failure has direct safety consequences. The challenge is not the availability of technology; proven systems exist. The challenge is execution: integrating thousands of components from multiple vendors into a coherent operational system within a timeline measured in hundreds of days.

The Systems Engineering Command Center (SECC) is an AI-powered artifact assurance platform developed to address this challenge. In this context, “assurance” refers to the systematic, evidence-based verification that engineering artifacts are internally consistent, traceable, and compliant with governing standards, a definition aligned with the INCOSE Systems Engineering Handbook’s treatment of assurance as a continuous process of building justified confidence in a system’s fitness for purpose (INCOSE 2023). Built using RAG and locally hosted LLMs, SECC ingests the full spectrum of engineering artifacts and performs automated analysis across the entire document corpus—detecting traceability gaps, semantic conflicts, structural inconsistencies, and cybersecurity misalignments, and returning scored findings through an interactive dashboard with drill-down to source evidence. This paper introduces SECC, describes its technical architecture and scoring methodology, presents evaluation findings from an initial proof-of-concept implementation, and discusses implications for T&E practice.

The scale of integration complexity on large government programs has grown faster than the tools and processes available to manage it. When two vendors develop subsystems that must interface, there is one interface to verify. With five vendors, there are ten potential interfaces. With ten vendors, forty-five. With twenty vendors, not unusual for a major national infrastructure program, there are 190 potential interface interactions, each capable of harboring undiscovered conflicts between requirements, specifications, or design assumptions.

Figure 1. Exponential growth in interface conflicts as vendor count increases. The number of potential interfaces follows n(n−1)/2, where n is the number of vendors.

This combinatorial growth is not merely theoretical. The FAA’s ATC modernization operates across a distributed execution model in which multiple vendors simultaneously deliver radar systems, voice switch equipment, and automation platforms across six geographic regions. Each vendor’s systems must meet individual performance specifications while maintaining interoperability across the regional network. Each interface represents a potential failure point where requirements, interpreted differently by different vendors at different times, can cascade into system-wide incompatibilities (Hayman and Keegan 2025).

Manual review processes cannot scale to meet this challenge. Systems engineers currently spend an estimated 30 to 40 percent of their working time, approximately 12 to 16 hours of a standard 40-hour week, on manual cross-referencing of artifacts, tracing requirements to architectures, and identifying conflicts between specifications. Despite this investment, manual reviews cover only a fraction of the total artifact corpus (INCOSE 2023). The remaining inconsistencies surface during integration testing or operational deployment, where the cost of correction is dramatically higher.

The cost differential is not marginal. A requirements inconsistency detected during document review may require hours of revision. The same inconsistency detected during system integration testing may require weeks of coordinated rework across multiple vendors. Discovered after deployment, it may drive field modifications that persist for the operational life of the program. The FAA’s Terminal Flight Data Manager (TFDM) program provided a documented example of this dynamic: late-discovered integration issues contributed to substantial schedule and cost growth during program execution.
The T&E community will recognize this pattern immediately. Defect discovery cost curves are a foundational concept in test planning: the cost of fixing a defect rises steeply with the lateness of its discovery. What SECC introduces is the application of this principle at the artifact level, moving defect discovery earlier than integration testing, earlier than design review, to the point where requirements and specifications are first drafted and submitted.

A secondary challenge compounds the primary one. Modern defense and civil aviation programs increasingly adopt outcome-based contracting models that define performance objectives rather than prescriptive technical specifications. This approach enables vendor innovation and accelerates procurement, but raises a specific validation challenge: without prescriptive interface specifications to check against, cross-vendor compatibility validation depends entirely on human judgment applied to an artifact corpus that no team can fully review manually.

The Systems Engineering Command Center (SECC) is an artifact assurance platform that applies artificial intelligence to the systematic review, analysis, and validation of systems engineering documents. Its architecture consists of seven integrated functional components providing end-to-end coverage from document ingestion through finding visualization and evidence export.

Figure 2. SECC system workflow. Seven sequential stages: Upload & Ingestion → PDF Preprocessing → Embedding & Vector Storage → RAG Analysis Engine → Issue Classification → Dashboard Visualization → Report Export.

Document Upload and Ingestion

Systems engineers upload PDF-based artifacts, ConOps, requirements specifications, architecture diagrams, ICDs, and V&V plans through a drag-and-drop interface. Upon upload, the system validates file types, assigns version identifiers, and logs metadata including filename, author, and timestamp. A cryptographic hash is generated for each document to support duplicate detection and version management, ensuring complete traceability from initial intake.

PDF Preprocessing and Conversion

Uploaded documents are processed by the Marker framework, a document preprocessing engine optimized for preparing technical text for LLM analysis. Each PDF is converted to structured Markdown format while preserving headings, tables, equations, and embedded figures. The system generates unique tags for every paragraph and section, enabling fine-grained traceability throughout subsequent analysis phases. It is noted that OCR-based conversion of complex or scanned PDFs can introduce formatting artifacts in the resulting Markdown, a characteristic with observable effects on finding accuracy that is discussed further in the Results section.

Embedding and Vector Storage

Converted document chunks are embedded using locally hosted text embedding models via Ollama and stored in ChromaDB, an open-source vector database optimized for offline retrieval. Each chunk retains associated metadata, document type, section title, and source tag, enabling precise context recall during analysis. Local hosting is an intentional architectural decision: for programs handling sensitive documentation, local execution eliminates any requirement to transmit artifacts to external cloud services.

Retrieval-Augmented Generation Analysis

The RAG analysis engine constitutes the analytical core of SECC. When activated, the engine retrieves context-relevant text from ChromaDB and submits it to a locally hosted LLM for focused analysis. The engine performs four classes of automated review systematically across all ingested artifacts.

Consistency checks examine documents for conflicting statements, mismatched system boundaries, contradictory interface values, or requirements that are individually compliant but mutually incompatible when compared across vendor submissions. Traceability checks verify that every requirement maps correctly across needs, functions, and test cases, and that no requirement exists without parent or child relationships in the hierarchy. Compliance checks validate alignment with governing standards, including International Council on Systems Engineering (INCOSE) guidelines, ISO/IEC/IEEE 15288 lifecycle processes, and domain-specific frameworks such as DO-278A for communications, navigation, and surveillance systems. Duplicate detection identifies duplicated or near-duplicate requirements by comparing semantic similarity scores across all ingested documents (Lewis et al. 2020).

Issue Classification and Taxonomy

Detected findings are automatically classified into four categories that reflect the most consequential failure modes in government program documentation:

• Syntax issues: Formatting errors, grammatical inconsistencies, and structural deviations from program standards.
• Traceability issues: Missing or broken links between requirements at different levels of the hierarchy.
• Semantic issues: Conflicting statements or ambiguous phrasing that could support divergent vendor interpretations.
• Cybersecurity issues: Safety and security misalignments and requirements that create unintended vulnerability exposure.

Evaluation of the classification engine across multiple LLM configurations revealed category-specific error trends worth noting for practitioners. Syntax issue detection was the most susceptible to false positives, particularly with higher-capability models. When the Marker preprocessing framework converts complex or scanned PDFs to Markdown, residual OCR artifacts, irregular spacing, fragmented table content, and hyphenation artifacts can be misread by the LLM as genuine formatting violations. This was the dominant false positive pattern observed with the GPT-4o mini configuration. Smaller locally hosted models (Llama 3.2, 3B parameters) exhibited the inverse problem: insufficient recall on complex semantic and traceability analysis, frequently missing multi-hop inconsistencies that required reasoning across disparate document sections. The mid-tier Qwen 3 (8B parameter) configuration demonstrated strong recall, with occasional citation errors that attributed a valid finding to an adjacent but incorrect source passage. These patterns informed the model selection guidance discussed in the Results section.

Figure 3. SECC issue taxonomy. Four-category classification framework showing representative examples of Syntax, Traceability, Semantic, and Cybersecurity issue types.

Each finding is assigned a confidence score reflecting model certainty and a severity level reflecting potential program impact (Critical, High, Medium, or Low).

Multi-Attribute Utility Theory Scoring

Findings are not presented as a flat list. SECC applies a Multi-Attribute Utility Theory (MAUT) framework (Keeney and Raiffa 1993) to compute a quantitative System Health Score for each analyzed artifact, converting the distribution of findings across severity levels into a 0–100 health metric. The weighted risk utility H is computed across four severity levels, with each level assigned a weight reflecting its relative operational impact: Critical (0.55), High (0.25), Medium (0.15), and Low (0.05). These weights ensure that a single critical finding significantly influences the overall score, while multiple low-severity findings have a proportionally smaller effect.

The System Health Score is derived as (1 − H) × 100, where higher scores indicate stronger alignment, clarity, and traceability. Scores above 85 indicate a Healthy artifact; 70 to 84 indicate Moderate concern; below 70 indicate an At Risk artifact requiring immediate attention.

The severity weights and health band thresholds were established through a combination of engineering judgment informed by systems engineering practice and sensitivity analysis. The Critical weight of 0.55 reflects the well-documented principle that unresolved critical-severity requirements conflicts carry disproportionate downstream risk relative to lower-severity findings (Keeney and Raiffa 1993; INCOSE 2023). The 85/70 band boundaries were validated through a Monte Carlo simulation across 1,000 trials spanning realistic finding distributions, confirming that the boundaries produce stable and discriminating tier separations across a wide range of artifact quality profiles. These weights and thresholds are configurable parameters. Program stakeholders can adjust both the severity weights and the health band boundaries to reflect program-specific risk tolerances, regulatory contexts, or contractual acceptance criteria. This gives T&E and program management teams the flexibility to calibrate the scoring model to their specific operational environment rather than accepting fixed defaults.

Figure 4. MAUT System Health Score visualization. Three-band display showing Healthy (85–100), Moderate (70–84), and At Risk (<70) ranges with severity-weighted inputs.

Dashboard, Reporting, and Evidence Export

All findings are presented through a web-based dashboard supporting program-level overview and finding-level drill-down. Users navigate from aggregate health scores to specific documents, from documents to individual findings, and from findings to source passages in the original artifacts, providing the evidence chain essential for T&E applications where findings must be traceable to authoritative source material. SECC exports findings in CSV, JSON, and PDF formats with evidence links preserved, enabling integration with program management systems, review boards, and compliance reporting processes.

SECC’s architecture was developed with the requirements of large government integration programs as its primary design context. Two program scenarios illustrate the application.

FAA Air Traffic Control Modernization

The FAA is executing the most significant modernization of the United States air traffic control infrastructure in a generation. The program encompasses the procurement and integration of radar systems, digital voice switches, ATC communications equipment, and automation platforms, all developed by different vendors, operating under different specifications, and required to interoperate within an airspace environment where reliability has direct safety consequences.

The artifact management challenge in this context is substantial. Vendor submissions include requirements specifications, ICDs, design documentation, and V&V plans that must be reviewed against the program baseline, against each other, and against applicable standards including DO-278A for communications, navigation, and surveillance systems. The volume of documentation generated across all vendors exceeds the practical review capacity of any program office team using manual methods alone.

SECC’s design specifically targets this class of problem. By ingesting vendor-submitted artifacts and performing automated cross-document analysis, SECC enables a program office to identify traceability gaps, semantic inconsistencies, and standards compliance issues across 100 percent of submitted documentation before those issues reach integration testing. The interactive dashboard supports review prioritization by severity and provides artifact-level traceability back to source passages, enabling targeted follow-up with specific vendors on specific findings.

In traditional practice, review cycles for major vendor submissions span weeks or months. With SECC, initial screening of a submission can be completed in hours, with findings prioritized by severity and linked to source evidence. Based on the parametric analysis described in the Results section, this approach has the potential to significantly reduce requirements validation cycle times while substantially increasing artifact coverage.

Distributed Multi-Vendor Integration: An Illustrative Application

The integration challenge described above is representative of a broader class of government program. Consider a large-scale defense command and control initiative involving multiple simultaneous vendor development efforts for radars, communications infrastructure, data links, and command automation platforms. Each vendor operates under outcome-based contract requirements defining performance objectives rather than prescriptive technical specifications. Each submits documentation using its own standards. The program office must validate that every submission meets program requirements and that the aggregated set of submissions is mutually consistent, a task that grows exponentially more complex as the number of vendors increases (Dahmann et al. 2011).

In this scenario, SECC can function as an assurance layer between vendor submission and program acceptance. At submission, vendor artifacts are ingested into SECC. The RAG analysis engine performs cross-vendor consistency checking—identifying cases where one vendor’s interface assumptions conflict with another’s, where requirements allocations are duplicated or missing, or where cybersecurity provisions are insufficiently addressed. Findings are returned to the program office within hours, with specificity sufficient to support targeted vendor feedback and accelerated revision cycles.

This application is directly relevant to the shift toward outcome-based acquisition models characterizing current defense acquisition reform (U.S. DoD 2020). When contracts specify outcomes rather than prescriptive technical approaches, the program office requires analytical tooling to validate that vendor solutions are mutually compatible, a task that manual review cannot address at the scale modern programs require.

The SECC Minimum Viable Product (MVP) was evaluated as part of a George Mason University SYST/OR-699 capstone project. The evaluation employed three document sets representing distinct systems engineering domains: commercial aviation operations, unmanned aerial systems, and residential automation. Each document set was prepared with intentionally introduced inconsistencies of known types and severities, a controlled seeding approach that allowed the evaluation team to assess whether SECC detected issues that were known to be present. This methodology provides a meaningful signal of detection capability under controlled conditions, though it does not replicate the full complexity of a production program artifact corpus where inconsistencies are unscripted and organically distributed.

Three LLM configurations were evaluated: Llama 3.2 (3B parameters), Qwen 3 (8B parameters), and GPT-4o mini, representing a spectrum from lightweight locally hosted models to cloud-accessible API models. Performance varied substantially across this spectrum. The Llama 3.2 3B model demonstrated insufficient recall on complex analytical tasks, frequently failing to detect multi-hop semantic inconsistencies that required reasoning across disparate document sections. The GPT-4o mini model produced the highest overall detection rates but generated a notable volume of false positive Syntax findings; these were attributable to OCR and Markdown conversion artifacts introduced during PDF preprocessing by the Marker framework, which were misread by the model as genuine formatting violations. The Qwen 3 8B configuration struck the most favorable balance: strong recall across Traceability and Semantic categories with a lower false positive rate, though it exhibited occasional citation errors in which valid findings were attributed to adjacent but incorrect source passages.

The evaluation used an 85 percent detection rate as the primary acceptance threshold, established as Non-Functional Requirement 4.1 at program inception. This threshold was grounded in engineering judgment regarding the minimum detection rate needed to provide meaningful improvement over manual sampling-based review. The Qwen 3 8B and GPT-4o mini configurations met or exceeded this threshold under the defined test conditions. It is important to note that the evaluation reported detection rate against seeded issues; it did not compute formal precision and recall metrics as defined in information retrieval literature. A structured precision/recall study, distinguishing true positive detections from false positives and quantifying recall against an exhaustive ground truth, represents a priority for future work and will be necessary before drawing conclusions about performance in production environments.

The efficiency findings are more straightforward to characterize. Manual review of a representative vendor submission, several hundred pages of technical documentation, requires an estimated 12 to 16 hours of senior engineer time while covering only a fraction of document content at the depth required for comprehensive traceability verification. SECC completed equivalent analysis in under one hour with 100 percent document coverage. A parametric cost analysis employing Monte Carlo simulation across 1,000 trials estimated that SECC reduces review cost by more than 89 percent relative to fully manual processes, with a corresponding reduction of 10 to 15 hours of engineer time per review cycle.

These metrics carry direct implications for program management. On a program with ten active vendors submitting quarterly documentation updates, the differential between manual and SECC-assisted review represents hundreds of engineer-hours per review cycle, hours that can be redirected from repetitive cross-referencing to higher-value analytical and design activities that drive program outcomes.

The relationship between per-finding accuracy and overall program coverage deserves explicit treatment. A common concern with automated analysis tools is that lower per-finding accuracy relative to expert human review makes them inferior substitutes. This framing, however, conflates accuracy on reviewed content with total defect discovery across the full corpus. Manual review, even when conducted by experienced engineers, covers an estimated 10 to 15 percent of the total artifact corpus. Automated analysis covering 100 percent of artifacts at 80 to 85 percent per-finding accuracy will surface a substantially larger total number of genuine issues than manual review achieving 95 percent accuracy on the 10 to 15 percent of content it reaches. The practical implication is that SECC is most effectively employed not as a replacement for expert review, but as a prioritization layer: systematic automated coverage across the full corpus identifies regions of elevated concern, which human reviewers then examine at the depth that expert judgment requires. This division of analytical labor addresses both the scale limitation of manual review and the accuracy limitations of current automated tools.

The T&E community has developed sophisticated methods for testing physical systems: hardware-in-the-loop simulation, digital twin evaluation, operational test and evaluation. These methods remain essential. They test whether a built system performs as specified. What they do not address, and cannot, by their nature, is whether the specifications and requirements that defined the system were internally consistent, complete, and mutually compatible before the system was built.

SECC introduces a complementary capability: artifact-level test and evaluation. Before a system is built, before a digital twin is instantiated, before integration testing begins, SECC applies systematic automated analysis to the documentation governing the entire downstream development and test process. This extends the principle of early defect detection, a foundational concept in T&E planning, to the earliest phase of the program lifecycle, operationalizing left-shift in defect discovery at the artifact level.

This capability has direct implications for T&E organizational engagement. Currently, T&E involvement typically begins after requirements have been baselined and initial designs completed. At that point, many consequential artifact-level defects have already been embedded in the program baseline, and the cost of correction is already significant. Earlier engagement, enabled by tools such as SECC, would allow T&E organizations to flag requirements quality issues, traceability gaps, and cross-vendor inconsistencies before they propagate into design, hardware, and test, extending T&E’s contribution to the phase where its impact is greatest.

This artifact-level perspective is also relevant to reliability growth and maturation processes. DoD policy and guidance on reliability growth management (MIL-HDBK-189C) emphasize that reliability deficiencies should be identified and resolved as early as possible in the acquisition lifecycle, before hardware production, and ideally before detailed design. Requirements and interface specification inconsistencies identified by SECC at the artifact level represent a class of deficiency that, left unresolved, will manifest as reliability and integration failures in hardware and software. Incorporating artifact-level assurance into reliability growth planning extends the defect discovery process upstream in a manner consistent with the early defect elimination philosophy underlying current DoD reliability growth policy.

SECC also addresses the outcome-based requirements challenge directly relevant to T&E. Outcome-based contracting creates a specific validation gap: how does a program office determine whether vendor-proposed technical approaches are mutually compatible when contracts do not prescribe technical interfaces? SECC’s semantic analysis across vendor submissions identifies cases where outcome-compliant solutions from different vendors make incompatible interface assumptions, providing actionable findings before integration testing reveals incompatibilities that are far more expensive to resolve.

The local execution architecture of SECC has a practical implication for adoption in security-sensitive environments. By eliminating the requirement to transmit program documentation to external cloud services, SECC can be deployed on programs where external data transmission is constrained or prohibited, directly addressing a common barrier to AI tool adoption in classified and sensitive program environments where T&E organizations frequently operate.

As the complexity and pace of government acquisition programs continues to grow, the gap between manual review capacity and artifact corpus scale will widen. Tools that extend T&E principles to the artifact level represent a forward extension of the discipline into the phase where the cost of correction is lowest and the leverage on program outcomes is highest.

The integration challenges facing major government programs are not primarily technology problems. The technologies required to field modern air traffic control systems, defense networks, and critical infrastructure exist and are largely mature. The challenge is execution: ensuring that systems developed by distributed vendor teams, under compressed timelines, against outcome-based specifications, will work together correctly when assembled.

The artifact layer, requirements specifications, interface control documents, design documentation, is where that challenge must first be addressed.
SECC represents a practical, deployable approach to that challenge. By applying RAG and locally hosted LLMs to the systematic analysis of engineering artifacts, SECC extends artifact coverage from the 10 to 15 percent achievable through manual review to 100 percent, moves defect discovery earlier in the program lifecycle, and provides program offices and T&E organizations with an artifact-level assurance capability that current programs require but traditional methods cannot deliver at scale.

The evaluation described in this paper represents an initial proof of concept under controlled conditions. The principal findings that mid-tier locally hosted LLMs can meet an 85 percent detection threshold on seeded test corpora, that the efficiency gains over manual review are substantial, and that a manual/automated hybrid approach addresses the limitations of each method individually, provide a reasonable basis for continued development. Future work should include formal precision/recall evaluation against production artifact corpora, integration with Model-Based Systems Engineering (MBSE) tools, expanded standards coverage for additional domain-specific requirements frameworks, and predictive analytics capabilities that identify high-risk artifact regions before formal review cycles.

The long-term vision is a persistent artifact assurance service operating continuously across the program lifecycle, providing real-time visibility into artifact health as program documentation evolves. The T&E and systems engineering communities are invited to engage with the SECC effort and contribute to the development of rigorous evaluation frameworks for this emerging class of AI-assisted assurance tools.

The author acknowledges the contributions of the George Mason University SYST/OR-699 capstone team, Gael Abboud, Yousuf Hamodat, Emmanuel Kwakye-Dompreh, Maha Raja, and Ethan Walter, whose development of the SECC minimum viable product provided the technical foundation for this work. The author also acknowledges Chuck Agava of the Federal Aviation Administration and Oscar M. Gomeiz of LS Technologies for their technical review and insights.

Dahmann, Judith, Rebovich, George, Lane, JoAnn, Lowry, Ralph and Baldwin, Kristen. 2011. “An Implementers’ View of Systems Engineering for Systems of Systems.” In Proceedings of the 2011 IEEE International Systems Conference, 212–217. Montreal, Canada: IEEE.

Hayman, Gene, and Charlie Keegan. 2025. “How the FAA Can Achieve ATC Transformation in Three Years.” 1–6.

International Council on Systems Engineering (INCOSE). 2023. Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities. 5th ed.

ISO/IEC/IEEE. 2023. Systems and Software Engineering, System Life Cycle Processes. ISO/IEC/IEEE 15288:2023.

Keeney, Ralph L., and Howard Raiffa. 1993. Decisions with Multiple Objectives: Preferences and Value Trade-offs. Cambridge: Cambridge University Press.

Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems 33:9459–9474

U.S. Department of Defense. 2011. MIL-HDBK-189C: Reliability Growth Management. Washington, D.C.: Department of Defense.

U.S. Department of Defense. 2020. DoD Instruction 5000.88: Engineering of Defense Systems. Washington, D.C.: Office of the Under Secretary of Defense for Acquisition and Sustainment.