My T&E Career, The First 25 Years

A few years ago, I sat at a table explaining to middle schoolers what we do in the test and evaluation (T&E) community, and one student tried to paraphrase the discipline, “So you try to break things.” After a chuckle, I realized these were 6th, 7th, and 8th graders, and that’s how they see it, maybe. The T&E discipline is unique, as very few of us at the career fair in middle school or high school said, “I want to be a tester.” Yet, I’m sure many of you will admit to playing with the Erector set, Legos, or plugging away at that new Commodore 64. Nor did we go to college to study T&E. We all fell into this discipline by some other means or path; I’m glad I did. The career field is challenging, ever-changing, and rewarding. I want to share how I got here and a few lessons I learned from my first 25 years in the (T&E) discipline, and I want to tie those lessons to International Test and Evaluation Association’s (ITEA)- nine values- Independent T&E, Timeliness, Scoped, Planned, Representative, Statistically Designed, Modeled, Measured, and Supportable.

My story started with a ten-and-a-half-year career in the Army as an Abrams Crewman, Platoon Sergeant, and Company Master Gunner. I added two years of T&E credit to my career as a tank master gunner, supporting the initial build-out of Ramjane Range in Kosovo and other small arms ranges in theatre to maintain crew and individual gunnery skills while deployed. Germany was a great place to be a tank master gunner; creating live-fire scenarios for platoon and company live-fire exercises in Grafenwoehr, Germany, was engaging and enjoyable work. Planning and forecasting ammunition and target requirements were critical to your ability to pass exam seven at the Army’s master gunner school. I didn’t realize it then, but those last few years in the Army planted the seed for my T&E career. Were we not creating T&E infrastructure to test and evaluate crew proficiency in an operational environment? As they say in the movie “Fury,” “Best Job I Ever Had.”

As mentioned above, we all stumbled into the T&E career field or, in my case, ran into it on a treadmill. I was on the treadmill in the gym in Friedberg, Germany, talking to a friend about getting out of the Army when a gentleman introduced himself and the fact that he was with General Dynamics Land Systems (GDLS). The gentleman on leave at the time, relocating from Ft. Hood, Texas, to Woodbridge, Virginia, offered me his card and discussed the new Expeditionary Fighting Vehicle (EFV) and how GDLS was looking to hire. The rest is history, as they say in Hollywood. I was hired by GDLS, ended my career in the Army, and started my bachelor’s in internetworking technologies, followed by a master’s in business administration. I worked initially creating the vehicle Battle Damage Assessment and Repair program; I designed the kit, the tech manual, and the vehicle’s unique parts and operating procedures. It was enjoyable, rewarding work where I learned much about the other engineering disciplines. I grew into an all-things survivability tester. I planned, resourced, and executed all sorts of testing for GDLS, from armor testing and signature management (acoustic, magnetic, visual, infrared, and radar) to live fire events. Live fire is where the passion skyrocketed; having been a vehicle crewman myself and after witnessing my first few live fire events, crew survivability and the testing to improve it were my calling at the time. The wars overseas drove significant testing; the range teams across the country doing multiple events daily to understand the threat and assess mitigating solutions were unappareled. At this point in my career, I began to understand the importance of T&E planning, both at the test event and programmatic levels. Linking test events to decision points and the backward planning process to ensure timely test results were in the decision makers’ hands before the decision point was vital.

If asked about our role in the T&E community, many of us will fall back on “To provide information to the decision-makers,” about the capability or systems we’re working with. That’s precisely what the EFV T&E community did. There were so many lessons learned from those test teams, whether related to amphibious vehicle testing, live fire testing, or signature testing; collectively, both industry and the government discovered a great deal from that program. Young test engineers failed, learned, and matured over several years. I was reassigned to Aberdeen Proving Ground, where I was fortunate to be back on the platform that started it all for me, the Abrams tank.

At some point, I realized I wanted more from my career as a tester. I had been fortunate to have experienced so much, traveled across the country testing various innovative products, and built a network and knowledge of the broader T&E community. I moved to the U.S. Department of Homeland Security (DHS), where I started as a Test Area Manager, with the same roles and responsibilities as an action officer with the Department of Defense (DoD) Director, Operational Test and Evaluation (DOT&E). The DHS acquisition and T&E communities are still young compared to DoD, with formal acquisition starting in 2008. When I arrived in 2015, the Department completed the final surge to get acquisition programs from 22 agencies and components aligned under a single Department. I hopped on board a fast train in what we call the Immigration and Border Security portfolio, covering programs from U.S. Customs and Border Protection (CBP) and U.S. Immigration and Customs Enforcement to the U. S. Citizens Immigration Service. I jumped at every opportunity to learn and grow at DHS; there is no lack of work in a Department with such a broad mission set. I completed a detail in Key West with the Joint Interagency Task Force South’s (JIATF-S) J7 shop, where I provided both acquisition, logistics, and T&E support to field situational awareness capabilities, increasing the Task Force’s ability to interdict drugs in the Caribbean and Eastern Pacific Oceans. JIATF-S is a great target-rich environment for those needing to test capabilities; they’re more than helpful. The detail gave me an appreciation for the broader strategic picture and joint operations across departments, agencies, and countries. Intrigued by the DHS mission, I applied for and was accepted to a program at Naval Post graduate School’s Center for Homeland Defense and Security Studies master’s, which started in 2017. This is a unique program as not all participants are from the federal government- first responders, public health officials, intelligence analysts, and many other disciplines from the Homeland Security Enterprise participate and broaden the student body’s understanding of the Enterprise and its challenges.

Four years ago, I was promoted to the Deputy Director position of the Immigration and Border Security portfolio within the DHS Science and Technology Directorate (S&T). While I knew and understood the portfolio, the relationships changed. New engagements with different program managers and executive acquisition stakeholders further broadened my understanding of T&E, acquisition, and the homeland security mission. We entered and crawled out of the COVID pandemic, moving to a new hybrid work model we are all still navigating. I volunteered to support Operation Allies Refuge in Germany as the evacuation of over 35,500 Afghans took place in weeks. Once again, the joint mission and working with soldiers, airmen, customs agents, and border patrol agents motivated me. One morning in Germany while walking a portion of the closed-off flight line in Ramstein, with equipment ranging from aircraft to forklifts, tents, and tablets collecting biometric data, it hit me again. Everything you had seen in this operation had, at some point, been through the hands of a T&E professional, and each professional worked on a team. What we do matters, has been planned and has an impact!

Somewhere along the way, I picked up the Federal Emergency Management Administration program portfolio. I was back on a range at Naval Air Station Patuxent River with the Integrated Public Alert Warning System (IPAWS) for their Electromagnetic Pulse test, a first for DHS. Testing a system with a critical continuity of operations mission against an emerging threat hand in hand with Navy testers was that motivational boost we all sometimes need. F18s buzzing over the range every day was also motivational.

Picture 1 – Integrated Public Alert Warning System (IPAWS) Primary Entry Point (PEP) Shelter

In an out-of-cycle rotation for our Coast Guard Detailee, we were left short a Deputy Director covering the Cybersecurity and Infrastructure Security Agency and Headquarters Programs. I volunteered to cover two portfolios until we could backfill the position. The additional portfolio allowed me an opportunity to work in heavy enterprise information technology and cyber programs. Once again, new relationships, new mission sets, and operational environments that can change constantly were just some of the challenges. I gained a better appreciation here of how acquisition and T&E oversight impact programs when we have a rigid set of policies and frameworks that have yet to keep up with the speed of need or relevance. The threat and operational environment change faster than our acquisition system can keep pace with at DHS.

The better portion of my career has been spent on the developmental test side, and now, at DHS, the operational test time has accrued over the years. We call it the T&E discipline, yet we often forget to mention the “E.” Testing is the fun, glamorous side, while evaluation is the challenging, sometimes dirty work. I’m currently focused on raising the bar regarding evaluation within DHS. We’ve rolled out evaluation frameworks into our acquisition programs that in 2015 didn’t exist, and we continue to refine those efforts, taking advantage of what our counterparts in the DoD have experience with while tailoring it for DHS.

ITEA Values

It’s important to note that the values ITEA has identified can be interpreted differently based on where you sit in the T&E workforce or even your career. I’ll share some of what that looked like in my career journey.

As a tester with General Dynamics and an original equipment manufacturer, you wouldn’t think we were concerned with independent testing, yet we indeed were. When an industry partner proposed a specific component, the first question we always asked was, “Who tested it?” Armor at the time was my focus; better performing lighter weight armor was the goal. I am trying to remember how many times I took armor coupons (a.k.a.samples) to the range to learn the solution wasn’t defeating our threat as advertised. Or, when we started testing temperature extremes after being in the climatic chambers, the industry partner looked on in amazement as the armor coupon failed (this one ties into testing in a representative environment below as well). Yet, we learned something together and were able to continue to grow.

Some 20 years later in my career, sitting with the DHS S&T’s T&E office, I look at independence differently. I’m more concerned about whether a major acquisition program has a qualified independent test agent capable of planning and executing a test campaign that implements all the ITEA values across the life cycle of the capability being developed.

Timely

I’m not sure the importance of timeliness T&E has changed throughout my career. It would help if you did that backward planning from the decision point to ensure you have a final evaluation product for the decision-makers. This could have been a preliminary or critical design review while I was with industry or now a DHS Acquisition Decision Event.

Scoped

The scope of T&E is directly tied to the statistically designed test. I’ve successfully developed good evaluation frameworks that decompose test requirements down to the data element. It’s hard to argue with an evaluation framework identifying exactly what resources a test manager needs to execute the test campaign. It is also critical when it comes to long lead items, requirements that may require the development of a modeling and simulation tool, or unique testing facilities with limited availability. I’ll also always support a test engineer who wants to make an excellent bottom-up list of resources. You can save yourself cost and schedule by having everything you need onsite before testing.

Planned

Planning T&E should be one of the values ITEA established; as it’s critical to the success of any test program. Early engagement with the test community is vital, but I doubt anyone reading this will argue the fact. There are differences in planning test events between the developmental and operational test disciplines. Live fire events and cyber events are different in their ways as well. The number of stakeholders increases as you move from developmental testing to operational and even more for live fire and cyber events. I’ve learned that every test should be preceded by a site survey, even if you’ve tested on the same range in the past. I had tested at Twenty-Nine Palms six times before this one event, and it was the event where we didn’t conduct a site survey. We showed up with the semis of test equipment, and unknown to us, the concrete ramp to drive the forklift or pallet jack down had been demolished as part of a construction project. The trucks were unloaded two days and a couple of thousand dollars later. Do the site survey-because surveys pay for themselves; always include one in your test planning.

Picture 2 – Cold Regions Test Center, Fort Greely, Alaska 2007

Representative

I briefly touched on this above when discussing the importance of independence. Testing in an operationally representative environment as early in the life cycle as possible pays big dividends. I’ll shift from my experience base here to future concerns. For DHS, the operational environment can change quickly. Often, we capture a snapshot of the current environment in time. The testing of artificial intelligence/machine learning, cyber, and various software capabilities creates challenges for the test community, not just from discussing the actual operational environment but from a final production representative configuration of systems that will continue to evolve. We’ll need to understand better how the configurations of these capabilities will grow and look at ways to implement a continuous evaluation approach.

Statistically Designed

My lesson for you here is if you’re not a statistics expert, and I emphasize an expert, know who these experts are. Try to understand the basics and importance, and then rely on the experts supporting you to run the numbers. Design your test based on a proven statistical method and ensure the stakeholders all agree to that approach. At DHS, we’re very fortunate to have established a partnership with the Air Force Institute of Technology’s Scientific Test and Analysis Techniques Center, and we have an excellent reliability expert on staff who can take complex reliability approaches and explain them in ways so anyone without that background understands the concept and the criticality.

Modeled

I learned the importance of models as I worked live fire, and we conducted specific testing activities to generate data to build out vulnerability models. The ability to run various threats across the entire vehicle with some degree of confidence was interesting to me. I also supported and utilized signature management models, collecting data to feed the model and then again using the model as another source of data the decision-makers could use. At DHS, the story is similar-we needed a model to support the tunnel detection efforts along our border to represent the various operating environments, geologies, and depths as just a few of the variables. Models and simulations will play a more significant role at DHS as we move forward with more capabilities requiring actual development and as a tool to explore system performance across an ever-changing operational environment.

Measured

I tell new testers to measure it and collect it. All of us have come back from a test and told ourselves, “I wish I would have collected this data or more of that data.” If the resources and range infrastructure allow for it, collect it. I’ve worked with crude mine blast deflection gauges that may not have been state-of-the-art but provided a secondary data source when other data was lost to save a test event. We need to understand test measures and be able to communicate those measures and associated data elements to decision-makers when we are scoping test events and requesting funding; they have a right to know why we need something.

Supportable

This one is simple for me. As an Abrams crewman, I understood the criticality of a supportable platform. My transition to General Dynamics allowed me to see firsthand all the work that goes into confirming a platform is supportable. The work early in the design phase to ensure parts are removable, reliable, and adequately spared is fantastic, detailed, and well executed by both industry and government personnel nationwide. With DHS, I realized how challenging it is to evaluate supportability and the importance of integrating the logistics team into test planning.

Additional Words of Wisdom:

Find a mentor and be a mentor, formal or informal. I’ve done both now. Several years ago, at DHS, I was a mentee. I gradually better understood the DHS mission space and expanded my network. As a formal and informal mentor, I’ve learned more while mentoring a young procurement analyst, logistics analyst, and, yes, even a test engineer. You get an opportunity to try new things through conversation, explore what motivates different people, and understand that we each have our own “why.” The best mentoring I received was on the tailgate of a pickup truck on ranges across the nation. From facility managers, senior test engineers, and program managers from industry, government, and even a few foreign allies, each shared their experiences, tricks of the trade, and wisdom they gained within the T&E discipline.

The December 2023 issue of the ITEA Journal contained an article jointly authored by two interns our office sponsored over the summer. After selecting the two detailees, we met with both to discuss what they’d like to accomplish and learn while at DHS. Since the internship was just over two months, we provided a survey-like opportunity on all of DHS and acquisition. Both interns were exposed to all the major components at DHS, receiving briefs on the various missions and technologies under development by each. They completed our DHS ACQ101 and T&E101 online courses, sat with our test managers, and gained a better appreciation for the T&E discipline. They sat through heated conversations as we looked at raising the bar when it came to the evaluation piece of T&E and refocused on “E before T.” Our goal was to plant a seed within these two interns; whether it grows within them, or they pass it along to someone they know, the T&E community will be better off. As a capstone, they enjoyed attending our in-person Symposium at Camp Springs and interviewed several Government and industry testers in their research efforts for their article.

Develop yourself and others. Take initiative. Nobody takes better care of you than you. Understand, learn, and research your organization’s mission. If you want it, you must go after it. I wonder if you could convince anyone in the T&E community there is a lack of work, whether it’s the day-to-day operations or the strategic initiatives our agencies have targeted. There is something out there to do. Find what you like and do it. Recognize gaps in your agency’s knowledge base.  My portfolio needed someone with a cyber background when we started evaluating cyber resilience within DHS. I worked through a cyber security certificate of electives fitting my mission needs from Georgia Tech. While I’m no cyber expert, I can support the T&E mission and talk programs through what they need to plan for, resource, and execute cyber resiliency testing. Balance your self-development with both technical and leadership opportunities. I didn’t have that balance early on; it may have helped me understand who I was as a leader in the long run. Allow those you are responsible for the opportunity to go to school and grow as well. I recently had a conversation with a tester in a component about maturing their workforce, and senior leadership didn’t support getting testers out in the field. We must create opportunities to grow the workforce.

Picture 3 – Understanding various operational environments 2019.

Be honest with yourself: if we don’t allow young test professionals to develop, they will leave and seek opportunities with another organization providing those developmental opportunities. As you climb your career ladder, also realize that your knowledge, skills, and abilities must expand. If you choose to move up the ladder, you’ll give up that time on a range for the business side, strategic planning, and workforce development conversations, to name a few. You must take the initiative, plot your course, and do what makes you happy while meeting your needs. Write an article for ITEA, write test reports, update your resume; you only get better at writing by writing, someone once told me. At the Naval Postgraduate School, they teach the “ready, fire, aim” concept – just write what comes to mind, and the magic comes in the many iterations of editing. A few years ago, I was part of a leadership development cohort sponsored by the DHS Office of the Chief Procurement Officer, which owns the department’s acquisition training mission. Our project led us to look at all the acquisition training, identify gaps, and provide recommendations on how to close those gaps. We realized that we rarely teach or mention anything about writing within the DHS acquisition curriculum. Yet we require programs to write an acquisition strategy, a systems engineering lifecycle tailoring plan, and a T&E master plan, to name a few.

Participate in professional organizations. The various international T&E Symposiums and workshops are great opportunities to develop speaking and presentation skills. The local chapters also have opportunities throughout the year, hosting guest speakers presenting on relevant T&E topics.

Work across industry, government, and academia. I’ve been fortunate that my career has spanned industry and government. I developed an understanding of what original equipment manufacturers are capable of, how government acquisition works, and the struggles and motivations of both sides. Navigating both is challenging; take the opportunity to learn both, and you’ll be much better off. I’ve also taken advantage of DoD test facilities, industry facilities, and ranges. They both have pros and cons, but you only get an appreciation for that once you’ve worked across the gamut.

The future– be flexible, adaptable, agile, dynamic, pick your adjective, and be it, as the future is uncertain. Our threat is often evolving faster than we can develop and field capability. Testing will always be the discipline that costs too much or takes too long to get something done; everything will stay the same. Paradoxically, testing saves time in the long run, as finding a problem early is exponentially better than seeing a problem after a product has been fielded. I’ll stick to what I know, and that’s DHS currently. We must understand our end users’ needs and get them involved earlier in testing. Get out in the field, understand the reality of the operational environment down on the border or out on a Coast Guard cutter, or sit behind a cyber analyst for a few hours or days to understand their work environment and mission. We need to understand and incorporate T&E as a continuum into our programs, as Mr. Christopher Collins and Mr. Kenneth Senechal outlined in the March 2023 issue of the ITEA Journal. Systems will continue to evolve, so let’s lay out a T&E campaign that evolves with the threat and capabilities.

Picture 4 – Riding along with CBP to gain a better understanding of the area of operation around JIATF-S.

Work to understand the culture you support, from the component level at DHS to the micro-cultures within each program office. Culture and relationships matter. At DHS, I realized early on the true oversight mission of the office comes with different hats. Actual T&E oversight still exists. However, there is also a coaching and mentoring hat for our younger T&E professionals working at the components. I enjoy standing at the whiteboard, mapping out evaluation frameworks and test strategies, and explaining more complex aspects of the T&E discipline to program managers.

T&E is a people business. Learn about people, what motivates them, how to communicate with them, and how to lead. Whether or not you’re in a leadership position, it doesn’t matter. At some point in the test environment, whether in a lab or on a range, there will be a need for an emergent leader to exercise leadership and take the initiative. Learning to solve problems and deal with conflict is part of human nature and our business. Understand the human capital aspects of the T&E discipline, how your hiring process works, how the job descriptions are developed, and how we can recruit and retain the latest emerging talent.

Picture 5—From left to right: Wayne Dumais, Jamie Wells (Director T&E at DHS S&T), Adam Martin, and Andy Navario following a naturalization ceremony in San Diego.

To summarize, we’ve all walked a different path to get where we are today. There is no right or wrong path through or into the T&E discipline- you choose it, you own it. Find and be a mentor, develop yourself and others, work across industry, government, and academia, be ready for the unknown future, and be in the people business. T&E is an essential aperture into the homeland and national security disciplines, which have merged over the last few years. Middle schoolers may think our purpose is to break things; we know it’s more complicated and not that simple, yet the T&E discipline is a rewarding career field.