The Folly of Fast
January 7, 2020 9:24 pm Leave your thoughts
Air Defense Systems
Air defense systems are of critical Importance to national security, as well as domestic and military air traffic control. My employer is an industry provider of these systems and I have worked as an engineer for many years developing and testing them. Our systems are designed to be operational 24/7 in some of the most extreme environments on earth and to do so for decades. They are complex, real-time systems with multiple programming languages, operating systems, hardware platforms and networks.
The seriousness of the mission and the complexity of these systems demands a dedicated team of engineers who collectively bring strong field experience, academic excellence, high ethical conduct and character qualifications, adherence to proven development methodologies, strong teamwork, thorough testing, ongoing and probing reviews of all work products, and a good working relationship with the end user. These attributes are vital to achieve success and are described in more detail below.
Qualifications and Preparation
A US Citizenship and Secret Clearance is required because of the importance of safeguarding our national defense interests. At minimum a BS in an engineering discipline is required. Many of my colleagues have advanced degrees from top engineering schools, extensive military experience, and have installed and tested defense systems in the field, often in very remote and harsh environments. Nevertheless, we all need ongoing refresher training as well as unique training for the new technologies and tools that are continually emerging.
Task
While the core product is the same for all customers, each customer has unique needs as specified in their contract that must be addressed for the defense system to be fully utilized.
Because of the complexity and seriousness of the task, a carefully thought out plan and well-proven set of phases is implemented. There are no shortcuts. There must be dedication and adherence to proven industry standards and practices.
Phase
- Contract or Statement of Work
- System Architecture
- Software Architecture
- Requirements Development
- Master Test Plan
- Requirements Analysis
- Preliminary Design
- Detailed Design
- Implementation – Software and Hardware Low-Level Test
- Integration Test with Sub-Systems
- Site Installation
- System Level Test and Site Acceptance
- Warranty Period and Ongoing Support Services
Teamwork
Ideally our program teams consist of a mix of seasoned veterans and younger engineers that bring energy, new approaches and the latest technology. Over the life of the project there are always new team members that we need to assimilate and others that move on.
Lively but respectful (usually) disagreements and discussions are common as we strive to work together to develop the best possible product.
Nevertheless, there is an awareness that despite the strength and experience of our team and our best collective efforts, these systems are large and complex and latent defects will inevitably be uncovered. New releases are offered and delivered to fix problems and provide improvements.
Reviews
Throughout the life of the program, each phases’ work products undergo thorough scrutiny. There are internal program management reviews and assessments from external agencies.
Every customer brings different assumption and needs. The customer does not just pay for and utilize our product – they are part of the team defining what the product looks like. It is vital that we get to know our customer well and have an open, trust-based relationship. These kinds of relationships take much time to develop.
We must continually ask ourselves these questions:
- Did the customer receive what they contracted and paid for?
- Are we only supporting so we can win follow-up business or are we proud of our product?
- Did we only meet the requirements, or did we exceed them?
- Beyond the numerous internal reviews, we also bring in outside auditors and evaluators that can objectively assess our work products and provide unbiased feedback.
Validation
The essence of testing is to prove that the product meets the contract requirements. There are many types of testing and together they give us the highest level of confidence that the system will meet and exceed requirements and expectations.
- Incremental Testing
- Rigorous Testing
- Longevity Testing
- Stress Testing
- Regression Testing
- Automated Testing
Summary
In all these areas we seek to find the sweet spot that optimizes both speed (completing the task on schedule and within budget) and accuracy (meeting all requirements) but we must “err” on the side of accuracy. Unregulated speed introduces errors. Internal schedule deadlines are never an excuse for shortcuts that bypass proven process. If we do not test and review properly at a lower level we will pay more later when problems are uncovered at a higher level, especially if it is during a customer demonstration or test event. Even worse are problems identified later during customer use that affect critical functions, possibly jeopardizing the lives and safety of those we are trying to protect.
This has made me wonder if any of you have a similar experience. Have you seen parallels with my concerns in your own life? Does your job, or some other part of your world, reflect my concerns? Have you ever had to choose between speed and accuracy? What did you choose, and why?