What 26 Years at NASA/JPL Taught Me About What AI Can Be Trusted to Do, and What It Cannot
Artificial intelligence can plan, compare, optimize and surface patterns at extraordinary speed. But consequential decisions still require something the system cannot supply on its own: responsibility for whether the answer should be trusted.
For most of my career at NASA’s Jet Propulsion Laboratory, I have worked on artificial intelligence systems designed for environments where failure is not an abstract concept.
Spacecraft operate far from the engineers who built them.
Communication can be delayed.
Resources are constrained.
Plans change.
Unexpected conditions occur.
And once a system is operating in mission conditions, there is not always a person standing beside it ready to correct the next decision.
That experience changes the way you think about the word trust.
Trust is not the same as confidence.
Trust is not a fluent answer.
Trust is not a demonstration that worked three times in front of an audience.
Trust is evidence that a system performs within understood boundaries, and that we know what happens when it leaves them.
That distinction matters enormously as AI moves into family offices, investment committees, enterprises and institutions.
At the Prestel & Partner Family Office Forum in New York, I expect one of the underlying questions to be the same one engineers ask in mission-critical systems:
What can this system safely be trusted to do?
AI is exceptionally good at some categories of work.
It can search enormous bodies of information.
It can compare alternatives.
It can identify patterns that would take a person far longer to surface.
It can assist with planning, scheduling, classification, simulation, summarization and scenario generation.
It can help a human investigate a problem much faster.
Those are real capabilities.
But the presence of capability does not eliminate the need for boundaries.
Generative AI can still confidently produce false information.
It can infer beyond the evidence.
It can reproduce assumptions embedded in its data.
It can generate an explanation that sounds coherent even when the underlying conclusion is wrong.
And when the output is presented in polished language, users can mistake fluency for reliability.
NIST’s Generative AI Risk Management Profile warns about this problem.
This is why I become concerned whenever I hear someone say, “The AI said…” as though the sentence ended the analysis.
It should begin another one.
What was the source?
What assumptions were made?
What information was missing?
How sensitive is the answer to the prompt?
Can the result be reproduced?
Can it be checked against primary evidence?
Who owns the decision if the system is wrong?
In engineering, we distinguish between what a system can do and what we are prepared to trust it to do.
Business leaders should make the same distinction.
An AI system may be perfectly appropriate for summarizing a hundred documents and completely inappropriate for making the final judgment about which investment to approve.
It may be excellent at generating scenarios and poor at recognizing which scenario depends on a false premise.
It may identify a pattern correctly while having no understanding of the organizational consequence of acting on it.
That is where human responsibility remains non-transferable.
One of the most dangerous ideas in the current AI cycle is that greater intelligence in the machine automatically reduces the need for expertise in the human.
I believe the opposite is often true.
The more capable the system becomes, the more important it is that someone understands enough to recognize when the output does not make sense.
That applies to technical advisors too.
No expert should be trusted simply because the title is impressive.
Ask what evidence the expert used.
Ask whether the expert independently verified the AI-generated research.
Ask what would change the recommendation.
Ask where the expert thinks the model is most likely to be wrong.
A serious advisor should welcome those questions.
At JPL, review is not an insult to expertise.
It is part of how expertise becomes reliable.
That may be one of the most useful lessons AI can carry from mission engineering into investment and enterprise decision-making.
Use the machine aggressively where it is strong.
Define the boundary where it is not.
Verify consequential claims.
Keep a human accountable for the decision.
And never confuse an answer that sounds intelligent with an answer that has earned your trust.
That is not skepticism about artificial intelligence.
It is how you make powerful systems useful.
_____________________________________________________________________
About the Artificial Intelligence Institute
The Artificial Intelligence Institute teaches people across ages, grades and professions how to remain in command of increasingly powerful intelligence systems.
Its proprietary behavioral architecture trains learners to move from cognitive offloading to cognitive amplification, questioning, directing, verifying and controlling the technology rather than surrendering thinking to it.
Think First. AI Helps Second.
The goal: turn advanced intelligence into a personal superpower without surrendering human judgment, authorship or control.
Learn more: artificial-intelligence-institute.com
Follow the Frontier
Follow the Artificial Intelligence Institute on LinkedIn, Instagram and Facebook for Frontier News Desk reporting, field observations, AI-readiness insights and emerging developments in human-AI interaction.
LinkedIn:https://www.linkedin.com/company/ethicalai-institutes/ | Facebook: https://www.facebook.com/profile.php?id=100094622704133 |
Family First Foundation
The Artificial Intelligence Institute grew from more than two decades of Family First Foundation work with children, families, schools and communities.
Follow Family First Foundation for the human, family and community side of this work and follow the Artificial Intelligence Institute for the frontier of AI readiness and human amplification.
Comments