Taking a Black Box to a Glass Box: Building trust with your AI
- Adam Jeffreson
- 4 days ago
- 9 min read
The power and uptake of AI has crossed a threshold, its capabilities and implementations are no longer constrained by what your model can do; the discussion is now what the model should do, and how you can trust it. The challenges are no longer about models being accurate, they are now about how you can validate that accuracy.
This has moved many organisations' concerns and controls from their models' ability to complete tasks to how they should complete them, how they should control their outputs, and how they can explain their actions.
With AI adoption and implementation increasing in all areas of business, we are all facing this new challenge; how do we explain how AI is answering our questions? How are we validating its reasoning? Perhaps most importantly, how do we guide its decisions and the choices it makes along the way to them?
The Model is a Black Box
“A Black Box AI is an AI system whose internal workings are a mystery to its users. Users can see the system’s inputs and outputs, but they can’t see what happens within the AI tool to produce those outputs.”
What is black box AI? IBM 2024
This is probably a situation we are all familiar with, you prompt your model, get a response and have absolutely no idea how it got there. This is to some degree by design, AI companies don’t want to expose how their models work to their competition, and even if they did, the algorithms and decisions behind them are so complex and generative that they may not even be able to if they wanted to.
“Either their developers make them into black boxes on purpose, or they become black boxes as a by-product of their training.” “ - IBM 2024
In the early days of AI, this was a major barrier to users and organisations being comfortable with using models. How can we trust what AI is saying if we don’t know why it is saying it?
What’s wrong with a Black Box?
For simple tasks, minor questions and code formatting, nothing. There’s a clear use case for just being able to ask a simple question and get a simple answer, not everything needs granular verification.
But what about when it does? Take finance for example, if your model returns a number, it is not good enough to simply say “that is what the model thinks”, you have to demonstrate why it thinks that.
"Finance does not run on answers alone - it runs on answers you can explain"
Clearly putting finance and business data into a model and blindly trusting the output is not sufficient, but numbers and calculations are logically verifiable data. What about when the output is more abstract than numbers?
Ethics, values, company standards are much harder to quantify, and even more difficult to validate. If you feed 500 CV’s into your model and ask for a shortlist it may give you a workable result-set, but how can you check these are really the right candidates? How can you be sure there isn’t an implicit bias in its evaluation?
Trust is also an important factor for your end users, decisions made by AI have been shown to be less trusted than the exact same result set when it is presented as made by a human (Trust, Attitudes and the Use of Artificial Intelligence). This means that ‘AI information’ can carry a higher burden of responsibility for self-reporting than the same data from regular reports.
These concerns are not without merit either, AI ‘Hallucinations’ are well known, and most users will have encountered nonsense or simply wrong outputs from AI suggestions. In some cases they will even learn to ‘cheat’ results by stepping outside their defined parameters to find a more efficient solution to a problem; A system tasked with detecting images of Horses was once found to have learned to look for a copyright tag associated with horse pictures instead of reviewing the actual images as requested (The Guardian).
Guideline, rules and the law
Governments and agencies are now catching up with the use of AI with a number of frameworks, guidelines and legislations. Depending on the context of your use of AI there is now a good chance you might have to be able to demonstrate how it reached a decision, or even the choices it made on the way to that decision.
Some examples of current legislations;
In Europe: The EU AI Act requires transparency to affected individuals as an AI obligation.
In France: The Loi pour une République numérique (Digital Republic Act) grants subjects the right to request and receive information pertaining to the implementation of algorithms that process data about them.
In the US: Illinois HB 3773, in force since January 2026, requires plain-language disclosure of which employment decisions AI influences, what data it uses, and where candidates can go with questions.
In the UK: The ICO has published AI transparency guidance that is increasingly treated as the standard of care, and Article 22 of the UK GDPR gives people the right not to be judged solely by a computer.
UK GDPR, Illinois Human Rights Act, EU AI Act, Loi pour une République numérique
These are just some examples, with many more in development. With the global-scale of applications and the web, it is much better to ensure as broad a system of governance and reporting as reasonably possible than to have to reverse-engineer decisions at a later stage.
Of course, accountability, verification and governance do not just occur because there is a legal or moral obligation to do so, trust in AI outputs demands the ability to inspect the inputs and test the outputs. This is where a Black Box can be turned into a Glass Box.
What is a Glass Box?
“A black box is an AI system nobody can see inside. Inputs go in, outputs come out, and nobody really knows why. A glass box is the opposite; the process is visible, the logic is understandable, and the people affected can see what's happening to them.”
“Glass Box AI describes an AI system whose outputs are accompanied by inspectable reasoning, decision factors, or traceable intermediate steps.”
A Glass Box model for AI is known by various terms; explainable AI, XAI, interpretable AI is essentially the implementation of a model to counteract the Black Box of machine learning, providing the ability of users to scrutinise automated decision making.
The goals of a Glass Box are transparency, interpretability and explainability. In very simple terms this means the model should be able to explain its path to a decision, the underlying basis for the decision and the data the decision was based on.
So why isn’t every single model a Glass Box already?
The issue is the quantity and speed of the decision making process in the models. It is simply not practical or even feasible to inspect the decisions made when AI reaches its output. To be clear, they are trying: OpenAI’s models share information about the steps they take to reach outputs, however, this is not a direct look inside the model, but a model-generated explanation of its own activity (IBM).
There are also ‘White Box’ algorithms such as linear regression, decision trees, or generalized additive models, but these are separate from the much more widely used deep neural networks of the most popular LLM’s.
Due to the potentially massive numbers of choices and regressions that underpin each decision in the conventional LLM’s, having a system of human review or a report on each part would completely negate any efficiency gained by using the model in the first place. Not to mention that due to the generative nature of the algorithms means they are constantly changing to the point where their initial designers may not be aware of how they are ‘actually’ making decisions.
As such the practical implementation of a Glass Box is generally more like picking up the Black Box and putting it inside another box.
What does a Glass Box look like?
There is no short answer to this, and it is going to be a bit technical so feel free to skip down to the implementation if you like.
The most simple way to define the Glass Box is to imagine it fully containing the Black Box in such a way that everything that is passed in is defined, and everything that comes out is validated. This is probably the easiest way to get explainability and traceability from an AI system, you provide it with a strict set of data, rigorously define its behaviour and ensure each prompt you provide has a goal and a test (or a series of tests) to verify that goal.
In a basic example for software development this could look something like; limiting your chosen model strictly to your local repo, turning off all external permissions, having clear and detailed agent and project skills, and a changelog and verification tests in place before prompting the model. This ensures that you have a clear source of the data, you can define the change and the model can reason why the change was made. Of course, this is not a perfect implementation under the definitions, but it is a practical one.
Things get more complicated when you move from something logical like a code base into something more abstract like a natural language model on a chatbot, or a system reviewing human-input or the people themselves, such as for CV filtering.
At this level the model has to consider how it will interpret the prompt it is given, what context the prompt is related to and what implicit requirements it must consider (Ethics, Business Rules, Legal Frameworks, etc). Once the model returns an output it must be evaluated (observed) against tests defined along with the interpretation rules so that the process and output can be verified.
To demonstrate this process the above white paper “The Glass Box Approach…” has an excellent ‘Running example’ in Pt.3 that is too long to include here, but essentially demonstrates how to formalise abstract concepts such as fairness and equality and to define tests for these to implement observability in a models’ outputs.
How can we implement a Glass Box?
Depending on your individual use case, you might not need or want to fully implement the Glass Box approach, but some element of it will be useful to all AI deployments.
A Glass Box approach is also highly contextual, a financial institution will require a very different approach to a development studio so we will stick to the general principles here, but broadly speaking there are some quite simple ways to move from usages with opaque processes to ones that are much more transparent and verifiable.
The first and probably most familiar will be to have clearly defined skill documentation with your models. Skills provide valuable contexts and instructions for AI that can go a reasonable way to providing the traceability that a Glass Box requires. Information on creating and implementing skills is dependent on your chosen model and the way it is accessed, but they are generally available in each provider.
These skills can also contain instructions for a changelog that the agent populates as it works, this allows for documentation in line with regression, and provides a structured way for a model to capture the steps it has taken to reach a decision and what was done when the decision was reached.
Another simple change can be to explicitly require the model to state its reasoning and logic before delivering its answer (for very specific verification this can even be written to a document, but these have a tendency to grow very large and consume a lot of usage). Tracking confidence and uncertainty is another method that can be used, in your agent instructions simply add a line for the model to report its confidence in the output, and ‘check points’ at certain thresholds so that users are aware of when the model is effectively guessing at an answer.
For more granular information the implementation of XAI frameworks such as Lime or SHAP that provide a visual output of the inputs considered for a model’s decision, along with the weighting that each individual contribution was given.
SHAP visualisations for a model for predicting House Values
These framework visualisations can then be fed back into the chosen model to add contextual specificity for each individual use case.
How to ensure you and others can trust your AI
The most simple way to begin the implementation of a Glass Box is to examine your current use of AI within your business. Evaluate your use case, available data and the given outputs.
Do you have enough traceability? Can each output be justified? Could the reasoning be documented?
If you find it is failing in any of these steps consider the creation of detailed skills, methodologies or frameworks to mitigate these failings. They can be shared at a business level, the project level or even for individual tasks.
When working with data relating to the general public, create a ‘Glass Box Statement’ that clearly defines:
What your AI does (and does not do)
How it influences decisions
What information it processes
The rights and options of people to have their data reviewed or removed
Having a cyclical review of your Glass Box is essential, nothing in AI or business is static, and ensuring trust is a constant process that starts with review.
Starting from strong foundations of explicability, transparency and interpretability will ensure your business and its use of AI has the best possible position for generating trust and confidence in your usage and its outputs.
For more information on using AI in your business, Dan Batty has written an excellent article on context engineering.
