Human Oversight

Human in the Loop

To use AI models responsibly, Human involvement is essential to check model decisions and verify that they behave as they should. The AI Act reflects this human-in-the-loop aspect at various points.

To understand different responsibilities, first we have to understand the division of two major roles. Providers develop, train or alter AI models. They have to set up structures and processes to provide supplementary information useful for interpreting model outputs. Deployers use AI models or integrate them into their systems. They are responsible for observation; verifying model decisions and scanning for potential risks. In many cases, the provider and deployer will be a single entity responsible for both facilitation and execution.

The essence of human oversight is best captured by Article 13: Transparency and Provision of Information to Deployers, detailing the information providers of AI systems have to make available to downstream deployers.

---

“High-risk AI systems shall be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured with a view to achieving compliance with the relevant obligations of the provider and deployer set out in Section 3.”

---

The very first paragraph of this article highlights the fundamental challenge: deployers need to be able to "interpret an AI system’s output", and their operation needs to be "sufficiently transparent". To do this, the deployer requires additional information accompanying the AI model’s outputs. The provider will need to implement techniques providing additional information about the output of the model. These techniques need to help the deployer understand on what basis an AI model made a certain prediction.

---

“High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.”

---

The first paragraph of the article on human oversight refers to these techniques specifically as “Human-machine interface tools”. There are several techniques available to interpret the output of AI models. Popular ones are model-agnostic, which means they can be applied to any model architecture. The most popular approach uses Feature importance. These techniques measure how important each feature - a single value of the inputs - is for the prediction made by the model, this is expressed as a number. These feature importance scores can be used to show which inputs are most influential in determining the model’s decision or prediction - in Explainable AI jargon, this is called an explanation. Across different techniques, there are subtle differences in the way they express importance. You can learn more about these differences in our article comparing two popular techniques. In order “to enable deployers to interpret a system’s output and use it appropriately”, a technique like this needs to be implemented by providers of high risk AI systems. To enable deployers to use these techniques, appropriate access and training also has to be provided

Global & local explanations

In AI Act article 14: Human Oversight, we find a paragraph detailing how “natural persons to whom human oversight is assigned” are to be “enabled” by providers of AI systems to execute their role. Before we get into it, let’s save some ink and rebrand this new profession as 'AI Compliance Officer'.

---

“[The high-risk AI system shall be provided to the deployer in such a way to enable the AI compliance officer] to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance.”

---

In order to “detect and address anomalies, dysfunctions and unexpected performance” the AI compliance officer needs to be familiarized with normal, functional and expected behaviour of the AI system. This paragraph refers to a need for high-level information about the model’s general decision strategy, referring to global explanations. they provide a first glance of the model’s behaviour on average, for inputs which are most common. A global explanation provides useful background context for the local explanations used to explain a single prediction.

---

“[The high-risk AI system shall be provided to the deployer in such a way to enable the AI compliance officer] to correctly interpret the high-risk AI system’s output, taking into account, for example, the interpretation tools and methods available.”

---

Local explanations are used to “interpret the high-risk AI system’s output” for a single instance - a single decision to be made. the AI compliance officer will have to use domain knowledge, combined with knowledge on the model’s behaviour on previous instances and globally, to determine whether the AI model’s prediction is in line with desired behaviour. Depending on their judgement, they may roll back or amend a decision, request further investigation, or, in the worst case; press the big red stop button.

Interactive visualization

Interactive visualization is a key component in the analysis and understanding of models, enabling you to visually verify model predictions and investigate different model behaviours. We can look at the Anscombe's Quartet to illustrate the importance of visualization. This is a set of 4 data distributions that are all equal in key distribution indicators such as mean & standard deviation. However, when visualized in a scatter plot they are strikingly dissimilar. We’re visual creatures. Numeric values often seem arbitrary whereas geometry comes very natural to us. The Datasaurus Dozen, an extension to the Anscombe’s Quartet, really hammers home this point. In an act of self-affirmation, it provides an elegant visual proof that close to any desired shape of data can be achieved while retaining equality in key indicators.

As technology progresses and our AI models become more capable, their decision strategies are increasingly complex. The purpose of AI Explanations is to reduce this complexity to something we can more easily understand. Reduction is the key word. To capture the full nuance of model behaviour the AI compliance officer will need to apply an investigative approach. Using past experience in analyzing the model, combined with global and local explanations, the AI compliance officer is able to form hypotheses about the model’s behaviour. Exploring explanations for synthetic instances, he is then able to verify and, if nessecary, adjust his hypotheses. In summary, using explainable AI tools we are able to automatically generate annotations with model outputs, surpassing the obscure mode of simply using the outputs of a black box model. The secret sauce is added by combining visuality and interactivity. This allows you to move from mere annotation towards a more complete understanding of the model’s outputs.

Fully enabling the deployer to best interpret your AI system is more than just a regulatory consideration. Because of the ‘black box’ nature of AI models, having people from different backgrounds dissecting your model provides you with immensely valuable constructive feedback. Constructing barriers to human oversight is detrimental to the development of high quality products.

Strategic subgroups

Another important piece of high-level information is the model’s performance on certain subgroups. These are clusters of inputs that share common characteristics. Subgroups help us identify any vulnerable group potentially drawing on the short end of the stick. A model may underperform for a certain subgroup due to imbalances in training data. Model’s may also exhibit discriminatory or otherwise unwanted behaviour regarding certain subgroups due to the presence of statistical correlations which are not rooted in causation. Looking at employment data, we might find correlations between gender and wages, however, we do not want our model making decisions that propagate or even exacerbate this relationship, since it is not based on causality.

---

“[The instructions for use shall contain] when appropriate, its performance regarding specific persons or groups of persons on which the system is intended to be used.”

---

Defining these subgroups can be done by hand, using domain knowledge and data analyses to make an educated guess. This often results in groups based on a narrow definition, perhaps on the basis of only a single feature, like age groups. We call these data clusters, these subgroups appear as clusters when plotting the data in a scatterplot. During training, a machine learning model will compress and, as a result, organize the information it ingests. With Xaiva’s unique approach we are able to cluster data based on the strategy applied by the model, utilizing the organizing principles it learned during training. These model strategies have a much more nuanced definition, allowing for more effective clustering of the data. They are constructed in relation to the task used to train the model, reflecting how different groups correlate with the target variable - often called the label. Most importantly, using this method we’re able to focus on the subgroups that are meaningful for the AI model. Using Xaiva’s explainable AI tools, greatly enhance an organisation’s ability to identify strategic subgroups. This ensures all potentially disadvantaged groups can be tested for, allowing you to rest easy.

This is a screenshot of the Xaiva Analyze component of our platform. You can see the model distribution - constructed from feature importance values - is a lot cleaner, with more prominent clusters and less lonely specks of instances. We selected a clear group - a model strategy - within the model distribution, this same selection is reflected in the data distribution. This clearly shows the model strategy varies wildly from any data cluster we can define based on data features.

---

“The post-market monitoring system shall actively and systematically collect, document and analyse relevant data which may be provided by deployers or which may be collected through other sources on the performance of high-risk AI systems throughout their lifetime, and which allow the provider to evaluate the continuous compliance of AI systems”

---

The provider is responsible for implementing a post-market monitoring system. Due to natural factors as well as the AI system acting upon its environment, the AI system’s environment will change during its lifetime. In case the AI system’s parameters are frequently updated, this may even result in harmful feedback loops. By monitoring how strategic subgroups and the performance on these groups changes over time, the provider is able to evaluate the AI model’s impact in a changing environment and limit model drift.

Xaiva

The Xaiva platform consists of two main components; Xaiva Analyze and Xaiva Explain. After the initial setup is done, Xaiva only requires access to the AI model. The platform needs to be able to query the model for outputs on synthetic inputs. This makes Xaiva integration very flexible. Depending on your needs, it can be implemented on-premise or hosted by us.

Xaiva Analyze requires a minimal fingerprint comprised of basic statistical properties of each feature - mere metadata - and can then be used to provide global explanations and to identify important model strategies. Data never leaves your hands. Some features of Xaiva Analyze, like the screenshot showing the data distribution we saw earlier, are computed in the browser. Decision strategies can be further investigated for each of these model strategies, providing lots of context for local investigations done in Xaiva Explain. Here, you get a powerful search engine for the complex manifold the AI model has fit on the data it consumed. Taking the prediction on a single instance as starting point, the user can easily investigate the model’s changing behaviour around this point. Seeing the effect of changing inputs on the model’s prediction, as well as the importance of each unique feature. All of this is achieved in a no-code environment. Adjusting search parameters on the fly can be done with simple mouse drags. Hyperparameters for different explanation techniques are accessible via drop-downs menu’s, but can also be hidden to simplify the UI for less experienced users. These features really speed up the investigative process of forming & testing hypothesis.

---

“[In identifying the most appropriate risk management measures, it shall be ensured] provision of information required pursuant to Article 13: Transparency and Provision of Information to Deployers and, where appropriate, training to deployers. With a view to eliminating or reducing risks related to the use of the high-risk AI system, due consideration shall be given to the technical knowledge, experience, education, the training to be expected by the deployer, and the presumable context in which the system is intended to be used.”

---

Accessibility of the Xaiva platform reduces the stress for a high level of “Technical knowledge, experience and education” required for quality human oversight, as well as a reduction in the necessary amount of “training to be expected by the deployer”. With the right tooling, forming accurate mental models - truthful to the nuance of our AI’s complex decision strategy - can be achieved outside the realm of trained data scientists.

Whether the deployers performing human oversight are a separate party or part of the same company, deployer access to explainer techniques as well as the AI model itself can be structured in several ways. In most cases the weights of the AI model are the product, providers will therefor often prefer to keep these private. Xaiva can be implemented according to whichever structure you prefer.

• It may be fine to share the entire AI model with the deployer, enabling the deployer to use the human oversight measures you are obligated to provide, as well as any other method or tool they see fit.
• You could grant access to the Xaiva platform itself, giving deployers the necessary space to investigate model behaviour, but keeping the model weights and datasets private.
• You can use Xaiva to generate a report providing information on the model’s behaviour globally, as well as automate provision of local explanation reports annotating model outputs.

Seemingly harmless algorithms may have huge consequences due to the sheer number of individuals impacted by its decisions. Granting more access to deployers to conduct human oversight will result in a higher chance of risks and other anomalies being discovered. This may sound scary, but it may also be the surest way to train AI models with competitive performance, that are resilient to outliers and adapt to changing environments.

We can help

Are you interested in exploring the Xaiva platform and understanding how it can enable human oversight to help you deploy AI in a more trusted manner? Reach out to contact@xaiva.io and we would be happy to schedule a 1:1 introduction. Want to expand your knowledge on the upcoming AI act? Consider reading our concise high-level overview of the AI act, soon to be published. In need of a deeper dive? We recommend the EU Artificial Intelligence Act Deep Dive by Deloitte. For highly specific questions, we recommend consulting the AI Act itself.