Guidance on Artificial Intelligence and Machine Learning Models in our TRE

Owned by Laura Ward
Last updated: Jul 31, 2024
4 min read

Introduction

Our TRE is operated within the Five Safes Framework and as such, we use statistical disclosure control practices to eliminate the risk of disclosing information when releasing information from the TRE (to ensure ‘Safe Outputs’). All TRE outputs undergo HIC disclosure control which ensures that confidential data is not disclosed accidentally. 

Background

Artificial Intelligence/ Machine Learning (AI/ML) models fundamentally change how we think of TRE disclosure control. As TRE providers, we need to understand more about projects from Users, ideally at the project initiation stage (i.e., the beginning). Likewise, Users need to understand more about our disclosure control process. Rather than looking for definite proof of individual level data as we would in traditional statistical disclosure control, we instead assess what the risk is of there being individual level data in the model. This document is to help researchers to understand the processes involved in AI/ML.

When using AI/ML models on sensitive health data, a researcher should consider the following points:

Implication of AI/ML models in your project plan

Statistical learning models such as AI/ML use algorithms that learn patterns from known training data. The patterns are often uninterpretable by humans. These can then be applied to new, unknown data to infer an outcome of the AI/ML model (or algorithms). With confidential data, because these patterns are uninterpretable by humans it raises additional challenges for TRE providers to assess the disclosure risks.

  1. We will work with you to inform assumptions made in the project quotation, requirements specification, etc. If the answer changes part-way through the project, then the project governance may need to be re-assessed, which may lead to additional resources and costs.

Disclosure risks associated with trained AI/ML models

Having awareness of disclosure risks associated with AI/ML models can help to ensure AI/ML models are developed or used with safe parameters in place to reduce and mitigate disclosive risks.

It may be helpful to list any approaches considered to reduce disclosure risk that AI/ML model(s) can pose and ensure best practise is applied (e.g. over-fitting detection). The Information Commisioner’s Officer https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/ai-and-data-protection-risk-toolkit/ is designed to raise awareness of the risks to individuals’ rights and freedoms caused by AI systems and to provide practical ideas for reducing them at each stage of the development and implementation lifecycle.

Do you intend to bring trained an AI/ML model into the TRE?

If so, AI/ML models trained on sensitive data need to be assessed to determine whether the model is likely to be safe in our TRE. Much like software tools, we will check any files for viruses or other malicious intent, but we must also be satisfied that data brought into the environment is allowed to be there. This may be evidenced by existing agreements, by showing the model does not contain individual level data, or by a formal statement that this is the case.

Do you intend to remove your AI/ML model from the TRE?

AI/ML models trained on sensitive data have the potential to be over trained and require careful checking to determine the likelihood of them being disclosive. We will use tools to ‘interrogate’ and ‘attack’ your AI/ML model to quantify how disclosive it may be. These tools will be used as part of an assessment to help ensure your model poses no risk and is safe to be released. It may not always be possible to guarantee that your model will be released.

Consider your training data

To improve efficiency, maintaining a description of the training data used, and how it was prepared for the AI/ML model is helpful. This will help us to conduct our disclosure checks, and help to identify mitigations to the risk of the model being disclosive. This could include analysis scripts explaining each step of the process (including any transformations and any feature selection processes), documented assumptions that have been made about the variables, aggregated features and if aggregation is able to be reverse engineered, and what data wrangling and analysis steps were undertaken. This could be presented as human-readable code that is well documented and explains hyperparameters and what the settings you have used does.

Safe outputs for trained AI/ML models

A well-documented AI/ML analysis pipeline will help us understand the process better and enable disclosure control checks more swiftly. We will attempt to assess how likely the model is safe to output based on the model type and the parameters used in training. Different models pose different risks. Knowing ahead of time helps highlight potential risks and identify suitable mitigations early.

We may run ‘membership inference attacks’ to determine whether a record was used as part of the training dataset of a trained AI/ML model or not. That is, whether a specific individuals' data may br present to an attacker, i.e. sensitive data is ‘leaked’ and this is considered disclosive. Most AI/ML models output stronger confidence scores when they are fed with their training dataset examples, as opposed to new and unseen examples. When we apply disclosure control mechanisms, we may use various tools and processes that are developed for testing disclosure risks (e.g., GRAIMATTER). Note that we will only use your model in the TRE to check for disclosive risks and data security, nothing else.

Contractual element (external to UoD/NHS Tayside)

We operate as part of the University of Dundee, and may require work between our legal department and the TRE User Organisation to agree data sharing or other contracts. Appropriate questions or language should be incorporated at the contractual stage. This is to protect both the TRE host organisation and the development or research organisation and protect intellectual property.

 

We reserve the right to not permit the AI/ML model to be released from the TRE

Consequences

There are various penalties potentially imposed for breaches in the data security, which are undertaken at our discretion. We prioritise prevention than statutory penalities, but may seek prosecution for any breaches in line with the outlined consequences in our TRE User Agreement.

Other resources

 Related articles