Home / News & Analysis / Audit of NHS Trust’s app project with DeepMind raises more questions than it answers

Audit of NHS Trust’s app project with DeepMind raises more questions than it answers

A third party audit of a controversial patient data-sharing arrangement between a London NHS Trust and Google DeepMind appears to have skirted over the core issues that generated the controversy in the first place.

The audit (full report here) — conducted by law firm Linklaters — of the Royal Free NHS Foundation Trust’s acute kidney injury detection app system, Streams, which was co-developed with Google-DeepMind (using an existing NHS algorithm for early detection of the condition), does not examine the problematic 2015 information-sharing agreement inked between the pair which allowed data to start flowing.

“This Report contains an assessment of the data protection and confidentiality issues associated with the data protection arrangements between the Royal Free and DeepMind . It is limited to the current use of Streams, and any further development, functional testing or clinical testing, that is either planned or in progress. It is not a historical review,” writes Linklaters, adding that: “It includes consideration as to whether the transparency, fair processing, proportionality and information sharing concerns outlined in the Undertakings are being met.”

Yet it was the original 2015 contract that triggered the controversy, after it was obtained and published by New Scientist, with the wide-ranging document raising questions over the broad scope of the data transfer; the legal bases for patients information to be shared; and leading to questions over whether regulatory processes intended to safeguard patients and patient data had been sidelined by the two main parties involved in the project.

In November 2016 the pair scrapped and replaced the initial five-year contract with a different one — which put in place additional information governance steps.

They also went on to roll out the Streams app for use on patients in multiple NHS hospitals — despite the UK’s data protection regulator, the ICO, having instigated an investigation into the original data-sharing arrangement.

And just over a year ago the ICO concluded that the Royal Free NHS Foundation Trust had failed to comply with Data Protection Law in its dealings with Google’s DeepMind.

The audit of the Streams project was a requirement of the ICO.

Though, notably, the regulator has not endorsed Linklaters report. On the contrary, it warns that it’s seeking legal advice and could take further action.

In a statement on its website, the ICO’s deputy commissioner for policy, Steve Wood, writes: “We cannot endorse a report from a third party audit but we have provided feedback to the Royal Free. We also reserve our position in relation to their position on medical confidentiality and the equitable duty of confidence. We are seeking legal advice on this issue and may require further action.”

In a section of the report listing exclusions, Linklaters confirms the audit does not consider: “The data protection and confidentiality issues associated with the processing of personal data about the clinicians at the Royal Free using the Streams App.”

So essentially the core controversy, related to the legal basis for the Royal Free to pass personally identifiable information on 1.6M patients to DeepMind when the app was being developed, and without people’s knowledge or consent, is going unaddressed here.

And Wood’s statement pointedly reiterates that the ICO’s investigation “found a number of shortcomings in the way patient records were shared for this trial”.

“[P]art of the undertaking committed Royal Free to commission a third party audit. They have now done this and shared the results with the ICO. What’s important now is that they use the findings to address the compliance issues addressed in the audit swiftly and robustly. We’ll be continuing to liaise with them in the coming months to ensure this is happening,” he adds.

“It’s important that other NHS Trusts considering using similar new technologies pay regard to the recommendations we gave to Royal Free, and ensure data protection risks are fully addressed using a Data Protection Impact Assessment before deployment.”

While the report is something of a frustration, given the glaring historical omissions, it does raise some points of interest — including suggesting that the Royal Free should probably scrap a Memorandum of Understanding it also inked with DeepMind, in which the pair set out their ambition to apply AI to NHS data.

This is recommended because the pair have apparently abandoned their AI research plans.

On this Linklaters writes: “DeepMind has informed us that they have abandoned their potential research project into the use of AI to develop better algorithms, and their processing is limited to execution of the NHS AKI algorithm… In addition, the majority of the provisions in the Memorandum of Understanding are non-binding. The limited provisions that are binding are superseded by the Services Agreement and the Information Processing Agreement discussed above, hence we think the Memorandum of Understanding has very limited relevance to Streams. We recommend that the Royal Free considers if the Memorandum of Understanding continues to be relevant to its relationship with DeepMind and, if it is not relevant, terminates that agreement.”

In another section, discussing the NHS algorithm that underpins the Streams app, the law firm also points out that DeepMind’s role in the project is little more than helping provide a glorified app wrapper (on the app design front the project also utilized UK app studio, ustwo, so DeepMind can’t claim app design credit either).

“Without intending any disrespect to DeepMind, we do not think the concepts underpinning Streams are particularly ground-breaking. It does not, by any measure, involve artificial intelligence or machine learning or other advanced technology. The benefits of the Streams App instead come from a very well-designed and user-friendly interface, backed up by solid infrastructure and data management that provides AKI alerts and contextual clinical information in a reliable, timely and secure manner,” Linklaters writes.

What DeepMind did bring to the project, and to its other NHS collaborations, is money and resources — providing its development resources free for the NHS at the point of use, and stating (when asked about its business model) that it would determine how much to charge the NHS for these app ‘innovations’ later.

Yet the commercial services the tech giant is providing to what are public sector organizations do not appear to have been put out to open tender.

Also notably excluded in the Linklaters’ audit: Any scrutiny of the project vis-a-vis competition law, public procurement law compliance with procurement rules, and any concerns relating to possible anticompetitive behavior.

The report does highlight one potentially problematic data retention issue for the current deployment of Streams, saying there is “currently no retention period for patient information on Streams” — meaning there is no process for deleting a patient’s medical history once it reaches a certain age.

“This means the information on Streams currently dates back eight years,” it notes, suggesting the Royal Free should probably set an upper age limit on the age of information contained in the system.

While Linklaters largely glosses over the chequered origins of the Streams project, the law firm does make a point of agreeing with the ICO that the original privacy impact assessment for the project “should have been completed in a more timely manner”.

It also describes it as “relatively thin given the scale of the project”.

Giving its response to the audit, health data privacy advocacy group MedConfidential — an early critic of the DeepMind data-sharing arrangement — is roundly unimpressed, writing: “The biggest question raised by the Information Commissioner and the National Data Guardian appears to be missing — instead, the report excludes a “historical review of issues arising prior to the date of our appointment”.

“The report claims the ‘vital interests’ (i.e. remaining alive) of patients is justification to protect against an “event [that] might only occur in the future or not occur at all”… The only ‘vital interest’ protected here is Google’s, and its desire to hoard medical records it was told were unlawfully collected. The vital interests of a hypothetical patient are not vital interests of an actual data subject (and the GDPR tests are demonstrably unmet).

“The ICO and NDG asked the Royal Free to justify the collection of 1.6 million patient records, and this legal opinion explicitly provides no answer to that question.”

Read more

Check Also

The Reality Ecosystem: What AR/VR/XR needs to go big

Tim Merel Contributor Tim Merel is managing director of Digi-Capital. More posts by this contributor China could beat America in AR/VR long-term Ubiquitous AR to dominate focused VR by 2022 AR/VR/XR are going to be big, but not everyone will win. Today’s market is a collection of related point solutions to specific problems, but not a fully functioning ecosystem. Not yet. For the market to begin to challenge incumbent platforms (particularly mobile), great tech alone is not enough. AR/VR/XR needs its own Reality Ecosystem. And there are a lot of pieces to the puzzle. Source: Digi-Capital (Note: This discussion focuses on consumer, not enterprise, AR/VR/XR markets) Active users For platforms to be platforms, they need active users. Lots of them. Table stakes are tens of millions; hundreds of millions are better, but billions are the ultimate goal. Today we’re all active users of a platform with four and a half billion active users — mobile. Early indications are positive for mobile AR, with Pokémon GO (tens of millions), Snapchat (hundreds of millions) and WeChat (billions) showing what is possible. Google’s Maps/Lens combo demoed at Google I/O also has potential. But it will still take multiple breakouts across the 20-plus mobile AR app categories to leverage early successes into a true platform. VR has different user dynamics, partly because of a lack of plurality, but also due to relatively limited scale and user attrition. One of the challenges for VR is a primary entertainment focus (games, video), which can be done more easily and cheaply on existing devices (even though they can’t rival it for immersion). Also, the social side of VR hasn’t really scaled so far — AltspaceVR was one of the biggest VR social apps last year with 35,000 monthly active users (“MAU”) when acquired by Microsoft after reporting it was closing. So while there is an active core VR user base, some casual users haven’t stuck with the platform for long enough. If Apple launches smartphone-tethered smart glasses as an iPhone peripheral (we’ve forecast 2020 for a few years now), we’ll get a better idea of what smart glasses daily active users (“DAU”) could look like. But Snap Spectacles (not smart glasses) and Google Glass highlight that cool tech goggles can end up in the closet. Magic Leap’s recent launch has also received mixed reviews. Smart glasses need to sort this out if they hope to scale. High-frequency users The most important economic lesson from mobile is “Frequency ∝ Revenue” (“∝“ means “proportional to”). In other words, high user frequency = money. For example, the top 1 percent grossing mobile apps deliver 35x the sessions per day of the top 5 percent apps. And going back to active users, lifetime value of the top 1 percent grossing apps is 20x that of the the top 5 percent. While it’s obvious, you need to hold on to users and give them something to do to make money. While there are big differences between AR/VR/XR and mobile, this remains a crucial dynamic. Mobile AR has shown what is possible, with Pokémon GO, Snapchat and soon Google Maps/Lens again standouts. While each has a different approach to user engagement, usage frequency is high. It’s part of why they’re so valuable. Some VR headsets get used less than once a day, with a significant proportion every few days, weekly or even monthly. Our User Strategy team’s product/market fit reviews for startups have shown this dynamic even when users love their VR apps. The words “evenings,” “weekends” and “holidays” come up, particularly for under-34 Snapchat demographic users. Not ideal for frequency. Again, with smart glasses it is too early to tell, but app developers might need a mental model closer to mobile than enterprise to get frequency to work. Lightweight, short-duration apps that are opened tens to hundreds of times a day could keep smart glasses on peoples’ faces when they’re ready for prime time. No pressure there, then. Critical use cases We think about use cases for new technology platforms in terms of valuable versus critical. Valuable use cases might be cool and technically hard to do, but either don’t fundamentally transform user experience or aren’t important to users. Critical use cases enable lots of users to do something they really care about, and that couldn’t be done in any other way. Critical is interesting, valuable not so much. Critical use cases for mobile AR are beginning to emerge, with perhaps the first being Google’s Maps/Lens combination revealed at Google I/O 2018. It solves a universal problem when you come out of Embarcadero Station and are told to go south — but where’s south? Google combined computer vision with mobile AR to show you exactly where to go, and even gave you a cute fox to lead you there. This produced a visceral response in the audience at the I/O Keynote, because it solves a problem we all share. And it couldn’t be done in any other way. VR is valuable — just ask Palmer Luckey. It’s also cool, technically hard to do and can take you to other worlds. But critical? Again, VR’s entertainment focus effectively makes it a subset of the games market, as well as other use cases, such as enterprise training. But critical use cases don’t appear to have emerged three years into the market. For comparison, Uber launched three years after the iPhone. As smart glasses are largely enterprise focused today, again it’s early for critical consumer use cases. The first could evolve from mobile AR, but they are more likely to come from native smart glasses use cases that only work for that form factor. Critical apps On average, Americans tend to use nine mobile apps per day, and 30 per month. Most download zero apps per month. This means critical use cases are not enough. They need to be features of critical apps we already use all the time, or something so insanely great that we might actually download it. This dynamic could be mobile AR’s secret weapon, with mundane use cases embodied in ubiquitous apps the possible winners. Again, think Google Maps (soon Apple Maps, too) or Snapchat. Outliers like Pokémon GO come from specific sets of circumstances and are hard to duplicate. There’s a reason why breakout hits are rare in mobile (not just mobile AR). Houzz proved mobile AR apps can drive an extraordinary 11x sales uplift for e-commerce, but again, this is a feature of an already successful app. Current mobile leaders could determine how mobile AR evolves even more than startup insurgents. The challenges for critical VR use cases apply to critical VR apps too. It’s hard to describe a VR app most people couldn’t live without, even if you love Beat Saber. It’s too early to tell with smart glasses again, but their critical use cases might need to be more than ports from breakout mobile AR successes. Cloud/data Many people in the industry are excited about AR Cloud, a persistent 3D real-world data layer for shared AR apps. It could become a key enabler for the Reality Ecosystem for both mobile AR and smart glasses. But discussions with AR Cloud startup CEOs indicate that critical use cases and monetization remain open questions for some. Google, Niantic (and again, Apple) have it figured out, but startups in the space must avoid phase 2 of the Underpants Gnomes’ business model. Google and Uber are proof that platform economics can take time to develop, so the excitement may yet be warranted. Blockchain has been said to be VR’s cloud. While this could be the future, it needs to avoid what Steve Wozniak and others have called the “Blockchain Bubble.” Source: Digi-Capital Data/analytics to support the industry have largely come from industry reports so far (including our own). But AR/VR/XR tools equivalent to Bloomberg, AppAnnie or PitchBook hadn’t emerged before Digi-Capital’s Analytics Platform was first seen at Google in May (before launching in July). Piper Jaffray Apple analyst (now Loup Ventures managing partner) Gene Munster describes it as “Bloomberg for AR/VR/XR” — so hopefully that helps fill the vacuum. Installed base For the Reality Ecosystem to succeed, massive installed bases for underlying hardware and software platforms are required. While this does not guarantee users downloading or using AR/VR/XR apps, without them, there’s little chance of success. Mobile AR is the front-runner, as Apple ARKit, Google ARCore and Facebook Camera Effects could deliver more than 900 million installed base by the end of this year, and approach three and a half billion by 2022. So while mobile AR has a lot of other challenges to solve (not least UI/UX), installed base appears to be a done deal. VR (mobile, console/PC, standalone) could reach 50 to 60 million installed base in five years’ time, but again, looks more like a subset of the games market than anything else. Smart glasses could produce tens of millions installed base by 2022 (again, if and when Apple enters), and result in a combined AR/VR headset installed base in the high tens of millions to more than 100 million in five years (or around 3 percent of mobile AR). Critical hardware Today’s critical hardware is the smartphone. It’s the first, most frequent and last thing most of us look at every day. So mobile AR has critical hardware already. And if we’re right about Apple rolling out TrueDepth sensors beyond the iPhone X, functionality could get better for a broader set of users. VR hardware is valuable, but its usage patterns don’t make it look critical yet. Again, as a subset of the games market, it seems to have found a deep niche audience without going mass-market. Smart glasses need an Apple-quality device (whether made by Apple or somebody else) to be critical. They could start as mobile peripherals, but a device capable of replacing your phone might be what’s needed. However there are major technical and packaging issues to solve first, so this could take a few years. Corporate and VC investment The Reality Ecosystem could depend more on internal corporate investment than startups raising cash from VCs. For example, Apple’s investment over the last four years to build its AR Cloud for Apple Maps (and a range of other potential applications) could be greater than the largest VC investments. While attention has been focused on monster rounds for Magic Leap and others, the corporate world could prove more important. However, VC investment will always be a major driver because it enables black swans like Google, Facebook, Uber, Tencent and Alibaba to emerge. They would not have reached escape velocity without great VCs bankrolling them, and the Reality Ecosystem should be no different. Talking with 30 leading VCs in Sand Hill Road and China showed a mental model geared toward mobile AR and computer vision in the near-term, and smart glasses in the long-term. VCs appear to be far less focused on VR. While AR/VR/XR saw more than $1.5 billion investment in Q2 2018 and more than $5 billion in the last 12 months, the shape of the VC market has changed. The shift to China (which could win AR/VR/XR long-term) with larger, later-stage deals, was twinned with fewer early-stage deals in the U.S. in Q2 2018. It will be interesting to see how this plays out through the year. If you build it, they will come… but who will build it? Ubiquitous AR (mobile AR, smart glasses) could drive $85 billion to $90 billion revenue by 2022, dominating focused VR (mobile, console/PC, standalone) with $10 billion to $15 billion in the same time frame. But this won’t happen without a robust Reality Ecosystem to support it. So who will build it? Digi-Capital tracks thousands AR/VR/XR leaders across mobile AR (gathering momentum, but more traction needed), smart glasses (too early to tell) and VR (still all to prove, three years in). While there are extraordinary startups like Niantic, some of the smart money is on incumbent platforms like Apple, Google, Facebook, Tencent, Alibaba and Amazon to form the bedrock of what is yet to come. None has emerged as the Reality Ecosystem’s true champion yet, so it’s all to play for. That said, Apple’s full-stack ecosystem, Google’s AR Cloud, Facebook’s social scale and Alibaba’s Online-to-Offline (“O2O”) dominance position them as strong contenders. There’s clear blue ocean between here and the end-game for the Reality Ecosystem, and a massive amount of work to be done. A fair wind and following seas could help, but luck and determination might turn out to be far more important.

Leave a Reply

Your email address will not be published. Required fields are marked *

Disclaimer: Trading in bitcoins or other digital currencies carries a high level of risk and can result in the total loss of the invested capital. theonlinetech.org does not provide investment advice, but only reflects its own opinion. Please ensure that if you trade or invest in bitcoins or other digital currencies (for example, investing in cloud mining services) you fully understand the risks involved! Please also note that some external links are affiliate links.