Copilot Vision on Windows 11 sends data to Microsoft servers • The Register
Microsoft is again throwing AI at Windows 11 to see what sticks, releasing features including the even more eyebrow-raising successor to its controversial Recall, a screen-streaming remotely processed backseat driver dubbed Copilot Vision.
“AI is changing the way we use our PCs,” Microsoft claims, and while some may agree, not all would say it’s a change for the better.
“Windows 11 is the home for AI,” it adds, “offering the most expansive and capable AI experiences for consumers today on Copilot+ PCs – with exclusive AI superpowers like Recall (preview), Click to Do (preview) and improved Windows search, as well as groundbreaking new ways to interact with your PC, like Copilot Vision on Windows.”
Copilot Vision is an extension of Microsoft’s divisive Recall, a feature initially sort of exclusive to the Copilot+ systems with a neural co-processor of sufficient computational power. Like Recall, which was pulled due to serious security failings and subject to a lengthy delay before its eventual relaunch, Copilot Vision is designed to analyze everything you do on your computer.
It does this, when enabled, by capturing constant screenshots and feeding them to an optical character recognition system and a large language model for analysis – but where Recall works locally, Copilot Vision sends the data off to Microsoft servers.
According to a Microsoft spokesperson back in April, users’ data will not be stored long-term, aside from transcripts of the conversation with the Copilot assistant itself, and “are not used for model training or ads personalisation.”
Microsoft’s vision for Vision is, in the words of the Copilot team, to eventually become “a true companion,” offering “a deeper understanding of your goals and the ability to provide clear, step-by-step guidance to help you accomplish them.”
While the screen snooping only happens when the user expressly activates it as part of a Copilot session, unlike Recall, which is constantly active in the background when enabled, it’s also designed to be more proactive than previous releases – which, for many readers, will conjure memories of Clippy and his cohort of animated assistants from the days of Microsoft Office 97 and onward.
At the time of writing, Microsoft was only offering Copilot Vision in the US, with the promise (or threat) that it will be coming to very specifically “non-European countries” soon – a tip of the hat, it seems, to the European Union’s AI Act. That doesn’t mean those outside its borders escape entirely, however. The company’s latest update to Windows 11, the only mainstream version of its operating system still in its official support lifespan, brings a selection of other AI features, including the operating system’s first “agentic” AI, which is now available to fiddle with system settings on your behalf.
Based on a local language model dubbed Mu, and only available to Copilot+ systems running on Qualcomm Snapdragon hardware with Intel and AMD support to follow, the agent promises to take action on the user’s behalf. Instead of simply searching for where to change screen resolution or connect a Bluetooth device, as in previous releases, the agent accepts natural language instructions – like “connect to my Bluetooth device” or “change my screen resolution to 1920 x 1080” – and offers to carry out the task itself, providing it knows how.
“Our goal was to create an AI-powered agent within Settings that understands natural language and changes relevant undoable settings seamlessly,” explained Vivek Pradeep, Microsoft engineer and vice president of Windows Applied Sciences, at the time of Mu’s beta release. “We aimed to integrate this agent into the existing search box for a smooth user experience, requiring ultra-low latency for numerous possible settings.”
“Managing the extensive array of Windows settings posed its own challenges,” Pradeep admitted, “particularly with overlapping functionalities. For instance, even a simple query like ‘Increase brightness’ could refer to multiple settings changes – if a user has dual monitors, does that mean increasing brightness to the primary monitor or a secondary monitor? To address this, we refined our training data to prioritise the most used settings as we continue to refine the experience for more complex tasks.”
What Microsoft has not stated is how, with a model specifically tailored to be small enough to run on-device, it is addressing the unresolved “hallucination” problem inherent to language models – the issue where, in the process of converting a user’s prompt to a stream of tokens and returning the most statistically likely tokens as a continuation, the tokens output by the model turn into an answer-shaped object that is, unfortunately, entirely detached from reality. For chatbot-style LLM implementations, that means a wrong answer; for agentic AI, which takes action itself, it can spell disaster, as a user of vibe coding platform Replit recently discovered to his dismay.
Microsoft has also released new actions for its “Click to Do,” available as a preview in countries outside the European Economic Area, which can listen to the user read passages aloud to improve their reading skills, use Copilot’s generative-AI capabilities in Microsoft Word with any selected text as a prompt, activate an “Immersive Reader” mode with adjustable visuals, text-to-speech, syllable-break functionality, and a picture dictionary, and to trigger messages and schedule requests in Microsoft Teams.
The company’s AI push doesn’t stop there, though. It has also added an AI-powered “Relight” feature to the Photos app, designed to simulate the effect of up to three virtual light sources added to an existing photograph. The once simple mouse-teaching tool Paint, meanwhile, gets its own AI update with a “sticker generator” to turn textual prompts into cartoonish imagery and “object select,” which aims to make it easier to select individual elements in a busy image. Even the Snipping Tool, which took over from simple screenshots, hasn’t escaped, gaining a Copilot+ exclusive “perfect screenshot” feature that resizes the capture area automatically based on screen content – plus a color picker tool, which arguably doesn’t require any machine learning at all.
The latest Windows 11 update isn’t all about AI, though given Microsoft’s increasingly desperate focus, you’d be forgiven for thinking otherwise. It also brings about the end of the classic Blue Screen of Death (BSoD), a familiar sight going all the way back to the original Windows 1.0 – with a brief switch to black in Windows 3.0, and Windows NT 3.1 being the first to have a “true” BSoD that displays critical system errors on a blue background. Now the BSoD is a Black Screen of Death, which at least retains the classic acronym.
The new BSoD represents more than just a fresh coat of digital paint. Microsoft says the new error screens will be less verbose and more readable, while the company has also added a system dubbed “quick machine recovery,” which replaces the earlier “Startup Repair.”
When the feature released as a beta earlier this year, Microsoft’s Riddhi Ameser wrote: “With quick machine recovery, when a widespread outage affects devices from starting properly, Microsoft can broadly deploy targeted remediations to affected devices via Windows RE Recovery Environment – automating fixes and quickly getting users to a productive state without requiring complex manual intervention.”
It may be a while before all these features trickle down to your desktop, though. Microsoft is making them available through its “controlled feature rollout” system – designed to prevent widespread outages by sending a buggy update to every user at once – over the course of the next month.
Those eager, for whatever reason, to jump on board are advised to enable the “Get the latest updates as soon as they’re available” checkbox in Windows Update; those who would rather not have more AI bloat in their operating system are given little option other than to look outside Microsoft’s offerings.
Microsoft unveils new Surface Laptop 5G
At the same time as pushing more AI at Windows users, Microsoft showed off new hardware in the form of the Surface Laptop 5G – featuring, of course, Copilot+ support.
Built around a 13.8″ display, larger than the company’s refreshed 13″ Surface Laptop and 12″ Surface Pro, the Surface Laptop 5G comes with a choice of Intel Core Ultra (Series 2) processors, each with a 40 tera-operations per second (TOPS) neural co-processor for local AI models, and an optional built-in 5G cellular modem – “available later in 2025” – which, the company warns, means “users stay continuously connected to Microsoft 365 Copilot and other cloud tools.”
More information is available on this Microsoft website. ®
