Have you experienced that the mobile phone listens to you and adapts the advertising to what you say? You’re not alone. The rumor has circulated for several years – but no one has yet managed to prove that it really is so.
But we know that the mobile phone listens to us. Or? The knowledge and willingness to adapt advertising to what you say near your phone has been around for a long time.
Already ten years ago, the company Amazon Technologies submitted a patent application for a technology that can listen for keywords* and adapt advertising to them. The company described the technology with an example: two people talk about what they have done on vacation, after which they receive advertising based on what they have talked about.
* Keywords are related to various topics or activities that the eavesdropper is interested in tracking or controlling.
The mobile phone records sound without a microphone
Three years later, Facebook is applying for a patent to use the mobile phone’s sensors to gather information about the user – in order to then be able to adapt its advertising. In the same year, the University of California also applies for a patent that points out one of the sensors: a so-called accelerometer, which can be used to listen for keywords.
Anyone who wants to eavesdrop on what is being said around a mobile phone does not have to do it via the mobile phone’s microphone but can do it via two other sensors, in addition to the accelerometer there is something called a gyroscope.
An accelerometer keeps track of which way the mobile is held to make the screen rotate correctly. It is often used in combination with the gyroscope, which can detect if the mobile is rotated around itself without being tilted. Signals from these two sensors can, at least in theory, be translated into text to, for example, recognize key words spoken by people near the phone, according to research.
- Location data: GPS sensors can track your location and movement patterns over time.
- Motion data: Accelerometers can measure how the device accelerates, moves, and vibrates, which can reveal the user’s physical activity or interaction with the device.
- Light conditions: Light sensors can measure the light level in the surroundings, something that the mobile uses to adjust the brightness of the screen or to understand the user’s environment.
- Proximity data: Proximity sensors can determine if the user is holding the device to their face during a call or how close an object is.
- Orientation data: Gyroscope can measure the rotation and orientation of the device, which can be used to reproduce the direction or orientation of the mobile in the room.
- Audio recording: Microphones can record audio from the user’s environment or to use voice commands.
But the fact that the desire and knowledge to eavesdrop on people near a mobile phone exists does not necessarily mean that it actually happens. Many people are convinced that this happens, usually because they receive an advertisement for something they recently talked about near the mobile phone.
Are apps secretly listening?
Myself* the other day I was in the lunch room when a company I had never heard of was mentioned. The next day, an advertisement from the company appeared in my Instagram feed. Suspected? Maybe not. A colleague who found himself in the same position as me has a son who works at the mentioned company. The position data in our mobiles showed that we were in the same place. Was it the connection to the son, via the colleague, that caused the advertisement to appear? Or was it because the mobile eavesdropped and “heard” the company’s name being mentioned? There is also the possibility that it was pure coincidence.
*Johan Frisk, article author
Stefan Alfredsson, senior lecturer in computer science at Karlstad University, who researches mobile broadband networks, has also experienced suspicious connections between calls and advertising.
– But if someone could prove it, it would be a sensation. Apps are subject to a lot of scrutiny, including by colleagues of mine who have researched health apps. We have not seen any examples of the apps making active eavesdropping attempts.
Spy on the mobile without a single click
So there is no evidence that installed apps are eavesdropping. But we know that mobile phones have been tapped.
The most famous example is the spy program Pegasus. which an Israeli company sold to a number of states, which planted it with journalists and others they wanted to wiretap.
Short facts: What is zero-click technology?
Pegasus spyware is known for its zero-click attacks, which means it installs on a device without the user having to click on any links or open any messages. The attacker instead exploits vulnerabilities in the mobile’s operating system or apps. But that’s not the kind of interception that’s accused of providing the basis for targeted advertising.
How eavesdropping is done – a trade secret
At the end of last year, the marketing company published CMG Local Solutions a blog post where they talked about technology for active listening. They claimed that they could intercept calls made via mobile phones and use this information to tailor marketing to companies.
The method involves identifying the keywords that a company is interested in, and then using AI to identify the words in conversations that are conducted near a mobile phone. When the post was noticed, it was removed. But in late January, the company published another post, this time about marketing for car dealerships. This is how they described the service:
Active listening technology picks up on the throwaway comments people say around their smart devices, deploying your ads straight to the very screens they’re spending their time on.
(Active listening technology captures what people say near their mobile, and places it in your ads that are sent directly to the screens where they spend the most time)
Exactly how this will be done is unclear, as well as whether the service has been used. The question is also whether methods to eavesdrop on people’s can really be used without the mobile owners noticing.
Check out what the apps have to offer
– To do this type of analysis, you either need to process the information on the device, which requires computing power, or you need to send a continuous audio stream to a server that does a text recognition. It would be discovered quite quickly because the data quota runs out or because the battery runs out faster, says Stefan Alfredsson.
Eavesdropping can also be detected by checking what the apps are up to.
– If we take the iPhone, there is an integrity report that the owner can activate. It logs which apps have access to what and which network activities they have access to. So there are good opportunities for the user to control what is happening in the phone.
Companies can also use an algorithm that can detect certain trigger words, says Stefan Alfredsson
– But again, it requires the microphone to be on all the time and collect the sound and to analyze the sound stream. Maybe you can analyze shorter sequences, but then you miss a lot.
Is the mobile eavesdropping? Probably not…
If the mobile does not listen to what you say:
Why are you getting ads in the mobile feed for something you talked about?
One explanation is the enormous amount of information that can be collected about you and that can be linked to phones and places you’ve been near.
In a conversation with a friend, one of you may have mentioned fly fishing, something you never otherwise talk about. When you then receive advertisements for flies in the mobile feed, you connect it to the conversation: The mobile must have intercepted the conversation. But in fact, the advertisement may be due to the fact that half an hour later you met another friend. She has a great interest in fly fishing, something you don’t talk about, but since the position data in your mobiles locates you to the same place, the advertiser “guess” that you are interested in the same thing. But how does it know your friend is interested in fly fishing?
Digital Footprints: When you interact online – visiting websites, shopping online, joining an interest group on social media, etc – companies can collect and analyze that data – then, for example, target advertising that seems to match your interests to your mobile feed.
Tracking Pixel: A small, invisible graphic is placed on a web page (or in an email). When you open the page or email, it is loaded from a remote server and sends information about you back to a third party, usually an advertiser or an analytics platform. It can be data about what you do on the site, your IP address or when you are there and what device you use.
– It is a continuous work of fine-tuning. What is new now is the advanced data analysis that can be done with the help of AI, which connects, for example, browsing history and purchase history. Anyone who sees you shopping for diapers can not only figure out that you are a parent, but also that in ten years you will have a ten-year-old, says Stefan Alfredsson.
Eavesdropping cannot be ruled out, say researchers
A couple of years ago, two researchers at the Technical University of Berlin published a study entitled: Is My Phone Listening in? After going through the arguments for and against, they concluded that it cannot be ruled out that a large-scale interception could take place without detection.
– As a researcher, you want some braces and a waist belt. We haven’t found anything to indicate it but can’t say it doesn’t exist. But there is a lot of active research into cyber security and so far no one has found proof.
This is how you protect yourself against eavesdropping
-
1
Anyone concerned about apps eavesdropping through the microphone can go into settings and turn off microphone access for the suspect apps.
-
2
Anyone who wants to be really sure that the phone is not listening, for example via one of the other sensors or with the help of an implanted program, simply has to stay at an appropriate distance from the phone.
Text: Johan Frisk on behalf of forskning.se
