Teaching old microphones new tricks

Sensor technology: Microphones are designed to capture sound. But they turn out to be able to capture other sorts of information, too.

Microphones exist in many shapes and sizes, and work in many different ways. In the late 19th century, early telephones relied on carbon microphones, pioneered by Thomas Edison; today’s smartphones contain tiny microphones based on micro-electro-mechanical systems, commonly called MEMS. Specialist microphones abound in recording studios; others are used by spies. But whatever the technology, these microphones all do the same thing: they convert sound waves into an electrical signal.
It turns out, however, that with the addition of suitable software, microphones can detect more than mere audio signals. They can act as versatile sensors, capable of tuning into signals from inside the body, assessing the social environment and even tracking people’s posture and gestures. Researchers have reimagined microphones as multi-talented collectors of information. And because they are built into smartphones that can be taken anywhere, and can acquire new abilities simply by downloading an app, they are being put to a range of unusual and beneficial uses.

That natural microphone, the human ear, is finely attuned to picking up certain characteristics in a person’s voice. It is not hard, for instance, to infer from a slight change in pitch when a friend might be under stress. Tanzeem Choudhury of Cornell University and her research team are building mobile-phone software that can be trained to do the same thing. That stress results in subtle changes in pitch, amplitude and frequency, as well as speaking rate, has been known for decades. But humans respond to stress in different ways and have different coping styles, says Dr Choudhury. So a one-size-fits-all smartphone app which analyses speech will not provide an accurate assessment of whether someone is stressed or not.
The sound of stress
Dr Choudhury’s solution is an app called StressSense. Running on a standard Android-based smartphone, it listens for certain universal indicators of stress but is able, over time, to learn the specifics of a particular user’s voice. It is unobtrusive yet is also robust enough to work in noisy environments, which is crucial if it is to be of practical diagnostic use. StressSense does not actually record speech, Dr Choudhury emphasises, but simply captures and analyses characteristics such as amplitude and frequency. In a paper published last year, the researchers concluded that “it is feasible to implement a computationally demanding stress-classification system on off-the-shelf smartphones”. Their ultimate goal is to develop an app that can help someone determine the links between irritating situations and subsequent responses. Your phone might realise before you do, for example, that your 8am meetings are the cause of your headaches.
StressSense is still in development. In the meantime, Dr Choudhury’s team has launched an Android app called BeWell that focuses more on overall health by looking at three metrics: sleep, physical activity and social interaction. These three metrics, Dr Choudhury believes, are important yet easily measured indicators of someone’s health. BeWell’s sleep-tracking feature guesses whether the phone’s user is awake or not by analysing usage, light and sound levels, and charging habits. Physical activity is monitored using built-in accelerometers for motion-detection. And social activity is measured chiefly by collecting snippets of sound that indicate that the user is talking to someone, either in person or over the phone.
Again, no actual words are stored, simply features of human speech, which the app can distinguish from background noise such as music or traffic. Certain changes in a person’s social interactions—a sudden drop-off, for instance—can be indicative of health problems such as depression, says Dr Choudhury. The idea is that the app could tip someone off to a change in their behaviour that might otherwise have gone unnoticed.
Microphones need not limit themselves to listening to the human voice, however. John Stankovic of the University of Virginia in Charlottesville is using microphones to capture heartbeats. Researchers in his group are using earphones modified with accelerometers and additional microphones that detect the pulse in arteries in the wearer’s ear. This makes it possible to collect information about the wearer’s physical state, including heart rate and activity level, which is transmitted to the smartphone via the audio jack. The researchers even created an app, called MusicalHeart, that analyses the wearer’s heart rate and recommends songs from a music library based on a heart-rate goal—faster to encourage a runner, or slower to calm someone who is feeling nervous.
Dr Stankovic says he is working with anaesthesiologists to develop a similar system that could use the calming effect of music to help people who are about to undergo surgery. The aim, he says, is to explore the potential for using a pulse-music feedback system to calm the heart, rather than simply resorting to drugs to do the job. In a separate project, Dr Stankovic is developing sound libraries that can be used to identify and classify certain types of sounds, from direct indicators of depression in the human voice to coughs, wheezes and lung function. Just as a phone can perform speech recognition, it might also be able to distinguish between healthy and unhealthy lung sounds. Dr Stankovic and his team published details of their proposed “physiological acoustic anomaly detection service” in a paper in April.
Working in a similar vein, Shwetak Patel, a researcher at the University of Washington in Seattle, has found a way to use a smartphone to measure lung function from a hearty blow on its microphone. He and his team have developed an iPhone app, called SpiroSmart, that simulates a digital spirometer, the device that measures the volume of air a person can expel from her lungs. Spirometers help doctors better understand the health status of patients with conditions like asthma, chronic obstructive pulmonary disease (COPD) and cystic fibrosis. But clinical spirometers are expensive, costing thousands of dollars, which makes them impractical for widespread home use. Dr Patel and his collaborators reckon that if people who need one could have a spirometer on their phones, they could better manage their conditions, and their doctors could spot problems earlier, in between check-ups.
But in deciding to use the microphone on an iPhone, Dr Patel had to rethink the design of a spirometer, which uses small turbines to measure airflow. Instead of building a turbine add-on to the phone, his team developed software that listens for acoustic features, such as resonance, that result from air being expelled through the trachea and past the vocal cords. These features, he says, directly indicate the volume of air moved from the lungs. In a paper published last year, 52 patients compared the mobile-based spirometer with a clinical one. SpiroSmart’s results were accurate to within 5% of those from the gold-standard clinical device. The app is undergoing clinical trials with patients diagnosed with COPD, cystic fibrosis and chronic asthma, and Dr Patel hopes to get Federal Drug Administration approval for SpiroSmart as a medical device by the end of the year.
Another of Dr Patel’s microphone-based projects uses sound to recognise hand-movements in the air, just as touchscreens can distinguish between different gestures. The software, called SoundWave, produces an inaudible, high-pitched tone from a loudspeaker that bounces off a nearby object, such as a hand. The microphone picks up the reflected sound, and audio-processing software can then detect different movements, such as waving your hand, wiggling your fingers, or making two-handed gestures. It does this by measuring the slight shift in frequency that results when sound bounces off a moving object, known as the Doppler effect (a familiar everyday example of which is the change in the pitch of an ambulance’s siren as it passes by). The advantage of this approach is that it allows existing devices (smartphones and laptops, say) to detect gestures, such as flicking your hand upwards to scroll through a document, without the need for any additional hardware.
Keeping an ear on drivers
 Expanding on this idea, Dr Patel is currently developing software that uses a smartphone’s existing microphone and speakers and enables the device to detect its position in a car. Just like SoundWave, it would produce inaudible tones that reflect off the car’s interior. It might then be possible to begin and end calls using gestures—or could, Dr Patel suggests, form the basis of an opt-in service that locks the phone from the driver, keeping him from texting or making calls while driving.
One drawback of using microphones for these various kinds of sensing is that keeping a microphone listening at all times, and running software to analyse what it hears, can consume a lot of battery power. But a new trick devised by Qualcomm, a maker of chips for mobile devices, may be able to help. Its Snapdragon 800 processors have a feature called Snapdragon Voice Activation that can wake up a gadget from standby mode at the sound of a voice command. When no voices can be heard, the device remains in a battery-sipping slumber. As smartphones continue to be put to unexpected uses, the humble microphone is ready to play its part.


0 yorum: