The Made at Google 2019 event saw the release of new consumer products by Google such as the Pixel 4 smartphone.
PHOTO: John General/CNN
The Made at Google 2019 event saw the release of new consumer products by Google such as the Pixel 4 smartphone.
Now playing
02:27
What you need to know about Google's Pixel 4 smartphone
Now playing
02:23
Square to buy majority stake in Jay-Z's Tidal
PHOTO: MyHeritage
Now playing
01:01
Watch old photos come to life using AI
Now playing
01:26
No, Tom Cruise isn't on TikTok. It's a deepfake
A Facebook employee walks by a sign displaying the "like" sign at Facebook's corporate headquarters campus in Menlo Park, California, on October 23, 2019. (Photo by Josh Edelson / AFP) (Photo by JOSH EDELSON/AFP via Getty Images)
PHOTO: Josh Edelson/AFP/Getty Images
A Facebook employee walks by a sign displaying the "like" sign at Facebook's corporate headquarters campus in Menlo Park, California, on October 23, 2019. (Photo by Josh Edelson / AFP) (Photo by JOSH EDELSON/AFP via Getty Images)
Now playing
02:36
Facebook to restore news in Australia
Now playing
02:57
Why Microsoft backs Australia's pay for news proposal
MADISON, WI - NOVEMBER 05:  U.S. President Barack Obama and rocker Bruce Springsteen wave to a crowd of 18,000 people during a rally on the last day of campaigning in the general election November 5, 2012 in Madison, Wisconsin. Obama and his opponent, Republican presidential nominee and former Massachusetts Gov. Mitt Romney are stumping from one 'swing state' to the next in a last-minute rush to persuade undecided voters.  (Photo by Chip Somodevilla/Getty Images)
PHOTO: Chip Somodevilla/Getty Images
MADISON, WI - NOVEMBER 05: U.S. President Barack Obama and rocker Bruce Springsteen wave to a crowd of 18,000 people during a rally on the last day of campaigning in the general election November 5, 2012 in Madison, Wisconsin. Obama and his opponent, Republican presidential nominee and former Massachusetts Gov. Mitt Romney are stumping from one 'swing state' to the next in a last-minute rush to persuade undecided voters. (Photo by Chip Somodevilla/Getty Images)
Now playing
01:26
Barack Obama and Bruce Springsteen team up for new podcast
Find Solution AI facial expression reading software mapping facial muscles to assess emotion
PHOTO: Shutterstock/Find Solution AI
Find Solution AI facial expression reading software mapping facial muscles to assess emotion
Now playing
03:54
How AI that reads emotions is changing the online classroom
This illustration picture shows the social media website from Parler displayed on a computer screen in Arlington, Virginia on July 2, 2020. - Amid rising turmoil in social media, recently formed social network Parler is gaining with prominent political conservatives who claim their voices are being silenced by Silicon Valley giants. Parler, founded in Nevada in 2018, bills itself as an alternative to "ideological suppression" at other social networks. (Photo by Olivier Douliery/AFP/Getty Images)
PHOTO: Olivier Douliery/AFP/Getty Images
This illustration picture shows the social media website from Parler displayed on a computer screen in Arlington, Virginia on July 2, 2020. - Amid rising turmoil in social media, recently formed social network Parler is gaining with prominent political conservatives who claim their voices are being silenced by Silicon Valley giants. Parler, founded in Nevada in 2018, bills itself as an alternative to "ideological suppression" at other social networks. (Photo by Olivier Douliery/AFP/Getty Images)
Now playing
01:25
Parler's website is back online
screengrab singapore contact tracing app
PHOTO: Government Technology Agency of Singapore
screengrab singapore contact tracing app
Now playing
03:07
How Singapore's contact tracing technology undermines citizen's trust
PHOTO: Boston Dynamics
Now playing
01:20
Spot the robot's new arm lets it jump rope (and do serious stuff)
Detail of a mans hand scrolling through Netflix on an Apple iPad Pro, taken on March 6, 2020. (Photo by Phil Barker/Future Publishing via Getty Images)
PHOTO: Future Publishing via Getty Imag
Detail of a mans hand scrolling through Netflix on an Apple iPad Pro, taken on March 6, 2020. (Photo by Phil Barker/Future Publishing via Getty Images)
Now playing
04:27
2020 was supposed to be the year of streaming. Instead, it was the year of Netflix
PHOTO: WCCO
Now playing
01:26
Man donates Nintendo Switches to kids in need with GameStop stock
NEW YORK, NEW YORK - JANUARY 27: GameStop store signage is seen on January 27, 2021 in New York City. Stock shares of videogame retailer GameStop Corp has increased 700% in the past two weeks due to amateur investors. (Photo by Michael M. Santiago/Getty Images)
PHOTO: Michael M. Santiago/Getty Images
NEW YORK, NEW YORK - JANUARY 27: GameStop store signage is seen on January 27, 2021 in New York City. Stock shares of videogame retailer GameStop Corp has increased 700% in the past two weeks due to amateur investors. (Photo by Michael M. Santiago/Getty Images)
Now playing
03:16
GameStop mania shakes up Wall Street
SAN ANSELMO, CALIFORNIA - JANUARY 08: The suspended Twitter account of U.S. President Donald Trump appears on an iPhone screen on January 08, 2021 in San Anselmo, California. Citing the risk of further incitement of violence following an attempted insurrection on Wednesday, Twitter permanently suspended President Donald Trump's account. (Photo Illustration by Justin Sullivan/Getty Images)
PHOTO: Justin Sullivan/Getty Images North America/Getty Images
SAN ANSELMO, CALIFORNIA - JANUARY 08: The suspended Twitter account of U.S. President Donald Trump appears on an iPhone screen on January 08, 2021 in San Anselmo, California. Citing the risk of further incitement of violence following an attempted insurrection on Wednesday, Twitter permanently suspended President Donald Trump's account. (Photo Illustration by Justin Sullivan/Getty Images)
Now playing
04:29
What impact could deplatforming Donald Trump have?
screengrab US social media
PHOTO: Getty Images
screengrab US social media
Now playing
04:35
Tech companies ban Trump, but not other problematic leaders
(CNN Business) —  

While Google’s latest smartphone, the Pixel 4, got top billing at a major launch event in New York on Tuesday, the unveiling of an AI-enhanced recording and transcribing app was perhaps the biggest surprise of the day.

Recorder is meant for recording meetings, music, lectures, and such. It can recognize and transcribe in real time what you’re saying and identify other types of noise like music and applause. The recordings can be searched for specific words. For example, you can search for “rainbow” and receive results that show where that word was uttered in each recording.

Recorder will come with the new Pixel phone — Google’s flagship handset line for showing off the latest features of its Android app. That phone starts at $799, or $100 more than the baseline iPhone 11 model, and ships October 24. Starting in December, Google will roll Recorder out to older Pixel phone models as well.

While Recorder may sound like a fairly simple app, Sherry Lin, the product manager for Recorder, told CNN Business it wasn’t easy to make its speedy transcription work without killing the phone’s battery life. Google had to figure out how to pack into the handset a lot of AI that’s usually tucked away on a remote server.

“Honestly, when we started out we weren’t sure if we could even ship,” Lin said in an interview Tuesday.

As countless journalists and college students know, there are plenty of apps for recording audio on your smartphone, and some of them, such as Otter.ai, use AI to translate chatter into transcripts, allowing you to do things like search the resulting recordings. Typically, if you want to do more than simply record a conversation, you’ll need an internet connection because much of the AI involved in analyzing and transcribing, say, a riveting lecture about Hegelian dialectic, tends to happen on a faraway server, rather than on your smartphone.

A demonstration of Google's new Recorder app, which will come on its new Pixel phone.
PHOTO: Google/Youtube
A demonstration of Google's new Recorder app, which will come on its new Pixel phone.

To show off how Recorder does this work on the phone, Sabrina Ellis, vice president of product management at Google, noted Tuesday during an on-stage demo of the app that the phone was in airplane mode.

Lin said the reasons for keeping all of Recorder’s operations on the handset are twofold: to help protect the user’s privacy by keeping the audio and related text on the phone, and to allow speech to be translated into text more quickly than if it first had to take a trip to and from a remote server.

Making the app usable on a phone was tricky, however, in part because it relies on multiple pieces of AI that can run down the handset’s battery and bog down its main processor. These include an AI model that is specifically aimed at transcription (a retrained, retooled version of the model that powers Google Assistant), one that works on search, one for inserting punctuation into transcriptions, and one meant for classifying sounds other than speech.

Lin said that when she and her team started working on the app in earnest in March, the transcription model — the app’s biggest chunk of AI — drained the phone’s battery life in less than half an hour and made it heat up.

“We were like, ‘We’ll never ship unless we ship an air conditioning unit with the thing,’” she joked.

Early on, the software also froze the phone and was simply too large to send to consumers via Google Play, the company’s online app store.

To shrink the AI behind the app, Lin said the team “pruned” the transcription model and trained it on capturing long-form speech (this was done, essentially, by feeding the AI lengthy recordings of things like meetings, interviews, and lectures from YouTube) and ignoring background noise.

Google's Recorder app, which will come with the Pixel 4, uses AI on the smartphone to make transcripts from audio recordings.
PHOTO: Google
Google's Recorder app, which will come with the Pixel 4, uses AI on the smartphone to make transcripts from audio recordings.

Lin said the app doesn’t use remote workers to listen to any user recordings — a longstanding industry practice with virtual assistants that has been changing in the wake of media scrutiny regarding privacy concerns. (An exception might be if a user reports a bug, such as a strange static sound, and gives explicit permission for the company to listen to a recording, she said).

According to Lin, the app defaults to saving all the recordings and transcriptions on the phone, and the data is subject to standard Android device encryption. The company can’t see any recording-related data unless you choose to export it to a Google product such as Google Drive or Gmail, she said.

One thing the Recorder team is now working on is figuring out who is speaking when there’s more than one voice on a recording, Lin said. Currently, the app records all audio as though a single person is talking, and she wants to figure out how to segment the transcribed speech by speaker.

“It’s one of those things where it’s so easy for humans to do and so hard for a computer system to tell,” she said.