Savage was 22 at the time and a youth minister at Woodlands Parkway Baptist Church outside Houston when the incident with then-17-year-old Jules Woodson occurred.
Highpoint Church issued a leave of absence for Savage to begin immediately, Chris Conlee, the church's lead pastor, said in a statement Thursday night.
Conlee said the church is asking "a qualified, independent, third-party organization to do a full audit of our church processes and Andy's ministry."
"Please know that we support Andy as a leader of our church, but we also understand this has been a difficult season not only for Andy and his family, but for our congregation as well," Conlee said in his statement.
"We want to maintain trust in both Andy and our church leadership that we are not only doing things right, but we are doing right things."
Conlee said that when the audit is finished, "we will continue to support Andy and his family."
During a Sunday service, Savage said he "regretfully had a sexual incident with a female high school senior in the church."
"I apologized and sought forgiveness from her, her parents, her discipleship group, the church staff, and the church leadership, who informed the congregation. ... I took every step to respond in a biblical way."
He said he "deeply regret(s)" the pain he caused the family.
The encounter, which he said happened before he was engaged to his wife, Amanda, was "dealt with in Texas 20 years ago," he said without elaborating, "but in the last few days it has been presented to a wider audience."
He added, "In hindsight, I see that more could have been done for Jules. I am truly sorry more was not done. Until now, I did not know there was unfinished business with Jules. So today, I say, 'Jules, I am deeply sorry for my actions 20 years ago. I remain committed to cooperate with you toward forgiveness and healing. And I mean that.'"
In a YouTube video
of the sermon, several congregants on the front row can be seen standing up amid the applause. Lead pastor Chris Conlee then leads the church in prayer.
"Holy Spirit, would you take this prayer and right now as thousands of people are praying in agreement, in unison, would you touch Ms. Woodson's heart in a way that only you can and heal her of the pain that was caused from this sin 20 years ago?" Conlee said.
A "disgusting" apology
Asked for her thoughts on Savage's apology, Woodson told CNN through victim advocate Amy Smith: "It was disgusting."
CNN does not normally name juveniles involved in sexual incidents, but Woodson identified herself in a blog post
two days before Savage confessed to the congregation.
Woodson declined to speak to CNN directly but she wrote that "there are triggers that take me back to that night, there are nightmares that haunt my dreams."
"I am a grown woman now and although it's been almost 20 years since everything happened, it still affects me to this day," she wrote.
Woodson filed a report with the National Center for Missing and Exploited Children last week, Smith said.
On Tuesday, a case was opened and a detective from the Montgomery County Sheriff's Office interviewed Woodson, the department said in a statement.
The sheriff's office said it had worked with the local prosecutor, who researched the 1998 law and determined the statute of limitations had passed, based on the law at the time of the crime.
"Using the current statute, we would have some possible options, but we are limited to the law as it was at the time of the offense in 1998," said the statement released by Capt. Dan Zientek.
"As a result, we are unable to investigate and seek justice to the full extent of what we normally would in such a case," it said.
that her "hope in finally coming forward with my story is not only that I can begin to get closure and healing for all that has happened to me, but more so, that my story might have a positive impact on others and effect positive change in how these types of situations are handled within the church."