The office of President Jair Bolsonaro announced Alvim's dismissal in a statement Friday, explaining that his position had become untenable despite issuing an apology.
"I repeat our rejection of totalitarian and genocidal ideologies, as well as any allusion to it," read the statement.
"We also share our total and unrestricted support to the Jewish community, of whom we are friends and share the same values."
In the video, which was posted to the secretariat's Twitter account but subsequently deleted, Alvim announced a national award for the arts and made the case for Brazilian art to have nationalistic ideals.
"The Brazilian art of the next decade will be heroic and it will be national, it'll be endowed with great capacity for emotional involvement and equally it will be deeply committed to the urgent aspirations of our people, or it will be nothing," he said.
Many Brazilians were quick to spot that the statement resembled a quote featured in Peter Longerich's biography of Goebbels, which reads: "The German art of the coming decades will be heroic, will be steelily romantic, will be unsentimentally objective, will be nationalistic with grand pathos, it will be both committed and unifying -- or it will be nothing."
Alvim responded swiftly, calling reports he had quoted Goebbels a "fallacy" of the left.
He said he would never quote Goebbels, but acknowledged there was "a coincidence with a sentence from a speech from Goebbels" in the video.
"It was a rhetorical coincidence," Alvim wrote. "But the sentence itself is perfect: heroism and the aspirations of the people is what we want to see in national Art."
However, in addition to the Goebbels similarity, the background music playing during Alvim's announcement is from Richard Wagner's opera "Lohengrin," which Nazi leader Adolf Hitler said had played a defining role in his life.
"At the age of 12, I saw the first opera of my life, Wagner's Lohengrin," Hitler wrote in the first volume of "Mein Kampf," adding it had helped him confirm his "deep rooted aversion" to the career his father had chosen for him.
"I wanted to become a painter, and no power on earth could ever make an official of me," he wrote.
Controversy over Alvim's video drew a response from the German embassy in Brazil, which called Nazi rule the "darkest chapter in German history."
"We oppose any attempt to banalize or even glorify the age of National Socialism," said the embassy in a tweet.
Rodrigo Maia, the speaker of the lower house of Brazilian Congress, said in a post on Twitter that Alvim had "gone beyond all limits."
He said the video was "unacceptable" and urged Bolsonaro to remove him from the post of culture secretary.
The Brazilian Israelite Confederation, the country's main Jewish organization, also spoke out against the video.
"To emulate [Goebbels'] view... is a frightening sign of his vision of culture, which must be combated and contained," it said in a statement that called for Alvim's immediate removal.
"Brazil, which sent brave soldiers to combat Nazism on European soil, doesn't deserve it," the statement read.
Alvim later apologized for what he described as an "involuntary mistake," saying he had been unaware of the origin of the sentence in his video.
The latest controversy follows a December video in which the head of Brazil's National Arts Foundation (Funarte) linked rock music to abortions.
"Rock music leads to drugs, which leads to sex, which leads to abortions," Dante Mantovani claimed in a post on his personal YouTube page.
"At the same time, the abortion industry feeds into something much more serious, which is Satanism," he added.