The NAACP Florida State Conference is urging the university president and board chairman of Bethune-Cookman University in Daytona Beach, Florida, to step down following the announcement of DeVos' speech last week.
The organization is claiming that faculty members have been threatened with termination and students could have their degrees withheld for speaking out against her appearance.
The university did not respond to CNN's request for a comment on the NAACP letter.
In addition, protests took place last week and more are expected on campus outside the hall where DeVos is scheduled to address those graduating on Wednesday. The anti-Trump group Indivisible, the NAACP Florida Conference and students plan to demonstrate at the event.
University president Edison Jackson has defended the invitation, however, saying it benefits students to hear from those with controversial ideals and differing beliefs.
"If our students are robbed of the opportunity to experience and interact with views that may be different from their own, then they will be tremendously less equipped for the demands of democratic citizenship," he said in a statement posted online.
The university has not responded to the NAACP Florida Conference's call for the leadership to step down.
DeVos initially received criticism after meeting at the White House with leaders of Historically Black Colleges and Universities and then issuing a statement praising those who started the institutions were "real pioneers" of school choice.
HBCUs and their supporters rejected the characterization as insulting, since the schools were founded during an era of segregation when black students were prohibited from attending mostly white colleges in the South.
DeVos later walked back her comments, saying black universities and college were not about choice but due to racism.
An online petition with over 6,000 supporters wants to stop DeVos from delivering the commencement speech, charging the secretary had reduced consumer protections for student loan repayment plans and amnesty programs -- something many of the Bethune-Cookman's students depend upon.
President Donald Trump over the weekend issued a statement saying, "Secretary DeVos chose an HBCU as the venue for her first commencement address to demonstrate my administration's dedication to these great institutions of higher learning."