Several Congressional Black Caucus members are in talks to get veteran Rep. John Conyers to resign amid allegations of sexual misconduct, several Democratic sources told CNN on Tuesday. Those members are trying to ease his exit without trampling on his legacy during his 50-plus years in the House. Conyers, a Michigan Democrat, is the longest currently serving member of the House of Representatives. House Minority Leader Nancy Pelosi is not leading the effort but is tacitly supporting it, according to Democratic sources. A staffer to a member of the Congressional Black Caucus told CNN that “there is a feeling among some of our members that we need to protect his legacy,” noting that Conyers is a founding member of the group and was a leading figure in the civil rights movement. The House Ethics Committee announced last week it had opened an investigation into allegations against Conyers after BuzzFeed reported that he settled a wrongful dismissal complaint in 2015 after allegedly sexually harassing a staffer. A separate woman told CNN Tuesday that Conyers made three sexual advances toward her when she worked for him in his district office in Detroit from 1997 to 2005. Conyers has denied wrongdoing in both cases and has repeatedly said he will not resign from Congress. On Sunday, he stepped down as the top Democrat on the House Judiciary Committee.