WASHINGTON (CNN) -- A Virginia congressman who said in a letter to constituents that without immigration reform "there will be many more Muslims elected to office demanding the use of the Koran" won't apologize for that and other remarks, the congressman's spokesman said Wednesday.

Republican Rep. Virgil Goode's letter also warns that without immigration reform "we will have many more Muslims in the United States."

Spokesman Linwood Duncan said Goode's letter was written in response to complaints his office received about Minnesota Congressman-elect Keith Ellison's request to be sworn in using the Koran.

Ellison is the first Muslim to be elected to Congress.

Goode's office released the letter to CNN Wednesday. In it, Goode wrote in part, "When I raise my hand to take the oath on Swearing In Day, I will have the Bible in my other hand. I do not subscribe to using the Koran in any way. The Muslim Representative from Minnesota was elected by the voters of that district and if American citizens don't wake up and adopt the Virgil Goode position on immigration there will likely be many more Muslims elected to office and demanding the use of the Koran.

CNN EXCLUSIVE: Wolf Blitzer will interview Congressman-elect Ellison, 4 p.m. ET, The Situation Room