It was he who in 1860, exasperated by what he considered obstructionism on the part of Chinese imperial officials, led a British military expedition into Beijing that destroyed and looted the emperor's fairyland-style Summer Palace and presented Queen Victoria with a Pekingese picked up from the ruins.
She said Chinese officials in London for last year's state visit by the president, Xi Jinping, had proved difficult and rude and had treated Barbara Woodward, Britain's ambassador to China, with particular disdain.
Perhaps they were getting their own back for Lord Elgin -- but it clearly rankled with the Queen, whose comments were of a candour normally associated with her husband or eldest son.
The comments came the day after David Cameron was heard, also on a Buckingham Palace video, describing
Nigeria and Afghanistan to the Queen as "fantastically corrupt." Not surprisingly, Nigeria's president, Muhammadu Buhari, was less than amused.
It's tempting to see all this as an example of old-style British imperial arrogance -- and the British mindset as the last great bastion of Empire still holding out against decolonization.
You could view the words of the Queen and her prime minister as merely reflecting the old prejudice that foreigners are intrinsically funny and not to be trusted.
Nevertheless, President Buhari himself admitted the scale of corruption in Nigeria and pointed out that he was himself elected on a ticket to tackle it.
There would, after all, be little point in holding an anti-corruption conference if it were not a major problem somewhere.
But Diane Abbott has claimed that Britain is in no position to point moral fingers at other countries as long as it owns so many of the world's offshore tax havens, themselves a relic of the British Empire. It's difficult not to catch an echo of Lord Elgin in the air.
Moral (and economic) high ground
Part of the image the British built up for themselves under the Empire was certainly that of a country which rose above the low moral standards they thought they saw in other parts of the world.
In India, the British governor-general, Lord Dalhousie, followed a policy of taking over one "corrupt" or "inefficiently ruled" Indian kingdom after another. He claimed to be acting in the interests of the ordinary people, but it was obviously also in Britain's interests to extend its territory in this way.
Similar drives against corruption or slavery also drove British colonial expansion in Africa and were open to the same accusation: that the British were claiming the moral high ground as a way of pursuing their own self-interest.
Modern China is an economic superpower and can behave with a sort of imperial arrogance worthy of Lord Elgin.
As governor of Hong Kong, Chris Patten was driven to distraction by the obstructive attitude of the Chinese government in the run-up to the 1997 handover -- and there were eyebrows raised at the heavy-handed behaviour of Chinese officials escorting the Olympic flame through London in 2008.
But Hong Kong was handed over in the end and the Olympic flame did make it to Beijing
George Osborne might have gone to considerable trouble to forge a "Golden Relationship" to encourage Chinese investment in Britain, but it is not a relationship of equals -- and steelworkers in Port Talbot, undercut by cheap Chinese imports, saw little in it to celebrate.
One has one's pride
Those who see the remarks from either the Prime Minister or the Queen as a product of hypocritical latent British imperialism are missing the point.
Pointing the finger at the moral shortcomings of the powerful is a way of maintaining a sense of identity and pride when your own power has gone, pretty much exactly as the Chinese did in Lord Elgin's day.
Perhaps we should be grateful that President Xi didn't loot Windsor Castle and present his wife with a corgi.