Why would you want to annex a territory whose population rejects you? It's a question that should be asked more often.

Spain, for example, has seized on Brexit as an opportunity to revive its claim to Gibraltar. Madrid has successfully lobbied Brussels to declare that no EU-U.K. deal will apply to the Rock except with Spanish approval. Gibraltarians are horrified, fearing that Spain will now demand concessions on sovereignty.

Spain ceded Gibraltar to the U.K. "in perpetuity" in 1713. It understandably resented the transfer, and tried to grab the Rock back during the American Revolution – resulting in a rather chivalrous siege, in which gifts of food and wine exchanged across the lines. After the failure of that campaign, the claim was maintained in theory, but not pursued in practice, until Franco revived it as a way to stoke Spanish nationalism. Between 1954 and 1984, Spain closed the border, forcing Gibraltarians to rely on air and sea connections to the outside world.

Unsurprisingly, the 30,000 locals – Llanitos as they are colloquially called – are reluctant to accept Spanish rule. Their territory has, after all, been British for longer than it was Spanish. In 1967, they held a referendum, opting for British over Spanish sovereignty by 12,138 votes to 44. In 2002, they voted on a Spanish proposal for joint sovereignty, and rejected it by 17,900 to 187. Yet every Spanish party, except the Catalan separatists, claims Gibraltar.

So, to repeat, why claim a population which doesn't want you?

You might be saying "Funny question for a Brit to ask, what with your empire and all." But there is a difference between anti-colonialism and self-determination. Sometimes the two coincide; sometimes they don't.

The only territories where the U.K. retains jurisdiction are those where a clear majority of the population demands it, such as the Falkland Islands. Far from hanging on to places that want independence, the British have pulled out of several places that wanted them to stay.

Malta, for example, which asked to join the British Empire at the end of the eighteenth century, held a referendum in 1956 to integrate fully into the U.K., and be represented by three Westminster MPs. Three quarters of Maltese voters supported the proposal, but London said no, and Malta duly became independent. More recently, Britain withdrew from Hong Kong – the part that had been ceded in perpetuity as well as the part granted on a 99-year lease – despite vocal opposition from local people, who preferred British to Chinese sovereignty.

I'm not suggesting that the British Empire was an altruistic or democratic enterprise. It was acquired from a variety of motives, some downright sordid. Most of India, for example, was seized by adventurers who aimed to displace local elites and grab their tax revenues. The acquisition of African territories, by contrast, was largely driven by anti-slavery campaigners and evangelical Christians, who aimed to raise them to a point where self-rule was feasible.

As the Cambridge historian Robert Tombs puts it:

"Paradoxically, there was little or no appetite in London for adding to the empire. Requests from inhabitants of Ethiopia, Mexico, Uruguay, Sarawak, Katanga and Morocco to join were firmly turned down… A British empire should at least be a free association, and many hoped that its constituent parts would eventually become independent, like the United States."

It was the American experience that had changed attitudes in London. Determined to avoid another breakdown, successive U.K. governments tried – not always successfully – to remain ahead of the curve, devolving more power to the colonies than they yet been demanded.

Like all imperial powers, the U.K. has entries on both sides of the ledger. It extracted resources from conquered territories; it repressed indigenous political movements; it could be brutal. At the same time, it built roads, railways, schools and clinics, and sought to prepare the ground for eventual independence.

Decolonization was, in most places, almost miraculously peaceful. There were exceptions: violence in India, Cyprus and, in particular, Kenya. Ireland was a special case, being seen as integral territory rather than as a possession, and there were hideous acts of repression there. But most Caribbean, African and Asian colonies were brought to independence without a shot being fired in anger. Few empires can make such a claim.

The idea that the U.K. would today claim any territory against the wishes of its people is absurd. That applies even to core parts of the U.K. Not a single English or Welsh politician, for example, disputed the right of Scotland to hold a secession vote.

Britain, in short, holds Gibraltar for purely democratic reasons. Who are the real imperialists here?

Dan Hannan is a British Conservative MEP.