The origins of the word 'toff' are obscure; Chambers thinks it may derive from 'tuft', the name given to a gold tassel once worn on a nobleman's cap in English universities, marking him out as aristocracy. But there's no dispute about the highly-loaded connotations of the word today: unearned wealth; invoiced education; limited experience of the lower orders; and, as Alexander McCall Smith notes in his novels about the Edinburgh bourgeoisie, a fondness for wearing cords the colour of crushed strawberries.

At the Crewe and Nantwich by-election last week the word toff was used as an insult, to be spoken with a sneer. Labour campaign chiefs believed it would tap into a visceral working-class prejudice against Tories as people of pampered privilege who neither know nor care how most folk live. It was an attempt to smear not just the Tory candidate as an individual, but Tories as a breed.

This weekend, as the by-election defeat is being used to bang another nail into Gordon Brown's political coffin, Labour big-wigs are queuing up to distance themselves from the anti-toff tactics used in Crewe. Most regrettable, they say. Counter-productive, they tut. Some even conclude from the result that the class war is dead, and that exploiting divisions between cloth caps and top hats should be consigned to the recycling bin of history.

My feeling is that this is premature, maybe by a century or two. You can't erase class differences that easily, nor the mutual misunderstandings and resentments that accompany them. And there is no escaping the simple fact that there are quite a lot of toffs in the Tory party – unsurprisingly given its history – and that many of them live down to the worst possible stereotypes.

I once called a senior Scottish Tory at home to get a quote on a story, and his rather grand wife answered the phone. After I asked for her husband she put a hand over the mouthpiece, but I could still hear her say: "It's for you, dear. No, I don't know who it is – but he has a rather strong Scottish accent." Here was a woman who lived in Scotland – most of the time anyway – who was apparently unused to dealing with those of us who are strangers to cut-glass Received Pronunciation. Proud Scot though I am, I'm sorry to say my muttered response was pure Anglo-Saxon.

Sometimes the Tory toff's tenuous grasp on modern life is akin to a kind of social autism. Lord James Douglas Hamilton, when he was a Scottish Office minister, was once making small talk with a civil servant and asked him where he lived. The official answered that he had a flat in a nice Edinburgh suburb, close to a good school and handy for the shops. "Ah yes," said Sir James, "but where do you live at weekends?"

Of course, not all Tories are toffs and there is a working-class strain of Scottish Conservatism. The Tenement Tories were the creation of Teddy Taylor when he was MP for the self-consciously respectable working class families who lived in the redstone tenements of Glasgow Cathcart. But Tenement Toryism doesn't have a hope in hell of making a comeback – its hang-em-and-flog-em populism isn't quite the image favoured by that cuddly David Cameron.

Last week's by-election result doesn't mean that Britain no longer cares about class. All it proves is that it's not as strong a motivator as the urge to give an unpopular Prime Minister a good kicking. It's a mistake to assume that Cameron can now stop worrying about being an old Etonian and how this plays with voters. Cameron can't help where he was educated – sending him to one of the most elitist schools in Britain was his parents' decision not his. What is wholly down to him, however, is his decision to surround himself as Tory leader with old chums with exactly the same pedigree. Until quite recently there were 13 Old Etonians on the Tory front bench, and more working in key party backroom jobs. Are they really all there on merit? This is bad politics for two reasons – it looks awful; and it fatally narrows the range of advice filtering in to the leader's office.

I don't believe you should judge someone by their social background any more than you should judge them by their religion, their ethnicity, their accent, their sexuality or their gender. To do so is simple prejudice. If you must judge – and who doesn't – then judge on actions and utterances.

The irony is that David Cameron needn't worry too much about being seen as disconnected from ordinary people's lives. His kitchen in Notting Hill may cost more than some people's entire homes, but the family life lived there is recognisable to millions across the country. What he needs is new blood in his blue-blood, toff-heavy leadership. Now, where are all those working class Tory MPs – the teachers, nurses and social workers who understand the public's instincts and who gave up caring jobs to devote themselves to the Conservative cause? Ah...