I am continually amazed and amused by the wonderful metaphors and similes generated by users of the English language. The phrase "Are you talking to me or chewing a brick?" is a 'hard-man' phrase, contrastive with 'wet-lettuce' phraseology. But after reading this week's New Scientist, I shall never view 'chewing a brick' in quite the same light again.

Edit:
For people who may be asking what does 'chewing a brick' actually mean ?  -

This phrase comes originally from East London.  It is hard man, tough guy talk.  'Hard' is opposed to 'soft'.  Firstly, a London Brick is a particular kind of baked clay brick that looks soft but is really quite hard.  It would take a really tough person to chew on one of those!  So if you want to challenge someone, you can insult them twice over with just one phrase.

By asking "Are you talking to me or chewing a brick?" you are pouring scorn by saying, more or less directly, that the person is mumbling.  You are also implying that, even if they are tough enough to chew a brick, you are tougher.  It is 100% fighting talk.

Footnote:  if you enjoyed reading this snippet of etymology, you may also enjoy this.