Originally Answered: People judge me by my clothes?
Don't change if you don't want to and you're not harming others. Which you are not. Also, I'd like to point out, that dude was never your friend in the first place if he's showing concern that you're "poor" by shunning you or talking smack (or both in this case, from the sounds of it) rather than being helpful about matters.
Personally, I have a lot of experience with the poor. I am nowhere close to poor, but I am heavily involved with the very poor and homeless in my community. A lot of them dress very, very nicely in what others perceive as "things I can't afford, they're getting too much on welfare!" and other BS when in fact, I, and others like me, buy much of their clothing. Usually by taking them to thrift stores on sale days as the bulk of the money my group brings in via donations goes to things like medical care, transitional housing, food, transportation and subsidies of various sorts we provide for our clients. So the manner of dress does not necessarily indicate that you are "poor" or "wealthy", it's just the way you dress because you like it or that's what you happen to have. If someone can't deal with that, then that's their problem, not yours.
As for why he cares - as I already mentioned, he really doesn't. He actually shouldn't care how anyone dresses at all, "friend" or not, unless he's caring in a helpful manner.This means he doesn't "care" so much as he enjoys being a judgmental douchebag. Remember that peek at his true personality and avoid him as much as possible. People like that are never worth your time or energy.