I believe that colleges do matter whether it be long term or short term. If the name or brand image of a college would not have been of any use, then i don't think that anybody would have dreamed or worked hard for getting admission in IIM, ISB, FMS, XLRI, SIBM etc. etc.
What do these elite management institutes provide that everyone wants to be in it?
These colleges have been around for a considerable amount of time to prove worthy in the eyes of the recruiters. They (recruiters) believe that a student from the above mentioned institute will be worthy enough to handle responsibilities rather than any "xyz" school of management who has its premises with an area of 2,000 sq. ft. and takes candidates who have barely cleared their graduation.
Many have said that in long term, it is one's performance that is going to matter. In any scenario, if one does not perform in the short term, then it don't matter even if he is a product of "The Wharton School, University of Pennsylvania" (The top business school in the world); he will be sacked. Long term is out of the question.
Studying in a recognized institute is advantageous as one learns and develops the required skills in right manner and will be provided a better opportunity to use them with the help of corporate exposure one gains due to the reputation of that institute.
It is a biased world and therefore, its not awkward if you sing in the same tune the winning side is.