University Rankings in Pakistan

Dr. Syed Irfan Hyder, January 5, 2004

https://www.dawn.com/news/1065476

This is with reference to the letter to the editor of Dawn, December 31, 2003 where the writer has highlighted the risks of jumping prematurely in the area of university rankings.

Development of a ranking system is a good idea, but coming up with a criteria that is complete, just, fair, valid, reliable and professional is a non-trivial undertaking as explained below. Hurriedly put together experiences of a few academicians in to a ranking system cannot be expected to be fair, valid or even professional.

It is surprising that with all the emphasis on R&D, there is a reluctance in investing in the research and subsequent development of the proposed ranking system. The ranking initiative is therefore headed in the same direction as the Model University Ordinance and Tenure Track initiatives. Recently a questionnaire was sent out by HEC that lacks in completeness as well as sufficiency. It is even without clearly laid out objectives and has typographical and structural mistakes. It is surprising to note the lack of thoroughness even in the booklet on Criteria for Establishing New Institutes and Universities that has been published and widely circulated. Institutions are threatened to be de-recognized if they do not conform to the booklet Criteria in five years!

A proper ranking system for an institution of higher learning should have the following characteristics without which it would not be implement able:

Just: It should be applicable to all the institutions of higher learning including public and private universities. It should be general enough to encompass variations in engineering, medical, general sciences, business and other professional disciplines. It should incorporate and accommodate all finer differences such as those between the doctorate in medicine and doctorate in philosophy. It should be able to cater to mixing and matching of different disciplines in some universities as well as directed specializations preferred by some others. It also must rationalize on the basis of the economic cost paid per student rather than the fee charged per student.
Complete: It should measure all the factors that contribute to the quality of the institute. It appears from the recent HEC questionnaire and the HEC criteria for setting up of new universities and institutes, that the questionnaire is biased towards brick and mortar evaluations. Evaluation of the following factors are conspicuous by their absence and must be included and given due weights: Financial discipline, accounting system, costing system, purchasing systems, grading systems, academic monitoring systems, exam and classes scheduling systems, syllabus management systems, fee management systems, faculty management systems, attendance monitoring systems, curriculum improvement systems, student records and student progress evaluation systems, examination systems, students complaints monitoring systems, industry interaction systems, etc.
Valid: Ranking system should be valid. That is, it should actually measure what it sets out to measure. Does it incorporate all the aspects of higher education? Does it measure the objectives of higher education for Pakistan? What does our economy require from the institutes of higher education. What is the economic demand for researchers, scientists, consultants, managers, office workers, professionals, academicians, responsible citizens from institutes of higher learning. What should be their proportion coming out from a given discipline. How often and how soon does the graduates switch from their area of specialization to an area of economic opportunity and what are the skills that they take into the new fields that they choose.
Criteria Testing and Pilot: Was a pilot done to identify reasonableness of the criteria. Was the criteria applied to a couple of universities to test its applicability, practicality and ability to yield the desired results. What shortcomings were noticed in the pilot application and how were they rectified. Unfortunately in Pakistan, government organizations tend to launch grand survey projects without the necessary pilot studies. One of the prime examples of failed surveys has been the business survey launched in 1998-99 for CBR and NADRA. We are still reeling from the lack of homework and pilot study done for the NIC project. More pertinent example is the Model University Ordinance that was applied without a pilot study.
Practical: Criteria should be realistically measurable. Is the asked for information readily available. Is it verifiable. In how many ways can it be interpreted. How much effort does it take to fill out the questionnaire. Would all institutes put a similar amount of effort. What would be the source of their data. Is the source of data reliable, authentic and comparable.
Reliable: Does multiple application of the criteria by different teams and different personnel and at different times on the same institute yield the same result.
Professional: Has the questionnaire been prepared using established research methodologies. Selection of the survey technique, selection of the questions, wording and format of the questions, selection of the respondents, estimation of respondents’ effort, estimation of surveyors’ effort, compilation effort, subsequent analysis effort and other related aspects need to be evaluated and addressed by professionals in business and analytical research.

A brief analysis of the issues raised above indicate, that HEC needs to do a major research in developing the criteria before it can be ready to actually start ranking the institutions. The development of the criteria would have to be done in an open and transparent manner with continuous interactions with institutions of higher learning to yield the desired results.

Dr. Irfan Hyder