Tuesday, May 7, 2013

Myth: Impact Factor Measures Real Impact

Myth: Impact Factor Measures Real Impact

One of the most interesting game being played in the universities is that of impact factor research. There is a whole industry that is now specializing in how to increase the impact factor. There are researchers who are publishing at the rate of one publication every month (or even every other week) in the prestigious ISI index. There are several researchers in Pakistan who have published over 30 ISI papers in three years. Where is this impact coming from? How can one do a unique original research that can produce a paper every few weeks, unless it is just a rehash of some existing work?

Impact factor is the holy grail. A game that researchers play. Simply stated it means that greater the references to papers in other papers, the greater the impact. Supposing a Pakistani researcher's paper gets cited hundreds of times in ISI research journals of obscure names and places, what impact would it have created on the society here in Pakistan and on this nation. It is possible for a group of researchers to collude and using the you-scratch-my-back-and-I-scratch-yours approach to jack up the count of citations. I am told that some researchers are really playing this game well.

One of my friends who did his PhD from a prestigious US university told me about a course that he took in USA that he later dropped. The professor told him to write a paper that is so dense and complicated that it should not be understandable by any one. This should be done by including some complicated equations and matehmatical models and should revolve around terminology related to a particular field. The only restriction was to include a few references to the work of some professors in Europe. When those professors' papers were analyzed, it was found that the papers were equally dense, incomprehensible and even non-sensical. These works are all published in reputable journals.

There is an interesting analysis  of computer generated non-sensical but completely grammatical and correct English papers that appear to be scholarly but totally gibberish which were sent for publication to various  journals/conferences and which got accepted and were published. There is even a website that helps you generate scholarly research papers complete with tables, facts and figures that get accepted in various conferences. Even major publishers of online journals had published computer generated research papers.

I know of some scientists who have over 800 publications in a career spanning over 40 years. If you do the maths i.e. number of days in 40 years divided by the number of publications, the average number of days in which a publication was produced comes out to be around 18 days (= 40 x 365 / 800). What originality can there be in such creative work? Mind you majority of these publications would be in worthy impact factor journals.

I think there is a fundamental problem in this kind of measurement. You are measuring the instrument with itself. Fever can not be measured with fever of other people, it needs a device called thermometer that measures expansion of (say) Mercury. Speed of a car can not be measured by using speed of other cars, it needs a device called speedometer. Height of a plane in air can not be determined by comparing them with other planes in the air, it needs an instrument called altimeter that uses another quantity. The issue in impact factor measurment is measuring a quantity with itself. Hence, the number of publications have increased tremendously. There are now services available that would publish your journal and articles and would get your journal and articles a high impact factor if you pay some thousands of dollars. 

The more fundamental question that may be asked is can we use this impact factor measure to quantify the contribution of Steven Spielberg, Disney, Steve Jobs, Bill Gates, or Sam Walton (of Walmart fame) to the US society.

Can this impact factor quantify the positive contribution of Abdul Sattar Edhi, Dr Adeebul Hasan Rizvi, TCF, and so many others who are creating a sizeable impact on society.

Can this impact factor quantify the negative contributions of the dictators, bigots and war-mongers or exploiters of environment and plunderers of the financial industry in Pakistan. The impact factor is hopelessly out of sync with society and industry.

The challenge for HEC in Pakistan was to define an impact factor that has relevance for Pakistan. It should have measured:
– Impact on the industry; new ventures, new products, new opportunities, new jobs, ..
– Social change, economic change, change movements

Instead of plagiarizing impact factor research and whole scale implementing it in Pakistan without thinking about its societal impact, we should have been prepared to do original research and should have researched to enhance the impact factor so that it can measure concrete changes brought about in our society. This should have been the role of higher education commission.

West is at a stage where the relevance of research to society has already been worked out and they can play this game. Pakistan is at a stage which has to ensure this societal impact as it spends its precious resources in research. Development of such an impact factor that quantifies contribution to our society is our need. It is not the need of West that has already transited from the stage, where we find ourselves at present.

What we need to do now is to develop indices that measure industry growth, job growth, poverty alleviation, health improvement and other such goals. Then we need to link the output of the university such as the contribution of the alumni to society, industry, and entrepreneurship. We should subtract from such positive contributions, all the negative contributions made by the alumni. There should be Hall of Fame for each university and then there should be a Hall of Shame. Let's measure the true impact. The impact on character of the graduates as reflected in their contributions to the society and industry.

People may argue that it is difficult, it may not be objective, it may not be measurable, what would be instrument, who would calibrate. But these are the challenges not the problems. Solving such challenges is the real impact factor research. If proponents of higher education would not do this, then who would!

See also:
Myth 1: Our backwardness is because of science and technology
Myth 2: There is mushrooming of educational institutions in Pakistan 
Myth 3: Public universities cost lower than private universities
Myth 4: Bigger infrastructure (land, building, equipment) means better education
Myth 5: Impact Factor research measures real impact

3 comments:

  1. Sheheryar Mohsin Qureshi commented
    (Yesterday 2:05 AM)

    Salam!
    Being related with education sector as well as research, after reading it, i felt i had written these words. Very true for those who have finally decided to be a part of this damned "race of publications."
    For the sake of information, i feel same is now happening in SCI indexed, SCI-e indexed and SSCI indexed journals. Or may be ISI index has now transformed into these indices by Thomson Reuters. Anyway, this is really thought provoking. Researchers have to decide the goal of research - number of publications or impact on society!

    ReplyDelete
  2. Muhammad Hanif (PhD Fellow at IoBM)March 2, 2016 at 6:58 PM

    The Institute of Electrical and Electronic Engineers and Springer, a publisher of scientific journals and research, agreed to remove more than 120 fake studies following an investigation by Cyril Labbe of Joseph Fourier University.
    Labbe spent two years analyzing research papers and discovered that more than 120 conference proceedings, as well as research papers attached to specific conferences, were published in 30 different journals, reports Nature. This occurred over the course of five years, from 2008 to 2013, and 16 studies were published in Springer journals and in journals published by the IEEE.

    The full news is available at the below link:
    http://www.ibtimes.com/fake-research-papers-how-did-more-120-gibberish-computer-generated-studies-get-published-1558725

    ReplyDelete