Performance Ranking of Scientific Papers for World Universities 2017 - Methodology


Sample Selection

This ranking system employs bibliometric methods to analyze and rank the scientific paper performances of the world’s top 800 universities. The selection of the 800 universities for inclusion in this ranking system was based on information obtained from the Essential Science Indicators (ESI). Of more than 4,000 research institutions listed in ESI, this ranking system first selected the top 900 institutions based on the numbers of published journal articles and numbers of citations. Non-university institutions were then removed from the list, and the project staff compared the remaining universities to those included in other ranking programs such as ARWU, THE, QS, and U.S. News. This resulted in 1,294 universities for this ranking system. Data used to assess the performances of the universities was drawn from ISI’s ESI and Web of Science Core Collection (WOS), which includes SCI and SSCI, and Journal Citation Reports (JCR).

The concept of authority control was employed to retrieve data indexed under different forms of a university’s name in the aforementioned databases – i.e. the official name, the abbreviated name and other possible forms of the name. This ranking system also considered the merging and splitting of universities (or different campuses in a university system) and included publications by university-affiliated institutions such as research centers and university hospitals. This effort ensured the accuracy of each university’s number of published journal articles and the subsequent citation statistics.

Some university systems have several campuses. A few campuses within a particular university system may have been commonly perceived as individual institutions. However, they are indexed in ESI only by the university system name. For example, the University of Connecticut system includes a main campus in Storrs and five regional campuses throughout the state. Furthermore, it also contains the Schools of Medicine and Dental Medicine at UConn Health in Farmington. These are all indexed under "University of Connecticut" in ESI.

This ranking system corrected this flaw by manually searching SCI/SSCI in order to identify the actual number of articles and citations of these articles produced by each individual campus. Likewise, this ranking system employed the same manual searching procedures to ensure that the measurement of each university’s Highly Cited Papers fairly represented the research performance of each individual campus.

The reader may notice that the publications counting method for some institutions in ESI changed this past year, and some universities and their affiliated institutions are now considered as single institutions where they were previously considered separate institutions. This led to an increase in the number of published journal articles. Moreover, journal articles are now indexed by publication year instead of database year. These two changes will affect our ranking results.

Indicators (back to top)

The 2017 performance measures are composed of eight indicators. These indicators together represent three different criteria of scientific paper performance: research productivity, research impact, and research excellence. Table 1 lists the indicators and shows the respective weightings for each indicator.

Table 1 The Criteria and Indicators, and Their Respective Weightings, Used for the Overall Performance-Based Ranking

Criteria 2017 Overall Performance Indicators Weighting
Research productivity Number of articles in the last 11 years* (2006-2016) 10% 25%
Number of articles in the current year (2016) 15%
Research impact Number of citations in the last 11 years* (2006-2016) 15% 35%
Number of citations in the last 2 years (2015-2016) 10%
Average number of citations in the last 11 years* (2006-2016) 10%
Research excellence h-index of the last 2 years (2015-2016) 10% 40%
Number of Highly Cited Papers* (2006-2016) 15%
Number of articles in the current year in high-impact journals (2015-2016) 15%

*Note: The timeframe of the three long-term indicators is consistent with that in ESI, providing cumulative data for the last 11 years.



Indicator Definition (back to top)

■Research Productivity:

The number of articles published in peer-reviewed academic journals is frequently used to indicate the productivity of a research institution. To objectively represent a university’s current and on-going research productivity, this ranking system employs two indicators: the number of articles in the last 11 years (2006-2016), and the number of articles in the current year (2016).

"Number of articles in the last 11 years" draws data from ESI, which includes 2006-2016 statistics articles published in journals indexed by SCI and SSCI. "Number of articles in the current year" relies on the 2016 data obtained from SCI and SSCI, which were extracted in April, 2017. We used to extract data in every January, but after a yearly observation, we discovered that papers published in 2012 would continue to be indexed by the databases into early 2013. Accordingly, we decided to postpone the data extraction time to April so the data would be more complete.

■Research Impact:

The number of citations of a particular academic article within a specific time frame is a commonly accepted indicator for that article’s impact. This ranking system considers both the long-term and short-term impact of a particular research and seeks to provide a fairer representation of a university’s research impact regardless of its size or faculty number. Thus, this ranking system measures research impact by the number of citations in the last 11 years, the number of citations in the last 2 years, and the average number of citations in the last 11 years.

"Number of citations in the last 11 years" draws 2006-2016 citation statistics from ESI. "Number of citations in the last 2 years" draws 2015-2016 citation statistics from SCI and SSCI in WOS, which include citation statistics updated to the dates of retrieval. "Average number of citations in the last 11 years" is the number of citations in the last 11 years divided by the number of articles in the last 11 years.

■Research Excellence:

This ranking system assesses each university’s research excellence by the following indicators: the h-index of the last 2 years, the number of Highly Cited Papers from ESI, and the number of articles in the current year in high-impact journals (Hi-Impact journal articles). "h-index of the last 2 years" measures both the quantity and quality of a university’s research via the use of the 2015-2016 SCI and SSCI data. Employing Hirsch’s (2005) concept of h-index , a university has index h if h of its Np papers in the last two years have at least h citations each and the other (Np – h) papers have ≦h citations each.

"Number of Highly Cited Papers" utilizes data from ESI, which includes statistics of "Highly Cited Papers" from 2006 to 2016. ESI defines Highly Cited Papers as SCI/SSCI-indexed papers that are cited most (in the top 1% of the total papers indexed in the same year) within the last 11 years.

"Number of articles in the current year in high-impact journals" employs data from JCR, which supplies the impact factor of each journal in its subject field. The impact factor of a journal is the number of citations of the papers published in that particular journal within the previous two years divided by the number of that journal’s papers within the previous two years. A journal with a higher impact factor means that journal has articles more frequently cited by other journals, thus suggesting its higher scholarly value. This ranking system defines high-impact journals as those whose impact factors are ranked in the top 5% of the total journals within a specific subject category. With high-impact journal lists derived from JCR, this ranking system is able to count the numbers of each university’s articles published in high-impact journals by subject.

Reference Rankings (back to top)

Adjusted reference ranking is presented to balance the overall ranking, which favors universities with greater number of faculty members. Four indicators significantly affected by university size are normalized by each university’s number of full-time faculty; these include the number of articles in the last 11 years, number of articles in the current year, number of citations in the last 11 years, and the number of citations in the last 2 years. This ranking system employs faculty numbers obtained from the following sources (listed by priority in usage): numbers of full-time faculty obtained from university official websites, numbers of faculty registered with each higher education administration, and numbers of academic staff of each university obtained from university websites.

Calculation Method (back to top)

The procedures for data processing are as follows: First, the project staff conducted authority control on the various forms of a university name and inspected all the SCI/SSCI bibliographic records in which the address field contained a form of the university name. An accurate number of the total articles from a university was obtained after removing duplicate records containing different forms of that university’s name. Second, using SCI/SSCI this ranking system obtained the total number of citations by adding the number of citations of each article from that university, from the article’s inclusion in SCI/SSCI to the date of our retrieval.

Based on the measurement procedures, this ranking system calculated a university’s score for each of the eight indicators. From 2013, the indicator "h-index of the last 2 years" has been further adjusted and differentiated utilizing number of papers and cited times for the field-based and subject-based rankings, due to highly concentrated distribution of h-index. Then, all indicators are normalized using T-score to obtain finalized score. Finally, the ranking system calculates the final score of each university using the indicator weightings presented in Table 1 and sorts the universities by their final scores. Universities with the same scores are sorted alphabetically. It should be noted that many universities obtained similar scores, and the slight differences in the final scores must be interpreted carefully. A university having a slightly higher score than another may not necessarily suggest its superiority in scientific research, as the two universities might be in very close proximity in the ranking.

 

References

Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.

Huang, M. H. (2008). Application of H-index for Research Evaluation at University Level. Evaluation in Higher Education, 1(2), 29-50.

Huang, M. H., & Chi, P. S. (2010). A Comparative Analysis of the Application of H-index, G-index, and A-index in Institutional-Level Research Evaluation. Journal of Library and Information Studies, 8(2), 1-10.


Copyright © 2017 NTU Ranking
No. 1, Sec. 4, Roosevelt Road, Taipei, 10617 Taiwan (R.O.C.)
Visits Count: 631834