National Health and Medical Research Council grant funding: can the process be improved to achieve its objectives?

2 October 2009

There is a fundamental flaw in the structure of the National Health and Medical Research Council (NHMRC) review panels. They are largely disciplinary in nature, but not functional. This means that basic, translational and clinical research applications are all in the one basket. Yet the objectives of these different research phases cannot be fairly compared.

Basic research seeks to identify and develop some new aspects of medical research. Translational research takes the results of basic research into the clinic. Clinical research is the ultimate endpoint of all research.

Ideally, basic research should be supported from the intellectual aspect only, on the basis that any new knowledge is good knowledge. On the other hand, the endpoint of translational research is to bring the state of the art to the clinic, whereas clinical research applies directly to patient care. How can these three different phases of medical research be judged by the same criteria and by the same reviewers?

Everyone would agree that clinical research is poorly served by NHMRC. Many would say the same about translational research. I expect that basic researchers are not so happy either, because there are so many applications in this category.

Can we admit that there is a problem within the NHMRC. If so, then what needs to be done?

A major change would be to categorise proposals by their research phase, ie basic, translational or clinical research. Then we can find out the spectrum of such grant applications. Money should be apportioned to fund these phases so as to achieve the right balance. What is the right balance? Perhaps each has equal importance and should have equal funding, although we need to allow for the fact that there are many more basic researchers. I expect that this would be far from the current funding situation, which does not seek to achieve any balance between the research phases.

Reviewers should be required to nominate both their discipline and research phase experience. Whereas discipline is the major criterion for reviewer selection for basic research, this diminishes in importance as we move to translational and clinical research, as long as reviewers have experience in these phases.

The different research phases actually have different endpoints, and it is inappropriate to judge them under the same endpoints. New endpoints can be defined as follows:

• Basic: new knowledge.
• Translational: new clinical approaches.
• Clinical:  impact on patient management and prognosis.

The composition of the review committees needs to be revised. Members should be selected by discipline for basic research, as is the case now. However, separate committees should be set up for translational and clinical applications. It is just as important that members have the relevant experience in these research phases as they have the disciplinary knowledge.

Once a score has been determined, projects are currently funded above a certain, fundable threshold. No allowance is made for the fact that the evaluation process has an uncertainty of perhaps 30% or more. Yet projects are scored to 1% (or less?) to establish the funding cut-off. This is not science. Perhaps it is better to ballot the marginal group than to so improperly misuse statistics at a government level.

The quality of the reviewing system is in need of upgrading. There are for too many junior reviewers on the panels. Professors should be capable of a higher level of reviewing, and if there are not enough professors to go round, they could farm off and supervise reviews by more junior staff and sign-off on them.

This brings us to the problem is anonymity. Why should a senior reviewer be afraid of an open review system? In this case reviews would be much more serious undertakings. We publish papers and risk our reputations, why not our reviews as well? No serious scientist would object to a critical review that is positive in approach, but picks up errors and omissions. We are usually grateful for such reviews, rare that they may be.

The easy way forward is to ask reviewers if they wish their reviews to be anonymous or not, and so move towards a more transparent system.

At present, applicants can nominate a non-reviewer, but without knowledge of who is doing the reviewing, this is a rather useless exercise. A better approach would be for applicants to identify low level reviews. If a reviewer receives, say, three such low level ratings, then they are dropped from the system. By the same token, an applicant who complains more than three times could have future complaints ignored, so complaints will be used sparingly. This has the benefit of giving an applicant a means of rebuttal of a low level review, which is otherwise denied. The current system denies the applicant a right to question low level reviews, but the reviewer is free to continue to wreck damage with more superficial reviews.

This increasing emphasis on journal ratings and citations needs to be moderated. How many citations did Einstein have within five years of his prime publications? I expect there were very few. Why, because very few people were working in the field. High citation indices belong to review journals for obvious reasons; everyone refers to reviews. Clinical journal papers may have a major impact on clinical practice, but few citations, because normal operating procedures are not publishable. Basic research papers may have high citation indices, because a new technique or concept is picked up by many laboratory researchers, which is fair enough. However, in many cases, they have may have no clinical impact at all. So why should clinical researchers be so disadvantaged by a system that is not applicable to their activity?

The task that NHMRC faces is enormous. Does NHMRC actually know how fruitful its funding is? NHMRC consumes a lot of federal funds, and now, regrettably, has the Cancer Councils captive to its review process. It carries a heavy responsibility, but is it actually doing its job in bringing new medical technology and practice into the clinic?

A reply to this letter, from the National Health and Medical Research Council, is also published online.