Shortly after special relativity was published and entered into mainstream physics, there was an experimental result that appeared to refute SR. I forget the details, maybe a violation of e = mc^2, but the incident is discussed in Pais' 'Subtle is the lord'.
Einstein said that if the empirical result were right then SR would be falsified, but that a wider range of empirical evidence and conceptual elegance convinced him that the most likely outcome would be that the anomalous result would be found to be in error.
Einstein turned out to be right.
So Phil, it is sometimes unwise to rush to judgement. A cooler head and a more sophisticated outlook is often called for.
Wouldn't you agree?
[Please don't try to say that my argument can be dismissed because it is based on one piece of anecdotal evidence, since the history of science contains a very large number of occurences of mistaken early experimental results.]
I agree with your analysis. While the NIH funding system will obviously make mistakes from time to time, it doesn't seem necessarily a flaw to me that its evaluations are not purely citation based. And from first principles;) it does seem that the authors have used a completely flawed measure of creativity.
@Oldershaw: Let's do it! Risk is lack of precedent, undesired solutions, political antithesis. Freedom is compliance.
Given n compactified dimensions at scale, active spatial dimensions are N = n + 3. Gravitational acceleration varies as 1/r^(N-1). Emergent compactified dimensions' radius (pdf, after Equation 5) is ~10^[(-32/n)-19] meters. Or, ~[(h-bar)c]/(M_s)(c^2)][(M_p)/(M_s)]^(2/n), "M_s" = M-theory energy unification scale, ~1 TeV, "M_p" = Planck mass, 1.221×10^16 TeV. Newtonian gravitation at Cm-248 nucleus' surface is 4.4×10^(-7) m/s^2. Five compactified dimensions emerge at seven times nucleus' diameter. Surface acceleration is now 1.5×10^64 m/s^2. Bullbleep!
Problems cannot be solved by theory that creates them. Test spacetime symmetry toward fermionic matter from cm to angstrom scales in existing apparatus. No precedent; chemistry does not constrain physics; inked paper hectares are correct. LOOK.
Also, creative work is not the only important work in biology. Both systematic work and luck are important. As an example, take deep brain stimulation for Parkinson's. It was based on work that was more systematic than creative. And it seems to work not because of reasons easily seen from the systematic work, but more because of luck. Nonetheless, latter could not be had without the former.
One of the current (and there are many) problems with medical research is that of guest authorship. Many highly-cited individuals had absolutely nothing to do with the actual research they are being cited for.
The example of cold fusion funding illustrates well, how the original research is actually valued with grant agencies - not to say about its usefulness.
I don't think this is a problem which only exists in the life sciences. It is very common also in physics that the guy-with-the-grant is named as author even though he might have had nothing to do with the content of the paper. There's been talk for some while to keep better track of which contribution came from which author, but I haven't heard if anything like this is actually on the way to realization. Best,
B.
4:02 AM, January 12, 2013
A recent issue of Nature magazine featured a study by Joshua Nicholson and John Ioannidis that looked at the citation count of principal investigators (PIs) funded by the US-American National Institute of Health (NIH).
Research grants: Conform and be funded
Joshua M. Nicholson, John P. A. Ioannidis
Nature 492, 34–36 (06 December 2012) doi:10.1038/492034a
Ionnadis is no unknown, he previously published a paper "Why Current Publication Practices May Distort Science" that we discussed here, and is author of the essay "Why Most Published Research Findings Are False".
The Nature article is unfortunately subscription only, so let me briefly summarize what it says before commenting.
Nicholson and Ioannidis analyzed papers published between 2001 and 2012 in the life and health sciences, catalogued by the Scopus database. They looked those who had received more than 1,000 citations by April 2012 and an author affiliation in the United States. They found 700 papers and 1,172 authors matching this query.
The NIH invites PIs of funded projects to become members of study sections. The purpose of NIH study section is to evaluate scientific merit. Nicholson and Ioannidis found that from the 1,172 top-cited authors only 72 were currently members of study groups, and most of these 72 (as expected) currently received NIH funding. However, these 72 top-cited scientists are merely 0.8% of all section members. Maybe more insightful is that they further randomly selected 200 of the top-cited papers and excluded those with authors in a study group. From the remaining top-cited authors, only 40% are currently receiving NIH funding.
In a nutshell, this is to say that the majority of authors of research articles in the life and health sciences that were top-cited within the last decade do not currently receive NIH funding.
That's as far as the facts are concerned. Now let's see how Nicholson and Ioannidis interpret this finding and what they conclude. In the beginning of the article, they are careful to point out that scientific success is difficult to measure and the citation count should be regarded with caution:
"The influence of scientific work is difficult to measure, and one might have to wait a long time to understand it. One proxy measurement is the number of citations that scientific publications receive. Using citation metrics to appraise scientists and their work has many pitfalls... However, one uncontestable fact is that highly cited papers (and thus their authors) have had a major influence, for whatever reason, on the evolution of scientific debate and on the practice of science."
However, towards the end of the paper they write:
"The mission of the NIH is to support the best scientists, regardless of whether they are young, old or in industry... Such innovative thinkers should not have so much trouble obtaining funding as principal investigators. One cannot assume that investigators who have authored highly cited papers will continue to do equally influential work in the future. However, a record of excellence may be the best predictor of future quality, and it would seem appropriate to give these scientists the opportunity of funding their projects."
Note how now authoring a highly cited paper is synonym for being an "innovative thinker" and "may be the best predictor of future quality". In fact, they go even farther than that by arguing that all authors of highly-cited papers should have their projects NIH funded (apparently regardless of what this project is):
"Funding all scientists who are key authors of unrefuted papers that have 1,000 or more citations would be a negligible amount in the big picture of the NIH budget, simply because there are very few such people. This could foster further important discoveries that would otherwise remain unfunded in the current system."
I find the above closing paragraph of the article simply stunning. They seriously argue that something must be wrong with NIH funding -- according to their elaboration it's a "networked system" in which "exceptionally creative ideas may have difficulty surviving" -- because the NIH does not automatically fund projects of authors with papers who gathered more than 1,000 citations within the last decade.
Now I know nothing about funding problems in the life sciences. Maybe they have a good reason to hold a grudge against NIH peer review practice. Be that as it may, the facts do simply not support their arguments. I am tempted to say it actually speaks in favor of the NIH that they do not pay so much attention to the citation count because, as the authors write themselves, it's a questionable measure: It measures not only innovative thinking, but also fashions and just usefulness (reviews and illustrative diagrams tend to gather lots of citations), it moreover picks up social dynamics, popularity of the authors, or the amount of secondary work that is created, irrespective of whether that work is particularly insightful.
Many top-cited works are created because somebody has been fast enough to jump onto a topic about to take off. Is that a sign for not being "conform", as the title of the article suggests? I am trying to imagine that somebody would argue that all top-cited physicists should get their projects funded without peer review. And would try to publish this as an essay in Nature.
posted by Sabine Hossenfelder at 12:52 PM on Jan 8, 2013
"Conform and be funded?"
11 Comments -
Here's a metric for funding that has probably never been tried overtly.
For Proposed Research: the number of definitive predictions made (which are feasible, prior, quantitative, non-adjustable and unique to the proposal).
For Funding Based On Past Work: the number of definitive predictions that have been verified.
Alas, it will never happen.
But man would it get rid of a lot of deadwood.
7:50 PM, January 08, 2013
And for experimentalists:
Proposal: definitive predictions to be tested.
Look-back Funding: definitive predictions successfully tested.
9:15 PM, January 08, 2013
Actually, if just one definitive prediction is falsified, the theory is wrong; it doesn't matter if others are confirmed.
6:11 AM, January 09, 2013
Shortly after special relativity was published and entered into mainstream physics, there was an experimental result that appeared to refute SR. I forget the details, maybe a violation of e = mc^2, but the incident is discussed in Pais' 'Subtle is the lord'.
Einstein said that if the empirical result were right then SR would be falsified, but that a wider range of empirical evidence and conceptual elegance convinced him that the most likely outcome would be that the anomalous result would be found to be in error.
Einstein turned out to be right.
So Phil, it is sometimes unwise to rush to judgement. A cooler head and a more sophisticated outlook is often called for.
Wouldn't you agree?
[Please don't try to say that my argument can be dismissed because it is based on one piece of anecdotal evidence, since the history of science contains a very large number of occurences of mistaken early experimental results.]
And have a nice day.
Robert L. Oldershaw
Discrete Scale Relativity
10:54 AM, January 09, 2013
I agree with your analysis. While the NIH funding system will obviously make mistakes from time to time, it doesn't seem necessarily a flaw to me that its evaluations are not purely citation based. And from first principles;) it does seem that the authors have used a completely flawed measure of creativity.
10:43 AM, January 10, 2013
@Oldershaw: Let's do it! Risk is lack of precedent, undesired solutions, political antithesis. Freedom is compliance.
Given n compactified dimensions at scale, active spatial dimensions are N = n + 3. Gravitational acceleration varies as 1/r^(N-1). Emergent compactified dimensions' radius (pdf, after Equation 5) is ~10^[(-32/n)-19] meters. Or, ~[(h-bar)c]/(M_s)(c^2)][(M_p)/(M_s)]^(2/n), "M_s" = M-theory energy unification scale, ~1 TeV, "M_p" = Planck mass, 1.221×10^16 TeV. Newtonian gravitation at Cm-248 nucleus' surface is 4.4×10^(-7) m/s^2. Five compactified dimensions emerge at seven times nucleus' diameter. Surface acceleration is now 1.5×10^64 m/s^2. Bullbleep!
Problems cannot be solved by theory that creates them. Test spacetime symmetry toward fermionic matter from cm to angstrom scales in existing apparatus. No precedent; chemistry does not constrain physics; inked paper hectares are correct. LOOK.
12:00 PM, January 10, 2013
Also, creative work is not the only important work in biology. Both systematic work and luck are important. As an example, take deep brain stimulation for Parkinson's. It was based on work that was more systematic than creative. And it seems to work not because of reasons easily seen from the systematic work, but more because of luck. Nonetheless, latter could not be had without the former.
1:17 PM, January 10, 2013
One of the current (and there are many) problems with medical research is that of guest authorship. Many highly-cited individuals had absolutely nothing to do with the actual research they are being cited for.
1:54 PM, January 10, 2013
The example of cold fusion funding illustrates well, how the original research is actually valued with grant agencies - not to say about its usefulness.
9:01 AM, January 11, 2013
Hi Andrew,
I am really puzzled though that Nature would publish such a shaky argument. Best,
B.
4:00 AM, January 12, 2013
Hi Juan,
I don't think this is a problem which only exists in the life sciences. It is very common also in physics that the guy-with-the-grant is named as author even though he might have had nothing to do with the content of the paper. There's been talk for some while to keep better track of which contribution came from which author, but I haven't heard if anything like this is actually on the way to realization. Best,
B.
4:02 AM, January 12, 2013