linking back to brembs.net





Welcome Guest
Username:

Password:


Remember me

[ ]
[ ]
[ ]
 Currently Online (30)
 Extra Information
MicroBlog
NeuroTwitter

[15 Feb 10: 11:11]
W00t! Editor's selection at Researchblogging.org: http://researchblogging.org/news/?p=966

[11 Feb 10: 01:02]
Pharyngulated! http://scienceblogs.com/pharyngula/2010/02/religion_adaptation_or_by-prod.php

[26 Jan 10: 12:28]
New Theme! What do you think? http://bjoern.brembs.net

[04 Dec 09: 08:25]
Rolled over 400 citations today... http://bjoern.brembs.net/citations.php

[17 Nov 09: 08:45]
Students! You tell them for 45 minutes why their papers have to be in IMRaD format and some still hand in garbled, structureless papers!

[28 Oct 09: 04:17]
The m.o. of university administrations: divide competence until you can never be mad at anyone, because there are always so many others who can be blamed.


Networking
Random Video
SciSites
GeoCounter
outils webmaster
The recent kerfuffle caused by Butler's article in Nature seems to have gotten quite a number of stones rolling. What I thought would be a rather slow process seems to be speeding up considerably because of Nature's rather unclever attack on PLoS. Cameron Neylon's blog post "What I missed on my holiday or why I like refereeing for PLoS One" points out what the real danger of PLoS One is for basically all traditional journals:
From an author’s perspective PLoS ONE cuts out the crap in getting papers published. The traditional approach (send to Nature/Science/Cell, get rejected, send to Nature/Science/Cell baby journal, get rejected, send to top tier specific journal, get rejected, end up eventually going to a journal that no-one subscribes to) takes time and effort and by the time you win someone else has usually published it anyway. It also costs the authors money in staff time to re-format, rejig, appease referees, re-jig again to appease a different set of referees. I haven’t done the sums but worst case scenario this could probably cost as much as a PLoS ONE publication charge.  Save time, save money, still get indexed in PubMed. It starts to sound good, especially for all that material that you are not quite sure where to pitch.
[...]
To me the truly radical thing about PLoS ONE is that is has redefined the nature of peer review and that people have bought into this model. The idea of dropping any assessment of ‘importance’ as a criterion for publication had very serious and very real risks for PLoS. It was entirely possible that the costs wouldn’t be usefully reduced. It was more than possible that authors simply wouldn’t submit to such a journal. PLoS ONE has successfully used a difference in its peer review process as the core of its appeal to its customers. The top tier journals have effectively done this for years at one end of the market. The success of PLoS ONE shows that it can be done in other market segments. What is more it suggests it can be done across  existing market segments. That radical shift in the way scientific publishing works that we keep talking about? It’s starting to happen.
Today's system of scientific journals started as a way to effectively use a scarce resource, printed paper. Soon thereafter, the publishers realized there were big bucks to be made and increased the number of journals to today's approx. 24,000. Today, there is no technical reason any more why you couldn't have all the 2.5 million papers science puts out every year in a single database. It doesn't take an Einstein to realize that PLoS One is currently the only contender in the race for who will provide this database. For all the involved, it is equally clear what the many advantages of such a database would be. Consequently, traditional publishers are rightfully concerned that their customerbase is slowly dissappearing.
I'm now not alone anymore in believeing that we are seeing the beginning of the end of traditional journals. The acceptance of PLoS One is a quantitative marker of this development and the positive reactions I get from virtually everyone involved (even scientific editors at traditional journals) underscores the numbers.
Precurser to this publishing reform was access reform: scientific papers are the result of publicly funded research and should be publicly accessible. This reform appears now to be well underway and will probably conclude in 2-3 years. Both reform movements have their base in the more general open science movement. The goal of this reform movement is to have full public access not only to the published papers, but also to the raw data, ideas and reagents for sharing among scientists. There are still plenty of problems which have to be worked out before open science can become a reality, if it is even feasible. One of the more easy to solve problems (one that is shared with publishing reform) is that of how to attribute credit. If we all publish in the same database and share ideas online, how can two scientists competing for the same position or grant be assessed objectively? Cameron Neylon just returned from a conference on open science and writes in his summary:
I am sceptical about the value of ‘microcredit’ systems where a person’s diverse and perhaps diffuse contributions are aggregated together to come up with some sort of ‘contribution’ value, a number by which job candidates can be compared. Philosophically I think it’s a great idea, but in practice I can see this turning into multiple different calculations, each of which can be gamed. We already have citation counts, H-factors, publication number, integrated impact factor as ways of measuring and comparing one type of output. What will happen when there are ten or 50 different types of output being aggregated? Especially as no-one will agree on how to weight them. What I do believe is that those of us who mentor staff, or who make hiring decisions should encourage people to describe these contributions, to include them in their CVs. If we value them, then they will value them. We don’t need to compare the number of my blog posts to someone else’s – but we can ask which is the most influential – we can compare, if subjectively, the importance of a set of papers to a set of blog posts. But the bottom line is that we should actively value these contributions – let’s start asking the questions ‘Why don’t you write online? Why don’t you make your data available? Where are your protocols described? Where is your software, your workflows?’
I would disagree here and argue that a multivariate portfolio is exactly what is required. Different universities/employers will focus on different aspects of a researcher and value some of his/her contributions more than others. I don't think there can be too many measures to capture the complexity of scientific output. I'd like to see an aggregating service, maybe based on services like  OpenID, where a flexible portfolio can be organized such that employers can easily search for the traits they are looking for and find or compare the people who maximize their efforts on these traits.
The analogous problem to comparing researchers is that of comparing papers. I have already written about this problem and I think it is easy to solve. I think most researchers would gladly pay for a service which has a track record of picking the most interesting, groundbreaking and well-done papers from the 2.5 million every year. Today's professional editors would be a great pool from which such services could recruit employees.
blush.png Like dino-oil, there's still some use in long-dead structures. grin.png

Posted on Saturday 19 July 2008 - 13:13:40 comment: 7
editors   science publishing   open access   open science   science blogging   science politics   PLoS One   

Comments
Cameron Neylon posted on 20 Jul 08: 03:52 : Journals - the dinosaurs of scientific communication
Guest


Reply to this
Bjoern, you make a good point about people being able to find those who can do the kind of things you are looking for. I need to have a think about that but the idea of encouraging rather than discouraging diversity in where people contribute is a good thing and obviously different employers will have different needs. My concern is born of a frustration with the apparent obsession with measuring everything. Can't we spend a bit more time and effort on measuring the world rather than trying to quantify ourselves to the Nth degree?

Cameron

Wobbler posted on 20 Jul 08: 05:29 : Journals - the dinosaurs of scientific communication
Guest


Reply to this
'Can't we spend a bit more time and effort on measuring the world rather than trying to quantify ourselves to the Nth degree?'

I think measuring a scholar's proficiency is (in)directly related to their proficiency with measuring the world. Which, in a world with limited academic positions and limited resources, is one way to improve the efficiency of measuring the world: by providing those that have proven to be good at it positions and/or resources with more impact.

Science is huge with so many different research fields and research skills. So I agree with Bjoern Brembs that more metrics, as long as they are valid, is probably more a good than a bad thing.

What I do not quite understand is the following statement though:
'One of the more easy to solve problems (one that is shared with publishing reform) is that of how to attribute credit. If we all publish in the same database and share ideas online, how can two scientists competing for the same position or grant be assessed objectively?'

Is that not one of the biggest bottlenecks of this whole thing? Measuring and accrediting? I definitely agree that scholars need the same place to supply materials. Rather than starting new tools and platforms every week and have the scholarly community scatter everywhere. It is harder to build a solid foundation/reputation in the latter case. And stability/consistency is what we should be striving for. That makes it easier for so it is easier for universities and grant institutions to assess and acknowledge their worth.

bjoern posted on 20 Jul 08: 06:42 : Journals - the dinosaurs of scientific communication
Comments: 161


Reply to this
If I understand correctly, both of your comments concern the way to attribute credit and compare different scientists. I'm a little old fashioned in that I believe still the very best way to do that is to actually read what they have contributed, be it papers, comments, blog posts, or what have you. However, I realize that this demand becomes unrealistic in today's overflow of well-trained and talented scientists. It is also plain to see that administrators will never have sufficient expertise to understand any of our contributions. Moreover, sometimes, you only want to have a very quick look at someone you think makes interesting work, or who you think might be a competitor. For all these situations, it is good to have a quick-and-dirty look (no, not what you think!! at how many papers, comments, blog posts, trackbacks, media mentions, books, citations etc. this person has.
With an aggregator service that puts all that on a single page, this is easy. For instance, this service could work similar to friendfeed: you register the RSS feed of you publications in the central scientific database and with this one registration you get the numbers of papers, how often each has been accessed, cited, mentioned in the media, etc. Then you add the RSS-feed of your blog and you have the number of posts, visitors, comments etc. Then you add the URL to each of the blog-posts you commented on.... I guess you get the picture. It's like an online resume builder. Mind you, it still doesn't replace reading what someone wrote, but at least it'll give you more nuanced picture of the person than the current paper and impact counts.

Wobbler posted on 20 Jul 08: 07:11 : Journals - the dinosaurs of scientific communication
Guest


Reply to this
'However, I realize that this demand becomes unrealistic in today's overflow of well-trained and talented scientists.'

Not to mention the growing popularity of OA. So the accessibility to both unrefereed and refereed papers is bigger than ever and will only continue to expand.

'With an aggregator service that puts all that on a single page, this is easy.' It is easy, and I have no doubt that these services will be made available, if they are not already. But it lacks accountability. Eventually, scholars will demand a more accountable "numbers game" that involves at least some type of structured scrutiny.

bjoern posted on 21 Jul 08: 03:42 : Journals - the dinosaurs of scientific communication
Comments: 161


Reply to this
I agree that properly accounting for "access" or "downloads" is still a problem, but citations/trackbacks, etc. can easily be verified. And even access can be defined almost arbitrarily as long as all the papers sit on the same database and thus access is counted in the same way for all papers. So I tend to think that accountability is a solvable problem.
Or did I misunderstand you?

Wobbler posted on 21 Jul 08: 07:00 : Journals - the dinosaurs of scientific communication
Guest


Reply to this
I meant accountability more in the sense of credibility

I am sure there can be accurate stats on all the things you have said. But they do not necessarily represent quality. They may represent popularity. But for instance, a paper that has been downloaded 100 times, you cannot be sure of the quality of the paper. Maybe people did not like it after downloading. And since there is no metric for the quality of a blog content, how many blog posts and comments someone has is also not a quality indicator.

Citation counts have the same issue with indifference to context. But scholars mostly assume that they cite based on the fact that they find it usable in a positive sense. And there is a bigger guarantee that they have read the paper and understand what they are talking about i.e. scholars to scholars. You do not have that with views or downloads. And the issue of providing (constructive) comments is also an issue that lacks real incentive for others to do. Rating systems might work, if standardized. But even with ratings, that are clearly indicative of context, it is difficult to guarantee the proficiency of the raters, assuming we are talking about blogs etc.

bjoern posted on 21 Jul 08: 09:10 : Journals - the dinosaurs of scientific communication
Comments: 161


Reply to this
Like I said (and maybe should have emphasized in the man post already), counting can never really substitute for actually reading the material. I don't think scientific quality can be put in any metric whatsoever, because quality is always in the eye of the beholder. Believing that scientific quality can be measured objectively will only lead to false hopes and misunderstandings, IMHO.

Submit comment
Subject
Username:
Comment:

Render time: 1.0831 sec, 0.7320 of that for queries.