Openness, replication, validation and the mark of quality in research… and knitting!

Whilst knitting (so I make no claim to have comprehensive notes), I watched a great talk on YouTube by the guy who wrote this paper:

Why most published research findings are false
John P A Ionannidis in PLoS Med (2005)

– Hedge fund managers don’t trust science: how do we know which science can be trusted?

– Looks like replication is an important aspect of science, for us to recognise quality

– Negative results should also be shared and lead to acknowledgement of contribution: there is a particular bias towards reporting of positive results in some disciplines. I think he said: “the analysis planned is different to the analysis published about half of the time” amongst the 60 or so research teams who responded to the author by sending requested protocols. And those who responded must presumably be amongst the most conscientious of researchers: the implication is that those who didn’t respond might publish analysis that is not what was originally planned, in more cases.

– Published articles should have published protocols associated with them, and there are a number of top journals who have now agreed that a condition of publication for articles about randomised control trials should have those trials registered already, before publication.

– Journals might have policies (I think: is this a sign of quality for authors choosing where to publish?), but are they always being adhered to? Not necessarily!

– When small studies’ results are published, the sensible thing is to wait for a larger study to confirm the findings.

– Transparency of data is important too. It sounded like he summarised a study where some top researchers tried to re-do the analysis in 18 papers from a top journal, and they could only replicate the results properly in two articles. There were various problems with the others, which ranged from a lack of availability of the data, through use of home-made and unavailable software, to an un-interpretable description of the methods.

– There are five levels for making research more open and more replicable (and thus more validatable?):

  1. Registration of data
  2. Registration of protocols
  3. Registration of analysis plan
  4. Registration of analysis plan and raw data together
  5. open live streaming

My reflection on it all was that my very act of knitting is a metaphor or even example for all of these themes, as my knitting is a form of replication. The knitting pattern was available for download for free on the knitters’ community site Ravelry, which is like open access publication, although you can buy individual patterns there too, and there’s frustrating, out of print stuff from books and magzines, too! Also, on Ravelry you can see pictures and notes from others’ projects that use the designs. This is partly replication, but also open, post publication peer review, as the project notes sometimes point out errors in the instructions. Sometimes, designers then admit to errors and release new versions. It’s also apparent that some designers have already engaged test knitters to try to avoid such a post-publication revision (pre-publication peer review). Some test-knitters might be paid, some are doing a favour for a friend, and some seem to do it for the wool!

I had difficulty interpreting my pattern in one or two places (perhaps because I was watching a fascinating video at the same time!), and had to fall back on my experience/expertise/creativity.*  But finally, I was able to produce a very nice little top, and is that not a form of replication that indicates the quality of the original designer’s work?

* I was using a lace yarn for a top that was designed for worsted yarn, and my gauge with 5mm needles was close but not perfect, so I was destined for a few modifications. I think that this is somewhat akin to data adjustment! And if it was a really negative result, I could list it on Ravelry as an “Ugh”, so I maybe I should suggest that Nature and Science start publishing “Ugh”s, asap!

Here’s a picture of what I knitted:

baccarat

Advertisements

SAGE Publications busts “peer review and citation ring,” 60 papers retracted

Important news from the Retraction Watch blog!

Retraction Watch

This one deserves a “wjvcow.”

SAGE Publishers is retracting 60 articles from the Journal of Vibration and Control after an investigation revealed a “peer review and citation ring” involving a professor in Taiwan.

Here’s the beginning of a statement from SAGE:

View original post 3,180 more words

Curating online content and recording information sources: tools I’ve used

I’ve mentioned in an earlier blog-post that the tool I value most for this at the moment, is Evernote. But there are some other tools I’ve had a good look at:

ScoopIt is also a pretty good curation tool, and if you use it often to discover content and link it up with Twitter (there’s bound to be an IFTT recipe), you can look more original on Twitter at the same time as creating something more visually attractive and useful for yourself than you could do with Twitter alone. The problem I’ve discovered is that your ScoopIt stories look out of date pretty quickly if it’s not a primary tool for you, and I can’t vouch for it being the best place to discover content: a better way to use it might be to investigate the bookmarklet tool.

Another such tool that I’m aware of is paper.li, largely because of one particular user who picks up on my tweets and reports on them there & tweets at me to alert/acknowledge them, which is a pretty nice, social way to curate/collate content and report on it.

I used Storify for collecting tweets relating to the Finch report on open access, and I still refer to the collection from time to time. I think Storify is particularly good at collecting tweets about a particular theme, but you can also use it to collect websites and material from other sources. Apparently, Storify also has a bookmarklet tool which I would use if I intended to invest more in Storify.

I also created a collection (or two) of academic papers on EndNote when I was at Warwick, and I exported and then imported the bibliographic data into Mendeley, for future reference. The reason I don’t use either Mendeley or EndNote so much these days is really that I’m not using so much scholarly content. If I were, I’d also want to investigate Zotero as an alternative: it’s a long time since I investigated it but it has a good reputation amongst researchers I’ve met. I note that EndNote’s Desktop version seems to remain the best at re-formatting your bibliographic data into the various styles for journal publication.

I used to use Delicious for website bookmarks but when it lost some features that I valued, I migrated my bookmark collection over to Diigo. Both of these tools, like Evernote, have handy content-adding tools for my browser toolbar (bookmarklets). My Diigo collection is nicely tagged but not maintained so much these days, because I prefer the way Evernote copies the content of sites. I once spent some considerable time weeding out dead links from my bookmarks, so it seems to me better to have a copy of content for future reference, in case the original webpage is moved/removed: apparently, Pocket can also do this.

Overall, the convenience of Evernote prevails, for me. It’s apparently a “productivity” tool and not only for content curation, although that’s how I use it at present: I know it’s more powerful. (I’m sensing that “productivity” is a keyword for folks at companies who provide these tools, especially Mendeley in their recent webinar for Librarians.)

Brian Kelly’s blog post on Evernote from Jan 2014 compares it to Simplenote, explaining why he’s sticking with Evernote. And if you want to explore productivity tools further, you could do worse than looking at the Libguide from the University of Minnesota on “Digital Academic Workflow tools”.

Looking good on social media

A great tool for this is Rebelmouse which picks up on my tweets. It prompted me to consider how to include more pictures into my blog posts, recently. And now it has got me thinking about how a researcher can “look good” on social media.

I blogged recently about what might incentivise a researcher to upload published outputs online, and one of the criteria was that their research should “look good”. I also blogged recently about my mini-strategy for social media  where I state as my first goal that I want a “professional-looking” profile. So really, it boils down to the question: What do I mean by “looking good”? And what does this mean to a researcher? Here are some answers along with my tips!

Great content

This is a top priority, to use social media to say stuff that I believe is meaningful and valuable, and sometimes to be a little original, at least within the community/network I belong to. I also try to write in a way that is accessible and friendly but not too informal. My pictures are… well, secondary.

For researchers, others might create content about your work, and being present in social media enables you to engage with them to build more great content. (This is where Altmetrics becomes of interest as you can use altmetrics tools to see what others are saying about your papers.)

Other people’s work provides a content source: you can review at great length or simply re-tweet, re-blog or collate others’ content. I think people who only re-tweet are essentially curating content for themselves whilst at the same time having a social media presence: it isn’t original work, but the collating and reporting role is pretty useful to us all. Twitter is great at the reporting role & reaching people, but not so great at curating, in my opinion. Read more about curation tools in a forthcoming blog post!

Academic (and authentic)

I’m not an academic, so this isn’t my own concern but rather one that I think researchers will need to display. Having said that, I do want to show a certain level of thoughtfulness and appreciation for academic ways. The LSE blog has some great advice for researchers on using social media as an academic. The concept of social media and its suitability for “academicness” is a really big topic so I won’t try to explore it all here!

Instead of wanting to sound academic, I want to sound experienced and well-informed but also authentic, reflective and exploratory. 

Active

This is a balancing act of choosing what to share and how, and not opening too many commitments for yourself! I’m active in the channels that matter to me, but I need to make it clear to anyone who stumbles across any out-of-date profile of mine, where I am actually active. My mini-strategy describes what I’m aiming to achieve in terms of activity levels with my blog, Twitter & LinkedIn and I’ve recently updated profiles on other sites to point out that I am active on those sites.

The tool I regularly use to curate content for myself these days is Evernote. And I haven’t yet chosen to share information about that activity, socially. I suppose that I’m active enough “out there”. I don’t want to flood other people’s feeds and annoy them! But if I wasn’t active in other ways, then sharing my curation activity would be a good time-saver.

Visually attractive

I choose profile photographs carefully! I want to look  reasonably professional and competent, but also to display a little personality and approachable-ness. I also try not to change the photos too often, but to keep them up-to-date, so that folks can recognise me. I’m not using my photo as a kind of “brand”: I like that when LinkedIn displays my WordPress profile it is with a different picture of me, so that maybe piques interest in the new source but still provides recognisable-ness for those who meet me.

Twitter and WordPress also allow me to use images on my home page/wrapper for my social media activity, and indeed in my content, along with formatting options for blog posts on WordPress. There’s a lot more that I could do on this front, so I’m really pleased that Rebelmouse provides a visually attractive view of my content with very little effort from me. Even though it’s just the same content as you would find on Twitter, I think it looks nice so perhaps I should promote it more… like mentioning it here 😉

Responsive

This is more important to me than “social”. I have been thinking quite a bit about whether I should tweet thanks for people who re-tweet my content, and how to respond socially online. I don’t think I’ve got the perfect protocol but as a minimum, I do aim to respond to anyone who tweets directly at me, either publicly or privately, and indeed to any comments on my blog. Ideally, I would also do this in a very timely fashion, but this is subjective and I’m often busy so I’m also a fan of “better late than never”! Also, I’m aware that some tweets at me are really more of a courtesy, an attribution or a citation, and so they don’t require a response except maybe a private thank you.

Thought-through & Linked-up

So I’ve got a blog and a twitter feed and I use LinkedIn and I have an ORCID and presence in Rebelmouse, on Slideshare and all sorts of other profiles besides. How do these all link up to each other and relate to each other? How can I make it easy and efficient for the person who wants to find out about me, to navigate through all of these profiles? I keep mentioning that I only update LinkedIn and that I’m active on Twitter & WordPress, so I don’t need to feel guilty every time Academia.edu tell me that another person has discovered my profile through a Google search. I’ve laid a trail! I’ve a feeling that ORCID is going to be the answer… at least, in the academic setting.

Popular

Tricky, this one… I mean, how do I look popular and why would I want to?! If I know that I’m creating good content, so why do I need the validation of popularity? But the opinion of one’s peers matters, and for researchers the attention from the wider community is an indicator of where to look for impact.

I do make it clear which measures of my social media activity I want to measure, in my mini strategy and this is a space I’m still watching. Basically, I’m interested in what measures are available for and from the tools/channels that I actively use, and whether they mean anything to me depends on what I’m trying to achieve.

How do I use this information to look popular? I don’t, but I see that LinkedIn, ResearchGate and other such sites are publishing attention numbers for me. Since a lot of the measures are new and people don’t fully understand or trust them, this is not a big deal at the moment because I don’t think folks are looking, but it might be in future. Depending on who is looking and whether or not one cares!

I expect that there are lots more aspects to “looking good on social media” that I could consider, but that’s my round-up for now!