Educational Scholarship: Measuring the Impact of FOAM

Educational Scholarship: Measuring the Impact of FOAM

I recently asked a prominent EM researcher “Does FOAM (Free Open Access Meducation) have academic value?” I don’t need to paraphrase his answer as it was quite concise: “No.” While he was elegant in his bluntness, you will probably not be surprised to hear that I disagree. This response led thinking and discussion that culminated at CAEP13 after I attended the Consensus Conference on Educational Scholarship.

What is Educational Scholarship?

The Consensus Conference was held to provide feedback on a group of educators’ definition of and vision for educational scholarship in EM in Canada. Its tenets were consistent with those in the literature (thanks to both of my reviewers, Teresa Chan and Michelle Lin, for pointing me to these references) and focused on three pillars: public availability, peer review and how well others are able to use it. For more literature on this topic, consider reviewing Boyer’s (1990) model of scholarship. Some prominent definitions include:

“We develop a scholarship of teaching when our work as teachers becomes public, peer-reviewed and critiqued, and exchanged with other members of our professional communities so they, in turn, can build on our work.” (Shulman, 2000)

“Educational scholarship refers to any material, product or resource originally developed to fulfill a specific educational purpose that has been successfully peer-reviewed and is subsequently made public through appropriate dissemination for use by others.” (AAMC, 2009)

“The systematic investigation of an issue which results in the creation of a product presented in a form that can be reviewed by peers for quality, and publicly disseminated for others to learn from and build upon.” (CAME, 2011)

FOAM clearly meets two of the three pillars with post-publication peer-review being the arguable criteria that makes or breaks the case for FOAM as educational scholarship.

Why does Educational Scholarship matter?

I started this blog primarily to meet my own educational objectives. However, over the last 7 months (time flies!) it has morphed from a curious educational hobby to something that I have become passionate enough about to spend several hundred hours on. I’d like to think that I am creating something that has worth beyond its intrinsic value as a learning tool for me.

I recall twitter discussions from many moons ago outlining the academic acknowledgement (or lack thereof) of the big producers of FOAM. If nothing has changed since, I believe Michelle Lin of ALIEM is the only person receiving substantial acknowledgement for her work (she holds the UCSF Academy Endowed Chair in Emergency Medicine Education). Am I better off plugging away in clinical research or systematic reviews?

Why should FOAM be considered Educational Scholarship?

I think FOAM should be considered educational scholarship because it is a novel method of knowledge translation that incorporates edutainment, engages a broad audience and allows for robust post-publication peer-review. Research and knowledge translation are two sides of the same coin: while FOAM would be nothing without the research it is based on, research is useless if it is not put into practice. Lauren Westafer and Scott Weingart delved into FOAM as knowledge translation  in much more depth in (Don’t) Mind the Gap and EMCrit Wee -Tacit Knowledge and Medical Podcasting, respectively.

As mentioned above, FOAM meets all of the criteria with the arguable exception of peer review. I know I’ve beaten that topic to death elsewhere (see previous posts A Commitment to Pre-publication Peer ReviewArguments for a Journal of FOAMFOAM: A Market of Ideas and Crowdsourced Instantaneous Review: The Peer Review of FOAM) but I need to open this can of worms once more because it led me to consider the use of impact and reputation as markers/surrogates of peer review.

During the consensus conference at CAEP13 online publication on websites like this one was dismissed from consideration as educational scholarship because “anyone” could post “anything” and it would be remiss to let it stand as scholarship with no publication standard. Leaving the problems associated with classical peer review aside, I think this is a fair point. Certainly, there is great variability in the quality of information found on FOAM sites and someone publishing poor quality work that nobody reads should not be lauded for their “scholarly activities.” However, there was also acknowledgement at the Consensus Conference that post-publication peer review was an important, but often absent, component of educational scholarship.

Following the conference I spoke to some of the attendees about the substantial post-publication peer review that many of the popular sites and podcasts enjoy. Scott Weingart addressed this on an EMCrit Wee -Tacit Knowledge and Medical Podcasting saying:

“If I get anything wrong or say something that (listeners) disagree with they write me or put show comments on and so I actually have a more robust peer review on my podcast than I ever would on any publication I would place in conventional journals.”

He noted that this robust review is primarily a result of his large and vocal audience. Other FOAM resources vary in the size of their audience on a spectrum of One (themselves) to Weingart (who doesn’t listen to EMCrit?). Why does he have such a big audience? Because his peers have spoken, deemed that he publishes good stuff and spread the word every time they end a statement saying “I heard about it on EMCrit.” With everybody listening to his words, they are subject to heavy scrutiny. The impact that his podcast has results in its robust peer review and because his work has survived this scrutiny his reputation has grown.

Presuming (perhaps prematurely) that a robust post-publication peer review process is sufficient to qualify as educational scholarship, a FOAM resource’s value as educational scholarship could be determined with surrogate measures like impact and author reputation.

How can we measure the Impact of FOAM?

The idea of measuring impact to determine the value of someone’s scholarly output is not new. The h-index and related metrics are used in academia to for a similar purpose: to quantify the impact of an academic’s publications rather than just their productivity. The h-index as defined by Wikipedia is:

“a scholar with an index of h has published h papers each of which has been cited in other papers at least times. Thus, the h-index reflects both the number of publications and the number of citations per publication.”

While these metrics are arguably a big improvement over systems looking solely at the number of publications, they are still criticized for many reasons. In particular, defining a single metric may result in too much emphasis being put on a single measure and there is a potential for ‘gaming’ the system by taking advantage of how it works.

As FOAM is criticized for its ease of publication, impact and/or reputation metrics would help to quantify the value of FOAM contributions. Greater impact/reputation would presumably result from contributions with greater scholarly value. While there would be no easy way to develop such a system, the idea has been discussed as a new model of peer review for the classic scientific literature and there are models that are functionally related. For example, a reputation system is used by Wikipedia that can accurately predict the quality of the contributions of its editors and Alexa web analytic software is able to rank websites based on their traffic and number of inbound links (citations?). Additionally, as I outlined in FOAM: A Market of Ideas, I think there is an argument to be made that the prominence of FOAM serves as a surrogate measure of their quality with the best work being shared the most often in social media and attracting more readers/listeners.

While there isn’t a perfect measure of impact in classical academia, creating a measure will be even more difficult in the multi-modal world of FOAM. How could podcasts be compared with blogs? Should they be? What about apps?  There is huge heterogeneity in this subset of the academic world – thanks to Michelle Lin for raising some of these questions in her review of this article.

Conclusion

Where do we go from here? Stay tuned this week for an experiment that I anticipate will be widely criticized, I will attempt to quantify the impact of some popular FOAM sites. Good idea? Bad idea? If you have any thoughts (or if you disagree with me completely regarding all of this), please comment, e-mail or tweet!

I’d like to thank Rob Woods, Brian Rowe, Jonathan SherbinoTeresa Chan, Joseph Bednarczyk and Ken Milne because this post represents the cyclone of discussions that I have had with them.

Thanks as well to Michelle Lin and Teresa Chan for reviewing and providing feedback on this post.

Brent Thoma, MD MA
Editor of BoringEM Simulation Fellow at Massachusetts General Hospital and Harvard Medical School Emergency Medicine Resident at the University of Saskatchewan
Brent Thoma, MD MA
Brent Thoma, MD MA
Brent Thoma, MD MA
Brent Thoma, MD MA

Latest posts by Brent Thoma, MD MA (see all)

  • http://emergencymedicineireland.com Andy Neill

    Cheers for the great thoughts Brent. I have doubts that we’ll ever be able to define impact that accurately. We’re not doing it particular well in the traditional model so I imagine we’ll not be able to come up with a good model for FOAM either!

    A lot of teaching (which is the focus of FOAM) is about persuasion, rhetoric and charisma – that’s part of what being a good teacher is. It’s also a big part of being popular, so by nature there’s a bit of a conflict there – is the biggest thing on FOAM big because it’s good or just because of the charisma, charm and rhetoric?

    While it would be nice to get some form of academic recognition for the hundreds of hours I put into this stuff, i have pretty low expectations that ever will be recognised formally. If I was to do this purely or academia I expect to be pretty dissapointed so my motivation remains largely 1) it’s fun 2) i get to meet good folk 3) i learn a lot.

    • http://boringem.org Brent Thoma

      Thanks for the comments!
      I guess my hypothesis is that things get big in FOAM without both the charisma and the quality. In fact, I’d expect quality to be held to a higher standard as a result of popularity. That said, it does not seem like that is the case in the academic world with prominent journals seemingly able to publish poor studies without a loss of influence.
      Quantifying impact with great accuracy may not be possible… but I still think we’d be further ahead if we give it a try so I’m going to give it a go in my next post! I’ll include your site.
      -brent
      PS – completely agree with your motivations!

  • http://want2bemd.tumblr.com Danica

    Hey Brent,
    Great post. From my time here in Edmonton, it seems the department of surgery at the U of A does recognize FOAM (even if they don’t call it FOAM specifically). There is a special “endowed chair” in surgical education, and while I don’t know everything that it entails, I can tell you that the current one spends a lot of his time creating high-quality resources for free. (I think he may get a budget from the university to do so, but a very small one, and they are struggling a bit with how to keep paying for the creation of more resources.)

    Also, in one of their year-end reports I read, in addition to having hours taught and students taught, etc., they had “podcasts produced”. So the faculty here are recognized for their work. I think it would actually be harder to NOT recognize them for it, since their episodes have been downloaded so many times in something like over 100 countries!

    Surgery 101 is expanding into videos, virtual cases, and more. I hope when the virtual case I saw on Wednesday is completed I’ll finally be able to remember the difference between transudata and exudate….?!?

  • / Gregor

    Great idea – bring that experiment on! ;)

  • Pingback: The Social Media Index (SM-Index): A Pilot Project - BoringEM | BoringEM