Monday, August 27, 2012
The traditional methodology for studying innovation in education may have been adequate at a time when only small changes were possible, when in fact one did change an aspect of the mathematics curriculum and keep everything else the same. But we need a different methodology altogether when we envision radical changes in education. Papert, S. (1990). A Critique of Technocentrism in Thinking About the School of the Future. M.I.T. Media Lab Epistemology and Learning Memo. Cambridge, M.I.T. Media Lab.
Monday, May 23, 2011

Badge Pilot - Phase 1 - Evaluation

I have several posts that I have been meaning to do over the last few weeks but there has been so much going on that I have been remiss. So expect a flurry of posts (or a few at least) from me in the next few days.

But to kick things off - we have completed the first phase of the P2PU and Mozilla School of Webcraft Assessment and Badge Pilot. It’s a mouthful and rightly so, since it was full of a lot of very cool stuff. These previous posts here and here give some background on the pilot but to quickly summarize, the pilot consisted of new assessments and badges for skills, values, community interaction and participation in the School of Webcraft. These badges are meant to be an alternative pathway to accreditation and credentialing that SoW community members can earn to demonstrate skills and then share with stakeholders like peers, formal institutions or potential employers to network, progress careers and/or find jobs. 

This initial phase of the pilot included 14 pilot badges (ones designed by us and aligned with specific skills, values and community behaviors relevant to web development) and a bunch of participation badges that came with the core system we were using for the dedicated badge environment, OSQA. The latter were meant to encourage and guide participation in the site as a question and answer forum. Since we were not using it as a true Q&A system, but instead simply leveraging the functionality to support the assessments and badge issuing, many of the OSQA badges were not relevant or achievable by users but some were, such as First Responder, Popular Answer, Editor, etc.

The full evaluation report is available here, but for those that don’t want to read a (titallating) 17 page report, here are some highlights below:

Goals of the Pilot:

  • Build proof of concept for a badge system for web development training 
  • Create and roll out initial taxonomy for types of badges 
  • Develop and roll out assessments that fit the peer and interest-driven learning environment
  • Get initial feedback and reactions from the community
  • Learn as much as possible that can be applied to later versions of the pilot or integrated solution
  • Prototype and pilot the open badge infrastructure

Key Findings


  • Overall: Participation was lower than expected, with only 52 registered users (in the dedicated badge environment) and of those, 21 active users (earned a badge, assessed work, etc.) We feel there are a couple reasons for this low participation: 1) communication and 2) lack of integration. 
    • Communication: From a communication perspective, this pilot was intentionally tightly controlled, mostly because we wanted to make sure that course organizers were prepared and that we had assessments closely aligned with relevant courses to encourage more active participation and assessing. But this meant we only touched a small portion of the wider Webcraft audience and did so through course organizers who rightly passed the message along (if at all) on their own schedule, so traffic and attention was intermittent at best. We intend to communicate to participants more directly moving forward so that we can ensure that they are fully aware and have all of the information (including why these badges are worth their time). 
    • Integration: On the integration side, as mentioned before, we used an OSQA system that is separate from the P2PU platform and thus required learners to log into a separate site (we built it so that they could use their P2PU account to reduce this issue but it was still a separate action they had to actively take). We plan to integrate the assessments more directly into the learning environment and experience moving forward to make it more seamless.


  • Overall: Feedback on the assessments was very positive and it seems like we are on the right track with authentic, relevant challenge-based assessments.
  • Types: Of the different types of assessments, we really only saw examples of peer assessment, which again were encouraging, with examples of constructive feedback and reworking of submitted work, as well as learners discussing how much they learned from the process of assessing peer work, but there was some struggle with ensuring that there were peers to assess submitted work. That incentive structure is still a gray area for us - we need to figure out how to attract quality people with the right skills to assess submitted work across the system. We will be exploring this more moving forward. We did not have any submissions for expert-level badges (see below) so we did not see any guru assessment, but hope to in subsequent rounds. There was some stealth assessment in the OSQA participation badges, but none of these were directly tied into the learning. 


  • Overall: The main feedback was that people wanted more badges to cover more skills which we totally expected and plan to build out further as we move along. 
  • Types: There was a good overall response to the types of badges we had and people felt it was important to have a mix of hard skills and soft skill badges, which we also know are important to badge consumers like potential employers, so we will continue down this path. 
  • Levels: There were no submissions for the expert badges which makes some sense given that all of the courses were entry level with some pushing into intermediate for some skills. We do feel the expert level badges are important to have as a goal or benchmark for people to work towards, but we will need some more advanced courses and active advanced community members before we will get more traction on the expert badges. 


  • Prototype: We were planning to run the first phase of the pilot with a prototype of the open badge infrastructure (OBI) that would allow us to port the badges from OSQA into the infrastructure, and then display them on other sites including the P2PU profile. But due to development cycles on both the OBI and P2PU platform, we decided to push this to the end of the second cycle, which will be in late June. 


Overall, the initial phase of the badge pilot was a positive step in the right direction for our assessment and badge work. We had initially planned on starting with 2 badges and ended up with 14 badges which allowed us to explore more types of assessments and badges in this phase. While participation was low, we learned a lot that we will apply to the next rounds in terms of communication and outreach, and have identified areas that need dedicated focus like driving more peer assessors to be actively involved.

Revisiting our goals, we met most of them by building and launching a quality proof of concept badge system, which included a basic taxonomy for badge types and various assessments approaches built around peer learning. We got some great feedback and interest from the community, as well as other stakeholders, and have some solid direction around future versions of our efforts. The only goal that we were not able to meet was the prototype of the badge infrastructure, which again, was pushed because of delayed contingencies on the development sides, but is targeted to roll into the second phase of the pilot. This will allow us to port the badges out of the OSQA environment and into the P2PU profile to give learners more control over sharing and using the badges in other contexts.

Overall, we feel that we produced a good proof-of-concept to build off of moving forward, and initial responses and observations indicate that it is important and valuable to continue to move in this direction.

Phase 2

We are rolling all of the stuff that we learned from this pilot into the second phase of the pilot which will launch in early to midJune and run through July 2011. Look for another blog post shortly detailing the plans for that phase of the pilot.

Over and out,