Monday, September 30, 2013

CSOL - How Did We Do?

The mornings are crisp and the leaves are turning rich colors here (you should come!)…it’s officially Fall. We’ve wrapped up the Chicago Summer of Learning and have had some time to reflect on how it went. 

It was helpful to start the debriefing process with a look back at what our original goals were back in what feels like years ago, but was only February of this year. Looking at the numbers through that lens has helped us start to thoroughly evaluate this summer. We, of course, want to supplement the raw numbers with more real feedback from youth and organizations and research, all of which is forthcoming.

THE GOALS

  • Challenge all Chicago youth to learn over the summer
  • Call on all youth serving organizations to collaborate and support learning initiatives over the summer
  • Highlight opportunities in the city related to STEAM

Internally, we also aspired to:

  • Reach as many youth as possible, especially low-income or minority kids, get them to earn at least one badge (goal: 100,000)
  • Provide pathways for youth to encourage additional learning and ‘leveling up’ (goal: 1000 kids leveling up)
  • Get as many participating orgs as possible (goal: 100)
  • Make the badges ‘worth something’ (goal: avoid all ‘participation’ badges, engage CPS and local institutions to recognize badges)

THE NUMBERS

The CSOL numbers:

  • Total orgs issuing badges: ~125
  • Total of badges available for earning: >1000
  • % badge types: 3% participation, 92% skill, 5% achievement
  • Total badges issued: ~150,000 (note: number includes some badges that are still in the process of being issued)
  • Total badges claimed to date: ~50,000
  • Total badges wish listed: >5000
  • Gender breakdown: of reported 53% Female, 47% Male
  • Race breakdown: of reported, 56% african american, 7% white, 6% asian, 29% latino/hispanic, 2% other
  • Age breakdown: of reported 26% under 13, 60% 13-17, 9% 18-24, 5% over 24
  • Average time on site: ~6 minutes
  • Total youth who earned city-level badges: ~600
  • Total youth who completed challenges, or successfully leveled up: ~300

THOUGHTS ON THE NUMBERS

What Went Well

When I look through these numbers, the initial reaction is wow. We can go down the goals and check them off across the board. Over 100 orgs and 1000+ badging opportunities represents not only a lot of coordination and collaboration that I am still shocked that we all could pull off in just a few months, but also an incredible coverage across the city and a heck of a lot of learning opportunities. The badges themselves were really great - with only a few orgs issuing badges simply for participation*, but most badges focused on skill development with robust assessment and criteria. The Chicago organizations really stepped up and came into this summer and this new concept of badging with open minds and a dedication that was astounding. Huge kudos to all of them - they are the ones with the content, experiences and relationships with youth that matter. The badges are just the recognizer on top of that learning. That said, the badges were very strong and DePaul University has already stepped up to say that they will accept some of the badges for credit in their institution, and CPS is working to define rewards and advancement that they can offer for the badges.

(*Note: this is not to say that participation badges are *bad*. There are many reasons why rewarding someone with an initial participation badge could be valuable, including motivating additional work, recognizing time commitment, etc. But we wanted to hold ourselves to a goal of having the majority of the badges tied to a deeper assessment, which they were)

Orgs issued over 100,000 badges this summer - that represents a lot of learning and a lot of youth. And there is diversity among those youth. Time on the site shows that youth (and parents) were spending time searching for things to learn or digging into the self-paced challenges that were available for youth on the site, and the wishlisted badges show that some youth were setting goals and into learning more.

Even though we had set a goal around 1000 youth leveling up, I was pretty worried about this one given the fact that many programs started later in the summer and often required multiple weeks of participation and learning before a badge was issued. In order to level up, kids had to not only earn a badge, but earn several within the same category (Science, Technology, Engineering, Arts or Math). This most likely involved doing a combination of local learning programs and online self-paced challenges. So I was pleasantly surprised to see that almost 600 youth got to the city level badges and had access to the challenges, of which almost half of those completed challenges, which were not easy and required commitment and hard work. 

What Could Have Gone Better:

The part I am less wowed by is the disconnect between the badges issued and the amount that have been claimed. This number is less stark as it looks but only about 50% of badges issued were claimed and added to Backpacks. To explain this a bit, badges were issued one of two ways: 1) through an issuing tool built by Mozilla called OpenBadger that had an admin interface for org representatives where they could issue badges to youth emails, the youth or their parent had to click through the link in the email to accept the badge and have it show up in the youth’s backpack.  2) Organizations with offline learning experiences could also hand out paper badges with unique claim codes printed on each. The earning youth then needed to log on to the website and enter the claim code to see it in the Backpack. 

There is still some research to conduct, including some surveys and focus groups that folks are conducting to learn about what worked and didn’t work for kids this summer, but its likely that the lower number of badge acceptance most reflects breakdowns in the overall user experience. Many youth haven’t checked their emails, or didn’t click through links of the original, text-only emails. Some might not have had access to a computer at all to claim or accept the badges. Others may have lost the paper badges - its likely there are lots stuffed down in corners of physical backpacks - before they could claim them. Another option is that kids just didn’t care about the badges, or didn’t understand the value, so that’s something we’re exploring more. 

While, as I mentioned, I was pleasantly surprised by the number of kids leveling up, we did not hit that initial goal of 10% of kids. Some of this can also be attributed to user experience design, some of this was the relatively short amount of time to learn and earn and level up. I also think there is an opportunity to design and promote challenges in a way that even more connects directly to youth interests and real opportunities. I’d love to see more challenges next year that show youth completing real projects for their neighborhoods or the city of Chicago. Or developing skills that are directly tied to career pathways, jobs and internships.

In general, the numbers were again, impressive, but I think we can do more. Through a combination of improved tools, better user experience design, more and better training and more marketing and communication, I think we can bring more organizations on board, reach more youth and see more accepted badges.

WHATS NEXT

There is a big assumption in that last statement: that we are doing another CSOL in 2014. I think there was a resounding consensus in a recent meeting in Chicago that the results of this year’s summer were compelling enough to demonstrate the value and promise of this work. So there will be a CSOL 2014, and in fact, maybe something more on the lines of CYOL (Chicago Year of Learning or something like that). Lots of details to work out on the goals, roles and specifics for that work, but given that we are building from a foundation and set of findings, as well as starting those conversations in September instead of a few months before launch, to me, means the possibilities are enormous and exciting.

We’ve learned so much this summer about what it takes to build a badge system at this level, how we can better the tools, who to invite to the table from the get-go, and more. Lots of work to do to get to a 2.0, but there are a lot of pretty clear marching orders. Look from more blog posts from me and the team on these lessons learned and thoughts for moving forward on our pieces.

Also, turns out that Chicago is and will continue to be a hub for innovation and advancement with badging, but another exciting development is that more cities are interested in building something similar to CSOL in their neighborhoods and I think we’re going to see not only more geographical reach, but also a significant amount of more learning and youth impacted in less than a year.   

Thanks,

Erin

Notes

  1. worldofe posted this