Friday, July 6, 2012

What the Numbers are Telling Us

We’ve been rolling things out in a big way and in my opinion, it would be a huge fail if we didn’t use this momentum, attention and summer code partyin’ to learn something about the stuff we’ve built - specifically for me and my team, about the learning projects. Enter The Survey. We have a survey that is available upon completion of a Thimble project* that asks some pretty basic questions about level of previous experience, fun, learning and if/where people got stuck. It’s not meant to be the most robust thing ever, but instead to do some temperature gauging during this first wave of users/learners.

*Note: we have another version of the survey (duplicate except for the word “Thimble”) for the DIY projects, which are the projects that point people out to different sites or tools. But so far, there are very few reposes there (~8) so this analysis is focused on the Thimble projects.

We have almost 150 responses so far, which is way lower than the estimates of folks using Thimble so far (more in the thousands), but not a bad response rate given that the survey link is a little awkwardly presented - right under the copy-your-link-to-your-finished-project field in the ‘Publish’ flow. 

Here are some highlights on what the numbers are telling us so far…

Findings

1) We’ve gone global

People are doing projects all over the globe. From the event registrations, we knew that we had events in 67 countries across the summer, but pretty cool to see this much activity in just a couple weeks.

2) Lots of existing experience.

We have a pretty even distribution of webmaking experience so far, with a slight advantage to those with more experience. This is a bit surprising since the Thimble projects are more targeted at entry level to intermediate, but its likely that some of this is due to the fact that we also just launched Thimble and have a bunch of people that are exploring it just to check it out. I’d also love to believe that there are mentor/facilitator/instructor type of people checking it out for the purposes of using it to teach other people these skills, but that question didn’t make this round of the survey (hindsight!!), so we’ll get those numbers in version 2.0. See below for a cross tab analysis of how this factor influenced other ratings.

3) Fun!

A whopping 74.8% thought that the project was fun and of that 39.4% said super fun. We are aiming for personally engaging, interest-based experiences so the sense of fun is an important piece. 

4) There was perceived learning.

63% reported some learning, with 25% reporting that they learned a lot. This is, of course, self-reported learning, not hard-core assessed learning* - but at this stage, again, to gauge the temperature of people’s experiences, I would say this is a pretty solid number, especially given that fact that over 50% came into the project with some or a lot of webmaking experience.

*Note: On the hard-core assessed learning: 1) we are building in more assessments that will use the work as evidence to validate that certain skills are demonstrated; but 2) all that said, again, ultimately we are after interest-based webmaking with some learning that happens in the process, so if people are engaged and able to make the things that they want to make, then I would call that a success without all the pre-post data hubbub…but we’ll do some of the latter as well next year.

5) People reported getting stuck.

One of our core design principles is to design for graceful failing - or said another, more direct way, don’t let people fail. By the numbers, it would appear we are there yet since 50% of people reported getting stuck. However, looking through the explanations, while there were a few that were overwhelmed by the code, it seems like most people were able to work through their stuck point:

"The </body> and </html> codes were missing and the webpage was showing errors and did not want to complete the project successfully. I figured it out at the end."

"I didn’t put an end bracket in the right place for the hyperlink.  Which is good, because it meant that I had to go back and figure out why it wasn’t working."

"In the beginning.  I had to read more carefully than just skim read."

That’s actually pretty promising then, because the projects were challenging but the learner had enough to solve the problem. It shows trial and error and tinkering which are also really important aspects to our learning philosophy.

There was a trend of ‘stuck’ responses about the publish feature which we need to investigate some more. Right now, when you click “Publish”, you get a URL which you can copy and share through Twitter, Fb, email, whatever but that doesn’t seem to be resonating with everyone. Responses included “I don’t think the publish worked right for me” and “Copy and paste WHERE??”. Apparently some people don’t understand that flow. So we’ll look into that more. 

6) Cross Tab == Cool

Of the people with no webmaking experience:

  • 70.9% reported having at least some fun, with 41.9% reporting a lot of fun.
  • 66.6% reported at least some learning, with 43.3% reporting learning a lot.
  • Only 10% (3) reported not learning anything at all.
  • 58.1% reported getting stuck.

Of the people with some webmaking experience:

  • 86% reported having at least some fun, with 43% reporting a lot of fun.
  • 68% reported at least some learning, with 15% reporting that they learned a lot. 
  • Notably, 0% reported not learning anything at all.
  • 36% reported getting stuck.

Of the people who came in with a lot of webmaking experience:

  • 34% though it was super fun, and 65.8% reported that it was at least kind of fun.
  • 48.7% reported some learning with 23.1% reporting that they learned a lot.
  • 35% reported that they didn’t learn anything at all.
  • 56% reported getting stuck.

6) Some great suggestions

How to make it more fun:

"More animal parts."

This was the number one request. The Zoo was definitely one of the most popular projects and people wanted more options to create more animals. I’ll take that as a notch on the side of ‘this was fun and engaging’. Also points to needing more projects with rich content and topics like this one. 

"When a project asks you to replace one image with another, it would be helpful if it supplied some links to alternate images."

This is an interesting suggestion. On the one hand, its a great idea and something that’s relatively easy to do, but on the other hand, again, part of the learning experience, and of web literacy as a whole, is being able to use the Web to find things that are interesting to you. I think what we definitely can do is provide better instructions on places to find images, how to properly attribute them, etc.

"More cat integration."

Well, duh.

How to foster more learning:

"Maybe a toolset of html elements and drag and drop as well for different html elements."

There were a handful of comments around this same topic of making it more WYSIWYG like or drag and drop. We intentionally built it this way, so that it is exposing the real code and learners/makers have to get their hands dirty in the code to make something. I think the potential for learning relevant webmaking skills is greater working with the real code…that said, we have to balance that with ensuring that the barrier is not too high. We are planning on doing some experiments with some different approaches with different abstractions with our DIY (read:non Thimble) projects later this summer.

"Maybe give the user an end result to aim for.  Can you make your document look like this [screen grab]  or create a page with a story or something and they have to create the next page."  

This is a good suggestion and we’ll consider it moving forward.

"The explanations when you click "read more" are WAY above a beginning coder’s head."
This is great insight - we’re pulling the hints from the MDN, which is awesome, but typically targeted at web developers. So we’ve started a “simple MDN" to start to write more basic descriptions. This is a community project - jump in and help out!

Takeaways:

  • We’re on the right path - the projects (+ Thimble), and thus this approach to learning, are providing some engaging and fulfilling experiences for a majority of people. 
  • Even though some of the feedback pointed to Thimble and the projects being too advanced for beginners, the numbers show that they reported the same amount of learning as those with some experience. The beginners reported it being a little less fun and got stuck a little more, but 43% reported learning a lot. I think even though the projects might feel overwhelming b/c of being dropped into the code, it seems like it was actually beneficial for people in terms of learning.  We’ve already started working on making Thimble even more accessible to entry level folks by integrating the X-ray Goggle functionality (you can now click on the right side preview pane and see things highlighted in the code on the left) and creating some more starter projects that focus on one or two elements at a time. It’s also worth noting though, that a lot of people with some and a lot of webmaking experience also reported learning and fun. I would guess that number would go up even more if (when) we have templates that are more targeted to various audiences and skill sets. So we’re on to something here. 
  • We need more projects that provide more interest-based access points for people - things like the Zoo project with rich content and compelling topics. We really want the community’s help with this so if you have an idea for a Thimble project, tell us about it.
  • We need more help and hints baked into the projects so that the barrier to entry is lower and less people can get stuck. 
  • We need to revisit the Publish flow to make that work for more people and (bonus!) use it as a teaching moment for those not used to sharing things on the Web.
  • We need more cats.

-E

Wednesday, June 27, 2012

Weekend of Learning

Last weekend, we launched the Mozilla Summer Code Party with the Global Weekend of Code, which was a resounding success. Over 100 events across over 30 countries, with over 1000 people learning to code. And there are at least 4x as many event scheduled for the rest of the summer. Now that’s a party.

Most of the partiers used our new webmaking tools: Thimble and Popcorn Maker. As I’ve talked about before, we have developed a number of projects and templates that sit on top of these tools and give learners an interest-based access point into some webmaking of their own, and of course, some learning in the process.

My role last week was on the Party Squad, monitoring the social medias and retweeting / posting / sharing all the awesome stuff that came through. It was an inspiring place to be in - just amazing to see all of the webmaking that was occurring. I wanted to reshare a few of the projects here. I know you’ve probably seen them all via our constant stream of blogs, tweets, posts, etc. last weekend but it’s worth taking the time to dig a little deeper to expose the learning that’s actually occurring underneath. 

Sell it!

Source / Built from the HTML Wrangler Project

This one may seem simple at first - it was built from what’s intended to be an entry level project for people that are total newbies to HTML and might be intimidated by a lot of code right away. The project introduces the basic concept of tags and comments, and then some text handling tags like <p>, <h1>, <h2> and linking. So there is actually a lot going on in there. And this maker not only mastered these HTML basics, but also added in text that was relevant and interesting to him/her, so that it became something that s/he made that mattered. And learned some HTML in the process.

Map

Source / Built from the Hack a Map project

This is another entry level one that requires the learner to fix a bunch of issues in the code. The fact that you see the page without big errors or floating <’s means that the maker successfully fixed all the issues. This requires you to understand the concept of open and close tags and other HTML syntax details, and then also introduces you to images and text handling. This particular maker decided to add a video to the page. The caption says “i put up this video cuz i figurd that lots of people would like it”. *smile* Thimble currently requires that videos be handled through iframes (because of security issues with the <video> tag) so this wasn’t a super easy thing to figure out how to do. It’s possible the maker pulled it from another project, but isn’t that the beauty of the Web? Of course, there are design aspects like how to surface the video above the map images, but that wasn’t a core part of this project and the maker was still able to hack it to add his/her own content. 

Bakery Bash

Source / Built from the Bakery Bash project

This project focuses on image handling, and helps makers learn the <img> tag, how to find and use image URLs and how to resize images, all under the theme of creating an invitation list for an awesome party with Loaf Cat. (If you don’t know Loaf Cat, ask the internetz) Makers are also asked to work with hyperlinks and link out to secret guests that will attend as well. This particular maker made this his own by using pics of NBA stars and cartoons (and a tomato?) and linking out to super heroes and characters. S/he even hunted down and edited the intro message to read “Porta Pottys Rock”. Silly? Yes. Legitimate HTML skills and interest-based making? Absolutely.

A New Creature

Source / Built from the London Zoo’s Awesome Animal Builder project

This is one of my favorite projects - it’s by the London Zoo and it asks makers to read about several endangered species and then create their own creature by moving HTML images around, and then wranglin’ text and lists to describe the new creature. There’s a lot on the page too, so you have to get comfortable sorting through the code to find the parts you want to move around and edit. And on top of all the awesome webmaking skills required, there are some great learning opportunities around conservation and biology. You can see here that the features and description of the new animal are pulled together from features of the real animals themselves, so the maker was internalizing those details about the real animals and mashing them together for the new creature.

TMNT Zombies

Source / Built from the Get Off My Lawn! project

This one is worth calling out because it requires some pretty advanced CSS positioning. The game involves using %, em and pixels in CSS to move objects to block zombies that are invading your lawn. Not only did this maker demonstrate positioning skills by moving everything to the bottom right corner, but also hacked the game completely and swapped out the zombie images for images of their own. I think some of the most compelling learning/making that we’ve seen is people that not only complete the challenge, but completely hack the process and page to make it something that they care about. Brilliant.

Gallery

Source / Built from ??

This one was just awesome. One of the things on the Thimble roadmap is a gallery so that people can share the things they built with other webmakers. Well this particular enterprising maker just used Thimble to build their own gallery highlighting the links out to the projects that their group made. It’s meta and it’s awesome. Maybe they started with a project and made it from there, made they just used the plain ol’ editor. Doesn’t matter how they got there at all, they knew enough or were able to hack together enough to make a page that suited their needs. Did I mention that I love the Web?

Robot Popcorn! (mmmmmm)

Source / Built with the Popcorn Robots template

Popcorn is our awesome web native film hacking initiative/technology/tool set and we just released an early version of Popcorn Maker that let’s makers build stories through video augmented by information that is relevant to them/their story. In this video, the maker adds a map of London and some pictures of their own, as well as some pop-up captions as the story progresses. This template also allows you to add comments that the robots will read in a robot voice. Very fun, but also some heavy duty skills required around video wrangling, open video, procedural storytelling, storytelling in general, information gathering and curating, etc. I particularly love that the maker started with a screen that says “I have no idea what I’m doing” and yet was able to add a bunch of personal touches and hacks to the video. 

Summary

So in summary, despite some of the surface simplicity of some of the projects, there is a lot of learning and web literacy behind them. I think we should be pretty proud of that. Of course, we don’t yet know what level of skills people are coming in with, and we hope to build more of that into the experience so that we can truly measure the learning. Anecdotally, we heard lots of people say “I am learning so much” and “I can’t believe I just did that”, and we have some basic survey results that I need to analyze and share soon. But above and beyond that, its hard to say right now how much people learned in a traditional pre/post kind of way…but from my viewpoint, we got 1000+ people engaged in making stuff about things that they care about online, and demonstrating a lot of webmaking skills in the process. I’ll call that a victory.

And with that, I leave you with this:

Source