Friday, July 6, 2012

What the Numbers are Telling Us

We’ve been rolling things out in a big way and in my opinion, it would be a huge fail if we didn’t use this momentum, attention and summer code partyin’ to learn something about the stuff we’ve built - specifically for me and my team, about the learning projects. Enter The Survey. We have a survey that is available upon completion of a Thimble project* that asks some pretty basic questions about level of previous experience, fun, learning and if/where people got stuck. It’s not meant to be the most robust thing ever, but instead to do some temperature gauging during this first wave of users/learners.

*Note: we have another version of the survey (duplicate except for the word “Thimble”) for the DIY projects, which are the projects that point people out to different sites or tools. But so far, there are very few reposes there (~8) so this analysis is focused on the Thimble projects.

We have almost 150 responses so far, which is way lower than the estimates of folks using Thimble so far (more in the thousands), but not a bad response rate given that the survey link is a little awkwardly presented - right under the copy-your-link-to-your-finished-project field in the ‘Publish’ flow. 

Here are some highlights on what the numbers are telling us so far…

Findings

1) We’ve gone global

People are doing projects all over the globe. From the event registrations, we knew that we had events in 67 countries across the summer, but pretty cool to see this much activity in just a couple weeks.

2) Lots of existing experience.

We have a pretty even distribution of webmaking experience so far, with a slight advantage to those with more experience. This is a bit surprising since the Thimble projects are more targeted at entry level to intermediate, but its likely that some of this is due to the fact that we also just launched Thimble and have a bunch of people that are exploring it just to check it out. I’d also love to believe that there are mentor/facilitator/instructor type of people checking it out for the purposes of using it to teach other people these skills, but that question didn’t make this round of the survey (hindsight!!), so we’ll get those numbers in version 2.0. See below for a cross tab analysis of how this factor influenced other ratings.

3) Fun!

A whopping 74.8% thought that the project was fun and of that 39.4% said super fun. We are aiming for personally engaging, interest-based experiences so the sense of fun is an important piece. 

4) There was perceived learning.

63% reported some learning, with 25% reporting that they learned a lot. This is, of course, self-reported learning, not hard-core assessed learning* - but at this stage, again, to gauge the temperature of people’s experiences, I would say this is a pretty solid number, especially given that fact that over 50% came into the project with some or a lot of webmaking experience.

*Note: On the hard-core assessed learning: 1) we are building in more assessments that will use the work as evidence to validate that certain skills are demonstrated; but 2) all that said, again, ultimately we are after interest-based webmaking with some learning that happens in the process, so if people are engaged and able to make the things that they want to make, then I would call that a success without all the pre-post data hubbub…but we’ll do some of the latter as well next year.

5) People reported getting stuck.

One of our core design principles is to design for graceful failing - or said another, more direct way, don’t let people fail. By the numbers, it would appear we are there yet since 50% of people reported getting stuck. However, looking through the explanations, while there were a few that were overwhelmed by the code, it seems like most people were able to work through their stuck point:

"The </body> and </html> codes were missing and the webpage was showing errors and did not want to complete the project successfully. I figured it out at the end."

"I didn’t put an end bracket in the right place for the hyperlink.  Which is good, because it meant that I had to go back and figure out why it wasn’t working."

"In the beginning.  I had to read more carefully than just skim read."

That’s actually pretty promising then, because the projects were challenging but the learner had enough to solve the problem. It shows trial and error and tinkering which are also really important aspects to our learning philosophy.

There was a trend of ‘stuck’ responses about the publish feature which we need to investigate some more. Right now, when you click “Publish”, you get a URL which you can copy and share through Twitter, Fb, email, whatever but that doesn’t seem to be resonating with everyone. Responses included “I don’t think the publish worked right for me” and “Copy and paste WHERE??”. Apparently some people don’t understand that flow. So we’ll look into that more. 

6) Cross Tab == Cool

Of the people with no webmaking experience:

  • 70.9% reported having at least some fun, with 41.9% reporting a lot of fun.
  • 66.6% reported at least some learning, with 43.3% reporting learning a lot.
  • Only 10% (3) reported not learning anything at all.
  • 58.1% reported getting stuck.

Of the people with some webmaking experience:

  • 86% reported having at least some fun, with 43% reporting a lot of fun.
  • 68% reported at least some learning, with 15% reporting that they learned a lot. 
  • Notably, 0% reported not learning anything at all.
  • 36% reported getting stuck.

Of the people who came in with a lot of webmaking experience:

  • 34% though it was super fun, and 65.8% reported that it was at least kind of fun.
  • 48.7% reported some learning with 23.1% reporting that they learned a lot.
  • 35% reported that they didn’t learn anything at all.
  • 56% reported getting stuck.

6) Some great suggestions

How to make it more fun:

"More animal parts."

This was the number one request. The Zoo was definitely one of the most popular projects and people wanted more options to create more animals. I’ll take that as a notch on the side of ‘this was fun and engaging’. Also points to needing more projects with rich content and topics like this one. 

"When a project asks you to replace one image with another, it would be helpful if it supplied some links to alternate images."

This is an interesting suggestion. On the one hand, its a great idea and something that’s relatively easy to do, but on the other hand, again, part of the learning experience, and of web literacy as a whole, is being able to use the Web to find things that are interesting to you. I think what we definitely can do is provide better instructions on places to find images, how to properly attribute them, etc.

"More cat integration."

Well, duh.

How to foster more learning:

"Maybe a toolset of html elements and drag and drop as well for different html elements."

There were a handful of comments around this same topic of making it more WYSIWYG like or drag and drop. We intentionally built it this way, so that it is exposing the real code and learners/makers have to get their hands dirty in the code to make something. I think the potential for learning relevant webmaking skills is greater working with the real code…that said, we have to balance that with ensuring that the barrier is not too high. We are planning on doing some experiments with some different approaches with different abstractions with our DIY (read:non Thimble) projects later this summer.

"Maybe give the user an end result to aim for.  Can you make your document look like this [screen grab]  or create a page with a story or something and they have to create the next page."  

This is a good suggestion and we’ll consider it moving forward.

"The explanations when you click "read more" are WAY above a beginning coder’s head."
This is great insight - we’re pulling the hints from the MDN, which is awesome, but typically targeted at web developers. So we’ve started a “simple MDN" to start to write more basic descriptions. This is a community project - jump in and help out!

Takeaways:

  • We’re on the right path - the projects (+ Thimble), and thus this approach to learning, are providing some engaging and fulfilling experiences for a majority of people. 
  • Even though some of the feedback pointed to Thimble and the projects being too advanced for beginners, the numbers show that they reported the same amount of learning as those with some experience. The beginners reported it being a little less fun and got stuck a little more, but 43% reported learning a lot. I think even though the projects might feel overwhelming b/c of being dropped into the code, it seems like it was actually beneficial for people in terms of learning.  We’ve already started working on making Thimble even more accessible to entry level folks by integrating the X-ray Goggle functionality (you can now click on the right side preview pane and see things highlighted in the code on the left) and creating some more starter projects that focus on one or two elements at a time. It’s also worth noting though, that a lot of people with some and a lot of webmaking experience also reported learning and fun. I would guess that number would go up even more if (when) we have templates that are more targeted to various audiences and skill sets. So we’re on to something here. 
  • We need more projects that provide more interest-based access points for people - things like the Zoo project with rich content and compelling topics. We really want the community’s help with this so if you have an idea for a Thimble project, tell us about it.
  • We need more help and hints baked into the projects so that the barrier to entry is lower and less people can get stuck. 
  • We need to revisit the Publish flow to make that work for more people and (bonus!) use it as a teaching moment for those not used to sharing things on the Web.
  • We need more cats.

-E