Crowdsourced User Testing : going for a sustainable strategy

There are a lot UI Improvements under way for Drupal 7. Wouldn't it be wonderful to be sure if these really are improvements and we don't overlook small details that make the entire building crumble? How about ongoing, small-scale, crowdsourced user testing. Leisa Reichelt issued a similar call some time ago to test the admin header. A pleasing number of people tested the header and helped checking the names of the top-level categories and testing (probably confirming) the usefulness of the entire concept.

What this is up to is a sustainable concept that will ideally be an ongoing process. Apart from this mid-term goal there is also a very here-and-now one: we need to urgently test the D7UX stuff before Drupal 7 comes out or before the time even for small changes is over.

You can help by:

  1. do testing and upload the videos
  2. find some participants, if you don't want to do testing yourself
  3. be a testing participant if you don't know Drupal too well

A perfect participant has some web experience, has mastered things like ebay product selling forms and maybe has some experience in other cms's or blogging software. No. 2 of the above list would mean another motivated tester has won some participants and could carry out the tests with them. Personally I am willing to do quite some testing, this can be a lot of fun :) We need interviewers and test participants, so what about building up an infrastructure where we have:

- always enough test participants
- always enough people who would like to do the interviews 
- always enough nicely described tasks anyone can just pick from.

The problem is often that only one of these three is there. Bringing them together is key. As long as this infrastructure is not in place: anyone that would like to participate please leave a comment on this post, we will get the ball rolling.

How to do the testing 

Get the latest D7UX special version via SVN from the google repository or if using SVN does not come in handy for you get a zip archive from here directly. I'll try to keep the zip file up to date. This version is tweaked a tiny little bit to get the icons into the shortcut bar that are very important IMHO. Icons have been postponed past code freeze, so they're not in the official version atm. Please report back if there are any issues with the zipped version. Don't mind the .svn and CVS folders in there: they are just inactive if you don't use SVN or CVS.

You need PHP 5 and a PDO-enabled version of mysql to install Drupal 7. Most online Linux machines at your Webhoster should be ready for that, while namely if you run Xampp on Windows, you need to fix mysql in Xampp.

There is one thing to watch when you got it installed (choose the default install profile thas is preselected): it appears as if you were logged in, but you are not. The javacript progress circle will run for infinity if you click anything in the admin header. Click on "Add content" in the left sidebar menu, the login block will come up, log in. Now you can start :)

Software and hardware

Pick a screen recording software. I don't know the free ones, but actually any will do. If you got something like camtasia on windows or silverback for the mac, you are privileged. But actually I find one needs only certain features: being able to cut the video, being able to denoise and optimize the audio and to export it into say .mov or any other format that vimeo accepts (it does not accept flash formats like .swf or .flv.) If you do not want to do videos or it is too much of a technical effort: no problem, do your interview and record just the audio or transcribe it from memory. Any real world testing is valuable, so we don't want to stop you by overcharging the technical side.

Doing the tests personally and best in surroundings familiar to the participant is sure best and the nicest. But you can also do your interview remotely with success: get Teamviewer (available for mac and windows). Get the full version yourself and send your participant the quick support version. Don't forget to zip the file, since every mail server in the world rejects mails with attached .exe files (don't know with dmg for mac). Connect to the screen of your participant, best as remote administrator so you could control his mouse if you wanted. As long as you stay outside the Teamviewer window with your mouse you won't obtrude the other side. What sets Teamviewer apart from the other solutions I have seen: No need to install, runs unobtrusively just from the .exe file. Leaves no traces on the client computer, works through all corporate firewalls.

Don't forget to show the remote mouse cursor (for some strange reason off by default), this is in the options shown at the top of your teamviewer window. Call the participant per Skype (or per skype out, if not possible per direkt Skype connection). This way you can with ease start your screen recording software and record the sound with good quality. Doing tests remotely is a viable alternative and opens up the personnage that is available to you a lot.

Let the fun begin

Plug in a microphone (this is for doing tests in person) You can use the built-in microphone of your laptop, but it might pick up the noise of the keyboard, which is often very loud on the recording and cannot be removed. If you don't use the keyboard but only the mouse, you will be fine. Make sure the volume is high enough, but you can easily raise it afterwards, if it is not too noisy. I bought a cheap lavalier microphone for 20 Euros that has battery as power, which is almost perfect. If the volume is low, this does not matter if it is noiseless or low-noise. This allows for increasing the volume to any desired level when editing the video.

Please, do the interview in English! Even broken english is much preferrable to a foreign language. This may be hard on some participants, but taking it slow and easy might help. Ask the person kindly if he/she allows uploading the video to vimeo or another public platform. Just don't record their faces to keep anonymity byx not mentioning names (and if you mention it, ask them for permission). Most people agree to uploading. If they don't, consider doing a thorough writeup of the video and publish this in written form.

Now you are set to go. Pick a topic from here or make it much simpler. Since we are mainly testing the admin overlay, very simple user interaction is enough. We want to see, how general tasks are affected by the new interface and the fact that we have an admin theme. So here some suggestions, of which two or three may be more than enough for your half-hour session.

I've found that I spend half of the time dioing small-talk with the participants. This will make them forget this is a user test so they feel at ease and act naturally. Half an hour is just about the concentration-span before everybody gets tense. The rest is introduction, outro and breaks (yeah, breaks! Though best to keep them short, every screencast software offers this option).

Suggestions for tasks:

  • Create an article and publish it fo the front page.
  • Go to the front page (ridicolous? Oh no, you will see...)
  • Find Content - let them go to admin/content/node and recognize the article they have just written
  • Edit the article from that list, and/or edit it from the node page
  • Be aware of a big issue: The "Where am I?" - effect. Do they understand what the overlay is, what's their live site and when they are in an admin or "action" area?

O.K. let's get more daring...

  • Choose a different theme for display (argh, there is only Garland and Stark right now... Nevertheless, do it)
  • Add the article to the menu. Either via the menu settings in the node form, or (very daring) through the menu admin page
  • Go to the user page and let's see what users we have.
  • Very daring: go to Blocks admin page and let them stare =8-0 Maybe ask what they think they can do here and try to make them try out and click around
  • Add some Tags to the articles (you need several articles) and click on the taxonomy term in node view to see where we go.
  • The most important: let your participant click around and explore for a while. Observe where the interface leads them and what reactions they show. Are they overwhelmed? Can they find a meaning in what they see? Do they feel motivated to act and try things out?

Most of these tasks are done either in the admin theme or in the admin overlay. The overlay (black sides and top) looks different from the pure admin theme (white with greenish gray), try to find out if this is confusing

Now edit your recording so unneccessary parts are cut off and the sound is agreeable. Register a free vimeo account (basic). No, we do not get money from vimeo, but the quality is better that on youtube and you can upload videos longer than 10 minutes.  Join this vimeo group and upload your video, best with some information what you tested and about the person that participated. Done. You have contributed considerably to UX improvement in Drupal.

If you want a bit more guidance, as said above comment on this post or contact me via e-mail, and we (the UX team) will provide you with short scripts for what and how to test.

The long term plan

Bojhan Somers is currently building a site to coordinate all the efforts better as a subdomain of We will put and link all the stuff from there and build the infrastructure outlined in the start of this post. Give us some time until it's ready. We can use the vimeo group and maybe even this post as a temporary home :P Maybe start a group at

Code just works or doesn't. Is there a binary indicator for user interfaces? Well probably not, but one can get quite close to that. A user succeeds in performing a task or not. Further limiting the target group to "Novice" "Some experience in Drupal" and some more levels of expertise can get quite consistent results. There is also a position in user testing that says five participants on the same issue are enough to get a representative picture. After that and surely before, reaction patterns start to repeat. 

Package icon d7ux-head.zip5.11 MB



I am interested in building on your tasks list above. I'd like to match against the issue queue, and zero in on some high priority tasks.

Larry Caboa

I'll like to get involved in this project too.. Please drop me a line when the group is up and running

ESL Classes Los...

I am also very interested in the completeion of this group! Please let us know when it is up and running!