In our daily contact with clients and during workshops general questions as to the methodology and project planning of Usability measures arise frequently. The following compilation will address some of them. You may also refer to our detailed descriptions of individual test methods for answers.
Please turn to one of our contacts in case any of your question remains unanswered.
"What is the basis of our assertions?"
Our methodology emphasises user interviews, task setting, and the observation of respondents while they are solving these problems. Based on the sum of our observations we are piecing together an illustration of interactions as if completing a jigsaw puzzle. Established Usability Principles (so-called 'heuristics') and experiences with similar systems form the background of our analysis.
Since many participants show great difficulty in verbalising their intuition, mere interview data can be relied on only in exceptional cases. Apart from that, more than half of the participants, men in particular, are reluctant to admit their use problems. The method of not only questioning the respondents but also observing them directly is the only way to obtain certain valuable data.
"How to recruit adequate participants?"
SirValUse recruits respondents in cooperation with specialised call centres and recruitment agencies. This allows us to recruit perfect participants for very extensive tests (e.g. 250 selected representative web users) as well as for small tests with very specific target groups (e.g. car drivers and internet users with a pronounced brand affinity, purchase intention, etc.).
"How many respondents are sufficient?"
Usability Tests geared towards a homogeneous target group merely require 10 to 12 participants to unearth the bulk of problems. If we already know in the run-up that the user group will consist of 2 dividable sub-groups, at least 16 respondents (a minimum of 8 per target group) will be necessary to reveal target-group-specific differences. In case special emphasis is given to the perception of the brand and user opinions concerning its design, we recommend to interview at least 20 respondents. To meet extreme demands, however, it may be useful to investigate up to 200 participants for a simultaneous testing of a multitude of services for a multitude of target groups.
The selection of groups for a test should focus primarily on the question which distinct user scenarios you expect for your product rather than asking which market segments are targeted.
If your budget or time merely allow of a small study, a test including 5 to 10 participants will certainly prove more efficient than none. Although small-scale tests might reveal only one portion of all discrepancies, they are very likely to detect possible major obstacles.
"Why don't you include more respondents?"
We tend to advise our clients to conduct frequent smaller tests throughout the development process, rather than focusing on a single test with a large number of participants. It is very probable that after your eliminating the biggest problems subsequent to the first test, a second group of respondents will detect totally different hitches, yet without stumbling over the shortcomings that have already been remedied. Thus, by implementing the first test in the very early stages of product development you will circumvent expensive wrong turns.
Experience tells us that the likelihood of unearthing grave interface discrepancies diminishes, if more than 10-12 respondents per target group are employed. Hence, it is not cost-efficient to conduct additional sessions, except if you wish to compile an all-embracing list that also considers very rare problems. However, when your offer's basic modes of operation (e.g. shopping cart implementation on a B2C e-commerce platform) are concerned or in case safety aspects are involved (e.g. medical technology) larger groups of respondents are indispensable.
"Are the results representative?"
The method of observing a small number of participants in a systematic way The method of observing a smaller group of respondents in a structured way is often considered to be diffuse because the compiled data appear to be non-specific and not 'solid' enough. Although we can understand this scepticism it is nonetheless due to a misunderstanding. The primary claim of User Experience Testing does not consist of predicting the exact frequency of certain problem occurrences in the total user community; we rather specialise in tasking our respondents with adequate problems and in observing them so closely that a most complete list of problems will be produced. Apart from such a compilation of problems we will also provide concrete suggestions for optimisation.
Only in exceptional cases our cooperation with customers in designing useful, usable, and enjoyable products calls for quantified probabilities of problem occurrences. Our in-depth analysis of user behaviour enables us to demonstrate the potential for optimisation in a fast and efficient manner. You are of course welcome to consult with us when in need of a standardised, quantifying Usability-Test.
Our marketing division knows our customers inside-out. "Why do we need Usability analyses?"
Amongst other things the expertise of marketing divisions focuses on categorising the user community according to customer 'types,' demographic segments, and geographical regions. Even though this information management proves useful for the recruitment of User Test participants, it is not suited to replace the analysis of the respondents' individual interactions with a specific interface. For that reason, our expertise in this specialised field of investigation will contribute to the success of your product.
"Why not organise focus groups instead?"
In market research products are frequently tested in focus or discussion groups. Focus groups are powerful instruments in dealing with controversial issues and with test objects that are well-known and relevant to each participant. From the multitude of statements in a group situation interesting conclusions as to user opinions and motivation can be drawn. However, the evaluation of complex software products that are unknown or hardly known to them often exceeds the capacity of focus group participants. Furthermore, respondents often display great difficulty in estimating how relevant a presented service might be to them.
Usability observations often focus on details, for instance, how a participant (mis)comprehends a specific characteristic of a certain service. In this context SirValUse favours individual interviews since they produce more diverse and exact data due to behavioural observation. The interviewer exercises much more control on the progression of the procedure and the respondents' statements remain uninfluenced by a group situation in which some opinion leaders may dominate the more diffident participants.
"What exactly does 'Thinking Aloud' mean?"
'Thinking Aloud' is a method that urges the respondents to utter their unfiltered thoughts while they are using a system. A concise introduction by the test head and a short period of familiarisation suffice to enable most participants to provide continual feedback without slowing down their interactions. This type of free thought flow is important to observers and interviewers as it improves their assessments of the participants' perception, or rather, evaluation of the system and of their emotional reactions to it.
My test budget is small. "How to save money?"
Especially if your budget is small we advise you to start testing at an early stage of the developmental process. A test will only develop its maximum leverage effect if conducted early on, before the participating parties have invested much time and money in the product.
It is also sensible to divide the sum for the User Tests up: two small tests with an interposed re-design will prove more effective than a single big test - a very efficient way to collect feedback.
The most inexpensive method, however, is to request an expert's report. Although you will not receive direct user feedback, you will profit from our thorough experience. We are able to provide compact recommendations for the optimisation of your product promptly.
"Why such an effort?," "Are enquiries not sufficient?"
A lot of people shrink from admitting to another person that they have got a problem. Therefore, when being interviewed, many respondents tend to embellish their incompetence to avoid appearing stupid. Also, a lot of participants are incapable of verbalising their intuition. This is a pity because Usability Tests thrive on revealing these difficulties because they provide crucial clues regarding a system's potential for optimisation. According to our estimates against the setting of diversified method tests more than 50 percent of the respondents overestimate their abilities, especially males. For that reason, particular emphasis is given to the method of observation. We draw our conclusions from watching what the participants do (not do) and from investigating what this tells us about the product.
The interview method is employed only if self-information is the sole way to obtain the desired data; in the course of online enquiries, for instance, we present questions concerning the participants' satisfaction with an offer or their perception of its image.
"When is the ideal time for a Usability Test?"
As a means to prevent costly mistakes user feedback ought to be collected early on. You will feel all set for the further development of your product if the concept for your target group is comprehensible and if you find the terminology and information architecture of the product logical (these issues could be the subject of an early test, for instance, with paper prototypes and card sorting techniques). In case you are testing a system that has already been implemented completely, many basic structures that are critical for the acceptance of the system cannot be altered anymore.
Moreover, we recommend to conduct frequent tests with several small groups of respondents instead of testing only once with a sizeable group of participants. This provides an opportunity to get repeated feedback concerning different versions of the product. As a result, the sophistication of the system will proceed faster thus optimising the price-performance ratio of such a test plan.
"How long does a Usability Test take?"
Depending on the preparation effort at least 5 work days will pass from the briefing to the test launch (exclusive of recruitment; a sample of 16 participants should be recruited at least 7 work days before the test start). The field phase of a test with 16 respondents will take approximately 4 work days in case a complete Usability Test of a functional web site consisting of several dozens of pages is to be conducted. Each individual test session requires about 90 minutes. Data analysis and the writing of the report will consume another 6 to 8 days. Additional analyses, such as experts' reports, or extensive video editing may claim a few additional work days. Altogether, a Usability Test of small complexity will be presentable after approximately 3 to 4 weeks.
"Do you have your own test lab?"
SirValUse runs a total of 10 fully equipped Usability Test Labs, so-called Labs which are distributed as follows: Hamburg (5), Munich (2), Frankfurt/Main (1) and Beijing (2). We are able to provide our customers with the opportunity to observe the sessions from customer rooms that are equipped with a one-way mirror and monitors. We strongly recommend that persons involved in the project watch as many sessions as possible on location; clients have reported that this type of immediate experience can be instructional and very intriguing.
Furthermore, SirValUse possesses a Mobile Lab Technology that allows us to conduct tests as required on different locations and in other cities.
Our camera technique is quite extraordinary as well. If you want to investigate a mobile service, for example, we are able to film copies of all kinds of displays without obstructing the participants in dealing with mobiles, smart phones or PDAs.
My designers are very experienced. "Why do I need an extra consultation with a Usability Agency?," "Doesn't this double the expenditure?"
In our experience there are many design teams who are capable of designing systems well, at least in formal respect. We do not dispute their skills and competence. Our speciality, however, consists of increasing effectiveness, efficiency, and user satisfaction. For this purpose we bring an expertise to bear that can generally not be found in the repertoire of screen designers and art directors: to observe users in a structured way and to analyse these observations in total set against the background of the principles of work psychology. Nevertheless, it would be misleading to construct an opposition of design and Usability. Successful interfaces profit from our making effective use of the knowledge which both perspectives offer about the complex of related problems. We have grown quite used to discussions about this.
We are the best specialists for our product. "Why involve an external institute?"
You are very familiar with your product. For that reason, we set great store by a detailed briefing at the beginning of the study. Nonetheless, after investing substantial initiative, a lot of money, and time you are likely to identify with the system thus losing your impartiality. From our position as external advisers we are able to open up new perspectives and to generate proposals. Our assessment is not committed to preferences of the development team; we rather consider ourselves to be the 'Users' advocates.' This attitude enables us to contribute to leveraging even deadlocked development processes on the basis of empirical data.
"Will we lose our brand image if everything is geared towards efficiency?"
It is imaginable to optimise systems for efficiency purposes exclusively, for instance, where people have to repeat the same operations over and over again with a high frequency as in professional data processing.
Yet, apart from a high efficiency ("Do users get a lot done within a short time?"), a good User Experience can only be granted if we focus on effectiveness ("Do users reach their goal?") and satisfaction, too. In compliance with Donald Norman's hypothesis that "Attractive Things Work Better" we put special emphasis on the emotional aspect of interactions: the Usability of products designed for relaxing and enjoyable occasions can be substantiated by means of a pleasant aesthetic design.
Frequently, the objects of our investigations are integrated into a product family, thus forming part of a brand appearance that is also communicated via print and broadcasting media. It is therefore important to us to establish data regarding the perception of brand values as well. If branding is a topic in your system, priorities will be clarified in the briefing discussion already for that will allow us to assemble recommendations that are perfectly tailored to suit your particular needs.
