Is this the end of touchscreens in museums? The use of touchless gesture-based controls
As museums and cultural institutions navigate the many challenges of reopening in a world where Covid-19 is still a real and life-threatening issue there is one question that I have seen come up time and time again: what do we do with all of these hands-on exhibits and touchscreens?
This question is especially tricky to navigate for institutions such as the Science Museum Group. Since the first interactive children’s gallery in the 1930s, interactivity and hands-on learning have been central to the groups approach in engaging audiences with science. And with good reason too. There are numerous studies showing the benefits of such experiences when designed correctly (despite being over a decade old the Exploratorium’s Active Prolonged Engagement (APE) project is still a must read for budding interactive designers). By providing visitors agency in their exploration of a topic we allow them to explore their own natural curiosity that ultimately not only leads to a deeper understanding of a topic but also shifts the power dynamic from the museum itself to the visitor.
Solutions to the problem have ranged from simply closing any touch exhibits altogether to issuing visitors styluses to turning everything into an app to designing new and novel ‘touchless’ interfaces. Each solution comes with their fair share of pros and cons. My usual response to such questions and challenges would be to do a decent chunk of audience research and prototype testing. However, in a world where I’m still confined to a makeshift home office and access to audiences for physical prototyping is near on impossible, I’ve decided to take a look at what lessons can be learnt from our existing research. In particular, what can we learn about those shiny (and potentially expensive) gesture-based interfaces?
Here are six lessons and questions to ask yourself before diving into the world of touchless gesture control.
1. Does it make sense?
With any exhibit, the first consideration should always be the visitor and the visitor journey. Who are your target audiences for the exhibit? How does your exhibit fit in the intended visitor journey? Does the interface work when considering how that audience interacts with exhibits, what mental models they are bringing with them? After considering the audience *then* you should consider the specific tech and interface to meet the needs of that audience.
By starting with the technology, designers often come up with elegant solutions that, in practice, do not work for the intended audience, and at worst confuse and frustrate. E.g. a VR solution to avoid a bunch of people sat in an enclosed planetarium space sounds great on paper when purely looking at it from a socially distancing and technology position but will be a failure if most of your audience group come in family units with young children who are excluded by a 13+ headset rating or it leads to lengthy queues of people waiting to have a go at the couple of headsets you’re able to provide.
Don’t lose sight of what your aims and outcomes are for the exhibit. An interface should complement the messages of the exhibit, not overwhelm them.
2. Positioning
The most common problem we have encountered with camera and sensor interfaces — at least those that require specific positioning — is visitors not knowing where to stand or place their hands in relation to the camera or sensor. More often than not, visitors will align themselves to the most prominent aspect of the exhibit — usually the display screen — leading to them either positioning themselves too close or too far away from the camera to ensure an image or gesture of suitable quality and accuracy can be captured.
It is vital that the camera or sensor is clearly visible to visitors as they stand in front of the exhibit, ideally in their line of sight as they face the display.
Where the exhibit is capturing whole-body images or silhouettes, by far the most effective solution is to place outline footprints on the floor at the position where visitors are required to stand. Our research has found that visitors have no difficulty understanding the intent of these footprints and do not require explicit instructions to use them. In a similar fashion, providing an outline image of a head on the screen helps visitors position their faces at the correct distance from the camera.
Cameras must be in mirror setting (as the visitor moves left the image moves right, as would happen if they were looking in a mirror). Otherwise it is impossible for visitors to accurately position themselves.
Lastly, I can’t stress enough — when designing any image or gesture-controlled exhibit think about how comfortable a position you’re asking all visitors to hold to engage with your exhibit. Asking them to hold their hand up in the air and keep it held in a specific position in order to read some text is a sure-fire way to make sure that content won’t be read.
3. Feedback
Another common problem we have encountered in our research is the difficulty alerting visitors when their image or gesture has been captured or registered. Where the visitors’ image or gesture is automatically captured, the exhibit must respond quickly. Any noticeable delay will cause problems as visitors tend to assume that the exhibit is either malfunctioning, needs some further action from them or that they have done the wrong movement and start trying a whole range of random gestures. If the exhibit does need a moment to calibrate, make sure this is clearly communicated to the visitor.
For exhibits specifically trying to mitigate visitors touching a screen, insufficient feedback to gestures may just lead to the behaviour you’re trying to avoid: visitors reverting back to what they know and repeatedly touching a screen with increasing frequency and strength as they try and make the exhibit respond. (As an aside we have ample observational evidence that turning touchscreens off doesn't necessarily stop visitors touching them in the hope that they will spring back to life)
The lack of immediate haptic feedback, that satisfying ‘click’ of a button being pressed, on top of the unfamiliarity of gesture-controlled exhibits mean visitors are also usually questioning whether they are “doing the right thing”. Solutions to this issue include both the use of sound effects as well as visual feedback on the display. It’s important to note however where the visitors’ line of sight is when they are interacting with the exhibit. Humans struggle to split their attention across multiple displays and interfaces (as frequently seen in dual screen exhibit setups), if their attention is on making sure their hand is doing the right movement relative to the sensor they are unlikely to notice a change on the screen off to the side.
4. What gesture to do
Touchless interfaces are not yet familiar to visitors and therefore it’s not immediately obvious how to interact. Even a ‘simple’ instruction to “swipe” can result in a number of questions and actions from visitors: Do I use my right or left hand? How big a swipe? How slow or fast do I need to go? Should I swipe side to side or up and down? Etc.
The most effective way to alert visitors to the use of gesture-based controls is a simple looped animation, lasting 2–5 seconds, which illustrates the required movement in relation to the sensor, accompanied by simple captions. Note however that visitors take these animated instructions very literally in terms of speed, timing and extent of movement. Therefore, any mismatch between the animation and the gesture that is actually required will cause problems.
Static illustrations showing the required gestures are less effective, in part because they fail to convey important information about the speed and extent of motion required, but also because they are less effective at attracting and holding visitors’ attention.
Compared to traditional touchscreen buttons, gesture-based controls allow for far less precision of selection and movement. As a result, active areas need to be larger and more widely separated than would be the case with traditional touchscreens. Gesture-controlled exhibits also need to be designed to cope with varied ways of performing a required action. If the specified action is too precisely defined, with no ‘forgiveness’ in the system, visitors will struggle to operate the exhibit.
The chosen gesture should be simple and make sense in the context of the exhibit and content. E.g. A “swipe” gesture will be more intuitive if asking to change a page of a digital book, less intuitive if selecting a specific element on that page. As mentioned in the first point, the gestures should make sense to the visitor. Asking visitors to learn too many gestures or perform complex movements will quickly overwhelm and the learning outcomes for your exhibit will be lost.
5. Taking instructions literally
As touched on in the previous point, visitors tend to take instructions very literally, especially in science museum settings. This not only goes for animated instructions but also written and spoken instructions and is particularly important to think about in cases where you’re planning on retrofitting a gesture-controlled interface on an existing touch exhibit. In one example at the Science Museum, when visitors were asked to ‘tap the dots’ in a touchless gesture interface, they assumed this meant that they had to physically touch the screen. Using the word ‘tap’ metaphorically in this case proved to be hugely misleading.
We also have examples of visitors following instructions very literally in more playful and immersive exhibits. In one example in an artistic immersive exhibit, visitors were able to move their whole bodies and influence the projected visuals in front of them. The intended experience was for visitors to experiment with different actions, they could jump up and down, move across the room, wave their arms, dance, anything and everything. In observations we found this openness to experiment was influenced by the instruction given by the member of staff. Those who mentioned closed instructions such as “stand here and wave your arms” led to groups spending the experience rooted to the spot and only waving their arms. Groups who received a more open instruction such as “stand here to begin with but then feel free to experiment with movements” saw a greater range of actions. This observation was only seen in adults, however. Younger children, typically, had no issue experimenting no matter what the instruction which brings us neatly onto the last point…
6. Performance anxiety
Performance anxiety — or the fear of looking stupid or getting things wrong in front of strangers — is a particular problem for immersive and whole-body exhibits. Although, we’ve certainly seen this as a barrier to engagement for hand-based gesture exhibits too.
The majority of times it leads to adults (we tend to see it less often with children, although still important to consider) being apprehensive to engage with an exhibit or try to keep their movements as small and subtle as possible to not draw attention to themselves. With the right reassurances, however, they can quickly get over this. By following best practice on the points above; clearly communicating the instructions, providing effective feedback, and giving explicit permission to play in cases of open-ended exhibits and experiences.
Visitors are also heavily influenced by the actions of other visitors. This can work in your favour. In the example of the immersive experience above, it only needed one person in the group to start experimenting beyond small hand movements to encourage others to do the same. It can also quickly exacerbate the barriers above: visitors are much more likely to copy the actions of the visitor before them rather than follow the exhibit instructions, even if they’re the opposite of what’s required.
So, are touchless gesture-based exhibits the way forward for museums? Possibly, but probably not yet. There are many reasons touchscreens are such a staple mechanism in making museums more interactive. They allow for a vast range of levels of interactivity, visitors are confident in how to operate them and over the years the hardware has become much more robust and cost-effective.
The 2020 Covid-19 pandemic has led museums to consider alternatives however, and that push to innovate should allow for a wealth of opportunity to research and improve the visitor and learning experience. As with all exhibits, it’s important not to lose sight of the audience. No matter what changes and solutions we decide on we’ll be continuously prototyping and researching with our audiences and building upon our existing knowledge base.
This article would not have been made possible without the comprehensive review of our research on digital experiences undertaken last year with consultant Ben Gammon and our Head of New Media Dave Patten. Thanks to them for the help trawling through the masses of research!
