Home > Vol 9, No 3 (2015) > Vallgårda

Temporal Form in Interaction Design

Anna Vallgårda *, Morten Winther, Nina Mørch, and Edit E. Vizer

IxD Lab, IT University of Copenhagen, Denmark

Interaction design is distinguished from most other design disciplines through its temporal form. Temporal form is the computational structure that enables and demands a temporal expression in the resulting design. Temporal form is what enables poetry. In music, temporal form is the composition of tones, pauses, and timbre arranged into harmonies and rhythms. In interaction design it is, for instance, the behaviors of pixels on a screen or the movements in shape-changing interfaces.

In this paper we show the power of working explicitly with temporal form in designing computational things. We give a nuanced account of what temporal form is in interaction design, and we look at related work synthesizing what we already know of the temporal concerns in interaction design and HCI. In the second part we present a design experiment through which we explore the experiential qualities of a set of 11 simple temporal forms by letting a series of expert designers reflect upon them. We borrow a framework from Boorstin’s film theory in which he distinguishes between the voyeuristic, the vicarious, and the visceral experience. We show how to use rhythms, complexity, gentle or forceful behavior, etc., to create experiences of “being alive,” being entertained, or being something that we empathize with. We end the paper by arguing how the temporal form in computational things enables richer experiences than static objects do.

Keywords – Computational Things, Design Research, Form-giving, Shape-change, Temporal Form.

Relevance to Design Practice – Developing an understanding for temporal form in interaction design can be compared to musicology or knowledge of physical forms. It gives us a better understanding of what we can do with interaction design, as well as a language to articulate choices and preferences.

Citation: Vallgårda, A., Winther, M., Mørch, N., & Vizer, E. E. (2015). Temporal form in interaction design. International Journal of Design, 9(3), 1-15.

Received March 30, 2015; Accepted November 19, 2015; Published December 31, 2015.

Copyright: © 2015 Vallgårda, Winther, Mørch, & Vizer. Copyright for this article is retained by the authors, with first publication rights granted to the International Journal of Design. All journal content, except where otherwise noted, is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. By virtue of their appearance in this open-access journal, articles are free to use, with proper attribution, in educational and other non-commercial settings.

*Corresponding Author: akav@itu.dk.

Anna Vallgårda is an associate professor and head of the IxD Lab at the IT University of Copenhagen. Her research focuses on developing a form-giving practice for interaction design following her work on the computer as a material for design and computational composites.

Morten Winther is an experience and interaction designer. He holds an MSc in Digital Design and Communication. In his work, he seeks to develop new design practices to explore the boundaries and new expressive potentials of technology. He is currently teaching various design courses at the IT University of Copenhagen.

Nina Mørch Pedersen holds an MSc in Digital Design and Communication from the IT University of Copenhagen, where she has specialized in tangible interaction design and digital aesthetics. Her projects mainly revolve around combining conventional materials with computer power to achieve new expressions.

Edit E. Vizer holds a BA in Art and Technology from Aalborg University and recently graduated with a master’s degree in Interaction Design from the Kolding Design School, Denmark. She is a practicing media artist and interaction designer who works on research-centered projects focusing on the experiential, on exploration of the aesthetic values and potentials of technology, and with the involvement of narratives and theoretical inquiry.

Introduction

Interaction design is distinguished from most other design disciplines through its temporal form. Temporal form is the computational structure that enables and demands a temporal expression in the resulting design. When programming computers we create a temporal form that then comes to expression through an output of actuators and other materials. Indeed, it is these material manifestations of temporal forms that enable our interactions with computational things, as digital computations in themselves are inaccessible. Thus, temporal form gives interaction design a kinship with temporal arts like music, dance, and film. While several interaction design researchers have already articulated this aspect, we have still to grasp the details of what it entails (cf. Hallnäs & Redström, 2001; Hallnäs & Redström, 2006; Mazé & Redström, 2005; Vallgårda, 2014). How do we experience temporal form in computational things? What significance does the temporal form hold for the overall experience? What is the relation between temporal form and the other form-elements in interaction design? What relations can we find between the expressions of computational things and our experience of them when it comes to temporal forms?

Temporal form is what enables poetry. In music, temporal form is the composition of tones, pauses, and timbre, arranged into harmonies and rhythms. In movies, it is the composition of actions and backgrounds moving stories forward. In poetry, it is the composition of meanings and rhythms. Temporal form holds functional as well as aesthetic power in the composition of the overall design—just as physical form does. The traditional view on temporal form in relation to computational things has been that of speed. It has been a matter of removing delays from hardware and software to enable instantaneous representation on graphical displays; it has been about speeding up computations to achieve faster results, etc. Certainly, the execution speed in a computational thing is important and will affect what we do with it, but there are more aspects of temporal form to be explored than speed. Material and tangible computing have changed what forms of interaction we design for and the materials we use. They have entailed that the contexts in which we use computational things have multiplied and that not all of those contexts are suited for speed as the only temporal expression. Indeed, it has become crucial to pay attention to the underexplored notions of timbre and pauses as well as the relations between materials and state changes. From what can we design our rhythms and harmonies?

Vallgårda (2014) has proposed to see temporal form as part of a trinity of forms that would constitute the form-giving concerns in the practice of interaction design, with the other forms being physical form and interaction gestalt. We have long understood physical form and its relation to materials. In industrial design schools, for instance, students are taught how any physical form can be broken down into cubes and cylinders, and that every other form comes out of the transitions between them (cf. Itten, 1975; Ching, 2010). We know about materials and their influence on form and expression (cf. Manzini, 1989), and we know about style of physical objects (cf. Semper, 2004). We also have an understanding of the interaction gestalt where concepts like affordances (Gibson, 1986) or signifiers (Norman, 2011) help us understand how physical form is linked to the interaction gestalt. Additionally, Dourish (2001) has succeeded in establishing all interaction as social as well as embodied. Indeed, what we lack most is an understanding of temporal form’s role in all this.

In the first part of this paper we underline the importance of working explicitly with temporal form in the design of computational things. We give a nuanced account of what temporal form is in interaction design. We then look at related work synthesizing what we already know of the temporal concerns in interaction design and human–computer interaction (HCI). In the second part we present a design experiment through which we explore the experiential qualities of a set of 11 simple temporal forms by letting a series of expert designers reflect upon them. We borrow from film theory Boorstin’s (1995) distinction between the voyeuristic, the vicarious, and the visceral experience, as it gives us a framework to qualify our analysis of the relation between the expressions of temporal forms and how they are experienced. Based on the outcome of the study, we argue how temporal form in computational things enables richer experiences than static objects do. Further, we summarize how the temporal form enables different kinds of complexities, each of which seems to evoke different kinds of experiences (e.g., how small rhythmic anomalies evoke a sense of the technology being alive, or how forceful behavior can evoke strong vicarious responses).

Temporal Form in Interaction Design

In simplified terms, we can talk of four temporalities at play in any interaction design: the temporality of the human, the temporality of society, the temporality of the computer, and the temporality of the input/output compositions (the physical form). The latter two determine the temporal expression of any computational thing, while the first two play into how we experience the temporal unfolding of a design. The temporality of the human is characterized by our ability to perceive the passing of time not just consciously but also in our embodied actions. We always experience the now in a simultaneous combination of what has just happened (retention), what is happening (impression), and what we anticipate is going to happen (protention) (Gallagher & Zahavi, 2012). The temporality of society is created through our mutual participation and interactions (Ingold, 1993). Yet these interactions happen in resonance with our environment—with the rhythms of nature, day and night, the seasons of the year, the tides, etc. (Ingold, 1993). The temporality of the computer is determined by a combination of the hardware and software. It is a combination of the internal clock regulating the execution of instructions in the processor and the designs of the algorithms being executed. Finally, the temporality of the input/output compositions is determined by the actuators’ and sensors’ transducive abilities as well as the responsiveness of the materials they are connected to (Vallgårda & Sokoler, 2010). The design of a computational thing or environment must inevitably take all these temporalities into account.

Basically, temporal form in a computational thing is created from a combination of the temporality of the computer and the temporality of the input and output compositions (cf. Redström, 2001). Yet, to help us explain what temporal form is we can look at music. Music is a temporal form. As Stambaugh (1964) notes: “In musical ‘form’ what ‘flows’ are tones, tone complexes, and patterns, and these are genuinely temporal in themselves” (p. 268). In computational form (the program) what flows is the changes of states, and patterns of state changes manifested as different voltage levels. The vast majority of these state changes are not intended for output and will thus remain only as part of the computational structure. The output, however, is adapted into physical form through various transducers and results in a spatial gestalt of the computational form (e.g., as pixels on a screen, as movements in a physical form, as audio, as temperature changes). Thus, unlike the pure computational form, this spatial gestalt is available to human perception. The temporal form is then the overall behavioral expression of a computational thing.

A musical score is comparable with a computer program before execution. The score and the program constitute the plan for what should happen. A key difference is that while it is possible to create music without a score (improvised) it is not possible to create any temporal form in a computational thing without a computational form (a program). Albeit the lines get blurred when working with genetic algorithms and adaptive AI, but even they need a programmed starting point. Thus designing computational things includes aspects that can be equated to composing music. Indeed, the design of the temporal form is always in the shape of a plan or a structure and not the actual execution of events. This becomes most apparent when the computational thing takes input either from interaction or from other sorts of background information. This input will likely influence the expression and quality of the execution, which can thus be precipitated but not actually known in advance. Thus the computational form is the structure that interprets the input and determines how and to what extent the input is reflected in the output. For instance, there may be a close relation between the input and the output both temporally and spatially, but they may as easily be separated in time and space.

Continuing with the musical analogy, composing tonal music, for instance, is more than just devising sequences of tones. It is as much about the temporal structure (i.e., the tempo and the pauses), about the timbre, the relation between the instruments in mind, etc. We can see the states and state changes of computation as equivalent to devising sequences of tones in music. As soon as we take the physical form into account, however, composing the temporal form of computational things becomes equally as loaded with potential variations and is equally as complex as composing music. Indeed, in the design of computational things we too need to consider aspects such as tempo and timbre as they afford certain interactive and communicative qualities. Indeed, it is in the composition of the temporal form that feedback and feed-forward cues are likely to be devised (cf. Wensveen, Djajadiningrat, & Overbeeke, 2004).

The choice of instrument in music can be equated to the choice of composition for the physical form—the composition of transducers and other materials. While in theory it may be possible to execute the computational form in many different physical compositions it will most certainly look and feel different and thus have different communicative and interactive qualities. In practice, however, the transferability of the computational structure will be limited because the transducers and the materials will enable and afford different temporal expressions. For example, the states and how the state changes are composed must differ dramatically to suit the respective qualities of LED light and Peltier-element-controlled temperature changes, simply because of the speed with which their effect comes to expression. Also, while we still work with states as the basic component, the transitions between them will be the most expressive in some material compositions. For instance, in shape-change compositions the transitions may be the primary expression and the states only perimeters that may never be reached (e.g., two endpoints in a rotation).

As described above, what is designed is a plan for an execution and not the execution itself, as that will happen in the future and typically repeatedly. It is a plan for how events should unfold and how input should be interpreted. Only the future user can really determine whether these interpretations and responses are meaningful. Thus the designer must always accept open-endedness. In a similar vein, some composers deliberately leave aspects of their music indeterminate for future orchestrations to fill out (Cage, 2009). Such freedom demands a heightened attention and participation of the musicians, which can enhance the energy and presence of the performance (Cage). This open-endedness can be compared to what we leave open to the users in interaction design (cf. Gaver, Beaver, & Benford, 2003).

This is an early account of temporal forms in computational things, which we assume will become more nuanced and expanded as we come to better understand its ramifications. With the study presented below we aim to begin to grasp what it qualitatively entails to design a temporal form, and more importantly how people perceive some of these temporal expressional qualities.

Related Work

Temporal matters have been treated within HCI and interaction design in a number of different ways. Some work has been focused on users’ perceptions in interaction, other work on the efficiency of the technology, and other work again on the use of temporal elements to communicate functionality or to achieve specific aesthetic experiences. Below we have attempted to outline this varied landscape.

Temporal Concerns in Interaction

The primary focus in previous studies of the temporal aspects of interaction has been on the perceived fluency in the interaction. In his work with experiential qualities Löwgren (2007, 2009), for instance, is concerned with “the degree of gracefulness with which the users deals [sic] with multiple demands for their attention and action” (Löwgren, 2009, p. 3). In a similar vein, several researchers have been concerned with interruptions (cf. Bailey, Konstan, & Carlis, 2001; Hudson et al., 2003), for example when an email program running in the background calls for the user’s attention by making a sound and showing a message in the corner of the screen. The concerns have primarily been to understand how such interruptions affect the users’ workflow, when they become stressful, and when they relieve stress. In related work, Huang and Stolterman (2011) study users’ duration of attention in their interaction with an email client, which they represent as graphs. Lundgren (2013) later proposes to nuance the concept of fluency as she introduces seven temporal themes (live time, real time, unbroken flow, sequential events, disordered events, juxtaposed events and branched versions) to assist analysis of temporal concerns in interaction. As they stand, however, the concepts in this list are not mutually exclusive and are thus difficult to use in a context different than that from which they were derived.

Temporal Concerns in Graphical User Interfaces

Temporal concerns have typically been a little more difficult to handle when it comes to the usability of computational things and environments. With the introduction of the graphical interface in the mid-1980s and the diminished latency of I/O units there arose the promise of dynamically adaptable user interfaces with the slogan, “Why should people have to adapt to systems, systems should adapt to people instead?” (Höök, 2000, p. 409). This, however, proved to be somewhat more complex than anticipated because the dynamics turned out to compromise the predictability and transparency of how the system worked (cf. Shneiderman, 1997; Höök, 2000). Thus the adaptive graphical user interface has been largely abandoned and currently exists primarily as platform-dependent adaptation and not as a temporal concern within a work session.

The dynamic or animated aspect of the GUI has, however, stayed on as an aid to communicating interaction possibilities either as feed-forward or feedback (cf. Wensveen et al., 2004; Harrison, Hsieh, Willis, Forlizzi, & Hudson, 2011). For example, Apple’s iOS includes a number of instances, such as the jumping icons that indicate they can be switched around—they are now in that mode of interaction. In their analysis of Kineticons, Harrison et al. (2011) lay out the communicative aspects of six types of motion metaphors: biological motion (human locomotion), gestures (deliberate human motion), organic motion (motion recognizable from nature), mechanical motion (toggles, sliders etc.), physical and natural effects (glass shatters, paper folds), and finally the cartoon conventions that include exaggerations and other manipulations of motions recognizable from nature. While the stringency of this taxonomy can be debated it is an attempt to articulate the qualities of different temporal metaphors in different uses of transitions between states.

Temporal Concerns in Ambient Displays

When Weiser and Brown (1995) introduced the concept of Calm technology they were, albeit implicitly, working with the temporal expression of computational things, for instance the Dangling String (Weiser & Brown). Ishii et al. (1998) later continued the exploration of the theme with their AmbientROOM. Here a host of information services were delegated to different objects and locations within an office. The information was displayed through different temporal forms of, for instance, changing light or airflow. In both cases, however, the temporal forms of the ambient and calm displays are not discussed explicitly, even if it seems obvious that to create something that is ambient and peripheral the patterns of change must play a significant role. Indeed, most commonly, temporal concerns in information visualization are paid little attention beyond basic observations of the designers’ own experiences (cf. Holmquist & Skog, 2003; Jafarinaimi, Forlizzi, Hurst, & Zimmerman, 2005; Mankoff et al., 2003; Stasko, Miller, Pousman, Plaue, & Ullah, 2004). For example:

Also, the display will change quite slowly, so that changes will usually not be noticeable unless one looks at the display for an extended period of time, or if there are long intervals between each time it is viewed. These are conscious design decisions that have been made to make the applications function more like a visually pleasing art-work, and less like animated computer graphics. (Mankoff et al., 2003, p. 170)

This is not meant as specific critique of Mankoff et al. (2003). It only serves as an example to demonstrate the extent and quality of reflections on the temporal form thus far.

Temporal Concerns in Computational Things

When shifting focus from seeing the computer as a thing to instead seeing the thing first and the computer only as a structural or material part, the view of the design concerns also changes (Vallgårda, 2014). This means, for instance, that the temporal concerns for computational things are not a matter of seeing interfaces or internal time constraints separately. Rather, they become integral concerns. When endeavoring to design with new materials and new modes of input and output, the accumulated knowledge regarding reaction times and conventional modes of interaction needs reevaluation. Fluency and ease of use may not be the primary objectives for the interaction gestalt; rather the new modes of expression may lead to entirely new areas of application.

As Maeda (2000) and later Mazé and Redström (2005) argue, designing the temporal forms of machines is new to the computational age. In the age of macroscopic mechanics temporal form was given by the speed of the mechanical devices, but computations happen at a speed and in a form and language that are impossible for humans to perceive. Thus we need to design not only the computations, but also how they come to expression.

Hallnäs and Redström’s (2001) work on slow technology was probably the first deliberate attempt to bring the temporality of computational things to the foreground through critical reflection. By working with delays, they challenged the expectations of their audience. They played on our general experience with computational things like laptops, smartphones, etc. as things that work more or less instantaneously and where a delay indicates something is wrong (e.g., the Internet is down or the execution of the program stalls for some reason). Indeed, simply by seeking the opposite of the predominant focus on speed in computational things they opened up a design space in which we could experiment with temporal form. With their slow technology they introduced a space for reflection (Hallnäs & Redström).

While we may readily want to explore the potential of this design space we also quickly realize the complexity involved in composing forms for things we have had little possibility to become familiar with because they are often not built yet. As a means to address this Frens, Djajadiningrat, and Overbeeke (2003) propose a combination of paper prototyping and “a timer to be able to time and scale the actions and reactions of the model” (p. 7). The purpose is to find a way by which a “desired feel for a movement can be composed by combining different materials and can be assessed at the handles of the haptic composer” (p. 7). Nilsson, Satomi, Vallgårda, and Worbin (2011) develop a combination of a graphical interface and a “musical sheet” as tools to experiment with the patterns of color change in the textile upholstery on an interactive piece of furniture. Here, the computations have been left as the last part of the design and the authors experiment with the piece of furniture as one would get to know a new musical instrument.

From a similar practice perspective Parkes and Ishii (2009) focus on the need for a vocabulary to express behavioral transformability. They deconstruct kinetic behavior into material (amorphous, layered, skeletal, rigid), mechanical (rotational, linear, radial) and behavioral (speed, acceleration, direction, twitter, delay, pattern) elements. They further state the importance of isolating “the perceptual response a motion can trigger, [and] simplifying the mental leap and engineering knowledge necessary to arrive at a potentially new and innovative design solution” (p. 370), but do not proceed to do so. Rasmussen, Pedersen, Petersen, and Hornbæk (2012) take this a step further and look at both kinetic parameters and expressive parameters. With the expressive parameters they begin to address some of the qualities for temporal form in a shape-change context. Based on analysis of a series of shape-changing interfaces/installations they divide the expressive parameters into “adjectives,” which refers to qualities (e.g., soft, peaceful, turbulent) and personality traits (e.g., happy, sad, angry); and “association,” which covers organic (e.g., anthropomorphic, nature) and mechanical behavior (Rasmussen et al., 2012). Building on this, Kwak, Hornbæk, Markopoulos, and Alonso (2014) devised a series of six shape-change behaviors, which they then subjected to evaluation by 18 students in a repertory grid study. They let the test subjects describe their experience by giving them a series of construct/contrast pairs and a number between 1 and 7 (Kwak et al.). The results suggest that (1) a segmented surface is perceived as playful; (2) a surface that approaches the test subject is more energetic; (3) the speed of shape change relates directly to perceived assertiveness, with (4) slow and retracting shape changes being perceived as calm and introverted; and finally (5) rotating shape change is perceived as mechanical. However, this collection of results seems somewhat arbitrary and hard to confirm through a repeated study.

As we can see from this body of work, dealing with the four temporalities in design of computational things is by no means trivial. Using the concept of temporal form, we believe, will enable us to get a handle on the design space—what levers and knobs we can work with. We still have little sense of the relation between the temporal form of the computational thing and the temporalities of the interaction gestalt. While Kwak et al. begin to study this empirically it is difficult to discern whether their results are relevant beyond the scope of their specific study since no attempts were made to understand the underlying explanation for the correlations. Thus in the following part of the paper we begin to study the relation between temporal form and interaction. Through an experimental setup, we specifically aim to understand the underlying relations between expression and experience.

Method

In this paper we have set out to explore the expressional qualities of temporal form for the practice of interaction design. While it is not possible to establish a formal link between an expression and how it is experienced, there is a degree to which we do have shared experiences simply because we have similar bodies, with similar perceptual abilities, and sometimes we even share culture and previous experiences (Goldman, 2005; Merleau-Ponty, 2002). For example, there is a reason why it is a common trait among Western pop songs to raise the pitch by a semitone in the B section of the song or why there are car chases in action movies.

In this study we chosen an experimental setup that was kept simple in all matters concerning physical form and interaction gestalt as a means to maintain a focus on the temporal forms (cf. Vallgårda, 2014). The physical setup was inspired by the shape-changing study by Kwak et al. (2014) as well as our own previous work on shape-changing textile (Dumitrescu, Landin, & Vallgårda, 2012) (see Figure 1). Here, however, we were not attempting to introduce new forms of shape change. On the contrary, we chose very simple motors and helical shape-memory alloys (SMA) and their direct effect on a cotton textile surface. We designed three boxes, each with two servomotors, two stepper motors, or two SMA elements. The two motor boxes were each given four different temporal forms—meaning programs that define a pattern of movements—and the SMA boxes were given three. Further, the study participants could control aspects of the temporal form through parameters such as speed, rotation degree, delay, or electrical current (in amperes) by turning a couple of knobs on the sides of the boxes. As study participants we invited experts, primarily interaction design experts, but we also included some with expertise in other temporal arts such as music and dance. The argument for choosing experts is that they will be better equipped to reflect upon and articulate what they see. Each was subjected to a one-and-a-half-hour think-aloud study. In the following subsections we describe in detail the physical setup of the study, including aesthetic descriptions of the temporal forms. We also account for the backgrounds of the seven expert participants, and describe the study scenarios as well as our subsequent analysis method.

Figure 1. The three boxes, the left-hand one powered by two stepper motors, the middle one by two helical strings of shape-memory alloy, and the right-hand one by two servomotors. All three boxes are 20 inches in length and width and four inches high. See also the temporal forms in action at https://vimeo.com/album/3639572.

Experimental Setup

In order to set up the experiments we had to first map out the design space they should explore. We did this by deciding early on on a physical form and an interaction gestalt. On top of this we then designed a series of temporal forms that would explore temporal aspects such as synchronicity and distortion. The forms further included different potentials to change elements such as speed, acceleration, velocity, delay, or intensity.

As the physical form, we chose to make an abstract setup in which we could focus on the aesthetics of the expressions rather than their ability to convey or perform a specific function. The choice of the white/light boxes was inspired by the “white cube” tradition of art galleries, used to present the artworks without their having to compete with other colors or patterns (cf. O’Doherty, 1999). While we know that no physical form can ever be neutral, we still believe that the box poses as little a spectacle as possible for the study. Thus we wanted the patterns of movement in the textile to stand out and the rest to withdraw to the background. The choices of textile and actuators were partly inspired by Dumitrescu et al. (2012) and partly by Kwak et al. (2014). We chose two types of rotating motors, one servo and one stepper. For the third setup we used helical shape-memory alloy (SMA). We simply attached the actuators to the textile so that the motors would rotate the textile, and the helical SMA would contract it. Technically, it was a simple platform to set up. The reason for choosing two similar actuators (the two types of rotating motors) was because, even if they technically perform similar rotations, the possible patterns of those movements would be different. This would allow us to explore whether the experienced temporal forms would also be different. Furthermore, we chose movements in a physical material over more traditional forms of expression in a computational thing, such as graphics on an LCD, light patterns in LEDs, or audio. The reason was a desire to explore that part of the design space where the instantaneous state changes that we have come to expect from traditional computational things would be suspended. Through removing this expectation we would be able to explore a larger range of expressions in which, for instance, transitions between states could play an important role. Finally, we chose two actuators per box. One actuator per box would have been a simpler choice but we quickly realized that the temporal forms would soon become too monotonous. Two actuators, on the other hand, would allow us the interplay between them and the use of the temporal form elements of synchronicity and asynchronicity.

As for the interaction gestalt, we also chose simplicity. A pilot study had shown that the study participants found it difficult to engage with the temporal forms if they had no means of interacting with them. Thus we introduced two turnable knobs per box. Turning a knob is so simple and familiar that it would not require further instructions. Thus we hoped the knobs would not distract from the temporal forms we wanted to study.

Only after completing the physical setups did we start to program their behavior and to explore our own perceptions of different temporal forms. We chose to restrict the parameters to two: synchronicity and level of control (see Figure 2). We had some initial notions that asynchronous behavior of the two actuators would result in more organic expression while synchronous behavior would appear more mechanistic. We further contemplated that the less direct (more distorted) control the user would have (through the knobs), the more organic and less mechanistic the expression would seem. These were however only initial notions meant to aid us in designing a variety of temporal forms. The four temporal forms these notions resulted in were only fully implemented on the two motor setups as the reaction time and nature of the shape-memory alloy forced us to make some variations.

Figure 2. Illustration of the four different temporal form expressions in the two motor driven boxes.
For instance, the first expression is a synchronized temporal form in which the participant has full control over speed and degree of oscillation, while in the fourth expression the temporal form is asynchronous and the participant’s control of speed and oscillation is distorted by a randomization factor.

Description of the Temporal Forms

The four temporal forms for the two motor boxes were derived from different combinations of synchronicity and level of control (see Figure 2). In the first temporal form, the two motors would be synchronous and their rotation would be a continuous oscillation from one extreme to another. In this case, one of the knobs would control how far the motor should turn—where its extreme point should be. The other knob would control the speed of the servomotor, or the acceleration of the stepper. In the second temporal form, the two motors would still be synchronous; however, we inserted a randomness factor in calculating the end position of the oscillation. This resulted in an abrupt movement pattern whereby the motion would sometimes go from one extreme to the other but most often it would change direction midway. The speed or acceleration would still be directly controlled by the position of the second knob. In the third form, the motors would be asynchronous, as we inserted a random-length delay between them, but the knobs fully controlled the degree of rotation on the one hand, and the speed or acceleration on the other. Finally, in the fourth temporal form the motors would also be asynchronous and the control of the end positions would include a randomness factor (see Figure 2). See also the temporal forms in action at https://vimeo.com/album/3639572.

Since the SMA elements work in a different temporal range than the motors—they react more slowly—it did not make sense to have the exact same setup for them. While we still explored the synchronous/asynchronous axis, the knobs would instead control the length of time the power was on or the level of current (in amperes) sent through the wires.

Thus in the first temporal form the two wires would synchronously contract and relax. One knob would control the length of the contracted state and the other the length of the relaxed state. (This form is referred to below as synchronous.) In the second form each knob would control the corresponding SMA wire and the current flowing through it (referred to below as amperes). Finally, in the third temporal form, one SMA would be turned on when the other was turned off and vice versa. In this case the knobs would determine the respective lengths of the states (referred to below as asynchronous). The temporal form factors at play in the temporal forms of the study are thus synchronicity, degree of distortion, speed, acceleration, and degree of movement; acceleration is present only in the stepper motor and to some degree in the SMA in the amperes setup. In the analysis we first look at which experiences the overall expressions foster, and only then break down the different experiences into the different form factors.

In total we explored 11 different temporal forms expressed in three physical forms (see https://vimeo.com/album/3639572).

Study Participants

As mentioned, the argument for choosing experts is that their phenomenal field is broader than a layperson’s (cf. Svanæs, 2013; Merleau-Ponty, 2002). So while they may not have a vocabulary to describe the experiential qualities of a temporal form, they still have a frame of reference in related matters, for instance for describing physical forms or interaction gestalts. Their expertise will enable us to ask them not just for their immediate experience, but also how this experience is related to what they see. Thus if they express frustration they are also likely to be capable of explaining what it is that makes the situation frustrating. This would provide us with reflections on the connection between expression and experience and thus ultimately give us a more nuanced picture from which we can begin to understand the temporal form.

We invited a total of seven experts from different fields. Four of the experts held a PhD in interaction design, design, or HCI, and after obtaining their PhDs had worked at least three years with research and teaching within the field. None of these were from our own research institution. They are referred to as IxD-1–4 in the analysis below. We further invited a ceramicist, who had done a PhD on the topic of developing computer-based graphical forms and behaviors as means to create new ceramic shapes, and had since continued this research for several years. He is referred to as the Ceramicist in the analysis below. We also invited a musician who held a bachelor’s degree in musical theory and was now finishing a master’s in modern culture. The musician had started out as a classical violinist as a child but now played drums in a band with professional engagements two to three times a month. She is referred to as the Musician in the analysis below. Lastly, we invited a professional ballet dancer who had been dancing for 30 years and professionally for 20. He is referred to as the Dancer in the analysis below. Among the seven experts three were female and four male, while among the interaction designers specifically, two were male and two female.

The interaction designers were chosen for the obvious reason of having a heightened sensibility for and insight into interactive artifacts. They had all been forced to articulate critiques of interactive artifacts in their professional careers in teaching and peer-review contexts and thus could be expected to possess both the critical eye and the vocabulary to express their thoughts on the intentionally abstract installations they were presented with. The Dancer and the Musician were invited because of their experience with other types of temporal aesthetics and we thus expected them to be able to provide a critical reflection based on their respective professions. Finally, the Ceramicist was invited as someone who had an ingrained experience with the relation between the temporal and the physical, albeit for him the temporal progression would always end in a solid state.

Study Scenario

Each participant would be presented with one box at a time with one or two researchers present (see Figure 3). Each session was audio recorded and video filmed. We would typically spend half an hour per box; however the first would usually get a little extra time for the participant to get used to the setup. The participant was asked to think aloud and only once in a while would the researcher ask questions like “What are you experiencing here?” “Can you describe the behavior you see?” or “Why does this make you laugh?”

Figure 3. One of the study participants explaining her experience of a form developing through the servomotor setup.

We would alternate between starting with the servomotor and the stepper motor but would always have the SMA in the middle as a means to clear the palate and ensure the session would not become too monotonous. We would also keep the order of the temporal forms for each box as they are described in the section above. The argument here was that there is a learning curve in how the boxes work and the first temporal form was the easiest to understand. While confounding of the experimental setups is probably inevitable it does not really matter since we were not interested in particular temporal forms but rather in the experts’ expressed experiences and their own explanations.

Data Analysis

All seven interviews were transcribed by the researchers and afterwards sensitized and coded by one of the researchers. We used open coding with the purpose of identifying key themes among the subjects’ descriptions of their experiences. In coding the transcripts we were looking for recurring themes, but particularly for any description of experiences that would enable us to link them to expressions. Obviously, however, the more participants corroborated such a link, the more detail we were able to describe it in.

Reflection on Using Experts

As expected, the interaction design experts demonstrated little experience with seeing and talking about temporal form in a design context. Thus their frame of reference for temporal behaviors arose from everyday encounters in nature and society. They did, however, rely on their expertise when analyzing the qualities of the relation between the temporal and the physical form and what this entails for an interaction experience. The Musician, on the other hand, had ample experience in analyzing temporal form in music and thus her frame of reference had much to do with rhythms and repetition, while she refrained from analyzing the physical form or the interaction. The Dancer read whole stories into the movement patterns in the boxes. Towards the end he would revisit the three boxes and tell a story of how one represented teenagers, another adults getting divorced and the third the serenity of growing old. He did not discriminate between the individual movements but looked at their capability as a whole. The Ceramicist, especially at the beginning, looked less for the movements and more for points where the movements created shapes he deemed worth preserving in ceramics. Later, he would shift focus toward the movements and their inherent qualities rather than what they could lead to.

Explaining the Experiences of Temporal Forms

What we are looking for here is to better understand the relationship between the experiences and expressions of temporal forms. We take as our starting point the experts’ experiences or their analyses of the temporal expressions and through these we try to understand what qualities of the expressions elicited those experiences. As a means to unpack this relationship we use Boorstin’s (1995) three ways to experience a movie: the voyeuristic, the vicarious, and the visceral. His point is that any experience will encompass some element of all three but with different emphasis. We have found that the experts’ experiences are well explained through Boorstin’s voyeuristic, vicarious, and visceral experiences even if the unfolding expressions of the boxes are not movies with intricate plots.

The voyeuristic experience is that of a “prying observer,” one who critically examines the world as it unfolds before her. The voyeur is captivated if the world is believable and the flow is intact; and she is entertained as long as there is an aspect of novelty. “The voyeur’s pleasure is the simple joy of seeing the new and the wonderful” (p. 12).

The vicarious experience draws on our empathic abilities and tendency to project and test what we see with our own reaction pattern. “If we see a face we have a natural, automatic impulse to define what the person behind the face is feeling, to test that emotion inwardly to see if it is suitable, and if it is, to taste it as our own. If it is not there … we will even try to fill in what’s missing” (p. 65). Indeed, the “viewer performs a dual role, empathy, yes, but something more, he projects his own feelings into the characters.… The vicarious eye sees with the heart” (p. 66).

Finally, the visceral experience is the immediately sensed experience of the unfolding events. The visceral includes experiences like harmony, anticipation, surprise, and even suspense, which in a movie can be how the current unfolding of events is colored by previous knowledge like a bomb on the bus, or a killer in the room.

Boorstin’s three views were not used in the moment of coding but are introduced now as a means to unpack the findings further. The concepts allow us to better articulate the coupling of expressional qualities with the experts’ expressed experiences.

Voyeuristic Experiences

The boxes and their behaviors are not movies. They do not already depict worlds that we have to believe. Instead, these abstract unfolding forms elicit our inherent need to make sense of what we see. The participants are forced to draw on their own experiences to explain what they see. We have chosen to divide these associations into two levels: those that serve merely as a frame to set the scene (a crawling worm or a contracting muscle), and those that are more vicarious experiences (a sense of being hugged or people fighting in the midst of a divorce). Here we will look at the ones that merely set the scene—the ones that arise from being a prying observer.

Is It Alive or a Machine?

The expressions associated with perceptions of the mechanical versus the biological are not easy to determine. The study indicates that discernable rhythms can elicit both types of association and that it comes down to the quality of the rhythm. Sometimes only miniscule differences determine whether a rhythm is associated with mechanical behavior or biological behavior. Breathing or heartbeat associations are not surprisingly strongest in rhythms in which the cadence matches our experience of both—meaning relatively slow and steady. That alone, however, does not seem to do it. Rather, apparently a hint of irregularity is necessary in order to rule out the machine from the association. Indeed, this was the case whether the setup was synchronous or asynchronous as long as a repetitive rhythm with a slight anomaly was discernable. Also, the speed seemed primarily to influence the kind of biological association at play.

I find that there is a certain breathing quality about them. Meaning, they hold a rhythm that is possible to discern. I don’t know if it is completely precise but at least they hold a rhythm which I find expresses something calm. (IxD-1, SMA, synchronous.)

There is still a rhythm over here but it is not machine-like anymore. It is a bit like there is a longer cycle of movements. It is no longer the same little movement. It repeats, it has more like that wave-like… (IxD-1, stepper, asynchronous w/ direct control.)

Well, it pulls in this sort of contraction again and then it relaxes. But it is more alive in a way because it isn’t a totally steady rhythm like the one we had before. (IxD-3, servo, synchronous w/ distorted control.)

Yes it is rhythmical. Of course it is rhythmical, but it is more unpredictable. There are some shifts happening which means that I cannot… I lack the right words regarding the rhythm—a shift that makes it less mechanical and more organic in a way. (IxD-3, stepper, asynchronous w/ direct control.)

And then the breathing, sky, and water. Again, the rhythm. Waves. I was more fascinated by the first two—not the second because it had that rhythm where it sort of moved back and forth within the lines as if it was lapping. Like a fetus lying in the abdomen lapping in the amniotic fluid. Whereas this is sort of mechanical. Before it made that lapping back and forth and it wasn’t… it was sort of in an irregular way. (Ceramicist, SMA, synchronous.)

There is something extremely calming about such a regular rhythm—this is a very slow one. It makes me think of something, pulse or heartbeats or something. (Musician, servo, synchronous w/ direct control.)

Now it feels like it is doing it all synchronous but sometimes it gets sort of out of beat with itself, which is rather—it’s a pretty great feeling. It sort of softens what it is doing. (Musician, stepper, asynchronous, w/ direct control.)

Personality

The participants willingly ascribed personalities and lives to the boxes when the interaction with them was distorted, but never when the participants remained in control through the interactions. Whether they were synchronous or asynchronous, fast or slow, dramatic or calm seemed less important.

The machine appears more in this jerky way. Yes it gets a personality. And this is because I don’t control it as much. (IxD-2, servo, synchronous w/ distorted control.)

Now it is a little unpredictable as if they get their own life. It is pretty cute in a way. Confused. As if they don’t really know what to do—a little indecisive. (Musician, servo, synchronous w/ distorted control.)

It is funny how they are out of sync because then I think it becomes more difficult to see some sort of regularity and it becomes more alive that way. (IxD-3, stepper, asynchronous w/ distorted control.)

It is unpredictable. In a way a little crazy or insane because it appears a little as a compulsion, I mean a weird distortion. Maybe it is because the fabric gets distorted and tense and it isn’t really possible to follow the idea. I mean there is no repetition or rhythm that makes sense in relation to one another. (Musician, stepper, asynchronous w/ distorted control.)

Entertainment

On several occasions the participants gave up interacting and merely stared at the behaviors. The expressions were sufficiently pleasurable and entertaining to engage the participants in a voyeuristic experience. The common denominator for the expressions was their complexity and unpredictability. Indeed, the participants only indicated this level of entertainment when the actuators were asynchronous or the behavior was distorted or both. Again, levels of speed, acceleration, and degree of movement seemed less important.

I feel less of a need to adjust it here. But just sort of look at it and see what they are up to by wringing that fabric.… It is crazy difficult to figure out what they are doing here but somehow still fascinating. Or, you get dragged into it. (Musician, servo, asynchronous w/ distorted control.)

This gets exciting because you really get to see how the material gets stretched to its limits. This is consistent with these two centers—it is fascinating that there are two. If there had been only one it would have been less complex and the complexity is what fascinates. You get a narration. You become curious. There is something temporal in it, right? It is exciting how it develops as we go along. It is a story getting narrated. Something is changing. There is something cinematic in the physical form. (Ceramicist, stepper, synchronous w/ distorted control.)

Well, what happens here? I don’t see a rhythm the same way. It is more unpredictable but still of course within the same frame of movements. The same folds in the fabric but much more unpredictable. Oy! Some completely unpredictable things appear. And now I’m less busy turning the knobs. There is sufficient activity without. Yes completely unpredictable.… There is a sort of aggression in this in a way. A wringing a forceful wringing some in a way is pretty satisfying to look at now when the material gets destroyed. The surface gets interrupted in a way. But I do not need to turn the knobs to have something interesting or meaningful to look at. At least this is interesting. There was a fun reaction at some point when it got really violent and I thought that it might tear but I also thought that it is as if something sort of fills my senses. (IxD-3, Stepper, asynchronous w/ distorted control.)

Obviously our choice of two centers instead of one was to enable us to create more complex forms but it seemed that this aspect was best achieved when they behaved asynchronously.

The asynchronous movements become more interesting. I have the sense that there is something more organic about this shape change here. It is as if the same shape is transferred from one to the other. There is something at play between the two now. This centerline appears—there is a pull, which I can’t recall from the synchronous behavior. (IxD-1, servomotor, asynchronous w/ direct control.)

This works pretty good now. When this pulls in and then afterwards the other pulls in. This makes it a kind of collaboration between them that fits. (IxD-2, stepper, asynchronous w/ direct control.)

What I find most interesting are the transitions. There was a nice one. Nice transition also between the two lines. How many steps can you see, right? At one point it is unperceivable. I lost it. I had a super nice transition before. The waves between the two are interesting. (IxD-4, SMA, asynchronous.)

These are some pointers on what to consider when setting the scene—when designing the expression. In designs where a function is made explicit these associations are likely to be diminished, yet independent movements have a strong hold on our imagination as the findings below further corroborate. While efficiency traditionally is more important than entertainment in interaction design when considering long-term use and perhaps even passive use like with ambient displays etc., entertainment may warrant an extra degree of aesthetic complexity. Indeed, breaking rhythms slightly or providing complex and unpredictable temporal forms seems to go a long way when designing for a vicarious experience.

Vicarious Experiences

While the associations described above were the result of consciously relying on experience, we saw a whole range of reactions that seemed to be more vicarious. They were immediate and strong empathic reactions, some of which even left the participants a bit embarrassed. They were typically visible in their faces or in their bodily movements (clenched fists, blushing, intense concentration, expression of horror or surprise) and the quotes below are their explanations of what had happened. While Boorstin (1995) talked of the vicarious experience in relation to stories acted by characters in movies, the vicarious experiences are here based solely on the abstract behavior of the textile in the boxes. Thus these vicarious experiences are predominantly concerned with simple embodied projections. However, once in a while more complex emotional experiences appeared, such as when participants described feeling caressed and hugged, or even blushed over imagined erotic behavior. Generally, it seems as if the force and acceleration of the stepper motor (Figure 4) elicited particularly strong embodied reactions and only in an asynchronous setting did the more orderly and even-paced servomotor (Figure 6) evoke vicarious sentiments. Reactions of a different kind, but no less strong, were also elicited by the slow and gentle movements in the SMA, which were stitched into the fabric (Figure 5).

Figure 4. Still of the rotation created by the stepper motor. The temporal forms in this setup seemed to elicit strong embodied experiences. See also: https://vimeo.com/144300679.

Figure 5. Still of a contracted form created by the shape-memory alloy (SMA). The stitching in the fabric fostered associations with scars while the temporal behavior was experienced as meditative and relaxing. See also: https://vimeo.com/144300673.

Figure 6. Still of the rotations created by the two servomotors. The temporality of this input output composition was apparently able to excite sexual associations. See also: https://vimeo.com/144300672.

When I sit here and see this I get an urge to clutch some clothing. I’m imagining that I was down there sort of grasping the clothes. That is actually a nice experience. It is a very physical sensation in the hand because I have done that with fabric before. It is very easy to relate to in a way. (IxD-1, stepper motor, asynchronous w/ distorted control.)

This probably has to do with how it wrings the textile and that it isn’t mechanical like the other one was. This one changes pace. It grabs and slowly lets go. It grabs slowly. Lets go slowly. It is very… It becomes more like hands. It is physical. There is a being underneath. It becomes like a being, right? Now I have to touch it! This one is more present. (Ceramicist, stepper motor, synchronous w/ distorted control.)

This is like a sort of erotic experience because it is a sort of caress or a surprise, right? When you touch one another there are some things that will sort of [audibly draws in air] give you a thrill. So you can turn away from it or you can take it in somehow. You have to—how to put it—it is something that suddenly fills your senses. There is something very relaxing about it, you could say. It is as if it is filled in a way and then you relax afterwards. It is like a muscle—it is relaxing in a way. It is as if you give a hug and this is the same. You can also talk about it in other ways but yes this is the same. (IxD-3, stepper, asynchronous w/ distorted control.)

Wow. What is this? I don’t know. It feels very bodily in a way. It has something to do with... I don’t know, but the way they intertwine and sort of cramps. (IxD-2, servo, asynchronous w/ direct control.)

You get an urge to put your fingers down in those. (IxD-2, stepper, synchronous w/ direct control.)

Go back! Go back! You are hurting yourself! … Phew! That was stressful! (IxD-4, stepper, synchronous w/ direct control.)

Here one participant is almost hurting because his association to the SMA is a stitched scar. Less dramatic but also affected was the Dancer, who described a sense of constriction (see also Figure 5).

This is so pronounced. I think that when you look at this healing laceration you can almost feel that sort of sensitive and delicate expression which expands around the stretch and contraction. You can almost feel that form as if it picks in a scab or a scar or something that is sewn together. Not just when you draw in it but also when you squeeze it together of course. It is because the surface is punctuated with the stiches and the irregularity around the seam wriggles back and forth in those curves. It creates a sort of fracture or crack, which is very, very skin-like. (IxD-3, SMA, synchronous.)

It’s because it’s straight and can’t really curve properly because it has to be attached to something that is vertical to work that… It feels constricted in a way. That it cannot… The movement is dictated by the mechanics so it doesn’t feel as free as this one [stepper], where you can get bigger variety of shapes. And maybe that’s why it’s… It’s not ugly or anything, it’s just not, you don’t feel as comfortable looking at it. You get a bit like “ah hi” uneasy. (Dancer, SMA, synchronous.)

After sitting and staring at the movements for several minutes in silence the participants noted about the SMA setup (see also Figure 5):

There is something almost meditative about sitting here and watching the same rhythm over again because it is so complex and dynamic. (Ceramicist, stepper, synchronous w/ direct control.)

Maybe it is because you get the urge to breathe with it. It is sort of the same rhythm. (Musician, SMA, synchronous.)

It is like the box is breathing. It’s like a calm way, when you sleep and you just go [inhales, exhales] without snoring. It is not snoring. It is more peaceful now than it was the first time. (Dancer, SMA, amperes.)

This is very lovely concentration exercise. It is very relaxing. (IxD-3, SMA, asynchronous.)

This one is so calming! You slow down. My interactions are slower compared to the other one. You feel it inside. (IxD-4, SMA, synchronous.)

Finally, several participants experienced a slight embarrassment in the form of blushing or being slightly flustered as they explained their sexual associations with the movements. Not all of them were confident enough in the situation to explain beyond the fact that it was a sexual association, but here are a few examples (see also Figure 6).

Oy! This is almost sexual in a way.… The thing is it has nothing to do with the resemblance of genitals. It is more about the movement. Which is more female masturbation than male masturbation. Yes there is something… Yes exactly more about masturbation. Am I anonymous here? (IxD-2, servo, asynchronous w/ direct control.)

That pace it is… yeah orgasm! Haha. (IxD-2, stepper, synchronous w/ direct control.)

It’s a naughty box. I am not going into a porn description now. (Dancer, servo, asynchronous w/ direct control.)

This gives me the impression of a kind of sexy behavior. It is because it is so relatively slow. It makes the effort of moving all the way into these shapes. (IxD-1, servomotor, asynchronous w/ direct control.)

While the white cotton fabric may have alluded to bed linen or cotton briefs we doubt boxes without movements would elicit similar sexual associations. Indeed, we were a bit surprised by the elaborate vicarious experiences these rather sterile white boxes elicited.

Generally, what explicitly evoked these different experiences is a little harder to pin down. None of them seems to have any correlation with the amount of control the participants had over the movements. The responses seem to have been based on the behavior of the textile, of the combination of the physical and temporal form. And while the more complex projections inevitably must be situated and subjective (depending on the state of mind, etc.) the fact that all of the participants regardless of expertise expressed these vicarious experiences does indicate something about the power of the temporal form.

Visceral Experiences

The participants’ visceral experiences of the temporal forms as they unfolded in the boxes were especially concerned with the harmony between the temporalities of the actuator’s behavior and the temporality of the material connected to it. The experiences were immediate reactions to the sensory input and thus less emotional than the vicarious experiences. Here, especially the IxD experts engaged in a critique of the behaviors and the design of the boxes as part of their reflection over their visceral experiences.

(Dis)harmony between the Temporal Form and the Material

The visceral experiences are concerned with the temporal form of the actuator’s movements in relation to the quality of the textile it is connected to. When the material is challenged to the edge of its anticipated strength or even when justice is not done to the apparent potential of the material it creates a tension. The participants all expressed aspects of this tension but IxD-1 explained it most fully.

I would rather want it slow. Not necessarily all the way down but when we have it about here I can better follow the shape changes in these stretches. I think it becomes difficult [turns up the speed]—my eyes simply can’t perceive the changes. There are too many beautiful shapes I’m missing here. It’s more about… cool that it moves but there are many qualities in the shape changes I don’t manage to catch. This is really annoying! Maybe it is just a bad use of motors and textile—this could be done through 2D images and sound. But at this pace I can hardly perceive the shape change. Maybe if the material were thick leather it would appear calmer. This looks really nervous—it does not suit this material. Maybe it is because this material is easily agitated in the sense that it reacts instantly to the slightest movement. (IxD-1, servomotor, synchronous direct control.)

It is a little bit like if you are dancing in a big ball gown then you shouldn’t dance the jitterbug because that is somehow not the best way to get something out of that gown. (IxD-1, servomotor, asynchronous distorted control.)

The fabric is stretched—the movement exploits some of the qualities in the fabric. It can be stretched and it gives in the interspace down here but without tearing anything. You don’t tear the fabric apart. There is a good interplay between the movement and the fabric here. (IxD-1, SMA, synchronous.)

This works with the slow pace and the fabric. This fits. The movement and the material fit much better. You experience it more naturally in a way. It is not because so much is happening here but in a way you think the rhythm in the material fits now. (IxD-2, SMA, synchronous.)

The harmony as well as the disharmony was also expressed as the root of some of the vicarious experiences described above. For instance, when the participants made associations to orgasms there must have been an underlying harmony or satisfactory correlation between the temporal form and the material. Or when one participant was afraid that the box would hurt itself because the textile was stressed to what appeared to be the limit.

Anticipating Movement

When we experience things in motion we quickly form an anticipation of where it is going—what likely paths it can take. When a movement deviates from these expectations it creates a friction resulting in a visceral experience of surprise or even frustration.

Now I want to see “go stop” “go stop.” But it is not. It is overlapping. (IxD-4, stepper motor, asynchronous w/ direct control.)

It is as if it embarks on a movement but then regrets it and instead stutters—it doesn’t follow through. Of course, this is because I have a sense there is a right way for this to behave. When it has embarked on this curly movement, and I recognize that this is possible, I want it to go all the way into this snail shape. This is frustrating. Now that I know that it could make such beautiful transitions between these shapes… It does not achieve the optimal where the optimal would be a perfect circle or a particularly beautiful snail. (IxD-1, servomotor, synchronous w/ distorted control.)

Here is actually a feeling that it completely takes on this form, right? Yes! This form that I have been talking about, the one that I think must be the purpose of them. They reach the perfect form and then slowly turn back. They actually turn all the way over. Ok this has to be the most optimal setting if you talk about going from one right form into the other right form. The other right form being when they are flat and uncurled. (IxD-1, stepper motor, asynchronous w/ direct control.)

I liked it better when they were both twisting the same way, because the shape there was more attractive to the eye because it gave sort of half of an eight. Whereas here it looks like the movement is cut. There is something missing, there is a connection missing. It looks like there is more tension, whereas before the movement looked, even though they were twisting the same way, the movement looked more connected. They looked more connected to each other. If you know what I mean. (Dancer, stepper, asynchronous w/ direct control.)

Working with the anticipation can be used to create a smooth impression and in a successful combination of physical and temporal form possibly also lead to a satisfactory experience. On the other hand, working against the anticipation can be used to gain attention in the interaction gestalt. For instance, the element of surprise can be used as an opportunity for action.

Generally, finding a good balance between the physical form and the temporal form is clearly not entirely straightforward. Our experience with materials and with their endurance and flexibility obviously plays a role in whether we are “convinced” by the combination or not. At the extremes, as with the anticipated movements, this can be used consciously to capture the attention of a user at the possible cost of frustration, or it can be used to create a more complete sensation.

Discussion

Based on this study we argue that temporal form in computational things enables richer experiences than static objects do. Obviously more is happening when things move and change state, but we also argue that the temporal form evokes qualitatively stronger responses in those who interact and observe. The voyeuristic experience is about being entertained, about experiencing something new; evidently this is easier to do when the possibility of change is included in the toolbox. Our participants expressed that they were entertained by looking at a white cloth exactly because it was stretched, crinkled, and twisted. While we did not study their reactions to static white cloth we feel fairly certain that it would not have proved equally entertaining. Thus the voyeuristic experience would have been shorter and possibly less memorable unless, of course, we had been Robert Rauschenberg. The vicarious experience is about getting emotionally invested; it is about living through someone or something else for a while. This experience can occur with inanimate objects but the temporal form undoubtedly creates a larger repertoire of means to evoke. Further, the—for us—surprisingly strong and elaborate experiences we encountered with the relatively simple temporal forms seems to suggest that the potential for evoking vicarious experiences through temporal forms is greater in quality and not just in quantity. However, further studies would be necessary to corroborate this. Finally, the potential for visceral experiences is increased with the introduction of temporal forms as they now include temporal experiences such as surprise, suspense, anticipation, apprehension, etc. And while we only evoked a few of these with our temporal forms we did evoke them and we believe others are more easily created in less abstract settings where contexts can aid expectations. In other words, temporal forms hold a qualitative power to create experiences. The temporal arts rely on it, but we are only beginning to comprehend the extent of this power in the context of interaction design.

This study has shown how small anomalies in a rhythm can create the perception of biological or animalistic beings. It has shown how non-initiated and erratic behavior will likely be interpreted as a sign of personality, and it has shown how increasing the complexity of a temporal form can capture the attention of the observer for a longer period of time. But it has also shown that the quality of the behavior matters, indicating that forceful and potentially uncontrollable powers evoke strong vicarious responses, as do delicate, timid behaviors, albeit of a different kind, but with orderly and predictable behaviors receiving less attention and eliciting fewer strong responses. Finally, the study has shown that in composing the temporal within the physical we play into a large repertoire of previous experiences and capacity for empathy, which will color the overall experience. Indeed, behavior cannot just be designed by itself but must be developed in relation with the materials it is expressed in, regardless of whether the aim is to create a tension or the opposite.

We see this study as only a first attempt at analyzing and articulating the potential of temporal form in interaction design. The study was deliberately kept abstract as a means to get a first handle on the temporal form in itself; however, the temporal form will never appear in and by itself. We have used theories from both music and film as means to help us unpack the temporal form. None of them is sufficient on its own, but while music theory fits well with the stringency of computation, film theory fits well with the complexity of temporal form expressed through actuators in different materials. We can even say that the temporality of the computer fits the temporality of music, while the temporality of a film fits the temporality of shape-changing expressions. We believe further studies may more meaningfully begin to look at temporal forms in the context of functional interaction design. Not just ambient displays, but comprehensive and intricate designs in which the material and physical forms extend beyond two-dimensional glass and plastic surfaces, and the interaction gestalt comprises more than look and point actions.

Acknowledgments

We owe a thank you to Lars Toft Jacobsen for his technical assistance and to Marisa Cohn and Naja Holten Møller for their comments on earlier drafts of this article. Finally we thank the anonymous reviewers whose comments and critique undoubtedly have enabled us to improve the arguments in the paper.

References

1. Bailey, B. P., Konstan, J. A., & Carlis, J. V. (2001). The effects of interruptions on task performance, annoyance, and anxiety in the user interface. In M. Hirose (Ed.), Proceedings of IFIP Conference on Human-Computer Interaction (pp. 593-601). Amsterdam, the Netherlands: IOS Press.
2. Boorstin, J. (1995). Making movies work: Thinking like a filmmaker. Los Angeles, CA: Silman-James Press.
3. Cage, J. (2009). Silence: Lectures and writings. London, UK: Marion Boyars.
4. Ching, F. D. K. (2014). Architecture: Form, space, & order (4th ed.). Hoboken, NJ: Wiley.
5. Dourish, P. (2001). Where the action is: The foundations of embodied interaction. Cambridge, MA: MIT Press.
6. Dumitrescu, D., Landin, H., & Vallgårda, A. (2012). An interactive textile hanging: Textile, context, and interaction. Studies in Material Thinking, 7(1), 1-13.
7. Frens, J. W., Djajadiningrat, J. P., & Overbeeke, C. J. (2003). Form, interaction and function, an exploratorium for interactive products. In Proceedings of the 6th Asian Design International Conference [CD ROM]. Tsukuba, Japan: Asian Society for the Science of Design.
8. Gallagher, S., & Zahavi, D. (2012). The phenomenological mind. London, UK: Routledge.
9. Gaver, W. W., Beaver, J., & Benford, S. (2003). Ambiguity as a resource for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 233-240). New York, NY: ACM.
10. Gibson, J. J. (1986). The ecological approach to visual perception. Hillsdale, NJ: Laurence Earlbaum Associates.
11. Goldman, A. (2005). The aesthetic. In B. N. Gaut & D. Lopes (Eds.), The Routledge companion to aesthetics (pp. 255-266). London, UK: Routledge.
12. Hallnäs, L., & Redström, J. (2001). Slow technology – Designing for reflection. Personal and Ubiquitous Computing, 5(3), 201-212.
13. Hallnäs, L., & Redström, J. (2006). Interaction design: Foundations, experiments. Borås, Sweden: The Interactive Institute and the Swedish School of Textiles, University College of Borås.
14. Harrison, C., Hsieh, G., Willis, K. D. D., Forlizzi, J., & Hudson, S. E. (2011). Kineticons: Using iconographic motion in graphical user interface design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1999-2008). New York, NY: ACM.
15. Holmquist, L. E., & Skog, T. (2003). Informative art: Information visualization in everyday environments. In Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (pp. 229-235). New York, NY: ACM.
16. Höök, K. (2000). Steps to take before intelligent user interfaces become real. Interacting with Computers, 12(4), 409-426.
17. Huang, C.-C., & Stolterman, E. (2011). Temporality in interaction design. In Proceedings of the International Conference on Designing Pleasurable Products and Interfaces (No. 62). New York, NY: ACM.
18. Hudson, S., Fogarty, J., Atkeson, C., Avrahami, D., Forlizzi, J., Kiesler, S., ... Yang, J. (2003). Predicting human interruptibility with sensors: A Wizard of Oz feasibility study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 257-264). New York, NY: ACM.
19. Ingold, T. (1993). The temporality of the landscape. World Archaeology, 25(2), 152-174.
20. Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbet, M., Ullmer, B., & Yarin, P. (1998). AmbientROOM: Integrating ambient media with architectural space. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 173-174). New York, NY: ACM.
21. Itten, J. (1975). Design and form: The basic course at the Bauhaus and later (revised edition). New York, NY: John Wiley & Sons.
22. 22. Jafarinaimi, N., Forlizzi, J., Hurst, A., & Zimmerman, J. (2005). Breakaway: An ambient display designed to change human behavior. In Extended Abstracts of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1945-1948). New York, NY: ACM.
23. Kwak, M., Hornbæk, K., Markopoulos, P., & Alonso, M. B. (2014). The design space of shape-changing interfaces: A repertory grid study. In Proceedings of the Conference on Designing Interactive Systems (pp. 181-190). New York, NY: ACM.
24. Löwgren, J. (2007). Fluency as an experiential quality in augmented spaces. International Journal of Design, 1(3), 1-10.
25. Löwgren, J. (2009). Toward an articulation of interaction esthetics. New Review of Hypermedia and Multimedia, 15(2), 129-146.
26. Lundgren, S. (2013). Toying with time: Considering temporal themes in interactive artifacts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1639-1648). New York, NY: ACM.
27. Maeda, J. (2000). Maeda@media. London, UK: Thames & Hudson.
28. Mankoff, J., Dey, A. K., Hsieh, G., Kientz, J., Lederer, S., & Ames, M. (2003). Heuristic evaluation of ambient displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 169-176). New York, NY: ACM.
29. Manzini, E. (1989). The material of invention: Materials and design. Cambridge, MA: The MIT Press.
30. Mazé, R., & Redström, J. (2005). Form and the computational object. Digital Creativity, 16(1), 7-18.
31. Merleau-Ponty, M. (1958). Phenomenology of perception (C. Smith, Trans). London, UK: Routledge & Kegan Paul. (Original work published 1945)
32. Nilsson, L., Satomi, M., Vallgårda, A., & Worbin, L. (2011). Understanding the complexity of designing dynamic textile patterns. Paper presented at the Conference on Ambience. University of Borås, Borås, Sweden.
33. Norman, D. A. (2011). Living with complexity. Cambridge, MA: The MIT Press.
34. O’Doherty, B. (1999). Inside the white cube: The ideology of the gallery space. Berkeley, CA: University of California Press.
35. Parkes, A., & Ishii, H. (2009). Kinetic sketchup: Motion prototyping in the tangible design process. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (pp. 367-372). New York, NY: ACM.
36. Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., & Hornbæk, K. (2012). Shape-changing interfaces: A review of the design space and open research questions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 735-744). New York, NY: ACM.
37. Redström, J. (2001). Designing everyday computational things (Unpublished doctoral dissertation). Göteborg University, Göteborg, Sweden.
38. Semper, G. (2004). Style in the technical and tectonic arts; or, practical aesthetics. Los Angeles, CA: Getty Research Institute.
39. Shneiderman, B. (1997). Direct manipulation for comprehensible, predictable and controllable user interfaces. In Proceedings of the 2nd International Conference on Intelligent User Interfaces (pp. 33-39). New York, NY: ACM.
40. Stambaugh, J. (1964). Music as a temporal form. The Journal of Philosophy, 61(9), 265-280.
41. Stasko, J., Miller, T., Pousman, Z., Plaue, C., & Ullah, O. (2004). Personalized peripheral information awareness through information art. In N. Davies, E. D. Mynatt, & I. Siio (Eds.), Proceedings of the Conference on Ubiquitous Computing (pp. 18-35). Berlin, Germany: Springer.
42. Svanæs, D. (2013). Interaction design for and with the lived body: Some implications of Merleau-Ponty’s phenomenology. ACM Transactions on Computer-Human Interaction (TOCHI), 20(1), 8:1-8:30.
43. Vallgårda, A. (2014). Giving form to computational things: Developing a practice of interaction design. Personal and Ubiquitous Computing, 18(3), 577-592.
44. Vallgårda, A., & Sokoler, T. (2010). A material strategy: Exploring the material properties of computers. International Journal of Design, 4(3), 1-14.
45. Weiser, M., & Brown, J. S. (1995). Designing calm technology. Retrieved from http://www.ubiq.com/weiser/calmtech/calmtech.htm
46. Wensveen, S. A. G., Djajadiningrat, J. P., & Overbeeke, C. J. (2004). Interaction frogger: A design framework to couple action and function through feedback and feedforward. In Proceedings of the 5th Conference on Designing Interactive Systems (pp. 177-184). New York, NY: ACM.

		International Journal of Design is a peer-reviewed, open-access journal. All journal contents are allowed to be shared and adapted in accordance under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.