Conversations about Embodied Interaction (with Desirée Solenberger)

Nahuel: I saw some of your experiments with movement, dance and letters. Can you tell us a bit about what you are currently working on?

Desirée: I am working on my master’s thesis on embodied interaction. I just finished the website documenting the initial research. My main interest is in movement, and how it translates to interfaces. For my thesis I put a special focus on typographic systems.

N: Your work seems grounded in choreography and intentional movement. How does that shape what you’re exploring, especially compared to other approaches in the field?

D: Last semester I was thinking about movement more through the lens of dance. I found it interesting to bridge dance and design. I’ve seen work by Luna Maurer and Crystal Zapata who both explore this intersection, and that really resonated with me.

I’ve been dancing my whole life, so bringing that knowledge into my work felt natural. In choreography, there’s this distinction between intentional, structured movement and improvisation, but both have intention behind them. What fascinates me about that distinction is: I see a lot of work in embodied interaction, like Zach Lieberman’s projects, where people move their bodies and something happens on screen. But I don’t always see a focus on what that movement actually is, or why that particular gesture matters. Because the field is still so new, there’s often not a standard way to perceive or categorize these interactions.

For my thesis, I want to bring that choreographic thinking into how I approach typography. Not just “what works, ” but what’s intentional about the movement, and how that intention shapes the digital outcome. You mentioned trying to build a vocabulary of basic gestures.

What initiated that project, and what patterns are you discovering?

N: It started somewhat by accident. I was offered to teach a course on body tracking, and through that I began exploring embodied interaction practically. But the more I worked on it, the more I wanted to understand the history. What have people done before? Is there a tradition here?

The challenge I kept running into was that there really are no standards yet. With traditional computing, you have mouse clicks, swipes … a basic vocabulary everyone understands. But with embodied interaction, every project seems to reinvent things from scratch. So I thought: there’s so much work happening right now, maybe some patterns do exist. Maybe we could collect them, document them, and build a basic vocabulary of motion.

It’s still very much an open research field. I work on it alongside other projects, when I have time. But I think it’s helped me focus. Now when I encounter work in the field, I’m collecting it, trying to structure it, and looking for commonalities. And technically, I’ve noticed interesting shifts too. A few years ago, everything was done with Kinect cameras for depth sensing. Now there’s this wave of people using 2D tracking with tools like MediaPipe or TouchDesigner, often combined with sound and MIDI.

It feels like the field is gaining traction and finding new directions. I began with direct integration of MediaPipe, a trained model to recognize body parts in a video or webcam input. Currently, I’m using ml5.js in a p5.js environment. I find the documentation is more accessible for visual-oriented people.

Are you also using ml5.js?

D: I’m also using ml5.js and webcam or video input to track human bodies. Have you heard of Ted Davis? He teaches interaction design in Basel and is adamant about creative coding. His classes have been an important introduction to rethinking our relationship to computers and digital-based design work. Although, it’s up to us students where we position our practice on the scale between digital and analog. Much of our work originates right in the middle.

N: One thing that attracted me to embodied interaction is that it’s not only happening on screen. All the expressions, gestures and motions that we’ve adopted and trained since childhood are relevant once again. Why not simply use those to talk to machines?

D: Yeah, so last semester I was thinking about this movement more in the dance context. Dance may be planned or intuitive, but following my previous training I worked with improvisation based on chosen movement qualities. Interestingly, I would still find a tendency towards screen-first thinking, often fitting movement in the box of the webcam’s field of vision. Recently, I’ve been exploring combinations of gestures, such as adjusted arm angle and pinches, or molding with your hands. I’ve also experimented with less controlled movements like blinking.

N: If the idea is to move past sitting and typing, then maybe controlled hand gestures are not such a difference. Motions involving the entire body could create maximum contrast. At the moment I feel it’s still a very open field. Of course there are also commercial applications already in surveillance or sports when it comes to training and optimizing your movements.

I had an interesting project with an artist from Portugal. He worked with elderly people and drawing. In the first phase they were drawing and painting. In the second they were exposed to digital tools, including AI and we created a face tracking tool where they could draw with their mouth and eyes. It was fun to see people with less mobility using these gestures to paint. This adaptation of embodied interaction could be a field adjacent to art therapy. There could be more funding than in the purely artistic field. I’d also like to see it more in museums as interactive installations.

D: What do you think makes an interactive installation successful?

N: Using the camera and interacting with your body is a great hook. But then the question is how to keep the attention of your audience. Is the goal short visual play, or is it about drawing them a little deeper, to keep them discovering and exploring. Is it possible to build something that creates meaning or adds value? Also, how intuitive is the installation, how much do you have to instruct your audience what to do? Can they discover it all by themselves? Again those are all open questions with no clear answers at the moment. It’s a bit like speculative design, showing possibilities without having to provide final answers. A great outcome for your thesis could be an exhibition where the audience can test your typographic prototypes.

D: I am experimenting with different camera setups. Sometimes I track live movement, also with a second camera. For some interactions it makes sense to see what’s happening live, but it can also be distracting. The other set-up I use are pre-recorded motion snippets. When you’re dancing, it’s almost impossible to watch a screen. I end up with standard movement phrases and can iterate over them with different graphic experiments.

N: That’s an interesting tension. The difficulty to focus on what you see on the screen and then the movement itself. With a digital mirror, you see yourself interacting, plus this layer can be abstracted. In the project I previously mentioned, the elderly people did not want to see their faces since this was distracting to them. We ended up using only color and a few keypoints for recognition.

D: It’s interesting to explore other ways of tracking too, like space and environment.

N: There are many people working with object detection. If you want to interact with something on the screen, usually it’s something that you put there statically, like a rectangle or a button, but if it’s real objects, it’s much more dynamic, right?

There’s one question I have to ask. What’s the role of AI in your work?

D: When I first started, I limited its use as much as possible. I watched coding tutorials and looked to understand the basics. When I got stuck, I would prompt the AI to help connect the sketch components. I started from very little experience with coding in September, so with AI I was able to learn but still create something at the same time.

N: Would you say it helped you on your learning journey?

D: Yes, in the sense that I could prototype what I was envisioning. But also no, in that it’s an easy fallback, instead of encouraging you to find different solutions. It’s an interesting conversation right now among students and our professors. Recently I’ve downloaded the in-terminal AI, Claude Code. As an experiment I wanted to throw every idea at it and see if it will work in the end.

N: Just to see how far you can go.

D: Yeah, and surprisingly, sometimes I get back something that I’m very happy with as a starting prototype in an afternoon or a few hours. When it’s a visual goal I’m going after, I want to prove that it can work. But then in the long run, as I get deeper into specific experiments or sketches I’m sure I’ll hit a wall where it’s really necessary to go back and understand exactly how things work. How do you approach this when teaching?

N: I don’t have anything against using AI, as it became an integral part of my work too. What I noticed is that some students were completely relying on prompting. I was surprised how far they actually got. Maybe not everyone needs to know how to code in detail. Other students though took a very structured approach. They built small prototypes, documented and compared them. They used AI as a learning opportunity to understand how the code really works. This is my approach as well. When I first started, there was only Stack Overflow and Google. I had people I could ask questions but of course they had limited time and capacities too. I would find a library with documentation but one small piece prevented me from going deeper. For me, AI can bridge these gaps.

What I also try to do is to build lots of small prototypes on isolated problems. I can just copy a few lines I wrote, document the code, and internalize it as my own vocabulary. Sometimes I also push the AI to keep going and vibe code. There is usually something new that I don’t quite understand. So I force myself to isolate that with AI, boil it down to its bare minimum and make it my own again. But there’s still no clear way how to work with AI and it remains an open question.

D: And then there’s Ted Davis, our professor, who’s against using AI when coding. He understands code as material, an art form, and a creative process. It’s a bit like saying you don’t want AI to make an illustration.

N: I can understand that. For me, AI should not dictate the final outcome, but it can definitely be part of the process.

D: I also see students that only use AI and its output and aren’t looking at the code.

N: I also understand all the ethical problems that people have with it. But still, I think it’s something really powerful, too powerful to just be ignored. One thing that it’s really good at is analyzing and mimicking language. Since code is a language it’s really made for that. It can analyze, teach, structure, and build for you. You have to find your comfort zone of how much control you want to give up. But I think there are ways to stay controlled and to stay focused together with AI. It’s a bit like concept art, right? Where you just write instructions at some point.

D: Totally.

N: In terms of the motion catalog, are there basic motions that became part of your work or that you think should be standard vocabulary? You already mentioned the pinch or molding with your hands, for example.

D: In one experiment, I would simply walk and track the changing distance between my wrist and hip. With dance phrases, I use a single descriptive word as the prompt. Movement quality is a big topic in choreography. Maybe I want my movement to be sharp, or flowing. I’ve been doing dance for a long time, so I can already imagine what that would feel like or look like. I prefer thinking in this abstract way. Some movements, like drawing with the wrist felt not so different from simply using mouse X and Y coordinates.

N: That’s probably why we see it so much. It’s natural to translate mouse movement as a first step since it’s learned behavior. It’s also interesting that some motions like clapping feel natural, but are quite challenging to track. But for molding, how would you even track that?

D: I was looking whether the hands are outstretched or caved in. I am usually thinking more of the individual anchor points than tracking coherent gestures.

N: For the end user it’s just a motion or gesture they do, but it’s hard to find a way of tracking and detecting sometimes. It usually ends up with measuring the distance between a few points. I also compared angles between points to get the eye direction for example. With angles you could rotate stuff too, or combine them with distance values.

D: Sometimes I would use combinations of XY positions to retile the video and make anchor points appear in different parts of the sketch. I like the idea of anchors not always having to completely align with the body. To fragment the body into different parts.

N: As a closing question : what do you think is the potential of this whole field? What would you like to see more and what do you hope to contribute?

D: What’s interesting to me is this idea of intuitively engaging with technology. In the field of graphic design I see a lot of potential to reintroduce the body into ways of making. This is something that was taken for granted before the digital era, but now we kind of forgot about it. It’s also about agency. Being aware of our movement gives us more power in these interactions. What we currently see in terms of tracking are mostly linear skeletons, which is already a very deconstructed view of reality. I’m curious, as these tracking systems get more and more sophisticated, what will be possible beyond this linear way?

N: I fully agree with everything you said. The reason I got invested in this topic is that I hope to see an alternative from sitting eight hours straight at a desk. Learning to use a keyboard first with all ten fingers as well as working with a mouse feels unintuitive to me. I hope the whole idea of using screens is a momentary thing that we did for a few years and can eventually stop. You also mentioned that the creative process is something where the full body is invested. I think that’s a great point. For painting, body motion is a big part of the creation.

I hope that this field enables us to express ourselves authentically again through using the whole body. I see it as an invitation for reconnection to our own body. We are so focused on analytical thinking. Maybe we can be more expressive again and thereby communicate better.

D: Movement is its own kind of language. Whether it’s something that people just do as automatic gestures, or as intentional movement in dance or theater. It’s kind of a daunting area since there’s so many possibilities.

N: Thanks a lot for taking the time! It was really interesting to talk to you today.

D: Thank you too. I really enjoyed it.