Conversations about AR (with Noa Joulin)

NJ: I am currently researching how AR can become a core component of the visual identity of a fashion brand for my master thesis. AR is an extension of the working creative canvas and I find that intriguing. I love the idea of bridging the virtual and the tangible and how it defies the laws of the real. If you want to make something fly or transform — you can. There's nothing with more freedom than a space with no rules. I really want to explore that! AR feels like an unexplored playground.

NG: I’ve played around with both Spark and Aero. For my master thesis project I ended up using Adobe Aero. It just clicked with me more. Spark definitely has its perks — you can craft nodes, add some neat interactions, and really make something production-ready. Aero is great for a quick, one-time mock-up. At the end of the day, the question is whether you want to deep dive into the technical aspects or if you just want to sketch out a concept.

NJ: It wasn't until I stepped outside the university bubble and connected with people in the XR (extend reality) industry that I got truly motivated. I started reaching out to professionals who work with virtual and augmented reality, and they got very excited. I had a few “Oh, this is innovative”-moments and I thought to myself: “Okay, finally!” I think because the technology is still so new, there's just a big gap yet to be filled.

NG: Of course that's a challenge you'll have to face. In the end you can do a nice project, but you’ll have to somehow explain and defend your concept to your professors: “This is what I did and this is how it's innovative” But I think it's good to explore these topics and you shouldn't be stopped by this.

I occasionally work with this company, called Prefrontal cortex, a design agency mostly focusing on 3D, AR, and VR. In one of their projects — ADEL & REBELLEN — they created 3D models of the city, to showcase historic events. They heavily integrate augmented reality and create applications using immersive storytelling. In this case, you find yourself in a medieval city and you can talk to all of these people.

A project I coded for them — STADT KLIMA — is kind of similar. They did extensive research and how to reuse the public space in a more environmentally friendly way. Then my part was to condense this into a website. We made the landing page with these different models and various audio elements, and you can jump between different models. There’s always the possibility of viewing them in AR. That way you can place the models in a public place that you are familiar with and see how they look like in life size.

NJ: I’d like to explore the graphic identity of a fashion brand with focus on the visual branding aspect. AR is starting to emerge in the fashion industry. I questioned how to integrate AR in graphic design and branding smoothly. Who can I speak to? What kind of audience would be receptive to it? Some of the most experimental projects I came across have that fashion context. How can I reflect that AR perception, that fashion concept in branding? It is rising and real after all. I want to make AR an integral part of the visual identity, it’s a bit like if you remove it, the identity isn’t complete. What happens if you do that? How does it change your creative process? How is it perceived and received?

NG: There’s this project I really like: We AR in MoMa. Basically they repurpose existing elements of the public space. The cool thing about AR is that you can you have this extra layer of digital information. Being digital you can change and update things very quickly, also it’s fairly accessible. In this project I like how they repurposed the existing exhibition space and art pieces of MoMa by layering their own meaning and interpretations on top of them. I can imagine that in fashion as well.

NJ: I was really drawn to your 36 days of type experiments, in which you meld AR with virtual type and body interaction. From your explorations, what insights did you gather about the interplay between the virtual and physical, especially with the body as interface?

NG: My colleague, Katja Rempel, and I developed this approach to embodied interaction during a workshop we conducted at Burg Halle. We began with a straightforward exercise: asking students to think about everyday gestures, like a motion they make in the shower or brushing their teeth. We then repurposed these bodily actions for the digital space.

In my view, that’s a good approach for kickstarting such a project. If you start from the perspective of, “I have a letter and I want it to grow using some bodily gesture,” it can feel counterintuitive. However, if you flip it around and ask yourself “what's fun to do with my body?” and then explore digital applications for that gesture, it's a more engaging approach. That’s exactly what we did. The tracking is done by media pipe. You can try it out if you go to Codepen: Media Pipe. It's developed by Google, so everything is open source.

The detection is powered by AI, leveraging machine learning. It scans input from your webcam, identifying and tracking specific points, like your hand and fingers. These tracked points can then be linked to digital elements. The same principle applies to the body, so there's potential for broader applications in embodied interaction or user interfaces.

If your experimentation centers on the body, especially with fashion, it naturally aligns with these experiments. Maybe it's a prototype, it doesn't have to be the final product, it can serve as a foundation. Visualizing it this way could be the starting point for crafting an interface.

In my experiments, most of the interactions are quite simple, I made the letter size change with finger movement, if it’s far away it's a large letter size, if it's closer, it's a small letter size. This is a very simple translation, but it could be applied to all kinds kinds things. There's someone in my course that built a body drum set: when you touch your shoulder, you hit the snare, if you touch your other shoulders, you hit the hi-hat. The body as interface is a really fun thing to explore!

NJ: Is there anything specific that initialised your journey within the AR context?

NG: I think I started with VR, but for me, it always felt a bit strange. You have these bizarre devices on your head, and it really cuts you away from reality. AR feels like the perfect blend between the physical and the digital world. With AR, you can still see your surroundings, you're not cut off from it. If you walk around a room with a VR set on, it feels like you could break your neck as you walk out the door or something. AR is just an extra layer of information, so you still have the ability to reuse and access anything that exists in the physical space.

I think, I was attracted to AR because of my master thesis topic, I wanted to create a unique learning experience. Many pedagogues say that the best learning experiences are built by integrating many different senses. If you can smell it, touch it, see it and hear it, it becomes a very impactful learning experience. AR allows you to do this mix of sensory and digital input.

Merge cube for example made a good compromise — You have this cube, it’s a holographic toy, you can touch it, feel it in your hand. So you have something physical, that means your hands can remember something haptically. Then with your phone you can access information, like a model of the universe. You can interact with 3D objects using augmented reality, integrating digital experiences with tangible interaction.

NJ: Was that an inspiration for you?

NG: I really like how they mix sensory with digital experience. If you combine the physical and digital world, it's just a stronger experience, with stronger imprint in your brain. Nevertheless the current state of AR still feels a bit limiting. You always have to have a device in your hand whether it’s a tablet or a phone. It's not 100% practical yet. But maybe at some point, when we will have extensions for our eyes or something similar.

NJ: There’s this company called Mojo Vision, they are developing AR contact lenses. Then there is AR glasses, of course, but they're just not very accessible or popular yet. I actually had to change some of my thesis text after the Apple glasses release, that just demonstrates how current of a topic AR is.

NG: But even though you can see both physical and virtual space with the new glasses, somehow it still feels like you would be deep diving in the sea. You're still cut off, it's not very practical. Probably at some point we'll have more intuitive ways of accessing AR. This means now is a good time for prototypes.

As designers we have a privilege. Before things gets applied in a wide commercial space, we can make all the experiments. We can try things out. These playful explorations might set new industry standards in the future. I think it's very important for us to go in crazy in a way. With new technology evolving — especially artificial intelligence — artists need to go crazy now and show what we can potentially do with this. Then there might be some really interesting commercial applications inspired by these experiments. But we have to do them, it is our obligation.

NJ: So, you think that AR has a space within the visual communication context?

NG: Of course, it's all about visual communication. It can be challenging, I also research a lot about user interfaces. In the web, I have to retain some sort of user interface. And in AR, or in 3D, it's just a completely new way to approach those interfaces, we have another dimension. So, what do we do now? Do we have flat patterns floating in space? Or do we have 3D icons? Or do we have something completely new? There might be some really cool ways to make these interfaces.

NJ: In your experiments you control things with your body and with your hands. That might be applicable in an interface.

NG: That would be awesome. I mostly did these things out of curiosity because I wanted to see how it works. But maybe it could really be a more intuitive way to create user interfaces.

There’s also this huge realm of barrier freeness and inclusivity. When I do web coding, I have to be more and more careful about inclusivity. There are some things that you have to respect when creating a website, it should work for blind people or people with hearing impairments. In that context, that body interaction really has potential.

I was recently invited by Rudolfo Quintas for a workshop he's conducting with elderly people. He wanted to test some of my tracking experiments with elderly people and see how they react to it. We want to look at this interaction idea and see if it helps people who struggle with digital devices, maybe it will lower the technological barriers for them.

NJ: Is research a big part of your design process?

NG: It always starts with curiosity. In many cases this leads me to unexpected places and I end up learning about stuff I never thought I would find.

NJ: During all this experimentation with AR and the body as interface , did you come across unexpected challenges or discoveries?

NG: During my master thesis I struggled a lot with the technical hurdles. I wanted to create working prototypes. You don't know what devices your user will have, if they have a very old phone, a nice smartphone or if they have a large display. That shouldn't be the decisive point but that’s why in the end I was aiming for a proof of concept rather than a working product. I just wanted to explore and show the potential of AR.

NJ: It's a topic so anchored in the present and future. I feel like it's hard to do something that is finished and fully operational. I have done some experimentation with Blender and Adobe Aero. I was exploring how I can put something in a space, explore scale, special interaction, can I walk around it? What happens if I touch it on the screen?

One difficulty that I face is that according to where you are, and according to what you have within your virtual space, the visuals will, more or less integrate seamlessly with that space. Sometimes, if you want something to be on the floor, but there is a lot of things in the way, it's just not going to look good. So, it becomes about finding the right balance and making it work in all kinds of spaces.

NG: I encountered this problem as well: you cannot really layer digital and physical objects just yet. Currently the digital layer will always be rendered on top. Sometimes you’d want something that’s interacting with tings in your surrounding, something real, but once the object is detected, the digital layer stays in the front which can be limiting.

NJ: In your thesis project you used AR markers. I personally came across three possibilities so far: The classic QR code, simple and basic but it might not be aesthetically what you want. The AR marker you mentioned. But for it to work smoothly, it needs to be clear, contrasted, and straightforward. Besides that, I encountered problems with depth. Distance when scanning remains of importance, that marker plays a vital role, if you’re too far away your device won’t be able to recognise it but also, if your visuals’ scale is big, there isn’t that possibility of exploring it from different angles or take a step back to see it fully, once the marker leaves your phone’s lens, the AR design does as well.

I few months ago, I took part in a workshop led by Eleonora Asparuhova and Adam Cooke from Manchester Metropolitan University’s SODA (School of Digital Arts) in the UK. The workshop was titled Deviant Diorama’s and we explored illustrated media and the dynamic storytelling experience of location in a blended media environment. We worked with augmented reality and at one point they gave me this little white and flat plastic round, which can act as an AR link but doesn’t require visuals.

NG: Is that like what you have on passports, chip enabled?

NJ: It’s the same technology, it’s called an NFC tag (Near-Field Communication). What you have in passport is an NFC chip. The NFC tag has an NFC chip inside of it as an antenna. The chip holds the information, and the tag helps send that information when a phone or another NFC device comes close. It uses radio waves for transmission.

An augmented reality experience often needs an app to run which can affect the viewing process of your audience. People often get discouraged or exhausted, the app system can become an obstacle. If you work with Artivive for example, you’ll need Artivive’s app to access an AR design. With Adobe Aero, you could use app clips which lets you use part of the app without downloading the whole thing, like a preview basically.

NG: Do you know how to code a little bit? You can use modelviewer as a starting point. I think it's comparatively simple. If you work with 3D, it will allow you to embed models to your website and then open in AR mode on your device. That’s a smooth way of publishing AR experiences. You could also use Spark AR to create Instagram AR filters. There are some restrictions with file size but the cool thing is that everyone can access it.

NJ: I’m really interested in the storytelling aspect of visual identity. You explored it in the educational context, I want to focus on a brand narrative context; how to shape and reflect the story of a brand. How do you utilise that platform in a way that you couldn't with tangible products?

NG: There are different approaches. One approach is the ‘We AR in MoMa’ exhibition we mentioned earlier. You take public space with some recognisable objects things could serve as image markers, like a poster in the subway, or maybe the outline of a specific building. You can then use those as markers to start your AR narrative. This can be an interesting way to create site specific experiences. By launching your narrative through physical reference points in the public space, you can make your users walk around and let them rediscover certain areas, buildings and objects in AR.

NJ: So the experience becomes faithful to its location. It’s like digital street art, the city becomes your canvas.

NG: The markers are one approach, but you can also go ahead without them and have your virtual objects just exist in the surrounding space, play with scale or whatever you want.

NJ: I want the AR layer to really reflect that crazy, colourful, and magical childlike imagination the fictional fashion brand is based on. The visual identity will be an experience introducing you to that world and concept.

My goal is to celebrate the freedom that we all once had as artists. I want to reconnect with the boundless and extraordinary, the otherworldly. That’s the aim. I'm bringing this very classic graphic design context of branding, but I feel that the idea behind the brand allows for a wilder approach.

NG: I think you could focus on creating these crazy AR visuals interacting with your body, or even create some kind of AR drawing platform, where someone could draw on a body.

NJ: I really like idea of creating an experience, of entering something new.

NG: Allow yourself some time to just go crazy and in all directions. I think that's where the interesting stuff really happens.

NJ: Sometimes I’ll start working with an idea in mind and I end up with something completely different. That's when it gets good in my eyes. I like to embrace my creative process and my mistakes.

NG: Usually when people think of AR, they think of face filters on Instagram. If you avoid that and create an experience, I think that will be already a success. Not that those filters aren't valid, but you can show potential use cases that go beyond social media marketing. I'm curious to see what you will make of it.