Zurich-based duo of architects, Maria Smigielska and CompMonks created a new development project on the series of mixed media installations of an analogue-digital ecosystem called Proteus. Proteus 4.0 focuses on interacting and communicating between a digitized material and humans through gaze-tracking and emotion recognition. The series has drawn from the complex behaviour of the ferrofluid material, which constantly changes its shape from dots to meandering stripes and coagulated regions through the invisible force of magnetic fields. The behaviour of the material is hard to control or simulate by nature and, as such, is taken as a value of diversity and exploration in the process of ephemeral and non-repeatable pattern generation in this generative and interactive artwork.

While former versions of the series were designing a single mode of interaction for a single physical space, the artists have now shifted to web-based interactions enacted by personal devices (personal computers, mobiles, tablets). The communication between the artwork and its beholders thus spans across the diversity of personal time dynamics and detaches itself from the walls of the museum. Embedded sensors in each of these personal devices also allow for a finer grain of face and gaze information, and a more sophisticated interaction powered by AI.

Conceived as an immersive and non-site-specific installation, it combines both the plethora of interactions happening on the web and its direct experience “on-site” with a dedicated interactive device. To extend the aesthetic vocabulary crafted across the series, this new version expresses the aspects of distribution through the design of spatially scattered displays embedded in a space frame. Individual display cells are complemented with reflexive and transparent panels across the space to render a unified spatial experience that occurs anywhere and anytime across the networked planet. Among them, one is actually interactive and enables the visitor to take an active part in it.

###################################

In Dialogue: Artist Talk with Tereza Teklić (Kontejner, Zagreb):

KONTEJNER: Both of you are architects by vocation. How did you start working on mixed media installations that use digital and interactive technologies?

Maria Smigielska and CompMonks: Architecture is transdisciplinary by design. It always conveys different streams of scientific and artistic discourses about our societies. The digital became an increasingly important topic of transformation in architecture since the second half of the 20th century. For most historians, it has a strong root in medieval and late renaissance thinkers. But schools and offices started to pick it up more seriously when design and fabrication softwares became more sophisticated. We both have been trained to work in a digital world and to think of the other doors it could open for architecture. Almost naturally, we went on to pursue that track and research on new technologies for architecture at large in the university context. But the language needed another dimension. It felt like a natural move to seek means of self-expression through media art. Because we could not find ourselves elsewhere to develop a more adequate vocabulary about the digital. It is, for both of us, a more open and empowering way to approach built forms of mixed-media objects. A different perspective from which we seek to address architecture.

KONTEJNER: Proteus, the project that you are currently developing within the EMAP residency, is focused on the complex behaviour of the ferrofluid material. What made you start researching this material in the first place? What have you learned about it by placing it in different frameworks, physical and digital?

Maria Smigielska and CompMonks: Ferrofluid is a synthetic and completely artificial material developed in the 1970’s by NASA as a rocket fuel that can be moved by magnetic fields in a weightless environment. The same magnetic forces that are used to move this material in outer space, became the vector we use to build interaction in grounded space. We see both forces and the material itself as one, exhibiting visually complex behaviours. From dots to meandering stripes and coagulated regions. For us it became a material allegory of the Greek god Proteus and its ability to shapeshift to avoid foretelling truth. The material itself has an unpredictable behaviour that emphasises uncertainty, and as such, became prevalent in the process of volatile pattern generations. Exposure to such an abstract and contrasted black and white visual brings out a constant search for signification through human visual intelligence. Not surprisingly, working with such physical material has been challenging to bring up to exhibition standards. First installations started small and we focused on intimate interactions. Progressively, we shifted towards a complete digitization of the ferrofluid. At first, it allowed us to begin exploring larger scales and richer spatial interactions involving collective experiences. But it also brought the idea of a layered aesthetics between the original analogue material and its digital surrogate.

KONTEJNER: Proteus is currently in its fourth iteration. How did the work and research change from its first iteration developed in 2018? What will be the focus of the newest version and how will it differ in terms of technology and set-up to the previous ones?

Maria Smigielska and CompMonks: The Proteus series has iteratively evolved along 3 main vectors: a spatial interaction moving between 2.5D displays, architectural objects and immersive enveloppes; a layered interplay between analogue and digital versions of the ferrofluid material; the diversification of forms of interaction from intimate to collective and distributed within the digital realm. Secondary vectors allowed us to be more playful in the creative process while exploring many different ways to produce a physical display of this material, either discrete or continuous, in various resolutions. We also explored a diversity of technological instruments to capture vision-based information and research on behavioural and physiological responses. As attention and gaze are tightly coupled, Proteus has been focusing on visual perception, including a face tracking system (Proteus 1.0), capturing neural signals through a brain-computer interface to couple machine and human intelligences (Proteus 2.0) or extracting visual regions of interest (ROI) based on skeleton tracking for similar coupling (Proteus 3.0). While former versions of the project always involved an interaction in a single physical space, we now seek to shift to a web-based interaction using any kind of personal mobile devices. On one side, embedded cameras in each one of them gives us higher resolution data of gaze and attention based on facial features. This allows us to refine our interactive and generative system. On the other side it gives us the opportunity to explore a completely new aspect of interaction for us that is scattered, distributed, and unconstrained to a single physical space. Interestingly, we also aim to produce a physical installation that expresses these aspects through the design of spatially distributed displays and reflectors supported by a space frame. Proteus 4.0 is the next milestone in our series as it expands the vocabulary we have been crafting together since 2018.