Startled fruit springing from sudden burst of reflective light
Swimming in the Light
Meta, Reality Labs studio: Everything as Input
Timeline
October 2022 - December 2022
Roles + Fields
AR technical researcher, ML generation,
Computer vision, environmental sensing
Mixed reality proposal
Methods
Unity, C#, RunwayML, Blender, Hololens, Arduino + sensors, semiotics research for: perception, privacy and power
This mixed reality prototype explores light qualities as environmental input: how would objects in mixed reality take on behaviors and responses of daily circadian rhythms?
(context)Computer vision systems are actively transforming our visual field into new forms of machine sensing and controlling, turning everything within its field of view into an input.
As environments are increasingly observed by autonomous cameras, there is an emerging algorithmic point of view to interact with.
How can we build and influence a technological value system as it expands into the social sphere?
(01)To first understand environmental computation, I explored algorithmic context recognition on a busy tourist street in St. Petersburg.
Using negative space formation amongst throngs of tourists, I trained an ML model to recognize when a picture was being taken, based only on crowd spacing.
In understanding this environmental signal that points to the action of a camera being raised, I found that ambient signals can be traced, interpreted and computed.
(02)
Informed by ambient crowd signals, I turned to light signatures as inputs to influence object interactions in mixed reality.
Through research, when lighting conditions in mixed reality degrades, objects are known to “swim”.
(03)
Implementing a light meter in Unity, I simulated various lighting conditions and light signatures of temperature (K) and luminance (lux).
Then I built a library of swimming primitives to interact in in these conditions.
Primitives swimming differently (shivering, hovering, nestling, bobbing, etc) in warm, fluorescent and neutral lighting.
(04)
Connecting the libraries in Unity to Arduino light sensors, the swimming behaviors now react in real time to ambient light signals.
Now able to recognize the time of day through light signatures, objects in mixed reality behavior correspondingly - nestling during the dim indoor evening and springing to life in bright daylight.