CodeKlavier meets Augmented Reality 📱 🌱

The CodeKlavier is a system which enables a pianist to code through playing the piano as a performative experience. The CK+AR (CodeKlavier + Augmented Reality) is the next step in our artistic mission of combining classically inspired pianistic, new media, and live coding performance practises into an innovative, united genre. In the CK+AR, the piano playing becomes the interface for coding and the coding unfolds as piano playing and as a visual AR experience, making the three mediums indistinguishable from one another (AR visuals + piano playing + live programming).

The coding is done in the CodeKlavier language, which implements lambda calculus to create numbers and arithmetic operations that can be translated into symbols to specify Lindenmayer systems. L-systems are formal grammars that consist of a set of symbols (alphabet), a starting point (axiom) and a set of production rules which define symbol substitution. The production rules are applied recursively to successive generations, creating a tree-like structure. These structures, together with additional information extracted from the piano playing, is then used to construct the AR visuals as well as a sonic scape.

Our inspiration for working with L-systems comes primarily from its roots in nature and organic growth which we would like to bring to the musical and programming paradigms. Whilst L-systems are already widely used in generative art and music, we are interessted in exploring how this fairly simple model can be used by the piano coder to express complex structures and how its rules will shape the pianistic improvisation. Furthermore, Lindenmayer’s work was done here at the University of Utrecht in 1968 and building upon it through artistic exploration makes the Netherlands a particularly special location for the development of this project.

We are collaborating with Patrick Borgeat for this CodeKlavier extension. Patrick is creating the augmented reality and L-systems engine.

Impression

Please watch our short trailer for an impression

Setup

The CK+AR is best presented as a combination of a live performance (15 mins), a pre or post-talk, and an interactive installation where the public can create their own L-system “trees” with assistance from us if necessary. However, this structure is flexible and may be adjusted in discussion with the creators and venue.

There are two primary modes of presentation:

CodeKlavier with the AR as visuals on a large shared screen: This requires minimal equipment and setup time where the AR visuals are projected on a large screen in the performance space. The equipment required for such a presentation is an acoustic 88 key piano, Stereo PA system (with subwoofers and monitor if available), large screen and powerful projector or large LCD screen.

CodeKlavier with the AR as a true interactive AR experience:

In this type of presentation the audience is free to move around at all times so as to experience the augmented reality visuals from different perspectives. Each audience member will be able to either download an Android app or use an Android smartphone or tablet device provided for the performance. The equipment needed for such a performance would be all of the above alongside the provision of ±30 android tablets or smartphones.

We would like to thank the Creative Industries Fund NL and our other sponsors for making this project possible. www.codeklavier.space