Real-Time Piano Onset and Velocity Detection

PUBLISHED ON MAR 8, 2023 — CATEGORIES: publications

I’m happy to present my last work, accepted at the EUSIPCO 2023 Conference:

Onsets and Velocities: Affordable Real-Time Piano Transcription Using Convolutional Neural Networks

ONSETS&VELOCITIES builds on some good ideas from previous literature to develop a convolutional neural network with several (IMHO nice) properties that allow to perform real-time transcription of piano onsets and velocities on modest hardware (e.g. a regular laptop), while keeping very good performance!

Below is a qualitative example. I invite you to read the paper for more technical details (and cite it if it is of any use to you 😎 ):

      title={{Onsets and Velocities}: Affordable Real-Time Piano Transcription Using Convolutional Neural Networks},
      author={Andres Fernandez},
      booktitle={{EUSIPCO} Proceedings},

The IAMĂșsica project

This work was performed in the context of the IAMĂșsica artistic research project, in collaboration with the amazing EulĂ lia Febrer Coll and Joan LluĂ­s TravĂ© Pla. The main goal was to “explore the usage of AI applied to real-time audio analysis to create new tools to support musical practice and bring a new quantitative approach to artistic research”.

We presented our project and conducted a live workshop at the Conservatori Superior de MĂșsica de les Illes Balears in September 2022:

At the top of the page you can find the presentation slides (in Catalan and Spanish). In order to conduct the workshop, I prepared a real-time software demonstration that looks like this:

Among other things, it allows to record and load audio files, extract their onset and velocity information, and analyze the velocity distribution and statistics for a given note selection as well as its time-alignment with respect to a grid. We demonstrated this functionality and discussed with music students and scholars possible applications and implications of this technology, which has only recently reached sufficient qualitative and practical levels (and we hope that our work contributed to that).

On a personal note, the IAMĂșsica project was a key đŸŽč opportunity to get acquainted with the full lifecycle of a research project, from conception and redaction of the grant proposal to the final divulgation stage.

It was a true pleasure to collaborate with Eulàlia and Joan, and the project also combined all things that I love: music, computing, research, divulgation, and now we add Mallorca to the list: ⭐ ⭐ ⭐ ⭐ ⭐ experience!

IAMĂșsica was supported by research grant 389062, INV-23/2021 from the Institut d’Estudis BaleĂ rics, and developed with the collaboration of the CSIB.

TAGS: audio, computer vision, gui, keyboard, live electronics, machine learning, music, piano, pytorch, transcription