Master 2013 2014
Stages de la spécialité SAR
Time series knowledge mining to uncover signal-symbolic relationships in musical orchestration

Site :Trac-Time series knowledge mining to uncover signal-symbolic relationships in musical orchestration
Lieu :IRCAM - Equipe représentations musicales
Encadrant : Philippe Esling Carlos Agon
Rémunération :Environ 400€/mois



Orchestration is the subtle art of writing musical pieces for orchestras. Hence, it requires combining instrumental properties in order to reach ideas of timbre coming from the blending and combination of individual spectral properties. Most of music pieces uses several instruments and its the knowledge and use of their different spectral qualities in combination that allows to create sought emotional effects that composers can sculpt and vary over time. Recently has rised the idea of computer aided orchestration, which encompass open issues from auditory perception, music analysis and composition, signal processing, computer science and combinatorics and represents a highly combinatorial NP-problem. Indeed, if we consider already the range of notes, dynamics, chords and playing modes provided by a single musical instrument, we can foresee that the number of combinations available with an orchestra is almost infinite. We recently proposed a new system for temporal and abstract computer-aided orchestration [1]. The idea behind this system is to use a sound target, ie. a sound example to match. The algorithm then tries to find a combination of instruments and notes (under a set of constraints) that can allow to reproduce any sound using only acoustic instruments. We use multi-objective optimization [3] in order to overcome the combinatorial complexity embedded in musical orchestration. Over this, we proposed a new approach based on time series analysis [2] that allows to account for evolving sound targets. In this internship we intend to push the topic orchestration further, by exploring the relations between signal and symbolic information layers. By using the latest research in multiple time series knowledge inference, we intend to uncover the relationships that may exist beween the spectral and symbolic properties of different musical lines.

Research project and aims

When multiple instruments produce different aspects of a musical orchestration piece, the measures of spectral properties might exhibit some form of correlations. Automatically evidencing such correlations from very large datasets of multivariate time series might lead to extremely powerful approaches. Particularly, the temporal di- mension can allow to explicit causality between observed phenomenon from different sources. Previous works have remained focused on the spectral reconstruction aspects of musical orchestration. Hence, the main goal of this internship is to extend the work previously done by taking a complementary approach, by automating the inference of relationships between signal and symbolic layers for musical orchestration. The idea would be to construct this knowledge incrementally from an automatic analysis on existing scores. Then a cross-analysis with correspond- ing signal recordings and audio features could allow to obtain deeper and more interesting relationships between symbolic and signal descriptors. This approach could lead to a powerful knowledge database on signal/symbolism relations. Therefore, a large amount of knowledge should be gathered and represented efficiently on both the symbolic and spectral aspects of instrumental capacities.

The final goal of this internship is to extract temporal motifs from data related to multiple instruments playing simultaneously. Hence, it introduces hard questions : what representations to associate to time series (symbolic, signal, hybrid) ? What sort of relations should be explicited between different sources ? How to modelize these relationships and perform reasoning based on this model ? Finally, interesting phenomenon might be more or less visible, depending on the temporal granularity of the representation. Hence, an interesting aspect of this work would be to study multi-resolution representations to model time series with different granularity levels.

  1. Gather a database of musical orchestration pieces with signal-symbolic knowledge for further processing. Further establish a proprietary format for working with the database.
  2. Analyze symbolic and numeric representation for multivariate time series, including the multiple granularities approaches. Find the most appropriate knowledge source and representation for further processing [7].
  3. Analyze the efficiency of time series knowledge mining approach [4,5,6] applied to multi-source and multivariate symbolic and numeric time series of musical orchestration.
  4. Analyze various learning and motif mining approach for multi-source temporal analysis.
  5. Propose extraction and learning algorithms to automatically find temporal motifs stemming from multiple time series simultaneously.
  6. Develop a working prototype of the proposed approach.
  7. Evaluate the proposal on the large signal-symbolic dataset of musical orchestration pieces gathered specifically for this study. Discuss and evaluate the outcomes of the prototype with musical experts and composers.

This internship project embeds stimulating questions from theoretic, academic but also industrial interests. The previous softwares are already used by a large community of music composers. Hence, the internship student will also be in regular contact with contemporary music composers in order to discuss potential needs and features.


[1] Esling P. “Multiobjective Time Series Matching and Classification”, PhD thesis, IRCAM, UPMC, 2012.

[2] Esling P. and Agon C. "Time series data mining", ACM Computing Surveys, vol.45, no.1, 2012.

[3] Esling P. and Agon C. “Multiobjective time series matching and classification” IEEE Transactions on Speech Audio and Language Processing, vol. 21, no. 10, 2013.

[4] F. Mörchen, “Time Series Knowledge Mining”, PhD dissertation, 2006.

[5] F. Mörchen, A. Ultsch. “Efficient mining of understandable patterns from time series” Data mining and knowledge discovery, vol. 15, no. 2, 2007.

[6] T. Guyet, R. Quiniou. “Extracting temporal patterns from interval-based sequences”, International Joint Conference on Artificial Intelligence (IJCAI), Jul 2011, Barcelone, Spain.

[7] Q. Wang, V. Megalooikonomou, and C. Faloutsos. “Time series analysis with multiple resolutions” Informations Systems, vol. 35, no. 1, pp. 56-74, 2010.