A real time performance system for interactively layered audio sequences Leonardo Aldrey Supervisor: Prof.
CONTENTS Abstract I. Introduction .........................................................................................................1 Motivation ..................................................................................................................................... 1 Other systems ............................................................................................................................... 2 Loop Machines..................................................................
Abstract Background: The cost is decreasing for including electronic equipment in modern styles of musical performance. This has opened up a wide variety of possibilities for creating new forms of performing live music. Some multi-instrumentalist musicians have developed techniques to record layers of audio, during their own performance to use them as background tracks.
I. Introduction Motivation The role of computers in current musical styles has become very important. From composition, to performance they represent a wonderful tool that has expanded the musical world to an endless number of possibilities. In musical performance, computers are used for many purposes such as sound synthesis, sequencing, processing effects, sampling, playback of background tracks, among others.
Other systems The following sections describe briefly the characteristics of existing systems. They are of special interest for the later section of the report where the needs for innovation are explained. Loop Machines Stand-alone machines (Boss RC-50) Figure 1 Boss RC-50 loop station This loop machine is considered to be one of the best loop machines available in the market. It consists of a pedal controller with built-in software to work as standalone equipment.
2. Loop or one-shot playback It allows you to decide if you want to play a phrase only once or if you want to loop it. 3. Background prerecorded samples. There is a built-in guide that provides background rhythmic to play along. 4. Tempo sync This option causes a previously recorded phrase to be played at the current patch tempo. 5. Pedal operation in single mode Description of the function of each pedal, depending on the configuration of each phrase.
5. Patch tempo: tempo reference for a number of functions including the guide, tempo sync, and loop quantize. 6. Guide settings: to determine the use of background rhythms or audio samples. 7. Patch name 8. Patch level 9. Patch phrase change (immediate/loop end) 10. Patch: fade in, fade out. 11. Simultaneous start: allows the user to determine if tow or more phrases will be started at the same time. 12. Patch inputs and outputs 13.
GENERAL FEATURES 1. As many layers as you want: Since it is a computer program, the storage limit is determined by the computer, which means that there is a great deal of capacity in the system. 2. Store a complete jam session: Guitar rig affords storage of all the layers that you create during the jamming session with the loop machine. 3. Pedal assignation: The pedals of the foot controller can be assigned to different buttons and controls of the system e.g. record, stop, undo, redo, etc. 4.
There are many features 3 but among the most important ones are: • • • • • • • • Recording, editing and mixing with up to 192-kHz and 24-bit processing power (pristine 32-bit floating point internal resolution) Intuitive handling and extensive customization options to accelerate workflow Projects port readily to and from all standard audio and video cutting systems Powerful multi-channel capability throughout the signal path 32-bit Audio Engine optimized for multiprocessor operation Up to 192 physical inpu
set the sampler to, upon ceasing recording, commence playback, looping, overdubbing, reverse direction, or my favorite, the old “To and Fro.” Scenario Another very useful instrument found in Reaktor’s library is “Scenario” which is a live tool mainly used as a sampler to trigger, play and transforms samples in a live performance. This instrument can be found in the User libraries 4 . Figure 5 Scenario: Mixing instrument design in Reaktor.
Discussions with artist Juana Molina (Argentina) Figure 6 Juana Molina (eminent multiinstrument artist who acted as reference for the project) To discover some the limitations and advantages of the system based on a set of loop machines, several discussions via e-mail were held with Argentinean singer/songwriter Juana Molina 5 . The different issues mentioned by the artist were: Latency: the artist expressed that some systems present latency (delay between the physical action and the system response).
Todos Santos (Venezuela) Todos Santos 6 is a Venezuelan band in the genre of electro-acoustic music. Their performances combine computer elements with the use of the computer for creating beats, playback of background tracks and sound effects. The band works with a software called Ableton Live which is a sequencer that has some special features for live performance. The discussions held with the band, were mainly to understand the features of the program as well as the limitations during performance.
II. Music Theory Aspects Musical structure The system developed through this project is based on certain principles of musical structures that are commonly applied in popular music. Some of these concepts are briefly mentioned below, for their importance in the implementation of the system. Sections Most popular music is based on a sectional form. The sections are usually Verse, Chorus, Bridge and Vocal runs (small variations of the chorus).
III. Design considerations Choice of programming platform Several platforms were considered for programming the “Scene Builder”. Among them were Pure Data, MAX/MSP, and Reaktor. Each had its advantages and disadvantages. Reaktor was chosen for the following reasons: Flexible programming environment: Reaktor has the advantage that it has many readyto-use instruments, effects, samplers, sequencers, etc.; and it is possible to open them and inspect how they were programmed.
Figure 7 Pedal controller for the hardware part of the project Features • 10 banks with 10 user-editable presets each • 2 expression pedals with freely assignable MIDI channel, controller number, and range • 5 MIDI program change and 2 MIDI controller commands per preset (simultaneously transmittable) • MIDI note-on commands transmittable for trigger and tap-tempo applications • 2 programmable relay-controlled switch jacks, e.g.
Usability “Scene Builder” is designed for the multi-instrumentalist musician who desires to perform songs by recording clips from different instruments and distributing them as sound layers to create a song. It could also be used as an advanced loop machine to perform with a band, but with the consideration that a machine will be playing audio so the musicians have to synchronize with it. The artist process: from composition to performance. Steps: 1.
time in configuring the system during the performance. Equalization, channel mode and evolution (Playback, Record, Overdub, Transfer, Do nothing), mixer configurations and other effects are stored for each scene. 2. Maximum parameter control with fewer pedals. Since the system is able to create a template of the configuration of the system along the song, the pedals are use only to control the transitions, while all the changes in the system are automatic during the performance. 3.
IV. Description of the system Overall structure SCENE NAVIGATOR Scene Navigator is the brain of the system. This is where the user creates the template of the song. There is a Scene memory which stores all the values that characterize each scene. The counter to read and write the audio files is implemented with two macros; Clock gate and Sample pointer which behave according to the status of the program and the performer’s use of the pedal.
Scene navigator blocks and description Panel SCENE’S MEMORY MIDI MIDI vector composition Actual Scene scroll bar Event table (storage) Song start Record to Playback or Overdub CLOCK GATE Playback or Overdub to record Different cases that generate changes in the “clock gate” signal Scenes change SCENE’S NAVIGATOR Five RPO lists per track RPO counter Three RPO Selectors INTERNAL STATUS Someone recording? Previous Mode Actual Mode Next Mode From recording mode? To recording mode? Transfer yes? (On
Figure 10 Scene’s Navigator Structure. Main blocks of the system 1. Scene’s memory: This is the memory of the system, used to store the configuration of each stage. A graphical description of this macro is shown in Figure 11. The main blocks of this macro are given below. • State navigator scroll bar: graphical interface for showing the actual scene.
Scene’s memory Actual Scene scroll bar System to store the behavior of the system in each scene Select scene to program Pedal button Panel element Variable Configure parameters of the panel for the scene Which will be the sequence of internal modes for each track? RPO mode T.0 RPO mode T.1 RPO mode T.2 RPO mode T.3 RPO mode T.
Figure 12 General blocks of the Scene’s memory 19
2. Clock gate: Generation of a binary signal that determines when the loop sweepers start and stop. Rising and falling edges are used to save and recall flags that determine if the state is occupied (with audio) and also its length. Loop sweepers will be explained in fallowing category. Figure 13 shows the logic of this component.
Figure 14 General Structure of the Clock gate macro.
3. Internal Status: This macro is in charge of developing the logic to let the user choose the evolution of modes that will occur within a scene. It is also in charge of taking all the decisions, during the performance, regarding the status of each channel and how they interact. The actual status of every scene is based on the RPO evolution list in the panel and other variables like the RPO counter, laps performed, etc (Figure 16).
RPO Button Pedal button Panel element Variable RPO INTERNAL STATUS =0 Song Start? a=0 RPO counter Resets after every Scene position change =1 a=1 RPO Counter = RPO counter + RPO Counter = RPO Modes List Scene 1 1 Record Record Play 2 3 Play Overdub 4 >5 Play Overdub Overdub Transfer Do Nothing RPO Modes List Scene 1 Record Play Play Overdub Transfer Play Play Overdub Transfer Do Nothing Figure 16 Effect of “RPO button” over a scene's internal status.
4. Next State initialization This macro manages the audio transfers between scenes. The user has several scroll bars in the panels to select the origin of audio clips that will be imported to the actual Scene • Transfer yes? Button: This button determines if audio will be transferred to that track from a previous scene. • Scene origin scroll bar: Scroll bar that shows all the scenes and where the user will determine the scene where the audio is going to be imported from. • Block: Next state Init.
5. Sample pointer system: This part of the system is in charge of the logic required to address and manipulate the audio samples stored in each track (Figure 18). Scene’s length and flag arrays This part of the system holds the description of each scene regarding if it’s occupied or not, and also the length of the scene. Contains also the logic to determine a scene’s length based on the length of the other scenes that constitute the background audio.
Oscillator Freq = Sample rate Read Loop Sweeper Pedal button Panel element Variable Sample pointing system Rising edge Clock gate Also resets when the user changes Scene Counter = 0 Laps performed =0 Counter increments Counter = Counter Max? Yes Restart counter Loop length? Counter max = inf Global Rec.
Internal Structure of the system that is in charge of storing and showing the length of a Scene. Internal Structure of the read Loop sweeper.
Audio tracks This part of the structure contains the audio tables (2 if working with stereo) that will store the audio for each track. There is also a logic needed to control the audio table’s inputs and outputs. This system is build through the fallowing blocks and it’s also shown in figure19 and 20: • • • • Write position: block that selects if the Read or Write counters will determine the read position. Only in Record and Transfer modes, the position will be determined by the write counter.
AUDIO TRACKS (One per channel) Storage function Matrix (999999, 30) Audio Table Properties Interpolation, Display units, etc Graphical interface Adjust the display to show current scene and zoom in and out according to the scene’s length Write enable Write position X Write position Y Read position X Read position Y Input Output Depending on the channel’s mode, if the user has selected to transfer data, and the number of laps performed within that scene, this macro enables o disable the write
V. Work in progress Things remaining to be designed or implemented 1. Be able to start song in scene different from zero. 2. Immediate/loop end modes to work. For the moment, only the modes of Record and Transfer work with immediate configuration, and Playback, Overdub and Do Nothing work with loop end. This new feature of the program doesn’t seem to be really important at the moment even though it will allow the musician to switch Scenes without having to wait until the Scene ends. 3.
Optimization possibilities Most of these ideas are intended to improve the design. Due to time constraints they were not included in this version of the program. 1. Group Scenes. The idea is to be able to create groups of scenes that are played chronologically. This will allow the user to create “Super Scenes” that will be loops of sequences of previous loops. 2. Create bitmaps and skins to improve the graphical interface. Some buttons could even be animated to make the interface less static. 3.
VI. Development history Performance needs The first stage consisted in analyzing performance styles in which the musicians combine the use of electronic equipment with their instruments or voice. The goal was to improve the quality of the show by allowing the musician to express their musicality with the assistance of technological equipment, without having the machine as the protagonist of the performance.
The substitute of the sequencer had to be a system able to store audio in an organized way, combined with a control logic to explore the different clips of each track that while being graphically intuitive for the user. Approaches rejected Record in tape decks: tape decks constitute an important CPU load, and therefore only few of them were allowed to be used at the same time.
should have the possibility to select the different modes and also the order in which they will be produced. These transitions should be triggered mainly by the RPO, PREV and NEXT buttons in the pedal controller. Approaches rejected The initial approach was to give the user the option to choose in a menu list, different RPO evolution modes. These evolution modes were for example Rec_Play_Overdub_Play_Overdub and the sequence was determined in a “Fixed Style”.
Important variables were created to make these decisions, some of which are mentioned below. • Global To Rec This variable determines if any of the tracks is going to enter a Record State. This is used for several reasons. For example if a scene starts transferring audio, and after this is going to enter in Record mode, the length of the scene should not be determine until the user closes the Rec mode. Therefore Global to Rec can avoid storing scene’s length that is not definitive for the Scene.
VII. Quality control tests and results This Section of the report shows the tests that were performed to the system. They are most of the possible combinations that could be made by the user. During this quality control test, most of the problems were solved to secure the functioning of the system. Single Track tests Internal mode evolution tests: (without transfer button pressed) Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Record Observations: Works Perfect.
Internal mode evolution tests: (with "transfer" button pressed, but Next Scene Init = Prev Scene) Mode 1 Mode 2 Mode 3 Record Play Overdub Observations: Problem The display is showing a wrong size.
Tests for two tracks Internal mode evolution tests: (Starting song) Track Mode 1 Mode 2 1 Record 2 Nothing Play Recor d Mode 3 Mode 4 Overd Play ub Mode 5 Play Play Play Test 1 Observations: Problem When pressing RPO button of the first track, since the system is new record mode in the same scene, the length of the first clip is and therefore the RLS = RWS Overd 1 Record Play Play ub Recor 2 Nothing d Play Overdub Overdub entering a not stored Record Play Recor d Play Overdub This config
Test 7 Obervations: 1 Careful in its use. When using a Second record mode within a Scene, a new length of the Scene is determined. Therefore if the second record mode is longer then the first one, the RLS will be equal to WLS and therefore pre-existing audio in the table will be played back. Alternative, to make second record equal os smaller or Split Scenes.
VIII. Appendix Terminology Template A form that contains all the configurations of the program for each scene. Represents the result of the programming stage of the software and it is used as the framework for the performance. Scenes Scenes are time intervals where the program has a fixed configuration and the musician is performing with the same instruments. Clips A clip is the audio contained in one track of one scene. All the clips in a scene represent all the audio contained in that scene.
Scene Builder by Leonardo Aldrey User Manual
2
TABLE OF CONTENTS 1 2 What is Scene Builder?............................................................................................ 4 System Requirements and Recommendations......................................................... 5 2.1 Hardware ..............................................................................................................5 2.2 Software................................................................................................................5 3 About the manual ...
Operation Manual 1 What is Scene Builder? Scene Builder is a computer program that allows you to create multitrack recordings during live performance. After programming your song, you will be able to perform with several instruments and record the different layers of your composition by using only a foot controller. It is designed for multi-instrumentalist musicians that intend to create their background tracks live.
2 System Requirements and Recommendations The requirements are the same as those for REAKTOR 5. A computer with the following minimum specifications is needed: 2.1 Hardware • • • • • Pentium III 1 GHz/ Athlon XP 1.33 GHz. 512 MB RAM Up to 1GB free HD space. A sound card compatible with Windows XP CD drive 2.2 Software • • Windows XP Reaktor 5 or later In addition Scene Builder requires a MIDI foot controller with at least 6 pedals and a MIDI interface.
5 Elements of the front panel 5.1 Output mixer • • • • • Equalization Panning Level Sends Solo/Mute This Mixer modifies the signal coming from the audio table into the outputs of the audio card. Its controllers are stored in the event table allowing further adjustments like equalize, pan, control level and other parameters in a different way for every scene. Figure 2 Mixer for the four tracks 5.2 Scene navigator # Scenes fader This fader determines the number of scenes that the song will have.
Current Scene scroll bar This is a scroll bar that lets you navigate the different scenes of the song. The position of the object changes when you press the “NEXT” and “PREV” buttons in the pedal controller. It has also a numeric readout to show the number of the Scene. Figure 4 Current scene navigator Planning buttons immediate/loop end, write, unplanned, flow, approve, and disapprove. The first four planning buttons are controlled in the front panel of the software.
Next Scene init: Audio origin scroll bar, transfer yes/no button, RPO evolution “Next Scene init” is a segment of the “Scene Navigator” that shows how each scene will be initialized. This initialization is done for all four channels in order to determine which will be their role in each scene. • • • • • Audio Origin scroll bar: This scroll bar lets you select from which previous scene you would like to import audio. To confirm your selection of audio to be transferred use the “transfer yes/no” buttons.
6 Quick Start Guide 1. Duplicate the original folder of “Scene Builder” that contains the ensemble and the table. 2. Change the name of the ensemble for the name of the song. 3. Open the ensemble with Reaktor 5 or later versions. 4. Connect your foot controller to the MIDI interface and also your instruments to the audio card. 5. Connect the output of your audio card to your speaker system or headphones. 6.
7 Programming The programming stage is when you will create the template of your song. Here you will configure the role of all the elements of the system for each Scene. For this you will have to compose your songs always thinking of what you are capable of performing, and how you will perform and distribute the different musical elements that constitute your song. 7.1 Audio Settings 1. Connect all the instruments that you will be using. 2.
Example: In the first your starting mode has to be record. The first time you press RPO the song will start and then when you press it again the channel will change to Playback and you will hear the last audio clip that you performed. If you press the RPO button, this will allow you to overdub, which means that you will be recording over the last clip in that scene. This internal evolution will allow to ad different layers within a scene. 6. Select the audio data transfers.
8 Performance You have now created a particular “template” that only needs to be filled with the audio signal you produce during your performance. All the different clips that you perform will assemble like a puzzle. The smaller pieces are the clips you perform. Several clips from different tracks, played at the same time, will be your scenes, and all the scenes together will build your song. 8.
9 Pre-configuration (Event table) This procedure is not needed if the user duplicates the original folder of the Scene Navigator. This folder already contains the table that you need to run the program. For each song created in Scene Builder, there has to be a folder in the computers hard drive, where the ensemble and event table are stored. The Event table is used to store the value of all controllers of the front panel for each scene. This table has to have a size of x = 30, y = 100.
10 Adding new effects to the audio tracks To add a new effect to the ensemble you have to: • • • Route the audio lines, from the input, through the effect and then to the input of the audio table. Select the parameters you want to configure different for each scene. activate Go to the properties of each controller and clicking on the icon the MIDI parameters and assign a controller note according to the “MIDI assignations for the Scene memory” shown in the appendix of the manual.
11 Appendix 11.1 Technical information about the program structure 11.2 MIDI assignations for the scene memory To store the values of all the parameters in the panel in the “Scene memory” of the program there must be assigned, for each control, a component in the MIDI vector of the instrument. To add an effect or other instrument to the program (and be able to configure it differently for every Scene), you will need to assign each parameter to the MIDI vector.
69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 Transfers yes? Track 1 Transfers yes? Track 2 Transfers yes? Track 3 Transfers yes? Track 4 X X X X NEXT button (pedal) PREV button (pedal) RPO button (pedal) Mute Track 1 Mute Track 2 Mute Track 3 Mute Track
19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 17
11.3 Terminology Template A form that contains all the configurations of the program for each scene. Represents the result of the programming stage of the software and it is used as the framework for the performance. Scenes Scenes are time intervals where the program has a fixed configuration and the musician is performing with the same instruments. Clips A clip is the audio contained in one track of one scene. All the clips in a scene represent all the audio contained in that scene.
