Sonification
Once we had a scoring framework in place, we followed it to code the data in the Excel spreadsheet. This Case Conversion file documents how language and expressions were converted into numerical values using the scoring framework.
We then removed the discourse analysis data from the spreadsheet, keeping only the numerical values. The result was the Data Breaches Sonification Data spreadsheet used to create the sonifications in TwoTone.
Demos
EXAMPLE 1
Example 1 is the default output in TwoTwo. As documented in the Example 1 Code Book, the default for any data set uploaded into TwoTwo is C major with diverse instrumentation, a duration of 4:47, and a tempo (BPM) of 60. Loading the Data Breaches Sonification Data spreadsheet into TwoTone will generate an identical sonification without manipulating any variables.
A significant amount of time was spent playing around with TwoTone. The program’s constraints can feel limiting initially, especially with the finite selection of instruments. A prime frustration was that early experiments produced very conventional musical compositions.
The examples below were created while I was becoming familiar with TwoTone. Listening even to snippets of each will make audible the ways I adapted to the constraints of the application.
Test 1
Out of the four, this is by far the most conventional composition. It is created this in C major and used diverse instruments to represent each variable.
Test 2
This example charts the experimentation with different tempos and major and minor keys, with an emphasis on experimenting with the different ways of manipulating the instruments and observing which of the instruments could produce intense sounds.
Test 3
Example three was a turning point in the process. This version is in D minor and there is significant experimentation happening with distorting the instruments. It also uses double bass on core variables like breach and perpetrators, which I transported to example 2.
Test 4
This example documents the full extent of attempting to stretch the limits of the instruments in TwoTone. To produce the distortion and droning sound, the track tempo was experimented with. The harp and trumpet were set to a tempo of 12x. These were important discoveries imported into examples 2 and 3.
It was through experimenting with TwoTone that I learned how to work within the constraints of the application. Notably, understanding the limitations of the application led to important insights for how to manipulate the instruments to produce different sounds to map the data onto, the outcomes of which are discussed in examples 2 and 3.
Further, experimentation enabled me to develop and refine a process. For example, very early on, I was not devising a code book for the sonification. However, it became apparent that the process was more important than the output, and rather than producing only one sonification that could encompass the project, it was appropriate to the experimentation to create a few examples to demonstrate the possibilities for sound data.
EXAMPLE 2
After months of experimenting with the data set in TwoTone, I decided the next step was to choose a key to work within. After careful consideration, I chose D minor. The notes in the D minor scale are D, E, F, G, A, Bb, C, and D. Traditionally, D minor is used to express melancholy, introspection, anxiety, uncertainty, despondence and similar states and emotional intensities.
Before the 20th century, associating musical keys with specific emotional or qualitative characteristics was a common practice. This understanding was part of the shared cultural experience among those who created, performed, and listened to music. For instance, when composers like Mozart, Beethoven, or Schubert wrote a piece in Ab major, known as the ‘key of the grave’, it was expected that many in their audience would recognize this association as well.1
The tonality of D minor in classical music is frequently linked with somber, melancholic, or even dark “humors”.1 This can be used to convey a sense of tragedy or sorrow, while in rock and pop music, D minor has been used to create emotional intensity. The Beatles’ Come Together (1969), Phil Collins’ In the Air Tonight (1981), New Order’s Blue Monday (1983), Wham!’s Careless Whisper (1984), Sade’s Smooth Operator (1984), Beastie Boys’ Root Down (1994), Radiohead’s I Might Be Wrong (2001), Beyonce’s Crazy in Love (2003), and Drake’s Hotline Bling (2015) are all in the key of D minor.
In the context of sonifying data, this historical practice of key association can be leveraged to enhance the emotional and qualitative impact of the auditory representation. By carefully selecting musical keys that align with the desired emotional response, data sonification can become a more powerful and intuitive tool for conveying complex information. In this example, D minor was an appropriate choice for the data set because it could augment and potentially distort the themes and observations to emerge from the discourse analysis. We observed many tensions in how data breaches were framed as crises and concluded that D minor could offer a heightened perception of the tensions, intensities and harmonies in the dataset.
As a first step, I considered which data would be assigned to which key and sound/instrument and why, how to represent linear and nonlinear data, and whether to stay within or defy certain conventions. I decided immediately to remove the ‘Year’ and ‘Date’ fields after the dataset was imported into TwoTone as I was not particularly interested in having this temporal data represented in the sonification least in part because the data was already sorted chronologically to chart changes over time. Removing these two fields allowed me to focus on the core narratives and themes that emerged through the discourse analysis.
With example 2, I chose a limited number of instruments to work with, specifically the trumpet, harp and double bass. Based on my previous experiments as documented in examples 1 to 4, I wanted to continue to manipulate some of the instruments to produce distortions. For example, setting the track tempo to 12x for the trumpet and harp resulted in a droning sound. I isolated the ‘perpetrators’ and ‘breach’ variables as the core narratives. Both are represented as a double base in D minor with a 2-octave range. The primary difference is that the perpetrators track has a tempo of 2x and the breach track has a tempo of 3x to produce variation and tension between the two. Together, the tracks produce the following:
‘Perpetrators’ and ‘breach’ anchored the sonification and produced a melody to assist with mapping the other variables to an instrument, key, scale, octave and tempo. I chose from the outset not to use arpeggios to diminish ornamentation. These were some of the initial decisions made to assist with structuring the sonification.
Below, is a detailed explanation of the reasoning behind the choices made for specific tracks. Following this section is the complete sonification
EXAMPLE 2.1 SIZE
The trumpet is at 12x tempo to represent size. At 12x, the trumpet produces a persistent low droning and changes to a higher pitch to represent large data breaches without the sound being too overbearing.
EXAMPLE 2.2 PERPETRATORS
Because the perpetrators and breach were the anchor narratives of the sonification, the double bass was chosen to represent these. Along with the drums, the bass is typically the instrument that provides the backbone to a composition. However, here, this relatively quiet instrument was applied to amplify a melody instead. The track uses 2 octaves to augment the pitch or the range of ways in which language is used to describe perpetrators without it overly dominating in the sonification.
EXAMPLE 2.3 COMPANY NAMED
The harp at 12x tempo represents the variable company named. At this tempo, the harp produces a machine-like sound. At a lower pitch it sounds like drilling and at a higher pitch it sounds like hammering. It is set within a 1 octave range to augment the binary coding (yes or no) we used in the scoring framework. The constant droning in the background sonifies the dearth in corporate accountability in news stories.
EXAMPLE 2.4 BREACH
As the companion narrative to the perpetrators track, breach is represented as a double base in D minor with a 2-octave range. While the perpetrators track is a 2x tempo, the breach track uses a 3x tempo to differentiate it from the perpetrators track and to differentiate between the two tracks. Like the perpetrators track, it is set within 2 octaves to augment the variation in the levels of intensity used to describe the breach.
EXAMPLE 2.5 DATA
Data is represented as a double bass in A minor with a 1 octave range and 2x tempo. These presets were chosen to produce the backbone to the sonification. The track is marked by noticeable sections of the same pitch with discernable variations. The data variable is the track that provides the backbone to the sonification, much like the way so many facets of contemporary life is characterized by, and through, data.
EXAMPLE 2.6 RISK
Like size, risk is represented as a trumpet at 12x tempo, but it is in E minor key. The trumpet was chosen for how it can augment the variability in how risk is represented in the research data. Further, because we understand risks and crises like data breaches as not exceptional to but rather an outcome of datafication, the persistent droning of the risk variable throughout is reflective of this.
EXAMPLE 2.7 COMPANY RISK
As the companion narrative to company named, company risk is represented by a harp at 12x tempo but in F minor to complement and differentiate it from company named. Although this variable has more variation than company named, it is still relatively monotonous. In turn, those few sections sonifying a documented risk to a company are discernable.
Example 2 Sonification
Example 2 leans into a notable distortion of nearly all sounds that could be recognized as traditional instruments, amplifying their effects to levels commonly heard in experimental genres such as PC music, industrial, or noise rock. This distortion is a deliberate effort to push the limits of the technology, resulting in the creation of new sounds that aim to put listeners in unexpected affective states. The deliberate distortion and amplification of the synthesized instrument sounds suggest the often overwhelming and disruptive nature of data breaches, while the contrast with occasional ‘pop out’ sonic moments can be understood as a representation of a sudden shift in how language was used to describe breach events, which often coincided with particular cultural or geopolitical moments in the West.
EXAMPLE 3
Example 3 was derived by using example 2 as the starting point. Only some of the tracks have been altered to observe the subtle changes between examples 2 and 3. If the instruments in the tracks constituting the melodies changed, what would the effect be?
Perpetrators and breach remained the anchoring narratives to which risk was added. Rather than use the double bass to represent the core narratives, I chose the trumpet at 12x. As previously noted, at this tempo, the trumpet produces a droning sound. The perpetrators and breach tracks are both in D minor with a 1 octave range. The only difference between the tracks is that the perpetrators track is set at a 12 x tempo and the breach track is set at a 6x tempo to differentiate the two tracks slightly. The risk track was not altered from example 2 and is represented again by a trumpet at 12x tempo in E minor. Together, the tracks produce the following: EXAMPLE 3 CORE MELODIES
I wanted to experiment with distorting the key melodies to consider how the framing of data breaches as security crises are also distorted and shaped by the social, political and historical contexts in which these events circulate.
Below, is a detailed explanation of the reasoning behind the choices made for specific tracks. Following this section is the complete sonification.
EXAMPLE 3.1 SIZE
In this version, size is represented by the double bass at 12x tempo. At this speed, the double base produces a persistent drone with some subtle changes in pitch with the most distinctive changes occurring when large data breaches are sonified. It is set at 1 octave so that the changes in size are noticeable yet subtle. In conjunction with the harp used for company named and company risk, a lazer-like sound is produced.
EXAMPLE 3.2 PERPETRATORS
Example 2 represented perpetrators with the bass guitar at a 2x tempo. In this example, the trumpet at a 12x tempo is used, producing a layered and rougher sound, which picks up the fluctuations in the intensity of language used to describe individuals and groups held accountable for data breaches. Along with breaches and risk, this track forms the anchor melody of the sonification.
EXAMPLE 3.3 COMPANY NAMED
The harp at 12x tempo represents the variable company named. At this tempo, the harp produces a machine-like sound. I chose to work within a 1 octave range to represent the binary coding (yes or no) we used in the scoring framework. The constant droning in the background sonifies the rarity of companies being held responsible. In this version, it elicited a meditativeness that I did not pick out in example 2.3.
EXAMPLE 3.4 BREACH
In example 2, the breach variable was represented by the bass guitar at 3x tempo. Because it is one of three variables with perpetrators and risk constituting the core melody, breach is also represented by the trumpet in D minor like perpetrators but at 6x instead of 12x to stress more of the melody and to differentiate slightly from the perpetrators track and allow it to stand out, which is appropriate given the project’s focus on data breaches.
EXAMPLE 3.5 DATA
Data is represented as a double bass in A minor with a 1 octave range and 2x tempo. These presets were chosen to produce the backbone to the sonification. The track is marked by noticeable sections of the same pitch with discernable variations. It was important to keep the data variable as the anchor to the sonification as a way of reinforcing how so many facets of contemporary life are characterized by, and through, data.
EXAMPLE 3.6 RISK
Risk is still represented as a trumpet at 12x tempo in E minor key. In this version, risk is paired with perpetrators and breach to observe how tensions and resolves compare across the three narratives. It is set at 1 octave so as not to dominate. Risk was paired with perpetrators and breach to listen to it not as exceptional but rather as new norm stemming from datafication. What do these variables sound like together?
EXAMPLE 3.7 COMPANY RISK
Company risk is represented again by a harp at 12x tempo in F minor, but it is set at an octave higher than example 2.7 to augment the droning. In turn, the few data points representing when a risk to the company was noted stand out.
Example 3 Sonification
This example features a foreboding, march-like rhythm reminiscent of Bernard Hermann’s film soundtracks. The rhythmic foundation emulates archetypal compositional techniques that convey tension, particularly through pizzicato on cello or bass2 (p. 94). Iconic soundtracks like “Psycho” (1960) and “Jaws” (1975), for example, used this technique innovatively. The surrounding instrumentation fills the sonic space around this rhythmic foundation, as different voices emerge unpredictably to create a shifting and uneasy soundscape; listeners may here dissonant ‘trumpet’ voices that disrupt the drone-like melodic lines, highlighting the track’s use of contrapuntal and polytonal qualities2 (p. 233). These elements aim to add layers of complexity and dimensionality that is distinct to that afforded by aural comprehension and which visual data often omits.
Our interpretation suggests this example emphasizes “breach fatigue,” with the sounds described above symbolizing the relentless, organized nature of cyber threats. The march-like rhythm reflects the systematic approach of cyber attackers, while dissonant interruptions and clashing polytonal elements represent the chaos and unpredictability of breaches, as well as the varied and sometimes disjointed responses to them. At a higher level, this sonification may evoke a heightened sense of both urgency and murkiness, mirroring the multifaceted and complex nature of cybersecurity discussions.