CALLUM COLLINS:
Exegesis Working Doc
GAME AND SOUND DESIGNER
July 2021
Learning FMOD and Unity Integration
FMOD will be the main DAW (digital audio workstation) for this research project. It has excellent integration with the Unity game engine.
​
Learning how this program works and establishing a workflow between my current DAW, FL Studio, Unity and FMOD will be the first step of this research. I will also be researching current Unity sound implementation so that I can compare the two workflows.
​
I started with the tutorials posted on the FMOD website:
​
QUICK START TUTORIAL:
https://www.fmod.com/resources/documentation-studio?version=2.02&page=quick-start-tutorial.html
​
UNITY INTEGRATION TUTORIAL:
https://www.fmod.com/resources/documentation-unity?version=2.02&page=integration-tutorial.html
​
These tutorials were easy to follow and quickly I learnt how to navigate the software. While working on the second tutorial I noticed the huge difference between Unity's default sound and the FMOD integration.
​
Unity requires more programming and objects to setup decent sounds while FMOD is very visual and the projects can be linked, making changes very easy to implement.
​
Unity Default Audio Prototype
After gaining a simple understanding of how FMOD works, I decided to look at creating a prototype with the default sound system with unity. This way, I would have a fundamental understanding of how sound should be implemented in a game engine. I can then adapt this prototype to use FMOD instead.
​
Following this tutorial; https://coding.degree/unity-audio-tutorial/
​
I was able to implement 3D audio with multiple effects such as;
-
Specialised sound with volume roll-off based on distance
-
Doppler effect for moving objects
-
Reverb zones
-
Music zones
-
Audio zones with proximity cutoff
-
Player modulated footsteps
Video showing all features implemented
Unity Audio Source Component
August 2021
Spatializing Sound
With my prototype created, I decided it would be interesting to explore sound within different spaces. Before starting this research, I believed I would be working primarily within the 2D platformer genre of games; however, this 3D first-person prototype inspired me to explore different areas.
Isometric Sound
The first space I wanted to explore was isometric. I enjoy isometric games and have wanted to incorporate them more into my research for a while, so I found a free low-poly character with animations to use on my nav-mesh surface.
​
I tested two different audio listening methods for this. One where the audio listener would rotate with the player character and one without rotation.
FMOD Studio Event Emitter Component
Iso sound test without rotation
Iso sound test with rotation
Usually, isometric games make use of 2D audio. I figured this was because 3D audio from the 'god perspective' would disorient the player. After creating my prototype, I found this to be a correct assumption.
​
I decided the next step for this prototype would be to try a top down perspective, referencing games like Darkwood.
​
I can also change the audio sources to be 2D but I'd prefer to continue exploring 3D sound for now.
First Person Sound with FMOD
To expand on the first person perspective, I wanted to use FMOD and experiment with different footstep materials and sounds. I found an old unity tutorial and followed this as it covered some more advanced features of FMOD as well as the footstep implementation I was looking for.
​
https://www.youtube.com/playlist?list=PLp4vT3ssm5SW0CjE3PEAj_BCUQN5gs8Rl
Viking Village prototype with broken footstep controls
The tutorial was not very helpful as it provided no sound files, so I had to source free sounds or record my own. I opted to start recording my own sounds and build a sound library. For this, I purchased a Zoom H5 recorder and started recording atmos and footstep sounds.
​
I took these sounds into WaveLab, where I was able to normalize and EQ my sound files. They are still a little rough, but they were good enough to start creating my footstep event in FMOD.
​
On the right is the FMOD Event, which uses a random pitch modulation on each sound and is controlled through parameters that should be changing in the game, but because of the tutorial's age, the script that controls this does not work.
​
My next step is to recycle this event and create a 2D platformer game that can use it.
FMOD footsteps event
FMOD footsteps random pitch modulation
2D Platformer Template
This has been the main timesink of the month. Rather than create a 2D platformer controller for one prototype, I created a template controller that uses a finite state machine. This way, new mechanics are added as 'states' which helps organise the code and makes debugging problems easier. This is because each state has its own script, meaning if there is a bug/problem, it's most likely in the new state script and not in any of the older code.
​
This also allows me to turn off certain mechanics. I can create several prototypes using this one template, switch off the elements I don't want, and change the ones I do. Since this research will most likely fall heavily into the 2D platformer genre, I figured this would be a good way to save time when creating multiple prototypes.
​
The majority of the code is based off this tutorial with new mechanics and template features being added by myself:
https://www.youtube.com/watch?v=78PLsHnRXiE&list=PLy78FINcVmjA0zDBhLuLNL1Jo6xNMMq-W&index=21
​
Covid-19 Lockdown
Auckland has been in lockdown alert level 4 and 3 for a total of 2 months now. The lockdown has impacted this research significantly in many ways. Firstly there is the lack of access to resources and equipment provided by AUT. With no access to a sound studio or different microphones, all sound recordings have had to be conducted in the field with a Zoom H5 recorder, (not a bad recorder but without access to different microphones such as shotgun mics and stands, recording quality has dipped significantly). Accessing equipment that is not provided by AUT has also been a challenge. Right when development of my 2D platformer template began, my computer broke and I needed to get a new one. Thanks to shipping delays and general part shortages, I was without a means of working on my project for just over a week.
​
Lastly, there is the mental strain of lockdown. Trying to find motivation or even just quiet time to work is difficult in a bubble of 7 people and there have been more than a few days where no progress has been made due to mental health.
October 2021
Despite the challenges of lockdown, over the last 2 months some progress has been made. A rough draft of my contextual review has been completed. This draft has all of the relevant ideas and just requires some paragraph structuring.
​
My platformer template has also been completed.
2D Platformer Template
Halfway through October and my 2D platformer template has been completed. The project has seen lots of development over these two months so I will detail its development in full below:
​
The basic player controller is based on the "2D Platformer Controller" by Bardent (linked above), using parts 20-27.
This includes the main mechanics of the game:
Jumping
Wall Grab / Climb / Slide
Wall Jump
Ledge Climb / Drop
Crouch
​
I have taken this code and edited it to work as a template, making all of the values for these mechanics modifiable, with the ability to turn individual mechanics on or off.
​
The player controller also features slope movement. This allows the player to move up and down angled tiles, with a feature to set the maximum height the player can climb. This will help the level design feel more immersive as the player can simply move through the world and not have to worry about jumping every time there is a small height difference in the terrain. This slope code was adapted from: "How to deal with slopes! in Unity 2D using built in physics" by Bardent. While this code is created by the same person as the template, they did not fuse together easy and this feature required many iterations and tests to function properly. This prompted me to move the entire project to GitHub, a internet hosting service allowing for version control and easy prototyping. Using GitHub I was able to create different "branches" of my main project files and test new features. If and when these features failed, GitHub stores all my file changes and I can easily restore my project back to the previous working version or simply remove an entire test branch if needed.
​
GitHub also stores all of my change history making it easy to refer back to previous work:
Data for Player Controller
GitHub slope history
I have also added my own mechanics to this template:
​
Object Pickups / Drops
Sprint
​
Both these mechanics were fairly easy to implement thanks to the finite state machine. All I had to do was add an object grab state and a sprint state. Both of these states inherit from the "Grounded" superstate which means if the player jumps or is in the air, they cannot sprint of grab objects.
2D Platformer Template: FMOD Integration
With the project migrated to GitHub, it was very easy to work on both the FMOD integration and the Slope code at the same time. I created a branch for prototyping my FMOD integration with the plan of using the games tilemap as a way of changing footstep sounds. The idea was to send a raycast from the bottom of the player, down at the tilemap, and retrieve the name of the tile underneath the player. This would then change the parameter linked to FMOD, telling it to play a different footstep sound based on the tiles name.
​
I recycled most of the viking village footstep project for this prototype:
FMOD footsteps event
Tile Identifier Code
Currently the sound plays when the player moves from one tile to the next cause it to be played a lot and out of sync. The next prototype branch will attempt to add an animating player character to the game which I can use to play the sounds correctly, (when the players foot touches the ground).
This prototype may also need a way to tell how fast the player is moving so that it can play the sounds quieter if the player is just tapping the move button.
New Inspirations: Diablo IV Quarterly Update
This month I found two new sources of inspiration for my project, the first was a video from Gamespot, "Diablo 4 Sound Effects and Design (Quarterly Update)."
​
This is a game I have been interested in for some time, and seeing an update revealing some of their sound design was of great interest to me. The first few clips show how the objects in the world are destructible. The sound design behind these objects is very clever as it appears to be physics-based; when the object breaks into pieces, each piece plays a sound based on what it hits and when.
​
Along with this, the video shows some interesting recording techniques in Death Valley. While outfield recording, I found some rocks that looked similar to the ones in the video, so I collected them for experimentation. I will return the rocks after I have finished recording sounds with them.
​
Lastly, they showcase their atmospheric tracks for different areas. This is a very high budget production, so taking inspiration from it is probably not recommended; however, there are some elements to these tracks I enjoy and may try to replicate certain aspects rather than the whole track.
New Inspirations: No One Lives Under The Lighthouse
This was a game that I had never heard of. "No One Lives Under the Lighthouse" (2020) is a game made by Sowoke Entertainment Bureau. It seems like it takes some inspiration from the film "The Lighthouse" (2019), but only so far as the level design and the themes of madness from isolation. There is a review of the game here, which covers most of what the game is about. What interested me the most about it was, of course, the sound design.
​
​
The game visually seeks to replicate a Playstation 1 style of rendering where textures are pixelated but are 3D with textures. It's hard to get used to, but it works; as mentioned in the review above, you always know where key buildings are despite the harsh pixelated rendering.
​
There are two main types of sound in this game; the atmospheric tracks and the players sounds. The atmospheric tracks are well mixed and give off an oppressive, eerie mood. The tracks typically only play at night, which furthers this idea. During the day, there are minimal sounds asides from the environmental ones like the seagulls or weather. When the player first arrives on the island, they can hear the crashing of waves against the rocks and the screeching of seagulls, and it is a perfect setup. I found myself sitting on the beach watching the rowboat leave while listening to the waves. (maybe it's lockdown, and I want to go to the beach and hear the waves).
​
Another thing that inspired me about the sound was how sounds were repeated and non-modulated. This is usually to be avoided but since the game seeks to mimic the old style of PlayStation 1, it works with the visuals exceedingly well. Hearing the same sounds as you climb the lighthouse tower every night to the same oppressive night track makes nighttime all the more daunting. This repetition is also using in the film. The maddening sound of the fog horn every night is similar to the sound of the night track.
​
If I plan on creating a minimalised pixel art game, I should consider using sounds with minimal mastering to try and achieve the same effect.
Experimenting with Field Recordings
Without access to the sound studio for foley recording, I have had to make use of my Zoom H5 and record sounds in the field. This gave me an interesting idea for my first prototype: A Walk in the Woods.
​
Behind my house is a short gravel path that leads down to the harbour. There are a lot of these little paths around, and since West Harbour is very 'hilly', a lot of these paths drop straight down through bush down to the water. Taking inspiration from the first moments of Limbo, I wanted to recreate the experience of walking down one of these paths at night.
​
I did some atmospheric recordings and attempted to make some more footstep sounds but quickly realised it would be easier to take one long recording of the experience and use that as a reference. I tried two different times, 10 pm and 3 am on the same day, and it was the one day where we had no wind at all, so both recordings are incredibly quiet. I will need to repeat the experiment on a windier day to get a different atmos.
After editing these sound files and taking a lot of inspiration from "No One Lives Under the Lighthouse", I decided I wanted to record some rough sea waves and maybe some seagulls for the final section of my prototype. Since the reference path goes down to the ocean, I could take this and dramatise it. On a particularly stormy day, I decided to attempt some recording. I knew the sound would likely be too distorted by the wind to be useful but figured it was a good experiment anyway. I made my way across the harbour rocks to a spot where the waves were present. These are not big waves at all, so I will probably plan a day at the beach for some better recordings.
​
To record the sounds of the wind through the harbour, I muffled the recorder by placing it, with its windshield on, in my bag and hunched down to block as much of the wind with my body as I could.
Muffling the sounds created a similar effect to compression where my sounds didn't have much depth and to get the sounds I wanted out of the recording I needed to use extreme EQ which didn't provide the result I wanted.
​
My plan is to now start planning my prototype so I have a better idea of what sounds I will need and can plan the recording days accordingly.
Prototype 1: A Walk in the Woods
I started by creating some sketches of my game level. I wanted the level to have a strong transition so I sketched an urban scene with elements that would produce sounds. I also created a quick moodboard to get some inspiration.
I used this sketch and my moodboard to try and come up with a pixel art style that wouldn't take too much time and would also fit the theme of the prototype. I first attempted to use photoshop to compress a quick drawing into pixel art but it didn't work as well as I needed so decided to start recreating the sketch in Pyxel Edit which is pixel art software. This is the first experiment below; I think I will try a couple more of these sketches and explore different colour palettes before working on my tilemap.
​
The other option is to begin working on my character. This way I can start adding my animations to my template and program the footstep audio and other effects while working on the environmental art.
Pyxel Edit Scene
Photoshop compression test
I started with the character design pulling inspiration from games with simple character designs. The point was to have an ambiguous character that would not draw the players attention and help them immerse themselves in the game.
Using the above reference I started putting together my character. I wanted the character to be cloaked and dressed in neutral tones to fit in with the dark atmosphere I was planning for the level.
Character Moodboard
Traditional pixel idle animation
New idle character
I did not like the standard pixel idle animation for this character. I felt it was too 'gimmicky for the prototype I was trying to create. I knew most of the atmos I would be creating would be wind and trees rustling as I had planned the sound of the level as I designed it; this meant I could create an idle animation influenced by the sound of the game.
​
I added in most of the remaining character animations and set them up in the game. This was pretty easy as I had already built the animations into the code; it was just a matter of adding the sprites and resizing the character colliders.
​
With the character design finished, I moved on to the level design. I decided to create another sketch, this time focussing on the forest section of the prototype.
Level sketch with sections and sounds outlined
My plan with this level design was to explore verticallity and create a level which has the player descending further and further down into some woods. I could then change the atmos and add more or less layers as the player descends. This downhill layout should promote caution, slowing the player down to get them to listen to the atmos and hopefully, engage with the environmental storytelling more.
​
Using this plan, some inspiration and some references images from the site where most of the recordings were taken, I started creating singular tree sprites with the intent of mashing them all together. This would allow me to parallax the environment with a perspective camera rather than use code.
Environment Moodboard
​
Colour grading each silhouette
Sprites placed on z-axis to create parallax with perspective camera
Sprite overlap and density
As I added my trees to the level, I would go back to pyxel edit and create the assets I was missing. Sticking to my 8 colours, I was able to quickly shape different bushes, trees and leaves to layer around the scene.
Playtest of section
I was very happy with this section of the level and since the next step visual would be adding more of the same, I decided to move onto my atmosphere design.
​
I decided to try and mix and master my sounds in FMOD as it has most of the tools I need and has the option to change settings and tracks in unity.
FMOD Atmos event
I made the 3rd track the base track as it was the most interesting and subtle of the tracks I had recorded in the field. It has trees creaking, a small amount of wind and some leaves rustling but nothing too extreme. I then layered it with the whistling I recorded from the harbour and some poplar leaves rustling in the wind. I added automation to the volume of these too tracks so that I could increase or decrease the volume of the mix live in unity.
FMOD Atmos event in unity
Custom script to control levels of mix
Prototype 1: After a (too long) break
After a definitely too long break partly due to covid and life-related things, I was able to finish my original prototype and create some sound implementation tools to help with iterations.
Prototype 1 Level Design
Prototype 1 Level Design Annotated
Some of the other features I have added to my prototype include animation based footstep audio and a UI link to my audio controller script which allows me to change and save mixing levels while playing through the level.
Movement animation events
Audio Controller on UI
Prototype 1: Evaluation
With the due date coming up soon, it was time to evaluate my prototype and decide if it's worth continuing with a 2D game as my research and making have led me to believe that 3D first-person would have been a better way to experiment with level design and atmospheric sound. Since all of this work is here already though I think it's best not to pivot my research and instead redesign my prototype, entering the iterative design methodology once more.
​
Also, with the assets and systems, I have created for this prototype it would make sense to continue and recycle as much as possible.
Prototype 2: Pre-Production
I think a good start for prototype 2 is to redesign the level with a larger scale in mind. I learnt from prototype 1 that you cannot cram level design techniques into a short section and hope for the same effect. Level and sound design take time to have an effect on the player. This creates the problem, of keeping the player engaged for long enough to experience this. Looking back at my research and case studies, I found that limbo's early level design is a good place to start as it takes the time to let the player learn simple mechanics like jumping and grabbing. I think it's time I introduced the mechanics I coded in the prototype and start my level by teaching the player what they can do and adding some challenges in the form of trap puzzles.
Prototype 2: Level Design Techniques
After doing some more research into level design techniques, I compiled a list of things to consider when designing my level so that I am not blindly drawing as I did with prototype 1. These techniques are:
Just like 3D grey boxing, we can start with a large block and carve away; we can begin with a start and endpoint and fill out in between
​
Knowing your metrics, player height, jump distance, and jump height is very important. This way, we can challenge the player to master their metrics
Always work with the reference viewport size. In the first prototype, I failed to do this, which led to scale issues.
​
Work closely with vertically, especially with the puzzles
​
Working in 2D means landmarks and mitigating awareness are not helpful (we cannot use weenies like journey; however, I think we could imply a destination using depth (abes odyssey?))
​
GMTK Mark Brown suggests looking at the 4 step level design process Nintendo has used for the Mario games:
​
-
4 step level design: taught, developed, twisted, removed. This works for multiple mechanics; however, with minimal mechanics, we only want the first 3 of these steps
-
Taught: no penalty for failure
-
Develop: introduce failure or risk
-
Twist: challenge the player with mastery or show a new perspective.
-
Removal: once the player has mastered a mechanic, there is no need to continue challenging them. We can reintroduce this mechanic, and the player will know what to do
-
​
I think teaching the player is a very important first step however, limbo breaks this first rule, and it works in its favour, as outlined in this article by GameDeveloper: https://www.gamedeveloper.com/design/level-design-in-limbo
​
​
"Limbo on the other hand, doesn’t have a ‘safe jump’. The very first pit is filled with spikes that gruesomely kill the character if you fall into them. What would otherwise be a big ‘no-no’ is fine in Limbo for several reasons:
1. There is no penalty for dying, Limbo doesn’t have lives or score.
2. You don’t respawn at the start of the level or a chapter, but very close to the location where you died.
3. It enhances the overall atmosphere of Limbo’s dark world where you mustn’t feel safe at any point.
4. There’s really not that many deaths that are unexpected, most of the times when you die you can understand why you kinda should’ve seen it coming."
​
I particularly like the idea that having the player die on the first jump enhances the dark atmosphere of the game so I will design my first section of gameplay to test both a safe first jump and a risky first jump and see what the result is.
​
Following the advice of Jessie Schell in A Book of Lenses, I started planning my level by looking at the interest curve I wanted. I took the time to evaluate the interest curve of the previous prototype for comparison.
​
1: Initial interest is generated by the art style and environmental audio. It continues as the player moves the player around the starting area learning what they can do.
2: The problem starts with the first obstacle in the game, the sudden drop. This is an unfair obstacle for the player as they cannot see the bottom and they don't know if there is fall damage or if they are even supposed to go this way.
3: If the player continues the interest can start to recover as they enter a new area of the map but it never reaches the original value because of the first fall.
4: Because there are no leading lines or narrative/mechanical elements encouraging the player to enter the cave area it does not increase and because of the simplistic tileset and lack of decorations, it actually drops the interest like the first fall.
5: We can recover a little bit of interest by having our first environmental storytelling element, the campfire. When the player enters the cave they can begin to hear the crackle of a fire and as they go deeper, they discover its source. I think the reverb zone and high-quality fire recording play a big role in the success of this area.
6: The ending of the level is incredibly boring. A much bigger scale is needed here in order to improve the intensity of the interest curve.
We can look at a generic interest curve from: https://www.researchgate.net/figure/The-intensity-contour-of-a-narrative-over-its-four-phases-i-interest-in-the_fig4_328118387
Here narrative has been broken down into 4 sections, usually referred to as a 4 act structure. We can apply this to our level design and overlap this structure with the 4 step level design used by the Mario games.
In the introduction, we want the player to learn the mechanics and give them time to adjust to the gamespace. Then at the end of the introduction section, we want to increase the intensity, so we begin to challenge the player to use the taught mechanics to solve simple puzzles or overcome obstacles. This can be as simple as teaching the player how to jump over a hole that they can easily climb back out of.
We want to develop these mechanics in the development section by introducing failure or risk. This can have differing effects on interest intensity, but usually, the intensity will stay at a medium level and not fluctuate. This can change dramatically based on the difficulty of the challenge presented. For example, if the challenge is too hard, the intensity will drop significantly and even though it will increase again when the player overcomes this challenge, it may not come back to the original level. In level design, we can start with spike traps to challenge the player to make jumps under pressure.
The climax section is where we introduce our twist. How can the player use this mechanic they have just mastered in a new way. This usually results in an "ah-ha!" moment for the player. We can use jumping to break through a floor and uncover a secret or new area for the player to explore for our level design.
Lastly, in the resolution section, the player has mastered the introduced mechanic and can wind down before beginning the cycle again. This relief is very important for the level design as some areas may have bigger climaxes than others; for example, there could be a boss fight in the last section, and the player needs time to soak in the glory of beating the challenge before being confronted with an even harder one.
Taking this into consideration, we can start designing a new prototype that focusses on the interest curve as the basis for the level design:
Adding start and endpoints for the prototype length gives us a block to carve away at. We can fill in the space between mechanics and puzzles that match the desired interest intensity. We can also consider what sounds will be present in each stage, affecting the intensity.
​
Using this method, I began by breaking down my level design into each section and following the curve, starting with the introduction section.
Here is a breakdown of my first level design concept which focuses on the curve of the introduction section as well as the game design principle of teaching the player basic mechanics at the start of the game.
​
In black is the level plan with all the obstacles sketched out and numbered:
1: This is the players starting location. It should be long enough that the player can take in the atmosphere before having to tackle the first obstacle, the jump, with no risk.
2: After the player completes the first jump we would usually add some risk to it; however, this game features a ledge grab mechanic as well, which we can teach them by making a jump that is just high enough that the player has to grab the ledge to get up.
3: This is where we can teach the player the wall climb mechanic. I actually think it would be better if we swapped this with number 5.
4: Here we finally have the jump with risk
5: Here, we teach the player they can jump on top of boxes which will be important when we later teach them they can pick up boxes and move them
6: Here, the player needs to pick up the box and put it between the spikes to make the jump
7: This is the culmination of all the mechanics the player has just learnt. Here the player will need to pull the box out from under the overhang and jump on top of it. They will then need to jump to the wall and climb up to the ledge.
8: This is the peak of our curve, and it signifies our position of high-intensity level and narrative-wise. This is the perfect place to introduce some of our environmental storytelling.
​
The red rectangle is the viewport size to use as a reference for building the greybox in unity.
​
In blue are areas where I could definitely introduce some environmental storytelling elements with more appearing as I build out the level in unity.
​
The yellow sections show where I will change my atmospheric mix. The first section will have a very quiet atmosphere with some tree creaking and crickets but little wind. The next section is where the player is introduced to risk so I will make the atmosphere tense and darker using more wind and louder tree creaking and maybe some mastering effects to make the sounds more uncomfortable. In the last section, the player has conquered the challenge presented to them and finally sees their goal. I would like to have a simple seaside soundscape for this with sea birds, waves, wind whistles and maybe some sounds from the lighthouse. Lastly, we have the bride, which will creak a lot, warning the player that it is not safe. Once it snaps, I can switch from the simple seascape back to the tense mix of the previous section.
This is the scene I plan to have at the top of the level. The lighthouse gives the player a goal and will act as a weenie as best as possible within a 2D gamespace. The bridge will collapse when the player crosses it which will bring us to the development part of our interest curve.
​
During the development section, it may be possible to show the lighthouse from different angles or have the light from it appear sporadically. Once we get to the climax section, the player should be positioned on the righthand side of this scene opposite where they fell at the start.
Quick jump there is supposed to be more documentation above. FOG HORN
​
Here is the reference:
https://www.youtube.com/watch?v=iHCmzvzCmhI&ab_channel=JJJamieson
​
