The Last Open Share:

The end of another academic year and with that OpenShare again for IxD considering last years exhibition I wanted this one to be good, better than last year and promoted more than last year.

So after discussing it on and off with my colleagues and with my added confidence after this year and more specifically projects such as Interactive Systems I had a want and need for this project to be better than the last year.

My motivation is from that when I was in first year, OpenShare had such a great turn out and as a first year I really benefited from it and I think it set me up in a great space and a great mindset for the coming years after it. So when it came to my hands I wanted to pay that forward and use this as an opportunity to show off the amazing work done at IxD.

My plan was three fold , first the posters and getting them out, second was the IxD Instagram and promoting the show online and third was word of mouth.

IxD OpenShare

Promo Work:

Firstly the poster. I thought that Interaction Design has great opportunities for icons and visual identification with the “IxD” short hand but before I had the chance to complicate it or over design it, I found a fitting clean yet digital font on adobe fonts and after running it by the others it was a winning success.

And the main focus of this personal ‘promotional project’ is the work that the students of our course produce so it felt natural to put it in the forefront and again with input from my colleagues a framing was decided and I made a template that would hopefully work for the visuals they provided.

Personal Work :

For my open share project it was between my expressive data project and XR project but with the time we have I think there would be too much work to do with expressive data to get it to the place I want but with the VR project it only needs a couple of minor tweeks and possible scripted events to get it where I’d like.

Extended Reality (XR)

Above is more of my mad scribbled trying to make sense of the ideas I had and trying to come up with new ones that would elevate the BlackLight demo to a higher quality.

BlackLight Logo Design:

As my mind was in the space of graphic design and giving things a brand I made a quick mock up of a possible logo for the finished project.

The name change was an odd one, so initially it was called BlackLight during the concept stage when the idea was to have a custom shader or the visuals inverted so that the light itself was a cone of black in front of the player. But when re-visiting the project for OpenShare and realising I might not have time to create my own shader I thought of renaming it ‘Light the Way’ as BlackLight might not make sense but then when I went to edit the Unity file I learned that I didn’t have the most recent build and didn’t have time to rebuild it. So with the file unable to be changed in time and the name being BlackLight on the makeshift title card of the game then BlackLight it had to stay.

Displaying the work :

So with the work itself frozen in its current state I wanted to give more time to the displaying of the work and the promotional aspect of the exhibition.

Insta Bookends:

To tie the instagram post together with the poster design, I thought some designed bookends for the post could be used when uploading other IxD images.

Troubles :

Ok so major/ not major problem. I do not have the most recent unity scene that is running on the headset. But, it is running on my headset at home, I have a finished that version. I would have liked to improve the current version of it but to do that I would have to rebuild at least half of the project in a day which I don’t think is possible.

As I do like this project and think it does have some legs to go further I might rebuild it during the summer in my own time improving and rebuilding it at the same time.

But with the time remaining and what I have at this moment I’ll present my work to the best of my ability.

Location :

I claimed the space closed off from the rest of the exhibition as to allow space for the people more enough and explore the space within VR.

And by complete accident the starting point of BlackLight doesn’t look a million miles away from how the space I have in the gallery looks.

Setting up the VR :

With my work being on the VR and it being self contained, it felt like there was very little to actually display as the work was contained within so with the time I have I’m trying to make more of a display for it.

IxD Promo Poster Collection :

The collection of posters for the exhibition, I was so happy how they turned out and was so proud of everyone that submitted such amazing images for the posters, it really shows the varied work from everyone on the course.

Posters around GSA:

With the posters designed and the deadline passed we got to work putting them out around the campus trying to hit high traffic areas around GSA.

BlackLight Screenshots :

During this time I played through BlackLight for two reasons, one was to re-familiarise myself with the application as well as getting stills from the game to display with the headset.

To be honest as soon as I put up the stills they felt silly and cheap so I took them down.

Stripping back the set up:

Before the opening of OpenShare I was just adding more and more to it, screen shots, decals, but after talking with my colleagues they reminded me that the art work is what is in the VR headset and that in that headset you are transported to another world and that world is the art, not the headset itself and to not “give it away”.

IxD Open Share 2024/25

https://videopress.com/v/9i1cyQmr?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=0&muted=0&persistVolume=1&playsinline=0&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Credit going to Blaise Treverton for this great video!

Reflection:

There is a few things that I would have wanted to do differently for this OpenShare and there are things I would have wanted.

I would have liked to be able to improve my previous project and get it to a better level and made sure it was gallery ready with things built into it like resetting the scene when finished or failed along with title screens and end screens. I would liked to evaluate my displaying of the work within a gallery to a higher level with either a projector showing gameplay or more of a thematically representing display.

I would have liked more engagement and energy from some of my collaborators.

But all that aside I am very pleased with OpenShare this year, I am happy with what I got done, I am happy with the work I did in promoting IxD and the show and I am so happy and proud of everyone that put in so much work and effort into the show and helping me produce and promote everything, I couldn’t have done it alone and thankfully I didn’t need to.

Thank you to the lectures, to everyone that helped and contributed and thank you to everyone that came to the exhibition.

Inspiration :

James Birdle

Autonomous Trap 001 (2017)

https://jamesbridle.com/works/autonomous-trap-001

With James Birdle’s work there was a number of things that interested me, the documentation nature, the real implementations of his experiments but the main take away was the humour in it and the lightness of his work even though his work and his discoveries have very serious implication to the world we live in.

Tom White :

“This is not a …” collection

https://chauvetarts.com/artist/tom-white

I’ve been a fan of Tom White’s work since I came across it in my research for DH&T in second year, and the same thing I enjoyed then was what I enjoyed now, that being his humour and his working relationship with AI and with this project I wanted to carry this relationship and humour through my work.

Philipp Schmitt :

Declassifier

https://videopress.com/v/pVcdBFbU?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://humans-of.ai/?img=8#15745,3139,3586

The Chair Project ( Four Classics) :

https://philippschmitt.com/

From Philipp Schmitt’s work these two are the ones that spoke to me most, again for the same reasons I liked Tom White’s work, this working relationship with machine learning and AI and using it as a tool or as a collaborator in the creation of the art. And as well as their sense of humour I also like the choice of not ‘correcting’ the work and leaving the digital fingerprints on it, be that the flaws or the imperfections.

Initial Ideas:

Bringing Depth :

Option One :

Option one would be a drawing machine allowing the user to use their tracked hands to draw with shapes and colours of their choosing in a blank canvas.

Option Two :

Option Two would be a generative art piece that elements of it could be controlled by the user infront of the screen using their body and limbs.

Generative Art Inspiration :

With these examples I’d like to emulate this level of detail and use of a cohesive colour scheme.

Like my control project I think allowing people to work within a selected colour scheme or a few colour schemes to keep an intersting and appealing aesthetic.

Ideally I use a regression model for this so that there can me movement and control between the different positions of the users hands.

I imagine a space for it to be used is like a hemisphere infront of the camera with trained points and then anywhere between those points would be guessed at.

Moving Forward :

I think I’ll go with option two for two reasons, firstly I feel that option one is very similiar to my control project from the previous year and dont want to tred the same ground or have others compairing the two.

Control

The other is that option two came to me more organically, I had a personal enthusiasm for it and I want to trust my gut more on this one and taking the note from Cat I belive that option two has more depth.

What I need to do to make it a reality :

• Train the skelton model to recognise the positions of the users hands.
• Create the generative art model that has variables that can be tied to the values of the skeleton model.
• Combine the model and the sketch.
• Set up webcam and test again, retrain if required.

BodyPose :

https://docs.ml5js.org/#/reference/bodypose

I think the best option for user interaction would be BodyPose detection, but thinking about it the hemisphere idea would be too inconsistent with the hands stright down the barrel of the camera and ml5 reading it.

https://videopress.com/v/anYrc0z5?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Tested above the snow angel idea shows the strongest areas within the space I had and with a screen in portrait and a webcam set up at the same distance it should have the same results.

Training with BodyPose:

With the ml5 BodyPose library and with the help from Cat we trained the regression/prediction model on my arm poses so that when people moved their hands in the view of the camera then it would control the predetermined values within the p5 sketch.

Visuals Inspiration:

https://openprocessing.org/sketch/492429

https://openprocessing.org/sketch/1615214

For these open processing sketches I loved the painterly quality of them as well as their use of colour and movement, finding the line between traditional style and the digital.

Tutorial:

With the visuals I had no clue, no idea but knew what I liked from open processing and when I found a tutorial inspired by the same thing I was I thought it could be a good starting point and by doing it step by step I’ll have a better understanding of what I’m doing and how I can change it how can I tinker.

https://openprocessing.org/sketch/1776463/

Tinkering :

Above is some screen shots of my tinkering and messing around with the tutorial code I followed along with. I need to remove my preciousness around working with code and continue tinkering in my own time.

Final Output:

https://videopress.com/v/mfbT7v6G?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://videopress.com/v/Lt9C4gfE?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://videopress.com/v/FSpRzBZ8?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://videopress.com/v/vJNzBFM2?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1&posterUrl=https%3A%2F%2Fvideos.files.wordpress.com%2FvJNzBFM2%2Fimg_9389_mov_std.original.jpg

The code :

// BodyPose Webcam Waves assembled by Nicholas McLaughlin.
// BodyPose traing code by Dr Cat Weir.
// Waves art code originally by Takawo on Openprocessing, original work 220717a by Takawo -- https://openprocessing.org/sketch/1615214 --.

//Thank you to Cat none of this word be possible or work without you! 


/* This code is for P5js that uses the Mljs Body pose library, isolating the arms of the users skeleton and then uses the values from the shoulders, elbows and wrists on each arm to control the amplitude and velocity of the modified Waves art sketch.*/



let palettesList = {
  0: ["#EFBDEB", "#B68CB8", "#6461A0", "#314CB6", "#0A81D1"],
  1: ["#2E1F27", "#854D27", "#DD7230", "#F4C95D", "#E7E393"],
  2: ["#BCC4DB", "#C0A9B0", "#7880B5", "#6987C9", "#6BBAEC"],
  3: ["#FFD289", "#FACC6B", "#FFD131", "#F5B82E", "#F4AC32"],
  4: ["#3A4F41", "#B9314F", "#D5A18E", "#DEC3BE", "#E1DEE3"],
  5: ["#CCFBFE", "#CDD6DD", "#CDCACC", "#CDACA1", "#CD8987"],
  6: ["#000000", "#CF5C36", "#EEE5E9", "#7C7C7C", "#EFC88B"],
  7: ["#E85D75", "#C76D7E", "#9F8082", "#8D918B", "#AD9B9A"],
  8: ["#EE6C4D", "#F38D68", "#EAF9FB", "#F662C9", "#17A398"],
  9: ["#000000", "#f2f2ed", "#000000", "#f2f2ed", "#000000"],
};

//------------- Waves Variables

let simplex;
let palette;

let xStep = 90;
let xFreq = 0.009;
let yFreq = 0.005;
let amplitude = 100;
let velocity = 0.001;
let waveCount = 40;

//------------- BodyPose Variables-----------------//

let webcam; // Variable to hold your webcam video

let bodyPose; // Variables to hold the body tracking values
let poses = [];
let connections;

let neuralNet; // Variables to hold the learning network
let state = "collection"; // A collection state to train for the values of the arm positions

let leftVal = -1;
let rightVal = -1;

let d = 10;
let f = 10;

//---------------- Load the bodyPose model----------//
function preload() {
  let options = {
    flipped: true,
  };

  bodyPose = ml5.bodyPose(options);
}

// Set up the canvas and the
function setup() {
  frameRate(10);
  createCanvas(windowWidth, windowHeight);
  simplex = new SimplexNoise();
  palette = palettesList[floor(random(Object.keys(palettesList).length))];
  noStroke();

  // background(255);
  ml5.setBackend("webgl");

  //listing all your computer's available media devices
  navigator.mediaDevices.enumerateDevices().then(gotDevices);

  // Create a constraints object and specify your webcam's deviceId, groupId, kind, and label (this will be printed to the console by the gotDevices function)

  webcamInfo = {
    video: {
      deviceId:
        "81986da6f580bdf305cb87f6a276115e084e99c333d938fd64c9bcac4577735e",
      groupId:
        "2defdc8c883929d6a70b71a9c5df6f5cd2e60555fb9e09aad9cdea3e6803a437",
      kind: "videoinput",
      label: "HD Pro Webcam C920 (046d:08e5)",
    },
  };

  // Setup your webcam using createCapture; using the constraints set above
  webcam = createCapture(VIDEO, webcamInfo);
  webcam.size(480, 480);
  webcam.hide();

  bodyPose.detectStart(webcam, gotPoses);
  connections = bodyPose.getSkeleton();

  let nnOptions = {
    task: "regression", // a regression model so that when the arm is between postions the model will take estimates.
    //learningRate: 0.01,
    //debug: true,
  };

  neuralNet = ml5.neuralNetwork(nnOptions);
  //neuralNet.loadData("2025-4-24_17-52-59.json", dataLoaded);

  const modelInfo = {
    // Set the info associated with the model
    model: "model.json",
    metadata: "model_meta.json",
    weights: "model.weights.bin",
  };

  neuralNet.load(modelInfo, modelReady); // Load in the custom model
}

//changing the model from the regression state to the prediction when its ready.
function modelReady() {
  console.log("Model Ready!");
  state = "prediction";
}

//load in the data instead of retraining each time.
function dataLoaded() {
  console.log("Data Loaded!");
}

// Print a list of all avaiable videoinput devices to the console
function gotDevices(deviceInfo) {
  for (let i = 0; i < deviceInfo.length; i++) {
    let currentDevice = deviceInfo[i]; // Get the current device info as a JavaScript object

    if (currentDevice.kind == "videoinput") {
      // Check if the current device type is "videoinput", i.e. a webcam

      console.log(currentDevice); // If so, print its info to the console
    }
  }
}

//key functions to then train the body pose model on how each arm position applies to each value.
function keyPressed() {
  switch (key) {
    case "w":
      leftVal = 10;
      rightVal = 10;
      console.log(key);
      break;

    case "q":
      leftVal = 10;
      rightVal = 10;
      console.log(key);
      break;

    case "1":
      leftVal = 100;
      rightVal = 10;
      console.log(key);
      break;

    case "2":
      leftVal = 200;
      rightVal = 10;
      console.log(key);
      break;

    case "3":
      leftVal = 300;
      rightVal = 10;
      console.log(key);
      break;

    case "4":
      leftVal = 400;
      rightVal = 10;
      console.log(key);
      break;

    case "5":
      leftVal = 500;
      rightVal = 10;
      console.log(key);
      break;

    case "6":
      leftVal = 10;
      rightVal = 100;
      console.log(key);
      break;

    case "7":
      leftVal = 10;
      rightVal = 200;
      console.log(key);
      break;

    case "8":
      leftVal = 10;
      rightVal = 300;
      console.log(key);
      break;

    case "9":
      leftVal = 10;
      rightVal = 400;
      console.log(key);
      break;

    case "0":
      leftVal = 10;
      rightVal = 500;
      console.log(key);
      break;

    case "t":
      state = "training";
      neuralNet.normalizeData();
      let trainingOptions = {
        epochs: 50,
      };
      neuralNet.train(trainingOptions, whileTraining, finishedTraining);
      break;

    case "d":
      neuralNet.saveData();
      break;

    case "s":
      neuralNet.save();
      break;
  }
}

// a training state for the regression model.
function whileTraining(epoch, loss) {
  console.log(epoch);
}

// a state for when the training is finished and change from a regression model to a prediction model based on the information it has been trained on.
function finishedTraining() {
  console.log("Finished Training");
  state = "prediction";
}


function gotResults(nnResults, err) {
  if (err) {
    console.error(err);
    return;
  }

  d = nnResults[0].value;
  f = nnResults[1].value;
  console.log(nnResults);
}

function gotPoses(results) {
  // Save the output to the poses variable
  poses = results;
  //console.log(results);
}

//Initially drawing the bodyPose points but now just used to draw the waves visuals when a body is detected.
function draw() {
  //background(255);

  /*
  push(); // Draw the flipped webcam image to the canvas
  translate(width, 0);
  scale(-1, 1);
  image(webcam, 0, 0);
  pop();
  */

  noStroke();
  
  //taking the required arm keypoints from BodyPose to train initially but now used to track the users arm keypoints for the prediction model

  if (poses.length > 0) {
    let inputs = [];

    for (let i = 0; i < poses.length; i++) {
      let pose = poses[i];

      let leftShoulderX = pose.keypoints[5].x;
      let leftShoulderY = pose.keypoints[5].y;
      inputs.push(leftShoulderX);
      inputs.push(leftShoulderY);
      //circle(leftShoulderX, leftShoulderY, 10);

      let leftElbowX = pose.keypoints[7].x;
      let leftElbowY = pose.keypoints[7].y;
      inputs.push(leftElbowX);
      inputs.push(leftElbowY);
      //circle(leftElbowX, leftElbowY, 10);

      let leftWristX = pose.keypoints[9].x;
      let leftWristY = pose.keypoints[9].y;
      inputs.push(leftWristX);
      inputs.push(leftWristY);
      //circle(leftWristX, leftWristY, 10);

      let rightShoulderX = pose.keypoints[6].x;
      let rightShoulderY = pose.keypoints[6].y;
      inputs.push(rightShoulderX);
      inputs.push(rightShoulderY);
      //circle(rightShoulderX, rightShoulderY, 10);

      let rightElbowX = pose.keypoints[8].x;
      let rightElbowY = pose.keypoints[8].y;
      inputs.push(rightElbowX);
      inputs.push(rightElbowY);
      //circle(rightElbowX, rightElbowY, 10);

      let rightWristX = pose.keypoints[10].x;
      let rightWristY = pose.keypoints[10].y;
      inputs.push(rightWristX);
      inputs.push(rightWristY);
      //circle(rightWristX, rightWristY, 10);
    }

    if (state == "collection") {
      if (mouseIsPressed) {
        let outputs = [rightVal, leftVal];
        neuralNet.addData(inputs, outputs);
        console.log(inputs);
        console.log(outputs);
      }
    } else if (state == "prediction") {
      neuralNet.predict(inputs, gotResults);
      fill(f);
      //circle(width / 2, height / 2, d);
      text("D: " + d + ", F:" + f, width / 2, height);
      console.log(d, f);

      randomSeed();

      let c = shuffle(palette);
      background(c[0]);

      let yStep = height / waveCount;

      for (let y = 0; y <= height; y += yStep) {
        push();
        translate(0, y);
        c = shuffle(palette);

        let gradient = drawingContext.createLinearGradient(0, height / 2, width, height / 2
        );
        gradient.addColorStop(0, c[0]);
        drawingContext.fillStyle = gradient;

        beginShape();

        for (let x = 0; x <= width; x += xStep) {
          let noise = simplex.noise3D(x * xFreq, y * yFreq, frameCount * d/2) * f/2;
          vertex(x, noise / 2);
        }
        vertex(width, height);
        vertex(0, height);
        endShape(CLOSE);
        pop();

        //a frame around the waves, the fill of it is still tied to d value so that I can see if its picking up a body from the webcam.
        rect(0, 0, width, width * 0.05);
        rect(0, height - 20, width);

        rect(0, 0, 20, height);
        rect(width - 20, 0, 20, height - 20);
      }
    }
  }
}

Reflection :

Throughout this project I have been unsure, unsure of my ideas of what I want to make, unsure of my abilities with code and all that outlook is something that I think has been creeping up on me throughout second and third year along with the idea that I’m not the ‘coding type’ but through the exposure to it within every project and more specifically this one I’ve came to the realisation that maybe my colleagues have picked it up quicker but to not let that deter me from continuing with my own learning and tinkering.

Within this project I might have not made my best work but have gained a better understanding of my self and my abilities within code and p5js and even through I might not use it for OpenShare or for the majority of my work in fourth year but throughout my life my goal is to practice it how others practice languages.

If I had more time with this project I’d have explored this rediscovery of code and the functionality of P5js and maybe got the output to a level that the visuals were more appealing and the interactions more focused to the situations discussed at the review, making it an ambient piece that comes to live with people walking past it and gently inviting them to play.

Return to Form

Design Domain Part One :

Chat with Paul:

What I’m Thinking :

I like the idea of having the 3 sculptures from part one and giving them 3 different outputs that lend themselves to each sculpture, whether VR, 3D printed, Axi Draw, screen-based, or even an audio output.

Each piece has its own strengths and weaknesses and considerations:

Form One :

Form Two :

Form Three :

Display Ideas :

As discussed with Paul, the three different forms can be a collection without a relationship to each other, and the only relationship I want them to have is to the original work but still seem to be part of the same family.

“Design is about the outcome; art is about the process.”

Ok, let’s work with what’s happening first in my head.

Form Three:

With form three, my first and strongest instinct was to have it in VR and allow the sculpture to be shown without a restraint to scale or gravity.

Form Two:

Chat with Gillian:

Again, I am going to Gillian for ideas when I hit a wall. So, after explaining that I had plans for two of the three sculptures; VR, 3D printed and nothing else so suggested that the logical next step would be to have high quality prints of the third from made, which seems obvious when pointed out, thats when you know its a good idea.

Problems Resolving itself :

So I planned to bring form three from blender and put it into a unity VR scene. But as soon as I put Form Three into Unity it started to struggle and even as I tried to brute force it into a VR scene the HMD would not run. But after replacing it with Form One , the sculpture that I had no idea what to do, the scene ran almost perfectly.

After Lucia had tested it, she had a great suggestion of two at different sizes, and after I implemented that, the lighting on each sculpture with some ambient music in the background the scene was were I wanted it to be.

https://videopress.com/v/1JNbd2JE?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://videopress.com/v/wLHV9rJ8?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Trying to print a second Form :

So with the models I’ve made, they are not viable for 3D printing and frankly I’m surprised that my computer only crashed at one point and didn’t burst into flames.

Ok so my Slicer does work, its just my models that are the problem.

After downloading a tape cutter thing from Printables to make sure it was not a problem with my software and that sliced in seconds.

At this moment:

At this moment of reading week I have been using the time to explore and problem solve the technological problems that are facing me with my design domain outputs.

For Form One I have it ready to go on the VR headset.

For Form Three I am currently rendering out high quality images of it to then get them sent to the printers at either the Reid or the Stow.

And Form Two in its current state is not viable for 3D printing so I can either continue to try and get it to work, I can try and model it again with more of a focus on it to be 3D printed and less madness or I can try and think of a different output that Form Two would let its self too.

Rendering out Form Three :

With a focus on the composition of the render and allowing Form three to have breathing room I think three well rendered outputs that show the shape and detail of the sculpture would be good.

I have tried to keep the overall image as close as I can to the original output as not to just slap some colour onto it, I’ve changed the material to a cool grey/ off white , ive build a background scene for the sculpture and included a simple soft spotlight, similar to the way Form One is displayed in the VR scene.

The next step when these renders are finished is to have them printed by experts at GSA.

Display Idea:

When having the form three final renders printed, also have HQ images of each piece of work that each form has came from to show a clear connection.

Also include a plane with the form one image within the VR space so that even when in the VR scene they can still compare it to the previous work.

Where I’m At :

Well first thing first. I’m an idiot. So initially the plan was to render out the images of Form Three and send them away to the printer with plenty of time and be more than ready for friday open studio but between the rendering finishing and the time I booked the printing consultation all the times were taken and the only avaliable time was 3:15 pm. So the printing will be late.

Learning AR :

So with the VR sorted, the renders sent away to the printing I started working on the AR. With a bit of messing around and trial and error I go it working with a simple unity scene and a trigger but again there was a problem.

I got the AR scene working on the Samsung tablet and reading the trigger but as soon as I implimented my model into the scene the application crashed.

I’ll swap out the model of a less strenuous model and make sure everything is working correctly and again I might just need to make a model that does allow it. Ideally I could use the model that I had made through the process in DD part one but if it wont work at all I might need to use smoke and mirrors.

Writing for Design Domain :

Form One : 

Form One started in Creative Coding 01 using processing in the first year of my time at GSA. It was one of the first instances where I had seen the power and creative freedom of what we can do here at Interaction Design. 

Please use the VR headset to explore the first of these digital sculptures.

Form Two : 

Form Two is taken from my Realtime Events scene using Unity from my second year of IxD. It was a creative outlet when I needed it, but it also taught me that we are only limited by our imaginations in the sphere of creative technology.

Please use the tablet camera to explore the second digital sculpture.

Form Three : 

Form Three began in Imagined Environments using Adobe After Effects in my third year. It also reminded me that the problems and troubleshooting we tackle here at GSA and IxD are not problems or roadblocks but opportunities to find new and creative workflows, processes, and outputs. 

Please enjoy these renders of the third sculpture. 

Set Up Trial :

With the vetrine reserved for my output I thought it would be a good chance to plan out the layout and get everything ready so that all i have to do is drop everything into it whenever its ready.

Planning it chronological order so that when the viewer comes up to the display they can get the order of the work instantly, well complying to western conventions.

Back to AR :

Banana Man Test :

https://videopress.com/v/16E8daS7?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

The trigger for AR is working and is contined within the application, even through we didnt go through it in the workshops I wanted to figure out this way so that the AR isn’t relying on a shakey internet connection whether it works or not.

It worked well, if a little shakey with the banana man model as a place holder.

Form Two Test:

https://videopress.com/v/FS6XMmNy?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

With everything else ready I had time to continue working on the the Form Two AR and getting it to work as much as possible.

With the trigger image of my unity image working with the placeholder banana man model I put the model of Form Two into the scene and it ran and seemed to work at a certain distance but getting close enough it tank the frame rate and would crash the scene.

I need to try it with a smaller model or just allow it to crash if I cant get a decimated model of Form Two.

At this point :

At this point Form One is in the VR headset ready, Form Two is working on the tablet with AR even if it is running very slowly. And Form Three is scheduled to be printed Friday afternoon.

Thursdays ToDo List :

• Put VR headset on charge.
• Put AR tablet on charge.
• Try and make AR model less taxing on the tablet.
• Set up lights for box

Friday ToDo List :

• Set up VR
• Set up AR
• Turn on lights
• Submit Name Card
• Collect Prints
• Cut out Prints
• Present.

Getting Ready :

With everything ready, well almost ready, I had time to consider the presentation of my work more and plan out the interactions as much as I could.

Gillian Saving me:

Gillian yet again saving me by getting the models into a state that can work within the tech we have like the AR scene.

There is some detail lost in models, which to be honest I expected because there was no way they would run on a tablet unless I get a brand new Ipad pro or a Galaxy S9. And as it still fits within my theme and practice for this project I think it means more that the AR actully works, even if detail is lost.

Morning of :

With the VR headset charged and ready, the AR tablet charged and ready and the prints due later on today I had more time to work away on the AR and impliment the new decimated models that Gillian has gave me and get the AR running even smoother.

Last minute change up :

During the final day I had the set up finished, or what I thought was finished, until I got the prints of Form Three back and had the thought that after going through all this effort for the prints they should be displayed. So after a lot of waiting around there was a lot of running around and trying to cut out the prints as perfect as I could and as quickly as I could.

Final Output:

What work was shown in the display:

• The discription of the work with a QR code that takes people to this very page and next to each output is a brief discription of the work and what project it was based on, along with a prompt of how to interact with each piece.

• The Meta Quest headset with the unity scene of Form One in it and the left controller to more around the scene freely.

• The Samsung Tablet with the unity AR apk for Form Two pre-loaded and self contained and the printed trigger image.

• The Blender renders of Form Three printed by the photography printers in the Stow building cut and displayed in the vetrine.

Return to Form :

https://videopress.com/v/lxQ1n7PV?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=0&muted=0&persistVolume=1&playsinline=0&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

For Design Domain I focused on two things, my own work and my pratice of always wanting to perfect everything and design a product. Within this project, my goal was to challenge myself to lean into the imperfections of digital mediums, the interesting forms and materials within them that would be difficult to replicate anywhere else. I would have never made them consciously, but by taking my previous work from each year and creating these sculptures by exploring a software I’m unfamiliar with and not trying to ‘fix’ them, I wanted to shake myself out of my usual way of working and then show off these ‘odd’ pieces with pride and confidence, allowing them the space they deserve to stand alone and be considered as works of digital sculpture.

Evaluation :

If I could do it over with more money and more time I would like to have each sculpture shown perfectly in their respective outputs, give more time and consideration to the space within VR and maybe give the user more control to explore the sculpture in a space with no restrictions.

I’d have improved the AR experince by maybe have the tabelt waiting in a better position and of course have the undecimated model of the form in AR allowing people to explore the detail and odd interior of it.

And for the third I’d like to plan better in the future to allow myself more time to display the prints better or even have time to think of how the renders them selves could be improved while still allowing for their form and material.

Initial Ideas :

With the first idea shown on the bottom left of the page I had the idea to meld both the digital world with the real world but not through a frame or portal that is often done but more like the work done by MarshmallowLaserFeast – In the eyes of the animal.

https://marshmallowlaserfeast.com/project/in-the-eyes-of-the-animal/

In the work ‘In the eyes of the Animal’ the audience is sat in a forrest before even putting the HMD on and when put on their head they see the scene they’re already sitting in creating the relationship between the real world and the digital. I’d love to have this relationship in my work.

The second idea at the top of the left page is an exploration of the idea that VR can give people experiences that they would never be able experience and can bring a sense of wonder to the audience that they might have never experienced, for example, space travel. But as not to make this project just a tech demo and more emotionally engaging, the scene would maybe have an impressionist style, an exaggeration of the real thing.

The third idea at the bottom of the same page is another interactive VR experience that uses light shadows and objects being implied by their shadows. Inspired by the feature of digitial objects in an AR space projecting AR shadows in the real world as well as the use of white space and black paint in the game The Unfinished Swan.

Idea four, noted on the left page is pretty much as it says there, a VR museum of dead technology and experiences that have either died due to technological advancements and a lack of support or an idea that was killed by those that had to pay for it, or even all of the above.

The museum would be a place for these 3D animation and effects to be shown in a virtual gallery.

This fifth idea is the concept of using materials that are not normally related to digital media to trigger a digital response such as a QR code made out of wood or moss or have a AR target made out of organic materials.

Testing AR :

https://videopress.com/v/BwDc5wCu?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Exploring Commercial AR :

Pokemon GO in the studio:

https://videopress.com/v/E8PlR8ld?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

https://videopress.com/v/BmXyfsqL?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Amazons products placement AR:

https://videopress.com/v/PDzYwwqQ?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Acute Art VR app :

https://videopress.com/v/3g0uogCj?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

My thoughts at this time :

At this time in the project, I keep thinking about the fun that can be had with AR and VR, this idea of making the real world magical and lighthearted and beautiful. Using technology to make magic happen.

Light hearted, joyful, magical, fun.

Finding joy in artwork:

In lights ‘Bird’ the grace and beauty is something that draws me in, with such a simple visual style the grace of a birds flight is captured.

The joy I found in ‘Joie de Vivre’ (Joy of Living) was the essence of play, the colourful figure in stark contrast of the grey architecture and business people of New Yorks lower Manhattan financial district.

Again as with ‘Joie de Vivre’ its the contrast between the work and the situation it sits, in ‘Seven Magic Mountains’ the outstretching harsh Nevada desert is harshly and unapologetically interrupted by these colourful standing stones.

Talk with Neil:

BlackLight Idea

Inspiration:

Unfinished Swan:

With Unfinished Swan the art style was a huge inspiration for me on this project, mainly for its visuals but also the way they introduce the main mechanic at the start of the game.

Shadowmatic:

I love the use of shadows to create different shapes, I’d love to work this into my project somewhere.

Schim:

With Schim I want to carry over its sense of warmth and joy in every situation, you work within the shadows in the game but at no point is it scary.

Superliminial:

With Superliminial, its the sense of joy and wonder in the majority of the game, using the one function of shifting and controlling perspective throughout.

Understanding Shadows:

https://videopress.com/v/G5S2DKD5?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

As the Unity scene will use shadows as its main interaction I think its good to look at them with a different focus, obviously we encounter shadows everyday but looking at them in a different lens or with a different context will give me a better of idea of how my scene should look.

Testing Light in Unity :

By turning off the mesh render but allowing the shadows still to be cast, it gave the effect that I wanted. And as it still has the mesh/ box collider then the player cant just walk through it.

Building a Scene:

My initial thoughts were to have a maze that was completely shadows but as you need the shadows to be cast onto something the white walls of the maze were made.

Planning Scenes and Interactions:

So with the basic concept of the torch sorted and the idea of a maze sorted I needed something for the player to do in the maze and with these interactions I wanted to use the functionality of VR and actions that were exclusive to VR.

Setting the Tone:

In the scene I want to spark joy through this unknown and adventure, that yes there is a little bit of fear but at no point are you in danger. Through not knowing what is around the corner but being called to find out I want to have a sense of adventure. Think Indiana Jones; the Mummy in movies or even the games Portal and Ratchet and Clank. All of these examples have fun, mystery and danger but at no point do they take it to horror.

More Primary Research – PowerWasher VR:

https://videopress.com/v/m9eDjhqd?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

So with knowing my concept and my functionality I went back to some personal research to see how other studios have done it and with Power washer Simulator it had simple concept, simple movements and functionality that was only in VR so the footage above shows myself messing around and exploring the functionality of VR again through the lens of my own project.

Building the initial scene and proof of concept:

At this point I was having trouble with the VR headset so to the best of my ability I began modelling my scene. During this process I had the idea that as the world would be using shadows then recognisability would be a big factor, if the player doesn’t have an idea of what the object is then there wont be any spark of joy or fun because they can not relate the object to anything in the real world.

The soundtrack:

After the previous project when audio was the main focus, I didn’t again want it to fall by the wayside and not give it the time it deserved. So as I didn’t think that I would have enough time to do it justice but I had an idea of tone I asked a friend of mine if he would like to make the music for it and he thankfully accepted. I have lots of very musically talented friends but I knew Michael would be the man right for the job.

Below is his Soundcloud if you want more.

https://on.soundcloud.com/38jtdNRKCKnNvwds6

All the prefabs:

As the scene developed and got bigger and bigger I used more and more prefabs with recognisable silhouettes to fill the scene. This step had a lot of trial and error, I was very conscious that with such a simple concept and such a minimalist art style there wasn’t really anywhere to ‘hide’.

Building the scene:

I’ll be honest at this point my documentation was a bit poor as it was a lot of quick ideas, quick changes and quick implementation that didn’t really seem like much at the time but looking back now it was the scene becoming a basic interaction and maze to demo that it is now.

Play Testing:

https://videopress.com/v/3APlbYig?resizeToParent=1&cover=1&autoPlay=1&controls=1&loop=1&muted=1&persistVolume=0&playsinline=1&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

Above is footage of my class mate Lucia testing out the scene. After this session of play testing with both Lucia and Mikhail I finished the final scene and removed the snap turn on the controller.

Note:	Fix:
Motion Sickness	Removing Snap turn
	Reduce walk speed
	Removing Vignette
Eye Strain	Increasing light temperature
	Lower Light Intensity
Lack of Ending	Adding in a final scene
	Defining a narrative

Planning the End :

Behind the scene:

Above is stills of the scenes in unity using a green wire frame of the prefabs to show the invisible elements.

Black Light :

https://videopress.com/v/SbTs235o?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=0&muted=0&persistVolume=1&playsinline=0&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

The Story of BlackLight –

In Black Light you find yourself in a odd space armed only with your wits and your torch.

Step into a world where time stands still, filled with intricate mazes and shadowy corners that feel strangely familiar. As you navigate through the labyrinth, you’ll encounter various structures and pathways that invite you to explore. Each area holds its own mysteries, waiting to be uncovered. Embrace the journey as you shine a light on

the stories hidden within the shadows.

Where are you ? What happened to the king ? What happened to the men on the ship ? And whats at the end of the maze ? What happened to the heros?

Reflection:

If i had more time and more resources I really feel like the concept has legs to stand on and could even be taken further to a full fledged game for VR, at this point it is a proof of concept or a demo.

If I was to do anything different or take it further, I would like to develop my own shader that gives the same effect as unfinished swan and lean into the reliance on the torch more.

Also one of the concepts that I didn’t implement was inverting the colours and making the white black but that would either need an after processing effect on the entire scene or another custom shader.

Research :

At this point I have no clear plan for this project but I want to keep it that was for as long as possible. At this point I’m exploring the links Paul provided for some inspiration.

Also the feedback from my summative assessment thats in my head at this point are trying to widen my search for inspiration and connecting a number of different media, I’m not going to force it if it doesn’t happen naturally but still keep it in mind.

The psychedelic visuals of the classic iTunes visualiser has a nostalgic flair to it and a part of me does love some of the visuals of it but it doesn’t feel like its 1 : 1 with whatever music is played on it.

Take away : However the audio is visualised or represented, its more appealing when its 1 : 1 and you can see what your ears naturally pick up.

An Optical Poem – Oskar Fischinger

With An Optical Poem the visuals have a clear relationship to the music played and convey Fischinger’s personal vision and relationship to colour and music. Even though he was restricted to the media of the times the emotion and energy conveyed still feel indicative of the music. The problem is that this is one mans vision and connection to the music and might not relate to others, that might relate different colours, motions or shapes to different movements or instruments in the piece.

Take away: An Optical Poem is one mans personal response to the piece of music and might not ‘line up’ with others ideas.

Electroplankton – Toshio Iwai

Electroplankton by Toshio Iwai really makes the act of discovering, creating and visualising music obtainable and accessible. The light hearted nature of the program removes the intimidation that some may experience when interacting with technology or music production. Even if it does become a bit mad at times its very forgiving for mistakes and discovery.

Take away: The interactive nature of Electroplankton allows the user to both create a relationship to the sounds they’ve discovered and see a direct correlation to their actions.

Personal Note : I want something that can be moving to the user and to help, in one way or another, create a realtionship between the viewer/user and the music they see/hear/create.

Test Pattern 100m Version at Ruhrtriennale – Ryoji Ikeda

Idea Sketch – Sound clock :

The sketch above in my note book is an idea about logging the sound waves of an area or room and logging it in the formation of the hours on a clock face to produce a collection of prints.

Inspiration – George Crumb, Spiral Galaxy (1972)

With Crumbs spiral sheet music, it sparked the idea of the music being shown through time and then the idea that the ambient noise of a location at the corresponding time being logged on the sketch, then producing a visualisation of the whole 12 hours.

Idea Sketch – Audio leading Audio :

With this idea the concept is to have an audio processing sketch that is driven by an external microphone that picks up the ambient noise of a room and controls wether the music plays or not. Initially the idea was to only play the music when the volume level of the room was below a certain threshold. But I actually think it would be more interesting to invert it and have the music only play when the room is loud enough adding to the noise .

Idea sketch – FFT Relations:

FFT images of two different songs mirroring each other on the same page.

FFT Relationship Development : Instead of two different songs its the same song but the audio waves on the left side of the image is the regular wav file and one on the right is the compressed mp3 file so that on the print you can see the relationship between the two and see what information is lost when an audio is compressed even if you cant really hear it.

FFT Relationship Development : The visuals of the audio could be either the lined edge justified frequency of the audio or the spectrogram visuals of the two audios and allowing them to overlap or reach out but never come into contact with each other.

All three ideas above allow themselves to be displayed in a number of ways and that can be explored in the development of this idea.

Idea Sketch – Three Visuals One Audio :

The first iteration of this idea is similar to the audio leading audio idea but with a video or movie only played when there is audio detected, playing on the convention of being quiet when watching a movie. This could be displayed as a projector and a mic.

Concept Test – Sound Clock :

Inspiration – Albert Bernal, Impossible music #9 

With Bernal’s mushroom cloud of musical notes, I was inspired less with the image its self and more with the density and overlap of the music notes used to make up the image.

Sound Waves + Compression :

Personal Thoughts :

At this point in the project I am lacking inspiration. I’ve naturally came to the idea of the printed audio visuals, a data visualisation of audio or audio decay. But I cant shake the idea that my work for this project could be more exciting or more interesting.

The Idea : Audible Decay

One idea or concept that is really interesting to me is the loss of information or loss of audio quality.

How can this lost data be shown or be represented to the viewer.

Digital Visualising Audio :

Fragile Territories – Robert Henke

https://roberthenke.com/installations/fragile_territories.html

With Henke’s work the striking visuals with the awe inspiring scale was appealing to me. The ability to be both still but still energetic and almost breaking out nature of his works in Fragile Territories.

Minimal Visual Inspiration :

Frank Stella :

With Frank Stella’s minimalist work from 67, I was drawn to the creation of shapes within the lines as well as the simplicity of the monochromatic colour scheme.

Within my work I was considering a monochromatic colouring as not to distract from the audio. A visualiser inspired by Stella’s work would react and shift with the audio without being distracting.

Dan Flavin – untitled (to jan and Ron Greeenberg)

In this untitled piece by Dan Flavin from 1973 I love the use of contrast in colour and the use of the space, the wash of colour with the inviting passageway of the blue inviting the user in.

In this project, as previously stated I don’t think I’ll use colour as not to distract from the audio as can colour carry its own connotations within a work.

Talk with Paul:

Don’t just deliver.

After my conversation with Paul this sentiment kept rattling around in my head.

I think I keep focusing more on delivering a finished piece or product and less on the creative exploration of the project. I’m thinking like a corporate designer and not like an artist.

Testing Audio :

Original Audio :

I chose the ‘pas de deux‘ by Tchaikovsky for a number of reasons. Firstly it is one of my favourite and most moving pieces of classical music and I was reminded of it through a number of examples that use classical music as well as Paul talking about how music “moves people” and how I was moved by this music.

Secondly I considered how that with my theme of audio data I was emotionally moved by a piece of music first preformed on 18th December 1892 at the Mariinsky Theatre in St. Petersburg, Russia. This piece of music being streamed over the internet from a server in the UK to my phone that is then sending those signals through a bluetooth connection to my headphones and finally to my ears eliciting a response.

Through my exploration of effects on Pas de Deux and toying with the effects on Audition it became less of destroying the music and more of isolating an element that would catch my attention and pushing that to its extremes.

Reverb to extremes :

Delay to Extreme :

⚠️ WARNING LOUD AUDIO at 0: 30 ⚠️

Different Samples :

Other than a beautiful piece of classic music I want to use other meaningful audios aswell.

Above is the first commercially released digitial audio track on PCM which is the jazz album by Steve Marcus and Jiro Inagaki.

I didn’t decide to take this any further but if this project is revisited then this could be taken further.

When looking for meaningful quotes, Mikhail had suggested Marshall McLuhan as a known and insightful voice in the world of technology.

Exploring Effecting the Audio:

At the most violent moment of the audio, the detached voices of Marshall McLuhan has been processed through the distortion effect to push them to extremes.

Above is the spectrogram is the digital router noise. I personally loved the visuals of the digital messages.

For the time stretched router noise I used the denoise to either extreme and syned up as the bed of the audio.

With Pas de Deux the Studio Reverb on after effects gave the piece of music an ethereal tone with the higher notes dragged out becoming a constant of the piece.

The screenshots above are the visuals of the archive footage using effects to make them unrecognisable.

Final Audio:

The isolated audio without the visuals of the piece.

Taking it further:

In my head I envisaged this final piece as a teaser reel of a gallery piece that would be more long form and on a larger scale.

Lost Global Village

https://videopress.com/v/GXIMDzJr?resizeToParent=1&cover=1&autoPlay=0&controls=1&loop=0&muted=0&persistVolume=1&playsinline=0&preloadContent=metadata&sbc&sbpc&sblc&useAverageColor=1

With my final piece the lost global village I took the 3 audio elements in the composition pushed to extremes with effects and the visuals are archived footage of 3D animation promo footage from the 90’s with the same treatment applied to the effects.

I had a personal outcome in this project to not just deliver a finished product but to explore.

Reflection:

Feedback Notes:

Personal Reflection:

I came into this project with pre conceived notions of using personal projects and producing a music track and using midi controllers but that felt like what I usually do lock into and idea.

So in reaction to this preconceived notion and after a conversation with Paul I tried to not deliver a clear finished product and which I think I failed at and should have leaned into it more.

If I was doing this again I would listen to Paul and Mikhail’s feedback and allow the work so go to extremes and not get in the way of the process and subdue it. To use one element and let it loop until it becomes ‘difficult’ and allow myself and my art to become comfortable with taking up space.