Author Archives: kennethf


Screen Shot 2015-12-16 at 6.44.11 AMThe Mouse of virtual reality, or The TomorrowLand Orb.

TomorrowLand PosterThe motivation for this project was to replicate the future technology that is displayed in TomorrowLand. Specifically, a scene where the characters are able to travel to any location and time with a sphere in the center of the room. They navigate the world by rotating and touching the controller. I wanted to replicate this controller for virtual reality and also explore the potential of creating a mouselike device for virtual-reality, Instead of a “touchscreen” like device.

Many people and companies are currently exploring gloves and gesture controlled interfaces for navigating a virtual environment. However, there may be some use cases for a devices that doesn’t require pointing or gesturing, but simply using subtle movements of the hand… like you can with a mouse on a traditional, flat screen.

Overview of Approach

The orb can be controlled by rotating the device as well as gestures on the device (pinching). You operate the orb standing with a virtual reality headset on, resting your hands on top of the sphere. Throughout the explanations below, I try to give a reason why I chose the approach I took over other alternatives.

Hardware – Mechanical

IMG_1462Mechanically, a 4″ diameter PVC pipe stands vertically on top of a base. Above the pipe, there is an un-noticable flower pot. The flower pot fits snuggly into the top of the PVC pipe. The flower pot is surrounded by a lamp shade (which is hot-glued to the base of the flower-pot). Inside the flower-pot is a confetti-like substance to fill the space.

Therefore, the only thing you can see as a user, externally, is the pipe and lampshade, with an orb resting on top of that. The components that can be seen from the outside were spray painted silver to give a futuristic look.

IMG_1461A hacked-mouse (see below) rests on top of the confetti. Finally, the orb sits atop the confetti and hacked-mouse.  Inside the orb, there is a half-sphere that rests at the bottom of the orb but is not attached. An iPhone is placed on top of the half-sphere, so that when the orb rotates, the phone stays at the bottom of the sphere without flipping over.

All of the hardware was purchased at Home Depot, and none of it required machining or power tools (since I do not have readily available access to either this semester).

Hardware – Electrical

The orb input is completely wireless. The rotation information is picked up by an upside down hacked-mouse (the buttons are removed so that the orb can not accidentally click, and as the ball rotates, the laser from the mouse is picking up the ball’s movement.

A user’s touch can be picked up by the camera on the iPhone inside the sphere (see software — iPhone, below).

Software — Overview

Rotational information from the mouse is sent to the computer via bluetooth, and then interpreted as translational motion. The touch information from the iPhone is sent “to the cloud”, the Mac then constantly checks to see if the touch information has been updated on the server (this is because direct communication from the iPhone to Mac is not officially supported with APIs by Apple, and is therefore very unreliable. The easiest way to do it is using UDP packets over USB — but I wanted the system to be wireless).

Software — iPhone

The iPhone runs an app I created to sense the pinch gesture on top of the sphere. OpenCV and Parse were the only two third party libraries used to create this app, both of which are available for free. The software loop is explained in detail here:

  • For a given frame of video:fingerDetection
    • Run a canny edge detection filter (result of two fingers shown in image to the right)
    • Check to see if edges appear to be moving towards a center, or away from each other
    • If either is true, make sure the gesture is sustained for about one second.
      • If gesture sustained for a full second:
      • Send a request to Parse to update the boolean variable stored on Parse’s servers.
  • Repeat

Software — Mac

The Mac uses a Unity application I created to handle the inputs and display the results to the Oculus VR headset. The rotation information is received via Bluetooth, and the pinch control is received using a constant check to the Parse server looking for a change.

Using C#, these values are converted into changes in the view.

Things Learned

  • I had not used Unity before this semester, so I learned Unity and C# language
    • Including how to create scripts, how to move objects in the view and change camera locations
  • I learned how to integrate multiple data sources into an application
  • I learned OpenCV for iOS (and in general)

Things to Expand On

  • In the future, this system could take more inputs (for example, different touch gestures besides pinch to zoom).
  • With better access to machine shops, the industrial design could be improved to more closely replicate the movie.

Useless Hope: The Discovery of the Lens


A contact lens that shows holographic projections of answers for students. These lens let them see out of any camera on any robot from the Uprising. This effectively allows the wearer to visualize any information from all the world’s knowledge in a instant. The children of the poor believe that robots and humans are distinct and mutually exclusive, but the wealthy have blended with robots but more than  The lens are very expensive, and can not be noticed by anyone who isn’t the student. Dynasties are created, since the wealthiest families gift these lens to their offspring, which all but guarantee successful and wealthy futures.



It was years later when I discovered the truth. The rhetoric haunts my dreams. Your poor life choices are beginning to take their toll, Gavin. Lies, all lies. My life choices had nothing to do with it. The truth was, there is no way to win. No way to get ahead. The game of life is rigged. Claire showed me a file on a device similar to the one she handed me all those years ago.

A file of instructions. Instructions for infinite knowledge and a guarantee to wealth and success. A free ticket to decades of childhood. The wealthy are not hardworking. The wealthy didn’t study. The wealthy don’t make better life choices, they make perfect life choices. They have all of the knowledge, and therefore all of the power.

They keep us in the dark. The message “work hard, study hard, and you can do better than your parents did” is worst of them all. It’s the lie that keeps the hope of a better future alive. An impossible future. I wish I had learned the lesson sooner. Or maybe never learned it at all. The class you are born into, is the class you stay.

Tomorrowland Orb: Part 2

Progression of Project 1

This project builds on my Project 1, which can be seen here:

Based on SciFifingerDetection

This project is most closely associated with Tomorrowland, seen here:

This project addresses some of the concerns and continues of the evolution of my project.


For the second part of my project, I addressed two major improvements of the orb.


Do Blade Runners have nightmares of electric sheep?

I’m not sure if it’s the passive nature of watching a movie compared to reading — or the different story choices, but both the emotional response and the questions in my head were quite different between the book and the adaptation.

The book hit a lot of major themes that many pieces of science fiction have since tried to address. Common notions of what it means human to the benefits of a robotic future are called into question. Another major point of discussion is around identity — especially with Racheal and Pris. However in the movie version, I was left thinking that the special effects and soundtrack were very good (especially the special effects for a movie from the early 80s), however I wasn’t left contemplating life nearly as much as after the book.

Even though there are no special effects in a novel, I felt that the novel did a much better job integrating technology into the story (besides simply the androids/replicants). The Penfield Mood Organ and Mercerism Empathy Box were both missing from the adaptation. I would guess that it because while these are cool technologies, they aren’t very flashy. They would require a large amount of description to allow the watcher to understand what was going on. The movie was so much more focused on the special effects and plot that I think incorporating these aspects would have slowed the movie down too much. It’s the same reason I think the movie did not have a focus on animals (buying, selling, and care taking) like the book did.

PS: When I heard “Nexus 6” in the movie, I immediately thought of the phone. Which, with a quick Google search, lead me to this interesting article about Philip K. Dick’s family suing over the name:

The Orb of Virtual Reality

IMG_1281Inspired by the movie Tomorrowland, I tried to replicate the orb that allows them to travel through time and space. I also think this will be a
great input device for destop virtual reality set ups, since it gives you extra dimensions.

This ultra minimalist design has no baring on what it looks like — since the tactile feedback is all you need when you are wearing a virtual reality headset.

Kenny Friedman

The Orb In Motion


  • Oculus VR: DK2, a virtual reality head set that tracks accelerometer and gyroscopic data for head mounted display positioning
  • Leap Motion 1: IR sensor that tracks hand positions, so you can see your hands while wearing the VR headset.
  • Orb: To create the orb, I used an empty light bulb casing, a pot for a plant, and confetti filler. All of which are very cheap at Home Depot. I also used a piece of wood and plastic for a stand, as you can see in the photos.
  • The orb is wireless, and uses a hacked-up & cheap, optical USB mouse to sense the movement of the ball.


  • Unity: using Unity for the 3D virtual environment.
  • C Sharp: using C sharp scripting to capture the orb movement and adjust the head mounted display.

Future Steps

  • Currently, a Google Street View like world is viewable in the virtual reality headset and in the world.
  • The touch controls are not implemented yet, as that would be a stage 2 or 3 part of the project.
  • I think a more sturdy base, and a bigger orb would be better than the current hardware — but the current hardware is very affordable.
  • The orb should be secured, but that restricts access to the development process, so I would not lock things down until the final stage.

What I Learned

  • I had never used Unity or C# before, so I learned the language and the 3D environment.
  • I had also never used a Leap Motion or Oculus, so trying both of those out took some getting used to, but are very cool.
  • I learned that simpler is always the better option. I tried many different steps to get the orb sensing, including: using

Links to the code will be available here later this week.