MoviePilot
by Nitin Shantharam, David Resnick, and Rebecca Maessen

What is it?
MoviePilot is an interactive gesture-controlled movie discovery system designed to be placed in movie theater lobbies.  The user performs natural gestures in front of a display in order to “pilot” through information about movies and play trailers.  MoviePilot provides a novel, fun, and informational service to theaters and moviegoers.

MoviePilot will work well in further expanding ubiquitous computing technologies in the moviegoing experience.  Theaters are already at the forefront of entertainment technology and recent years have seen the incorporation of ubicomp with e-ticketing in theaters, and websites like Fandango and Movietickets that enable mobile access to movie showtimes and e-ticketing.  MoviePilot builds on this already thriving ubicomp platform, is relatively low-cost, and has a high upside for both theaters and customers.

Screenshots

What does it do?
MoviePilot allows customers to explore detailed information – movie poster, description, MPAA rating, user rating, trailer, run time and show times.  The movies shown can be customized for any theater to display the movies in the box office and movies coming soon.  We believe that MoviePilot will fill an informational gap in movie theaters and provide theaters with an activity-of-interest to attract more customers.

The display would be quite large so that many people could gather around to see it while the system is able to recognize one person as the “pilot”.  MoviePilot adds the benefit of discovering a movie on a big screen with their friends and strangers around them–a primary reason they came to the theater in the first place instead of watching alone at home.

How do you use it?
The pilot is “captained” by the system by holding her hands up to the sides of their head until she sees circles, representing her hands, appear on the display. Once recognized by the system, the pilot can “take control” by moving her hands next to each other with her palms facing the screen.  The control-state is indicated by the grey circles turning white.  When she drags her hands to the right, MoviePilot browses one movie to the right.  Same goes for dragging to the left.  This is how to navigate through the selection of movies.  To play a trailer, the user drags her hands up.  To exit the trailer, drag them down.  Help information is presented in a persistent box along the bottom of the screen and details all the gestures the system recognizes.

What technologies does it use?
MoviePilot uses the Microsoft Kinect’s 3D camera, the NITE/OpenNI framework to generate motion tracking data points, and the Processing programming environment to code our custom software.  The Kinect consists of an RGB video camera, IR video camera, IR illuminator, and microphone (not used in this project).  We use NITE’s user recognition and skeleton tracking abilities to generate 3-axis data points for the user’s hands.  We stream the hand location data into Processing where we implemented the gesture recognition, graphical display, and movie content information.

This video was created in order to train people in a language of vibrotactile expression I call Vibespeak. It uses my chair apparatus from Undulating Flux. The video goes through how to use the Wiimote to control the hand and foot motors. You can have the system hooked up to the actual motors and see/hear/feel the vibrations as you go, or you can use the onscreen visualization for a basic guide to see if you’re doing the movements right.

In the future I hope to develop Vibespeak further so that people can use it remotely and without needing to have my specific hardware (opening the system up to people using whatever DIY actuators they want).

My group project in a User Interface Design and Evaluation class focused on the comparative evaluation of two mobile location sharing systems–Google Latitude and a research prototype Android app called Longitude.  The main points of consideration were privacy, social pressures, and context-specific etiquette with friends, family, and workplace. The ultimate goal of this study was to prepare the two applications for a formal usability test.

The first task was to analyze the existing test scripts and user scenarios.  From this analysis, we revised the scripts and began revising Longitude accordingly.  We performed two rounds of pilot testing and continued an iterative revision process of the test scripts and location sharing app.  Finally we presented usability-test-ready scripts and app along with a results analysis.

Our studies showed that people found the Longitude prototype more useful and more considerate of their privacy needs, which in turn caused them to feel that they would be more likely to use Longitude.  Females were more interested in finding opportunities to meet up with people and both males and females over age 25 were more interested in meeting up with people they’ve been separated from as opposed to coordinating a pre-arranged meetup.

Our “client” was HCI PhD student Xinru Page.  She created the project with guidance from Professor Alfred Kobsa and assistance from masters student Chung-yi Cho.  My team members were Arjun Bhattacherjee, Jackie Doong, and Raminder Singh Goraya.

You can download the music for Undulating Flux by saving this file to disk or click “play” below to listen now.

Download audio file (undulating_flux.mp3)

View my presentation of Undulating Flux at Dorkbot SoCal (video and pics).

Checkout the article about me and my vibrotactile work in Coast Magazine.

Read features on Undulating Flux in Gizmodo and Engadget.

———————————————

Undulating Flux by David Resnick
With Vibrationist Nadia Ahmed

Project Advisors: Simon Penny, Beatriz da Costa, Chris Dobrian

With assistance from: Sheron Wray, Tom Jennings, Garnet Hertz, Eric Mesple, Ian Hattwick, Dana Friedman, Domingos Begalli

Presented in an exhibition of graduate work
ACE 2010: Transmissions from Inside

Beall Center for Art + Technology at the University of California, Irvine
May 13 May 20, 2010

Arts Computation Engineering program
University of California, Irvine

Testimonials

“You’ve invented a new medium.” -Simon Penny, professor of Arts and Engineering, founding director of ACE, UC Irvine

“I felt like there was no delay between the dancer’s quick movements and how the chair responded.  You could get lost in the multi-sensory experience and at times the chair seemed to anticipate the dancer’s movements instead of react to them.” -Meg Cramer, PhD candidate in Informatics, UC Irvine

“It took me just the first 10 seconds to realize the dancer’s motion and the motor vibrations were perfectly synchronized.  It was interesting to experience the way the dancer interpreted the music, which implied to me the potential to share the personal experience of music with multiple people.” -Isshin Oikawa, MFA in Integrated Composition Improvisation and Technology, UC Irvine

Theoretical Background

Undulating Flux is my first-year project in the Arts Computation Engineering program at UC Irvine. The overall goal of Undulating Flux is to bring the participant into what psychologist Mihalyi Csikszentmihalyi termed a flow state. A flow state is described as a state of being wherein action follows upon action according to an internal logic that seems to need no conscious intervention by the participant. One experiences it as a unified flowing from one moment to the next, in which one is in control of actions, and in which there is little distinction between self and environment, between stimulus and response (Csikszentmihalyi 1975) Another term for this state, now common in both sports and business, is being in the zone.

My hypothesis is that being in a flow state enables a heightened sensory experience for the participant and leaves one in the end feeling expanded and yet present, energized and yet relaxed.

The areas I engage with Undulating Flux are artistic performance, gaming, massage therapy, and sensory assistance. Because this iteration of Undulating Flux was presented in an art gallery, it focuses on artistic performance.

I set out to explore three questions relating to these areas:

1) What does it mean to feel music and movement together?
2) Can haptic sensation facilitate a flow state for the participant and even be therapeutic?
3) What are the potential applications of a feeling-enhancing system?

How It Works

Undulating Flux explores these questions by setting up a transduction chain wherein a vibrationist sends intense music and motion-synced vibrations into the participants body. The participant experiences stimulation of their eyes and ears, as well as a massage-like feeling in the rest of their body. It is through this sensory immersion that Undulating Flux attempts to induce a flow state.

Each wiimote controls one handrest motor and one footrest motor. The position up and down in the pitch plane controls the handrests while the position side to side in the roll plane controls the footrests. I designed the controls in this way so that the vibrations that the participant experiences are a sort of mirror image of the vibrationists movements.

The vibrationist and I worked together to choreograph the vocabulary of the vibrational dance language, which is intended for improvisation. Drawing on influences from Afro-Brazilian, Indian, and flamenco dance, we modified the straight-up moves, which produced chaotic results in the vibrations, and adjusted the wiimote position tracking, so that we could have more precise vibrational controls. The vibration rig became a sort of instrument and the dance became the technique with which to play it.

The technology behind this project centers around the Nintendo Wii remote and the Max programming environment. The wiimote data is streamed into Max wherein parameters are set to control the intensity of each vibrator individually. A Max package called Maxuino is used to send the control data to an Arduino microcontroller, which is connected to the motors.

I composed the music specifically for this project using multiple rhythmic subdivisions and evolving timbres to provide the vibrationist with musical material well-suited for transduction. The driving pulse, jagged staccato strings, and thick slow-moving bass are examples of this. The piece was composed entirely electronically using synthesizers and samplers, with the exception of my throat singing (heard in the intro) which has been filtered to sound more like a synthesizer.

Undulating Flux is a starting point to working with transduction chains and haptic feedback. It only begins to answer the questions I set out to tackle. Future research in the areas of biofeedback, synaesthesia, game technology, philosophy of mind, and Eastern medicine will help inform my work towards developing the efficacy of flow state inducing systems.

This is the first step towards building an interactive sound and light physicalizer called the Enhanced Vibration Sound & Light System (EVSLS). The system will use intense physical vibration in realtime interactive sync with sound and light in both artistic and therapeutic applications. In this proof-of-concept, I made audio frequency control motor vibration using a microphone, Processing, Arduino, VN05, a big motor, PVC pipe, and foam.

It’s hard to distinguish the motor speeds on video (you’re supposed to hold the pipe and feel it, not watch it) but you can hear the speed changing if you listen closely. Maybe in the future I’ll add an easily noticeable visual cue that indicates motor speed.

This was my final project for the fall quarter Hardware Intelligence and Microcontrollers classes in the Arts Computation Engineering program at UC Irvine. Special thanks to professors Simon Penny and Tom Jennings for teaching awesome classes, Eric Mesple for helping with the hardware, Ian Hattwick for filming and being my first user, Sen Hirano for being my second user, my uncle Larry Revit for his expert feedback, and my parents for being supportive as ever.

The Glow is my current trip hop/electronica/R&B band (previously Marsh Meadow).  We’re  in the songwriting phase and plan to start playing out as soon as we have enough songs.  I’m really excited about the overwhelming positive feedback our music has been getting.

The band is:

Daniella Gruenspecht – vocals, lyrics, synths

David Resnick – synths, sampler, MIDI triggers, guitar, bass

David CasT – guest saxophone on “Jordan’s Moods.”

(use the left and right arrows in the audio player to change tracks)

I got into electronic music and composing with electronics in my junior year at Bard College.  Until then I had exclusively written for orchestral instruments or rock band arrangements.  Then I heard Morton Subotnick’s Silver Apples of the Moon and it changed everything for me.  From that piece, I understood that, with electronics, I could craft any sound I wanted and discover sounds I never dreamed of.

The first track, Little Flowers Meaty Monsters, was my final project for my first digital composition class that I took at the McGill University Conservatory of Music.  I collaborated with Alex Colwell, who did much of the drum programming, and a jazz guitarist who played the intro (but I can’t remember his name).  It was fun writing the music and then experiencing digital editing and sound manipulation, as well as programming the synths.  If anyone out there wants a lyric sheet for the “poem” I recite in the beginning, feel free to contact me

The second track, Porpoise, is often thought of by people as “creepy” sounding.  I URGE you to listen to it with a meditative mind and I promise you won’t find it creepy, but rather deep and exploratory.  I recorded three synth parts separately, attempting to remember how the first part sounded as I played the next parts.  It was an interesting compositional technique to try out.  The third synth part was a bass instrument, which I also edited as a fourth layer with automated delays and reverb.

The first clip uses captures of three different shows.  Two of the shows were from March/April 2008 at the Opium Garden during Winter Music Conference in Miami; one for the Aquabooty party and one for the MN2S house music label.  The third VJ show was with the Gemini Trio at the Pearl Gallery in Stone Ridge, NY in August 2008.

The credits are:

David Resnick – original video, laptop, midi controllers, mixing, master editing
Erik Jacobs – original video, source footage capture/editing/processing, laptop, DVDs, kaoss pad, mixing

Music: “Move for Me” by Kaskade

Video © 2008 Love Intelligence Group
Some source footage is courtesy of Vello Virkhaus.
This clip is for promotional purposes only.

The second clip is from recordings made during live shows from May-December 2005 in New York City and Miami.  It appeared on Love Intelligence Group’s second VJ demo reel and remains one of our strongest pieces to date.  The clip uses live performance footage of Fela Kuti and mixes in various original LIG footage as well as other found footage.

The credits are:

David Resnick – original video, some source footage preparation, laptop, midi controllers
Erik Jacobs – master editing, original video, most source footage preparation, DVDs, kaoss pad, mixing
Gabe Shalom – original video, some source footage preparation, laptop, midi controllers

Music: “Let’s Start” by Fela Kuti & Africa 70
Video © 2006 Love Intelligence Group
This clip is for promotional purposes only.

The third clip is also from recordings made during live shows from May-December 2005 in New York City and Miami.  It, too, appeared on Love Intelligence Group’s second VJ demo reel and remains another one of our strongest pieces.  The clip uses very eclectic and esoteric source footage and some of our own original footage as well.  Enjoy the Tibetan horse racing! (kudos to Erik for finding it)

The credits are:

David Resnick –  original video, some source footage preparation, some master editing, laptop, midi controllers
Erik Jacobs – master editing, original video, most source footage preparation, DVDs, kaoss pad, mixing
Gabe Shalom – original video

Music: “Now Is The Time” by The Freestylers
Video © 2006 Love Intelligence Group
This clip is for promotional purposes only.

(use the left and right arrows in the audio player to change tracks)

I co-produced, co-engineered, co-mixed, and co-mastered the Jeff Marx Quartet’s album Treading Air Breathing Fire on Soluna Records.  The musicians are Jeff Marx–saxophone, John Esposito–piano, Ira Coleman–bass, and Peter O’Brien–drums.  It was recorded with Paul Antonell at the Clubhouse, Rhinebeck, NY, mixed with Russell Frehling at the Studio at the Deep Listening Space, Kingston, NY, and mastered with Scott Petito at NRS Studio, Catskill, NY.  It was an awesome experience working with musicians, engineers, and studios of this caliber.  I was quite a youngin at the time, but I think I earned my respect–listen for yourself.

The credits are:

David Resnick – original video, laptop, midi controllers, mixing, master editing

Erik Jacobs – original video, source footage capture/editing/processing, laptop, DVDs, kaoss pad, mixing

Music: “Almost Ready” by Dinosaur Jr

Video © 2007 Love Intelligence Group
This clip is for promotional purposes only.

Ram is based on a section of the Ramayana wherein Rama meets Sita, Ram gets distracted by a deer and runs off, Sita is kidnapped by Ravana’s demons and taken to Lanka. Rama goes in search of her and, on the way, enlists the help of Hanuman and his army. They find their way to Lanka, battle Ravana’s demons, and rescue Sita.

Ram was created out of a discussion that Michael and I had in class, about what it would be like to create a dance piece intended for video and to use the video medium to do things with dance that you can’t do on stage.

Music composed by David Resnick
Choreography by Michael Braverman-Scult
Directed by Daniel Stone, David Resnick
Photography by James Curcio
Edited by Daniel Stone
Costumes by Sarah Dudzic and James Curcio
Sound engineered, mixed, and mastered by David Resnick
Produced by Evolving Media Network

Dancers: Michael Braverman-Scult (Rama), David Resnick (Hanuman), Dages Keates (Sita), Christi Mueller (Demon), Meagan Leatherbury (Dear, Hanuman’s army), Angelina (Demon).

Musicians: David Resnick (synthesizers, guitar, drums, drum programming, digital manipulation), Daniel Stone (vocals), Ehren Hanson (tabla), Bill Ylitalo (saxophone, gamelan), James Curcio (bass), Megan Bodane (flute), Mills Baker (drums), Molly Zenobia (vocals), Sarah Dudzic (vocals). Gamelan: David Resnick, Bill Ylitalo, Daniel Gibson, Meagan Leatherbury, Nikki. Improvised mass: David Resnick, Daniel Stone, James Curcio, Sarah Dudzic, Jonathan Metnetsky.

The first piece, Elunia, was for choreographer Ariel Grossman.  I started out by taking a string sample from David Homan’s ballet, Romeo and Juliet, and then added other instruments of my own creating a whole new piece around the sample.  Ariel ended up creating an even larger work out of Elunia called Moonsong which incorporated another piece of mine, Leel Ach Be’loh, and also the work of Cybele Kaufman and David Patterson.

The second piece, Ereneme, was for choreographer and York Dance artistic director Lucy Streuver. I composed everything on the guitar and then refined the melody for violin.  It was performed live with Amy Hondo, violin, and recorded with Lisa Liu, violin.

The dancers are Ariel Grossman, Kiley Durst, Amanda Kwolyk, Courtney Rottenberger, Brianna Smith, Shayne Staley, Claire Cholak, Sarah Kapchinske, Andrea Brown, and Anna Brown Massey.

(use the left and right arrows in the audio player to change tracks)

UltraSpace was a live improvised electronic music duo comprised of myself and Ehren Hanson. Ehren and I have been friends since 1997 and our musical collaboration goes back to 1999.  We’ve inspired each other in many ways and also taught each other a lot about music.  In the case of UltraSpace, Ehren told me he wanted to start integrating electronics into his tabla playing so I helped him get started using synths and the computer to make music.  Pretty soon we were improvising together and UltraSpace was born.

In the three clips above, you will hear pieces that were recorded as live improvisations without any loops or sequencing.  The credits are:

Venus Flytrap

David Resnick – synth flute

Ehren Hanson – M3X synth

Floppy Bop

David Resnick – Handsonic drums

Ehren Hanson – M3X synth, Absynth

Tabla Hop (clip)

David Resnick – Handsonic tabla and drums

Ehren Hanson – M3X synth, Absynth

Here are some photos from various live sessions.

(use the left and right arrows in the audio player to change tracks)

UltraZen was a production team I had with Molly Zenobia.  I had played drums in Molly’s band for a couple years and then helped her with some of the electronics programming for a big multimedia show of hers, so our great friendship and a rapport for creative collaboration and were already established.  Molly got an offer to score the soundtrack for a film, Last Grave, and she called me in to work on it with her.

As is normal in the film world, there was a really tight deadline so I moved my studio into hers and we worked for about 2 weeks straight.  It was fun having a setup like that; all of our instruments and gear ready to go the moment we had an idea, with nothing else to worry about except making music for this film.

We made some pretty cool sounds and a few really good songs, two of which you can listen to above.  The credits are:

Mr. Robot Earns His Wings

David Resnick – drums, bass, synths

Molly Zenobia – vocals, lyrics, keyboard, synths

Magnetic Detraction

David Resnick – drums, bass, synths, vocals

Molly Zenobia – vocals, lyrics, synths

Here are some photos from the Last Grave recording session.

Marsh Meadow is my folk pop band that has recently shifted style to become an indie-electronica band, The Glow.  We started recording an EP and then after the first song was completed we decided to just focus on the new material so we never finished it.  I think this one song, All Out, is really good though, and I hope we get to record the other folk songs someday.  So enjoy this song for now and stay tuned for the indie-electronic music….

The band is:

Daniella Gruenspecht – vocals, rhythm guitar

David Resnick – electric guitars, bass, synth

(use the left and right arrows in the audio player to change tracks)

Ultrasonix was a band of mine that played live improvised electronic music.  I had been using electronics to make music for a while but never in a live setting.  This was my first exploration of manipulating electronic instruments and controllers, using software looping and sequencing, and improvising electronic music all in real-time.

The lineup was:

David Resnick – handsonic, samplers, laptop (synths, sequencing), effects

Erik Jacobs – octapad, samplers, synths, effects

Jack Ryon – wind controller, samplers, laptop (synths, sequencing), effects (Jack left the band in June 2004)

We recorded three live albums that were really just recordings of our rehearsals, but then again our rehearsals were like performances so who can say what the difference is.  The albums were titled Rehearsals 3/04, Rehearsals 5/04, and Rehearsals 8/04.

Here is a mix of photos from our performances and studio work.

photoshoot in a field in New Paltz, NY (2003)

Download audio file (northern_line.mp3)
Download audio file (fallin_me_down.mp3)
Download audio file (across_the_heart.mp3)
Download audio file (red.mp3)
Download audio file (vanity_highs.mp3)

(use the left and right arrows in the audio player to change tracks)

The Beams is my alt-country project with Brandon Smith. It started out with Brandon playing a few songs for me in my dorm room in college and I whipped out a guitar and played along.  The result was instant chemistry.  We decided to record an EP, and thus, The Beams was born.

The 6 song EP called Two Weeks In May was really fun to make and sounded good, so eventually we started work on a full length album, Across The Heart.  Being a multi-instrumentalist, I recorded many of the band parts myself (the first of what would become a staple in many of the productions I worked on).  I also produced and engineered the album, with assistance from Brandon and Daniel Mostovoy.

We pooled our talented friends together to lend their skills on the recording.  The credits for Across The Heart are:

Brandon Smith – vocals, lyrics, rhythm guitar, keyboard

David Resnick – electric guitars, drums, bass, keyboard, vocals

Daniel Mostovoy – mandolin, bass, vocals

Matthew Loiacono – mandolin, banjo, lap guitar, vocals

David Homan – piano

Dana Barnett – vocals

Tara – vocals

Scott Petito – double bass

Here is a mix of photos from various Beams concerts, recording sessions, and our presskit photoshoot.  Daniella Gruenspecht joined us on vocals for the Baggott Inn performance.

with special guest Todd Reynolds – violin, electronics

Visuals by Love Intelligence Group: David Resnick – laptop, keyboard, DVDs
some footage prepared by Erik Jacobs (Love Intelligence Group), Tomas Ceddia (Aquabooty), and VJ V2 (VSquared Labs)

These clips were taken from a live performance of Geminii with Love Intelligence Group at the Pearl Gallery in Stone Ridge, NY on August 16, 2008.  This concert was part of the Pearl Jamzz Concert Series brought together by multi-instrumentalist David CasT to explore improvisation in its deepest forms.  The group layers the interactions of live rhythm, sampled grooves, and instrumental solos with live visuals.  The improvisations move through electronica, acid jazz, hip-hop, drum ‘n bass, funk, and ambient.

Click here to view the embedded video. Clip 1 features fast-moving visuals, but the rhythm that the speed creates with the material used actually has a 1/4 time sub-rhythm that syncs up with the music quite nicely.  The music is electronica-y with a rhythmic pad provided by effected violin and then a saxophone solo on top.

Click here to view the embedded video. Clip 2 features a color-sparse cut-up looking foreground element moving in place over very soft, blurry, more lush backgrounds.  The music is a pleasant ambient that evolves into an idm groove with jazzy overtones.

Click here to view the embedded video. Clip 3 is the most intense of the bunch.  The visuals are wild and freaky with both abstract and figurative elements.  The music has a harder funk/hip-hop groove and an acid-jazzy multi-octave effected saxophone solo.  For me, the visuals in this clip do the best job of syncing with the music–I remember hitting my keyboard wildly during the performance.

Click here to view the embedded video. Clip 4 does not contain my favorite visuals, but the violin solo is just so fantastic that I had to include it.  There’s a distinct middle eastern flavor to the music with a hard world-funk groove that evolves into a glitchy ambient section (where the visuals work better with the music).