Nonny de la Peña: When Virtual And Digital Become Reality – Virtually Speaking Episode 63

Nonny de la Peña is regarded as one of the most influential pioneers in virtual and augmented reality and immersive technologies and journalism. In 2022 Nonny received a Peabody Award, introduced by the Oscar Award Winning Director Alejandro Iñárritu, and in 2022 she was also inducted into the SXSW Hall of Fame. Nonny was named Wall Street Journal Technology Innovator of the Year, A Wired Magazine #MakeTechHuman Agent of Change, and she has been called “The Godmother of Virtual Reality” by Engadget and The Guardian. Fast Company named her “One of the People Who Made the World More Creative” and she was one of CNET’s 20 Most Influential Latinos in Tech for her pioneering work in immersive storytelling.

Her latest endeavor is as the Founding Director for Narrative and Emerging Media at Arizona State University, campus located in Downtown L.A. As a Harvard Graduate, a New America Fellow, Yale Poynter Media Fellow, and winner of the Knight Innovation Award, Nonny is widely credited with creating the genre of immersive journalism. She is currently a member of the BAFTA virtual reality board, and is a former correspondent for Newsweek.

Nonny’s virtual reality work has been featured by the New York Times, BBC, Mashable, Vice, Wired and many others. Showcases and Talks around the globe include the Sundance and Tribeca Film Festivals, The World Economic Forum in Davos, The Victoria and Albert Museum, Moscow Museum of Modern Art, and Games For Change. She has more than 20 years of award-winning experience in print, film and TV and her spatial narratives are consistently met with critical acclaim.

Video of Alejandro Gonzalez Iñárritu presenting the Peabody Award to Nonny:  https://www.youtube.com/watch?v=Xqoz7KZdyaY&t=0s

For more on Nonny or to book her to speak:

https://slot22.silverline.dev/portfoliotype/nonny-de-la-pena/

—

Watch the episode here

Listen to the podcast here

Nonny de la Peña: When Virtual And Digital Become Reality

Joining us is Nonny de la Peña, regarded as one of the most influential pioneers in virtual and augmented reality, immersive technologies and journalism. In 2022, Nonny received a Peabody Award introduced by the Oscar award-winning director, Alejandro Iñárritu. In 2022, she was inducted into the South by Southwest Hall of Fame. Nonny was named a Wall Street Journal Technology Innovator of the Year and a WIRED Magazine #MakeTechHuman Agent of Change. She has been called the Godmother of Virtual Reality by The Guardian and other media.

Fast Company named her one of the people who made the world more creative. She was one of CNET\’s 20 Most Influential Latinos in Tech for her pioneering work in immersive storytelling. Her latest endeavor is as the Founding Director for Narrative and Emerging Media at Arizona State University. As a Harvard graduate, New America Fellow, Yale Poynter Media Fellow, and winner of the Knight Innovation Award, Nonny is widely credited with creating the genre of immersive journalism.

She\’s a member of the BAFTA Virtual Reality Board and a former correspondent for Newsweek. Her virtual reality work has been featured by The New York Times, BBC, Mashable, VICE, WIRED, and many others and showcases around the globe including the Sundance and Tribeca film festivals, The World Economic Forum in Davos, Victoria and Albert Museum, Moscow Museum of Modern Art, and Games for Change. She has more than twenty years of award-winning experience in print, film and TV. Her spatial narratives are met with critical acclaim. Please join me with the incomparable Nonny de la Peña.

—

Nonny de la Peña, PhD, thank you for joining me here on the show. How are you doing?

Thanks for having me. This is fun.

You\’ve been busy. 2022 has been a crazy year for you so far. You have accomplished two things that people wait their entire lives to accomplish, a Peabody Award and a Hall of Fame award at South by Southwest.

It has been an extraordinary spring of 2022. It\’s unbelievable.

We\’re going to have to dive into that a little bit. I met you years ago. I remember an experience that you wanted me to remember in what you are all about, which is creating this immersive experience for people to feel things in a new way that they couldn\’t do before. You, as a journalist, were working with Newsweek. You came into this idea of using virtual reality.

I went to your offices and this situation with real audio from an actual assault by a boyfriend to a girlfriend or a wife. I went into that house and felt it. I couldn\’t even finish it because it was so powerful. I got what it meant for domestic abuse victims in a way that I never could have without you. Thank you for that and what you\’ve been doing all these years. Tell me a little bit about how that all started for you and leading you up to where you are now.

We began a little bit later. Although I was a print journalist, I also became a documentary filmmaker. I wrote stuff TV shows. I was always thinking about ways to put people on the scene and have them connect to real stories. In 2007, along with the digital artist named Peggy Weil, we built a virtual Guantanamo Bay Prison in Second Life. That\’s the OG of the metaverse. It was an interesting project because the real Gitmo was off-limits to most citizens and the press. Suddenly, we found ourselves inundated with visitors trying to learn more about the infamous prison.

After that, I was like, \”This technology could be used for all kinds of journalism.\” I then began my journey. The next huge hit was at the Sundance Film Festival. I had the first VR piece that they had ever shown, Hunger in Los Angeles. That was followed up by the first VR piece ever shown at the Tribeca Film Festival called Use of Force. It\’s about border patrol violence. Since then, I\’ve done stuff on Syrian refugees and LGBTQ homelessness and worked with the Japanese-American National Exam on the incarceration of Japanese-Americans. I have a piece premiering at Tribeca, believe it or not, on undiagnosed Lyme disease and the unfair treatment of patients. It has been a crazy career.

I\’ve got a cool new hat where I am now the Founding Director for Arizona State University on a new center on Narrative and Emerging Media here in Downtown LA. It\’s super cool. It\’s an old Herald Examiner building. It\’s old journalism and now I\’m bringing the new journalism. Even cooler is it was built and designed by a woman named Julia Morgan, the first woman to ever have the license to practice architecture in the United States. It was the first building and we\’re going to do more firsts in this building. It\’s fun.

There is a lot to unpack there. That\’s an incredible journey. The most important thing that people need to understand is you also developed VR to a point where you could use it more. You 3D printed your first wearable lenses. You see this emerging VR technology that you saw coming and a major use back when you started. How many years ago was that when you started the VR quest?

The first one was in 2007 but the goggle journey started working out of a lab in Barcelona. It\’s a piece people don\’t know as much about that puts you in the body of a detainee in a stressed position. I had all this Freedom of Information Act material. I was invited into the lab of one of the most amazing thinkers, like Mel Slater and Maria Sanchez-Vivez in Barcelona. They invited me to build this piece with Peggy.

[bctt tweet=\”Making stuff is hard, but making good stuff is even harder.\” username=\”calentertainmnt\”]

That led to the first article about immersive journalism using virtual reality for a first-person experience of the news. That was published by an MIT journal, Presence. It\’s still the second most downloaded article that they have. That was in 2010. That was followed up by Hunger. During that period, Palmer Luckey was a lab intern at USC. We only had this $50,000 pair of goggles.

When Hunger got into Sundance, the head of the lab, Mark Bolas, was like, \”You can\’t take this anywhere.\” That became the first attempt with Palmer Luckey making the goggles, driving our truck across the US, and crashing into my hotel room. Nine months later, Palmer went on to start Oculus Rift, which Facebook bought.

That\’s the founder of Oculus. You worked directly with him. When you say Hunger, I want to bring it to everybody\’s attention, it was homelessness in LA. Somebody who had diabetes didn\’t get the food fast enough. He succumbed to a diabetic coma right there on the streets of LA. You had the real audio. Was it a 911 call?

No. The hungry people and a lot of families too, I wanted to make the invisible visible. I used invisible technology to create more visibility. I had an incredible intern. Together, we were going up to food banks to record audio. I thought we were going to record a moment where food banks were running out of food all the time. What does that mean for a parent to tell their kid, \”There\’s no food. We have to go.\” What I found in recording that audio was that parents would go silent and drop.

My intern was at this scene when suddenly this whole thing erupts. It was crazy. The guy drops into a diabetic coma. The paramedics were horrible to people. The way paramedics treated people is astonishing. The food bank lady was overwhelmed, \”There are too many people.\” In the midst of the chaos, somebody tries to steal food. There were no GoPro cameras. Using that audio, we then rebuilt it digitally.

I spent $700 of my money because I had this vision. I had to get it done and learn how to be a little bit more of a C# coder. I begged and borrowed a lot of favors from some extraordinary people in that period, not just Palmer Luckey. A guy named John Brennan, who I\’m still very close with, built all the virtual cameras for Ready Player One and The Lion King I. I was on The Lion King II and he\’s a genius. He played in a mocap suit every one of the characters in Hunger and cleaned up that file.

A mocap suit is a motion capture suit.

It\’s an incredible project with incredible people who all came together to recognize these technologies of the future.

These stories needed to be told. These invisible people needed a voice and also a visual. When you\’re experiencing this, which is what I felt in that house with that domestic violence 911 call that you also built, reenacted, and created for people to experience, you don\’t feel it. It\’s over there. It\’s something you read about. It\’s awful and you move on but when you\’re in it or when it\’s your family, friend, or loved one, it\’s one thing.

If you\’re there in the presence of that situation with the man falling into a diabetic coma and you can feel it and see it, it means so much more to you because it brings humanity to the listener, viewer, or reader. This is something that we never could do in the past. You see a whole future for being able to tell stories and also to get people to empathetically buy into or relate better to situations that they never could before. Is that your end goal here?

It\’s that connection. I\’ve done stories like 24 Hours of Crack. It was a cover story we did at Newsweek on crack cocaine. I spent time in a crack den writing about it. An incredible female reporter named Martha Gellhorn, who was married for a little bit to Ernest Hemingway, called it The View From The Ground. That was the goal when I was writing stuff.

Digital Reality: Writing stuff and making documentary films are good, but none of them convey the impact of feeling like you are there, that you\’re embedded in the scene. The view from the ground is incredible.

I made documentary films but none of it conveyed the impact of feeling like you are there and that you\’re embodied on the scene. There\’s that feeling. When Hunger opened up, we had no idea and the reaction was extraordinary. We would end up staying open five hours more than we\’re meant to. Our lines were around the block the entire time. It was a moment when I recognized that this crazy idea had worked.

That\’s all the way to the Peabody Award that you received in 2022 and the wonderful director who was the person who they chose to induct you or make a presentation for you. What was his name?

It\’s Alejandro González Iñárritu. He is one of the top directors out there. I introduced Alejandro to VR. He did an incredible piece that won the first Oscar for VR. If you\’ve watched that, it\’s also nice. He thanked me on stage at that moment in that award ceremony.

It was at the Oscars. What was it called?

It\’s Carne y Arena. That piece was with a vision that Alejandro had after coming to see Hunger. He would come to see my pieces and slowly raise money to build his piece, which has been traveling. Alejandro kindly agreed to introduce me and my Peabody Award. He made Birdman and The Revenant.

It\’s life-changing.

It\’s the compliments from him.

It\’s the words that he used to address you and explain to people how you affected him in the video that I\’ve seen. He\’s so well-spoken and so well describes who you are and what impact you had on him and others. It was beautiful and well said. I\’m sure that must have been pretty awesome. You\’ve been surrounded by incredible and awesome people.

You don\’t have to name-drop but every time I hang out with you, these are amazing people who are all fans of yours and want to help you do this. It\’s an interesting place we\’re in because the metaverse is starting to be spoken about so much more. You\’ve been living in and creating the metaverse for years. What\’s your take on where this is all going? What\’s the new vision that you have because you always hit it right on the mark?

I come from a genuine place on this. I grew up in Venice, California, and went to Venice High. I ditched plenty to go to the beach and hung out with my favorite hoodlums and sometimes still do. I was a smart kid. I got into Harvard when I was seventeen. I had this boyfriend who would come over the summer before from Ohio State who was like, \”You need to apply to Harvard.\” I was like, \”I\’ll do that.\” I didn\’t expect it. I was seventeen. I had never been to the East Coast in my life.

When I showed up at school, I had no coat and boots. I had to do this always in my bootstrap way. The way I put it is I figured that and this out. When I\’ve got an idea and I sink my teeth into it, I am that dog with a bone until I make the thing that I want to make. Immersive journalism was it. I had people and fingers in my face going, \”That\’s not journalism. You can never do that. That\’s never going work.\” The pushback on it shocks me.

[bctt tweet=\”What\’s going to engage people and build teams is a sense of shared experience in these spaces that don\’t have to be limited to a conference room.\” username=\”calentertainmnt\”]

You understand the pushback on some things but when I\’ve heard about the pushback with you, it doesn\’t make any sense. The other funny thing is you\’re the most likable person somebody could ever meet. That\’s why you have so many great people who are your friends and are helping you. It\’s shocking to hear that you had any resistance from anyone.

I can be stubborn. It\’s true. I\’ve had amazing people work with me. The list of folks who have come through Emblematic and helped create these pieces is a team of producers, engineers, and artists who are willing to join me on this vision. I\’m grateful. I know that people put in hard hours sometimes in ways that nobody should have to. It is the film biz. Making stuff is hard. Making good stuff is even harder.

What about the metaverse? You\’ve been living in it and creating it. It\’s funny because I was thinking about talking to you and I said to myself, \”She has been creating that. She probably is the number one person people want to consult with them on how to make a room or a town.\” What\’s your vision of it? How can it be helpful? What do you think is going to happen initially and then maybe in the long run?

We\’re going through this funny period that is a mistake but there are probably some interesting thing that comes out of it. We\’re in that moment where we went from theater stage to film. I watched His Girl Friday. It\’s a great film but it could easily be all set in different theater stage designs. It\’s all static in rooms. We\’re doing that in business. We\’re trying to recreate conference tables and chairs and have people gathered that way without realizing that the space doesn\’t require that. What it requires is experience.

As far as the metaverse, you\’re going to have these large gathering spaces and meet with people and there will be small ones. We made one during the pandemic where we gathered in my team. It was a platform floating in the sky. We had these mics. If you clicked your controller, it would go loud so people would know who was speaking. There were some balls around. I didn\’t know how it started. Somebody threw a ball over the edge.

We\’re all looking over the edge to see when it would stop falling because we were up in some infinite space. Pretty soon, we\’re all throwing balls over the edge. It was this joint funny and weird experience that we\’re having together in the space. That\’s what I mean. That\’s what\’s going to engage people. It\’s going to build teams. It\’s a sense of shared experience in these wonderful spaces that don\’t have to be limited to a conference room.

There\’s a lot of opportunity for being completely creative and creating an experience. That\’s very memorable.

It\’s going to be interesting as we transitioned to these different things. Education is going to transform. You\’re not going to expect kids to sit there in the seats. They will be able to move their bodies through spaces. I\’m working with my husband. He wants to take his students into the Sistine Chapel. I\’m like, \”I\’ll make that work.\” I\’ve been building this software that you may or may not know of called Reach.love.

That software is a button-based system to let people make their facial content without knowing how to code. It was a rough ride for me to have to learn how to be a C# coder. I never got great but I got enough to make my stuff and then understand what other people could do. I\’m trying to make something that\’s much more intuitive, shared across all devices, and browser-based so you don\’t have to download anything. I\’ve been working on that. We should be deploying our beta in the summer of 2022.

What is it that ASU hired you for? What are you doing with them over there in Downtown LA? What\’s that all about?

It\’s super exciting. President Crow at ASU has gone all-in on immersive.

Digital Reality: Education is going to totally transform. We can expect kids to just sit there in the sheets and be able to move their bodies through spaces.

When we say immersive, we\’re talking about immersive journalism.

No, we\’re talking about immersive.

That\’s immersive everything. That\’s metaverse and VR.

It includes the metaverse, augmented reality, AI-driven narratives, machine learning-type exercises within virtual reality, and different experiences that come with machine learning.

I\’ve been reading all these articles because Google probably knows that I\’m interested in them. AI is writing your jokes, scripts, content, and ads for you.

I\’m going to do an adventure on AI narrative in my new building because what\’s going on is super interesting. This idea of being in a transportative place as a group is your metaverse. The metaverse is when you\’re in a virtual space that is shared. Sometimes it\’s immersive content with a one-on-one experience and sometimes it\’s larger. It doesn\’t matter. All those things are built with the same underlying technologies and building blocks.

That\’s what Arizona State University is embracing in a huge way. As part of that, I was hired to create a new center on Narrative and Emerging Media. It\’s our graduate program. It\’s going to be launching in the middle of August 2022. Our applications are going to be going live if you look for Narrative and Emerging Media. We have got some scholarship money to give away to start this first cohort because we got approval. Our applications are going up late.

I\’m going to have an interesting center on research and haptics and how we use physical body devices. For example, for Project Syria, you would wear haptic vests and feel the bomb going off in Aleppo. I had a deaf woman go through that. She engaged with me for weeks about it even though she couldn\’t hear anything and also with the head of the festival for months. This idea that we could use haptic for storytelling and make it more inclusive is an interesting project, so there\’s that.

We have the skunkworks to build stuff and help grow startups and endeavors with private students. In this incredible event space, I\’ll be doing a trailblazing women\’s event in late August 2022. We will be hosting a series of amazing things throughout 2022 by me in Zócalo. They\’re there now. It\’s such a rich environment. I can\’t even believe it. It\’s super fun.

We have the most extraordinary state-of-the-art equipment. We have a virtual production stage and dozens of game engine-ready computers. It\’s ready to engage with these new immersive technologies and help grow our system. It\’s a very inexpensive Master\’s program relative to other places. Therefore, I\’m able to diversify the demographics and make opportunities for people to enter this nascent but growing-like crazy field.

That is amazing and exciting. There\’s so much to talk about but I do want to start with two things that I\’m curious about. The Master\’s program is not just for journalists.

[bctt tweet=\”Engaging with these new immersive technologies can help grow our system.\” username=\”calentertainmnt\”]

It\’s for anybody interested in narrative and emerging media. We have a virtual production stage. If you know how The Mandalorian was made, if you go watch that video, we have that technology in our building to teach students how to go out into this field that\’s growing like crazy in Hollywood. It\’s about narrative. My expertise tends to be journalistic. I have hired this extraordinary woman named Mary Matheson. She\’s also a genius in the field who started a program similar to this in London.

I was able to approach her from there. She worked directly with me. She\’s this intrepid reporter who cut her teeth in the mountains with guerrillas in Columbia. She\’s extraordinary. There is a strong journalistic bent here. It is a joint endeavor with the Cronkite School of Journalism as well as the new Sidney Poitier New American Film School. We want you to come in and think about how you can use emerging technologies to tell narratives.

That\’s amazing. My other question was when you talked about wearing the suit, not only are you wearing the glasses. Let\’s pretend you\’re deaf or blind. You can still feel and hear. Is it ever going to be too much? Are the older people who are in their 40s and up not going to be able to handle it or is it all for the younger people? Is there going to be a point where you\’re going to play a video game or watch a video virtually with virtual reality and augmented reality? Is there going to be a point where you can\’t tell the difference between reality and it\’s going to look so real? We already have deepfakes. What\’s going on with all that?

It\’s not an irrelevant concern. We\’re always going to have to keep an eye on the most ethical frameworks to process stuff. I use deepfakes in my piece going in Tribeca on Lyme disease for purpose. I use it to cover a 30-year span. The doctor I follow uses early photographs. We go to the time with them and me as a reporter. WITNESS, which is one of the most amazing journalism organizations, did work using deepfakes to let people in Chechnya go undercover and tell what was going on and protected their identity. It felt seamless.

There\’s a positive thing with this culture but there are serious issues. Everybody\’s smartphone has a wider camera. The new ones all have wider cameras. What are you going to be filming when a bomb goes off? Are we going to let people step over the bodies? How are we going to ethically think about what material is going to be journalistically provided? There are going to be some things that are going to get out the same way that the gunman in Buffalo live streamed that video.

My son said to me, \”When I talked about that, everybody I knew had watched it. I started to watch it and I stopped.\” What does it mean? Are we going to have to teach critical thinking? There are very important questions. One key course in our Master\’s is an ethics course. We will grapple with some of these issues with privacy. These are important questions you\’re raising.

We are coming out of the pandemic. You\’ve got the new school, the new location, the Peabody, and a lot of things. 2022 is a huge year for you. It\’s a pivotal point where everything you\’ve done has led up to this moment. It\’s the end. This pandemic feeling is going away. All the live in-person events and back-to-work is happening. What are you most excited about in terms of technologies and also intersecting with your job and your place in all of it?

There\’s this time in which people felt like VR had gone up and down or something but suddenly, there\’s this explosion. Meta or Facebook has three new types of goggles out. I know that Apple has taken their hardware guy and put him onto their AR/VR team. God knows what Apple is going to show up with soon.

They\’re going to have the glasses.

These are bone conduction audio. These are my Bluetooth glasses. I do all my phone calls and listen to music on this. These are going to be my AR/VR glasses too. I love these glasses. I can ride my bike and still hear the cars coming. They’re safer. I understand Google is now stepping back into the game too.

I\’m not going to ask you to say who has the best VR glasses.

Digital Reality: It’s delightful that all these artists, thinkers, creators, storytellers, and educators are starting to embrace these technologies for their practice.

I\’m going to go away from the hardware and talk about, \”What then?\” That\’s what I\’m most excited about. I\’m most excited about all the incredible creators out there, the beautiful things that are going to get made, and the extraordinary experiences we\’re going to have. I\’m already seeing this explosion of women getting involved, which is fabulous because it was very different certainly at the beginning of a lot of other technologies. I am most delighted about the idea that all these artists, thinkers, creators, storytellers, and educators are starting to embrace these technologies for their practice.

It\’s easier for people to get involved in creating, entertainment, movie making, journalism, and all of it because everybody has access to phones. Everybody has the technology in their hands to do it. It\’s not like you have to be in the mailroom and work your way all the way up through this major studio anymore. You can do it, post it, submit it, and contribute to it. I imagine the metaverse will be very open to a lot of different people\’s interpretations of places and things to see there.

Experiences to me are going to be interesting. I know that there\’s going to be propaganda and the dark side. Humans can be dark but I believe that we\’re going to have some very joyful and beautiful connecting experiences through these new technologies.

Here\’s one last question for you before we wrap it up. I\’m excited to know this. Have you heard of any businesses or companies or have you thought of any ideas that a business is doing in the metaverse or with VR at least that is revolutionary and that you think a lot of other businesses should do, could do, or will do?

There are platforms that are coming out. Spatial is an easy one. It\’s fun.

The meeting space is probably the main area or the collaborative hangout place.

Honestly, whatever we think of Facebook, they have brought in some extraordinary talent. Some of the technologies that they\’re deploying are very interesting. There\’s a digital twin of yourself in these spaces. There\’s some interesting work being done there. Businesses are starting to recognize that this is a useful place for people to gather. The other thing is Mozilla Hubs. Hubs is fun. It\’s another metaverse earlier before Facebook\’s Horizon. It\’s easy to create your space, come together and share. Mozilla has been ahead of this curve. They\’re coming back in as well. I\’m super excited.

I\’m super excited always about you as a thought leader, speaker, journalist, and educator. I know that you are super excited about the school and everything that you\’re about to do. I only can imagine where you will be in the future and the things that you will be creating. Thanks so much for taking the time to come on here with me and talk about all of this. I enjoyed it. It\’s always a pleasure, Nonny.

Thank you so much, Chris. I appreciate it.

Important Links

• Nonny de la Peña
• Narrative and Emerging Media
• Peggy Weil
• Second Life
• Oculus Rift
• Reach.love
• Project Syria
• Mary Matheson
• WITNESS
• Spatial
• Mozilla Hubs

About Nonny de la Peña

Nonny de la Peña, founder of Emblematic Group, was selected by Wired Magazine as a #MakeTechHuman Agent of Change and has been called “The Godmother of Virtual Reality” by Engadget and The Guardian. Additionally, Fast Company named her “One of the People Who Made the World More Creative.” for her pioneering work in immersive storytelling.  At Emblematic Group, she uses cutting edge technologies to tell important stories—both fictional and news-based—that create intense, empathic engagement on the part of viewers.

A Yale Poynter Media Fellow and a former correspondent for Newsweek, de la Peña has more than 20 years of award-winning experience in print, film and TV. De la Peña is widely credited with helping create the genre of immersive journalism and her virtual reality work has been featured by the New York Times, BBC, Mashable, Vice, Wired and many others.  Showcases around the globe include the Sundance and Tribeca Film Festivals, The World Economic Forum in Davos, The Victoria and Albert Museum, Moscow Museum of Modern Art, and Games For Change.  She was also “in the room” when the Oculus Rift was first created.

Love the show? Subscribe, rate, review, and share! https://slot22.silverline.dev/virtually-speaking/