How do you make a Virtual Reality Glove? – Smarter Every Day 191
Articles Blog

How do you make a Virtual Reality Glove? – Smarter Every Day 191

August 25, 2019


Hey, it’s me, Destin. Welcome back to Smarter Every Day. I want this video to be long and I want it to get down into the weeds and just air out and let me get as technical as I want to, so I’m gonna mention this sponsor right now upfront, which is different, I know. But I want to say thank you to 23andMe for sponsoring this episode of Smarter Every Day. I’ve been to their lab too, its a really cool place. So, I’ll tell you more about their at-home genetic test service later in the episode but, for now let’s talk about Haptic Gloves. In the last episode of Smarter Everyday you got to see me interact with the Haptic Glove and it changed my mind about VR (Virtual Reality) and here’s the deal, when you’re a start up company, it’s a double-edged sword, right? You want people to know that you’re doing these great wonderful things so you can get investors and stuff like that. But conversely you don’t want them to know the “secret sauce” because you gotta maintain your competitive edge, right? So, the fun thing about this is when I got down to talking to the engineers at like the fundamental Haptic Pixel level on how this thing works, you can see their eyes dartin’ around the room, like they freak out. The reason they freak out is, because they haven’t been permitted to talk about how this glove works in forever apparently and then this dude walks in with a camera and sticks a camera in his face and asks him very technical questions, its really fun to see their reaction but, I was able to sign some paper work and we were able to work out a deal so that I can take you right up to the edge of proprietary information, but not cross that line so that they can still maintain their edge. So after I met the founder Jake and the co-founder Bob, they turned me over to the head of engineering ops, Lathe, to asks questions. And in the last video we went through what the experience was like, this time I just wanted to focus on how everything works I decided to just go for it and ask Keenan what kind of actuators they used to simulate touch. -So, I am going to wear a glove under the glove.
-Correct, its just for sanitary purposes. -I am really excited right now. -So the question I have is, if… -if I need… if I touch on my hand right here and I am supposed to feel the presence of something there. -In order to have an actuator in that location you have to have a ton of different valves for example if you’re using pneumatics. -or you know, linear actuators if you’re using
electro-mechanical actuators.
-Yes. -So how do you get around that problem? -We describe them as like, Haptic Pixels and each Pixel would be its own… pneumatic actuator -and they are arrayed over your hand. -Okay, I’m ready for it. Yeah.
-Okay. -How many of these exist? -Umm… -I don’t think I can share that. -Okay (both Laugh) -Cool. -So we’ll just pull this down and I’ll help you -Put these on -Ah, so you’re aligning it to my finger. Oh, wow. -Yup. -Well, what are these? -So, we… -We, uh…, ahh… -They don’t want to talk about anything! -Okay, so we developed a custom motion tracking solution that uses both optical and magnetic tracking -to get really accurate positions
of each one of your fingertips.
-Yes. -And then we do some crazy stuff in the software and using all that information we can figure out where all your fingers are in space. Yes and what I learned from my class I’m taking at school is, there is a lot of cordinate transformations happening in my hand right now. Right? Are they happening in my hand or are they happening off my hand in the processor? They’re happening in the computer I know the math and that’s awesome! Alright lets do it! After running the simulation I learned one of the most important concepts, it’s called the two point threshold. You’ve had to have done a study on this. You figured out the specific bladder density that you need in order to trick the brain knowing that it’s not gonna go photorealistic and feel like complete imersion. Is that true? Yeah. There’s actually a medically known metric called the two point threshold. Which is basically if you take two pencils and poke somebody on the skin, how close together do those get before they can’t tell that it’s two seperate points? I keep calling them bladders what’s the actual word? Ah…we call them tactors. So the spacial resolution of the tactors just has to be inside of the two point resolution of the human hand. It’s a little more complicated than that. I mean the two point threshold is one of very many aspects of touch and I am not the most qualified person to talk to you about that. and we do have a couple of folks here that can talk for a very long time about it -So I’m Destin.
-Hey I’m Adam. So how do you develop software for something that has never been done like this before? like do you create you own language for the glove? So…the way the software works is that it takes information from the physics engine and then we have to figure out how to render that And you know it’s pneumatic We have to figure out how to open and close valves in the right order and all that sort of thing. So the software does that. So you have 150-something valves or whatever it is you have to control, and you have to control it almost real time, You know your latency don’t you? -Yeah
-Yeah So, and you’re not gonna tell me that I’m assuming. -I’m not talking about that (both laugh) So you have to, It has to be quick enough where a human can think that they’re actually engaged with it. So there is a threshhold there. There is a threshold and they’ve done studies on it. They found that anything longer than a quarter of a second is too long. You lose the temporal connection between the things. -But if you go below like..you know.. half that then the brain
-You’re fine? Yeah your brain doesn’t know. They’re discussing about whether or not a I get to see a tactor I really wanna see a tactor (laughs) Who do I talk to to see a tactor? Do you think..do you think Lathe will let me see-? Ok. -We’re looking at the website for that.
-They’re looking at the website. Because thats one of the rules we were given If its on the website then we can- we can show you. Gotcha. Ok cool. Have you ever met someone that you just click with? You’re on the same wavelength instantly? This is Mark. The other engineers call him “Mr. Glove”. This is the guy that knows all the things The fun part about this conversation is if you watch Mark’s eyes, it’s clear that he just wants to “geek out”, but he can’t, because someone set up this proprietary borderline that he can’t cross. Anyway, this is one of the most enjoyable conversations I’ve had in a really long time. -So that’s Mark.
-Hi So…this is it huh? Yeah, and they’re really pretty simple. As you said they’re pnuematic, as you guessed they’re a bladder. So right here I’m looking at the end -So I’m seeing…how many is that, 12?
-Mhm What do you call- so this is a tactor. So what are these individual dots called? We actually call the individual dots tactors and the whole thing is a panel. So we group a bunch of tactors together on a panel, it makes it easier to route air to them, makes it easier to support them, and we can control each of them individually. Look I know we’re in the weeds here but Mark’s about to
describe a bunch of neat little things about your hands. So if you’ll check out
your hands and do these little activities as he’s describing them, you’ll understand things that you’ve never even thought about before. Like the squish of your skin as you touch a hard object. If you will use your hands to do
them as he’s talking you will understand the sense of touch on a much greater level. So this is the spot where the research stuff that you have to get into
kind of moves away from the traditional electrical mechanical engineering and
kind of moves into how the brain works. It moves into ergonmics, it moves into
human factors, and it moves into psychometrics Psychometrics…I don’t even know
what that means It’s how people process things.
It’s how your brain works for sensation. I don’t know if you’ve ever seen
something called the sensory homunculus it’s a picture of a person scaled relative to how many nerve endings they
have in different parts of their bodies. It’s basically a surface density of your
your tactile receptors. -Is that a good way to say it?
-It’s a very good way of saying it. These panels, we think a lot about the huge hands part of that. They’re incredibly sensitive. And so figuring out how to design a piece of
equipment that is high fidelity and capable enough of tricking those sense
organs is a lot of what our challenge is. -Okay so you’re inflating
this particular tactor here right?
-Right -It’s like a balloon right?
-Mhm If I were to touch this table with my hand right now, okay Touch the table. So it’s not a step function it’s a curve, but it’s not linear either. -Right.
-Because as you press in on
something there’s thickness to my skin and so it starts as a line and then it ramps up and then it gets harder and harder and harder right? -Absolutely.
-Is it an exponential curve? Do you know what that curve is? You don’t have to tell me what it is. You’re hitting on a great question which is what that approach and that that curve of displacement looks like. Remembering that your fingers are sense organs. One of the ways they sense is by stretching, right? They’re deformed. And you actually feel that deformation. So there’s “what is your skin doing?” Which is, it actually gets stiffer as you push on it harder and harder -And that’s part of why we have such good…
-Dexterity?
-Right. The other part of it is “what’s the
material you’re pushing into?” In some cases, if that material is soft, well that’s gonna deform as well. So you can have slightly different approach. -Yeah
-Like we’re sacks of watery meat stuff and so the relative difference between the
hardness of the object you’re touching is going to affect the attack time -I don’t know what term you
used but I would say attack time-
-Rise time. -Rise time?
-Mhm This is all making sense. There’s also little things like how well does your glove fit? That’s why when I put the glove on, it felt like there was… a physical thimble that goes on the end of my finger is because you wanted a tactor to be aligned with my finger just like that, right? -Right
-Is that because
you have to have something to react against? -Correct
-Is that why you did that?
-Yep When you’re touching something, it’s not just the feedback you feel on your skin, but also the feedback you feel in your joints, your tendons, as you push
against it. So you want to make sure to combine both of those. You’re absolutely right. And I’ve never thought about that at all. If I grab something…if I grab this. And I squeeze it, I can tell how big it is
because of where my tendons are. And I can tell how hard it is because of how hard I’m squeezing it and it’s deflecting my skin between my bones, and all that goes into…it rolls up into one transfer function… …to give you the
answer of what object is in your hand.
-Right. And that’s where things get
complicated. Your world revolves around boundary conditions doesn’t it? Like mathematical boundary conditions. Pretty much yeah. Boundary conditions and contact mechanics. So what is this right here? This tape that I’m seeing? It looks like a tape measure that’s pulling back. That’s the force feedback system. That’s what stops your hand from closing when you grab an object. -It’s a boundary condition.
-Yes. It’s a boundary condition. So instead of pushing on this side of the hand, you’re actually pulling from the other side of the hand, and that’s what stops the hand from moving. Correct And so you have to have some kind of braking mechanism back here… So, can you crush things? Like if you have something virtually can you set a resistance? Like if I were to touch
something hard versus something soft, can you vary that force feedback? (nervous sound) -Yes.
-Okay. Yes. Okay.
(laughter) These guys are like: “everything’s proprietary just wear the glove, stupid boy” I think that one was just a complicated answer. That was a complicated answer? That one is really complicated. So something that matters here is, the entire volume… -There’s a time it takes to do that compression.
-Right. So there’s a lag associated with…
It’s basically the elasticity of the volume right? What’s the word? well there’s a question of “how fast you can push air through to this guy?” and that gives you
how fast you can run that tactor. So it’s this balance between having enough
pressure to activate the system in fast time, granted you’re going through a
small tube. So the effusion rate of the gas is important.
So you’ve got all that figured out. And there’s mechanical lag associated
with the effusion rate, or the head pressure is what it’s called. Yeah. With the velocity of the fluid. The feedback diagram, or the the block diagram of how perfecting this looks it’s gonna look like a plus and a minus
between you two guys right? And include some electrical engineers in there for good measure. We don’t- nobody cares
about electrical engineers let’s be honest (laughter)
Just pretend I didn’t say that. I’m with Jeffery an electrical engineer. So Jeffery I know how the tactors work. Or at least I’ve seen them. I think I figured it out. -You felt them?
-I felt them I make sure we have good information about where the
fingertips and the other parts of the hand are, so that we can turn on tactors
at the right times to the right pressures. Dude that’s awesome. I mean those valves that you’re controlling, can you talk about that at all? I can’t talk about the type of valve we use, but the timing and the hydraulics problem that you mentioned are things we work on pretty frequently. It’s important to get them inflated the right amounts at the right times. We kind of get a pass on how accurate that is
just because of how slow sensation is. -Right
-You know people’s perception of touch especially is not nearly as fast as things like
vision or sound. -Especially thermally. I’m assuming that’s
something you guys are…
-Thats particularly slow. Ed was telling me about the thermal
component to this. Both of these objects are 73 degrees but
that metal feels way colder. Yes when you’re touching these objects you’re not just feeling the temperature of the objects, but you’re feeling the heat transfer between
them and your hand. And that’s because… The reason the metal feels colder is because
it has a higher thermal conductivity than the wood does. -So it’s like sapping more heat out of your hand.
-That’s exactly right yeah. You’re gonna model different types of objects by making more water flow or float a different temperature to pull more heat out of your fingers
that’s basically what you’re doing. Yeah that’s that’s exactly right and it’s important that we don’t just flow water at the right temperature, but we’ve flow it at the temperature
that you makes you think you’re touching the real objects with
different thermal conductivities. Adam showed me another demonstrator where a
dragon flies across the room and breathes fire on my hand which I can feel both thermally and like touch, and then it breathes ice on my hand, which felt cold. Now what’s clear to me is there they’ve got to be
using fluid at this point, because that’s the only way you can
remove that much heat. So I think there’s a tactor for touch
and a pad for thermal. but I think they’re separated
within the 2-point threshold. They wouldn’t tell me.
I wasn’t permitted to look. But one thing that was really interesting, If you switch from hot to cold, it (snaps fingers) happened really fast. So they have to dump the hot
from the lines and get cold in there fast. So there’s a lot going on here
and I want to know mechanically how they do this. This is a heat exchange problem. -This was number two?
-This is number two, you tried number three. Awesome so this is second generation. Ok so it’s just one hand stationary in location. Oh god that makes so much sense. If I were to develop this, I would have started at
moving the hand in space and then add the haptics to that but instead you
started with the haptics We knew people could do motion capture — that’s
been done before — so we wanted to start with the thing that hadn’t been done.
-Clever keep your thumb beside your hand as you go in How are you doing that? I would expect
that that’s fluid in there or that’s actually a solid that I’m touching.
You’re not allowed to tell me? I’m looking at Lathe right now just to see. (offscreen) It’s hot and cold fluid
– Gotcha it’s a chunk of snow it’s a snowball oh
it’s a dragon oh it’s gonna burn my hand I don’t think it likes being poked oh
shit that’s awesome okay that’s good so you guys like Game of Thrones so it’s a
combination of the tactors firing and temperature control. so you haven’t
implemented temperature control into– -Not in the glove, the glove does not have thermal Is that a goal? I don’t think I can talk about that
(Laughter) That’s awsome, so so what we’ve learned is it’s how far
the tactors are fired, the rate at which they they are fired, so it’s not just the rise time.
The timing is important, not just the rise time but the the impact timing is
important the heat transfer with fluid which is something obviously they don’t
want to talk about obviously they’re working on it that’s amazing that all
that stuff actually tricks your brain it does a really good job of tricking your
brain which I did not expect and I did not like VR up at this point seriously
that’s pretty cool we’re living in the future haptic gloves you’re touching
stuff in virtual reality I’m about to tell you about the sponsor which is
23andme 23 pairs of human chromosomes where the name comes from if you don’t
know about 23andme the way it works is you go to 23andme.com/Smarter. I’d appreciate it if you use that link you get this vial you spit in it and you
send it off you can do one of the two kits you can do the health and ancestry
kit or just the ancestry kit but if you have any questions about where this goes
I did a whole video at their lab the cool thing about this though you get
this report online you can learn stuff about your body about your family it’s
it’s amazing but here’s the deal the Food and Drug Administration the FDA
just recently approved 23andmes ability to release new genetic marker
information and I don’t want to go into the specifics of what that is
but it’s stuff you care about and this is new like this is future stuff okay so
go to 23andme.com/Smarter and at least read about the new information that 23andme can release to you because the FDA allowed it it’s cool stuff so I
would appreciate it if you participate in the future with me and go 23andme.com/Smarter
you’re smart people you know how this works: that helps smarter every day.
I hope you love this cuz we’re stinking in the future like right now
and it reminds me of when I was a kid and used to watch Beyond 2000 with my
dad I feel like I’m getting to make that that show now, which is cool. Anyway,
please consider going to 23andme.com/Smarter that’s support Smarter Every Day.
Please consider subscribing, if you enjoyed this video, if not, that’s no big
deal. Thanks for coming along on the ride with me. Anyway, I’m Destin, you’re getting
smarter every day. Have a good one, bye! So I feel like I sufficiently understand at
least a little bit of the system yeah thank you very much! It’s Mark, right?
Yeah.
So I have found the wizard for the glove, it’s Mark.

Only registered users can comment.

  1. They've probably already through about this, and there is probably a good reason for it, but why not use water instead of air in the taptors ? Wouldn't the compression delay be significantly reduced ?

  2. @16:56 I'm guessing they would have to use a carrier medium like carbon as itis one of the best natural radiators. maybe utilizing something like a carbon nano-tube mesh system to ramp up surface area?

  3. Man, I love your videos; I found your channel recently and have been soaking it up. I love moments when you, or an interviewee are about to mention something which I had never thought to consider for various different reasons, but makes perfect sense (e.g. being able to feel the difference between two equal temperature objects which transfer heat at different rates). I wear a stupid grin as I proceed through your videos; the genuine enjoyment in your videos as you explore these different subjects is fantastic and infectious. Keep up the good work friend, in the mean while, I'll be digging through your back-log. Cheers.

  4. Very interesting! That stuff will need regular maintenance in the long run, which is not too nice for consumer products.

  5. The hit and cold is probably a "termochip" ,if you invert the polarity of the termochip, you get hot or cold from the same side of the chip. Fluids will be slow but gas may be an other solution.

  6. 12:00 If they use air I would be highly surprised since hydraulic system would be way more controllable, faster and with higher precision.

  7. How would you build a handheld tractor beam that is controlled by the brain that you could pick a person up and moved in there's a challenge for you let's see if you can do it

  8. I wonder if they'll use several tiny Peltier modules to generate the sensation of touching something hot or cold in their future gloves.

  9. Watching your videos literally fills me with elation for this wonderful world and how it all comes together, how with science, we can break it up and put it all back together. We can put Humpty Dumpty back together again.

  10. as an amputee i think this technology will be very usefull in prosthetics because one of the main reasons for extensive therapy is because prosthetics cover sensory nerves

  11. For the hot and cold liquid differences, could it be possible that the different temperature liquid is in every other tactor? Instead of air, they use liquid to inflate each tactor and Similar to how an LED panel that has different color temperatures have a warm and cooler colored LED every other bulb?

  12. Just cool tech things. It's so deep that engineers just don't want to split any key information. But well done video. Learnt something new

  13. VR + AR + Haptic gloves is the future, the 21st century future.
    Their using peltier with fluids.

    God bless, Proverbs 31

  14. This was a really fascinating video and I wish that company all the best, cause that tech is amazing.

  15. Another problem you have to take into account is nerve damage. I lost the nerves in the bacl of my left hand to fire damage. So the skin there is just dead feeling. Imm curious how they would overcome something like this if even possible

  16. 7.18: this is exactly how I use to feel when I had an extremely high fever & delirious but I could never explain it. Feeling big & small with fear from audible & visual weirdness of it. (Sensory Homunculus) as a kid, it's terrifying.

  17. These guys act like they work for skunk works, Area 51, or (DARPA).Sorry,that's classified information.Can't talk about it. : I

  18. Couldn't the heat and cold be handled by peltiers instead of liquid? Could allow for rapid changes in temperature.

  19. Nearly everybody is wearing sort of less the same blue checkered-striped button up shirt thing at some point in the video.

  20. The feel of temperature is maybe stimulated by thermoelectric plate. it would be more rapid when it come to change in temperature then water.

  21. Why are y’all getting mad at them for not saying a lot because they are just protecting themselves from people taking their project. Cause they want the patent and many awards if this becomes successful

  22. Matrix style vs full haptic suit. What's first?

    Haptic will come first. Matrix style I believe is still a bit of a ways off unless they go for invasive physical contact and reading of fired neurons as opposed to passive MRI, cat scan, or EKG

  23. If they say something that they are not allowed to, they would feel like Mark rufalo, or Tom Holland from his company, spoiling every important detail hahaha

  24. Imagine if they where open about the project in ways…. It probably be completed by now. They need to come out with gloves. This glove is nice but Overkill. This isn't Player One yet

  25. Love how HaptX guys were squirming over sharing that they had essentially GPS mapped a hundred plus "pixel actuators" (Which is … Brilliant to say the least) using optical sensors and magnetic fields (or, some other term that's way over my head)This technology would not only revolutionize synthesized touch, but the real world applications are endless, from Macro to micro this technology could really change things. I really need to follow these guys.

  26. More my first question before putting the haptic glove on would of been…. “Can this glove snap my fingers off?”🇦🇺😂😂😂😂 “We can say that until you sign this waver”… Pun intended😂😂😂

  27. This was an amazing video! One thing this did was strengthen my belief in how amazing our bodies are! The size and scope of what they are trying to recreate our bodies do so efficiently and compact. I really enjoy your content and the fundamental base that you stand on, keep the great content coming.

  28. Honestly it was kind of easier to understand them when they talked about the glove. More simplistic than technical. Easier for people who dont know alot about the subject but who might want to get into it.

  29. 23 and me, hmm isn't it weird how we're all now willingly handing over dna samples to a company that is ran by google. just don't.

  30. This video was pretty much the most informative video I've ever seen on youtube. So much info so quick really awesome!

  31. Throughout you could see all of them in almost physical pain of holding back information that they so want to talk about and explain because you can tell this is their baby and they have worked very hard on it…

  32. Damnit this was super cool. I fully understand the IP thing. I deal with the same thing at my job, but damnit. I wanna know!!!! Also Destin you're very good at asking the correct questions.

  33. When he talks about the thermal transfer in the glove and thinking its gotta be liquid cooling and heating. To be able to do that. I honestly first thought about the electrical process auto makers use to heat and cool seats in vehicles. Take my dads Ram 1500, he got all the bells and whistles. This was the first vehicle I had ever ridden in that had not just heated seats, because thats been around for a long time. But also cooling function as well. I had gone to school to auto and Diesel training. I am Ford certified. But I never worked on fords. I worked on Cummins engines and freightliner trucks. My dads truck has cooling option for his seats and when you hit the button its pretty quick about removing the heat from my seat. You do feel the difference quickly. And its all done by how they direct electricity through these modules in the seats. If they run electricity one way through it the seats will heat up. If they run electricity the other way the seats will cool. I don't remember the exact names and details. But I at least remember this much. So that's why I first thought they might be using a similar system. But to my knowledge the modules are big enough to hold in one hand. But its the special material or alloy they use to do the heating and cooling thats important. So I guess they could come up with a way to make it fit in a glove.

  34. Gotta give it to these guys they went well out of there way to your answers and that struck me as very cool

  35. Hi, Destin. I THINK I have an answer for the thermal interactions… Peltier junctions. If current flows one direction the hot and cold sides will be one way. Reverse the current, and they flip over! Thus, you'd have one side of the junction touching your hand, and the other some kind of heat sink. This is where just a steady flow of water would be sufficient… you don't have the lag of the flow, or lags in changing the temperature, etc… the Peltier junction is against your hand, and can change very quickly, depending on the current flow and direction. By changing the "attack/decay/sustain/release" of the current, they can make the temperature change quickly, slowly, etc, and adjust your perception. NOTE: I have NO CONNECTION AT ALL with the folks at HAPTX, this is just MY THEORY about how this works… based on the limited information given here, and my own GEEK factor. 🙂 Just FYI, I absolutely LOVED this video and the one before it… these are AMAZING and are my favorite of all the "Smarter Every Day" vids. 🙂 God bless!!

  36. These guys are way to protective. Technology moves so fast this is already old news. I understand they dont want people to copy them but dont have dustin come interview you then because he will ask and you guys go erm can i answer that makes you look dumb.

Leave a Reply

Your email address will not be published. Required fields are marked *