Hey, it’s me, Destin. Welcome back to Smarter Every Day. I want this video to be long and I want it to get down into the weeds and just air out and let me get as technical as I want to, so I’m gonna mention this sponsor right now upfront, which is different, I know. But I want to say thank you to 23andMe for sponsoring this episode of Smarter Every Day. I’ve been to their lab too, its a really cool place. So, I’ll tell you more about their at-home genetic test service later in the episode but, for now let’s talk about Haptic Gloves. In the last episode of Smarter Everyday you got to see me interact with the Haptic Glove and it changed my mind about VR (Virtual Reality) and here’s the deal, when you’re a start up company, it’s a double-edged sword, right? You want people to know that you’re doing these great wonderful things so you can get investors and stuff like that. But conversely you don’t want them to know the “secret sauce” because you gotta maintain your competitive edge, right? So, the fun thing about this is when I got down to talking to the engineers at like the fundamental Haptic Pixel level on how this thing works, you can see their eyes dartin’ around the room, like they freak out. The reason they freak out is, because they haven’t been permitted to talk about how this glove works in forever apparently and then this dude walks in with a camera and sticks a camera in his face and asks him very technical questions, its really fun to see their reaction but, I was able to sign some paper work and we were able to work out a deal so that I can take you right up to the edge of proprietary information, but not cross that line so that they can still maintain their edge. So after I met the founder Jake and the co-founder Bob, they turned me over to the head of engineering ops, Lathe, to asks questions. And in the last video we went through what the experience was like, this time I just wanted to focus on how everything works I decided to just go for it and ask Keenan what kind of actuators they used to simulate touch. -So, I am going to wear a glove under the glove.
-Correct, its just for sanitary purposes. -I am really excited right now. -So the question I have is, if… -if I need… if I touch on my hand right here and I am supposed to feel the presence of something there. -In order to have an actuator in that location you have to have a ton of different valves for example if you’re using pneumatics. -or you know, linear actuators if you’re using
-Yes. -So how do you get around that problem? -We describe them as like, Haptic Pixels and each Pixel would be its own… pneumatic actuator -and they are arrayed over your hand. -Okay, I’m ready for it. Yeah.
-Okay. -How many of these exist? -Umm… -I don’t think I can share that. -Okay (both Laugh) -Cool. -So we’ll just pull this down and I’ll help you -Put these on -Ah, so you’re aligning it to my finger. Oh, wow. -Yup. -Well, what are these? -So, we… -We, uh…, ahh… -They don’t want to talk about anything! -Okay, so we developed a custom motion tracking solution that uses both optical and magnetic tracking -to get really accurate positions
of each one of your fingertips.
-Yes. -And then we do some crazy stuff in the software and using all that information we can figure out where all your fingers are in space. Yes and what I learned from my class I’m taking at school is, there is a lot of cordinate transformations happening in my hand right now. Right? Are they happening in my hand or are they happening off my hand in the processor? They’re happening in the computer I know the math and that’s awesome! Alright lets do it! After running the simulation I learned one of the most important concepts, it’s called the two point threshold. You’ve had to have done a study on this. You figured out the specific bladder density that you need in order to trick the brain knowing that it’s not gonna go photorealistic and feel like complete imersion. Is that true? Yeah. There’s actually a medically known metric called the two point threshold. Which is basically if you take two pencils and poke somebody on the skin, how close together do those get before they can’t tell that it’s two seperate points? I keep calling them bladders what’s the actual word? Ah…we call them tactors. So the spacial resolution of the tactors just has to be inside of the two point resolution of the human hand. It’s a little more complicated than that. I mean the two point threshold is one of very many aspects of touch and I am not the most qualified person to talk to you about that. and we do have a couple of folks here that can talk for a very long time about it -So I’m Destin.
-Hey I’m Adam. So how do you develop software for something that has never been done like this before? like do you create you own language for the glove? So…the way the software works is that it takes information from the physics engine and then we have to figure out how to render that And you know it’s pneumatic We have to figure out how to open and close valves in the right order and all that sort of thing. So the software does that. So you have 150-something valves or whatever it is you have to control, and you have to control it almost real time, You know your latency don’t you? -Yeah
-Yeah So, and you’re not gonna tell me that I’m assuming. -I’m not talking about that (both laugh) So you have to, It has to be quick enough where a human can think that they’re actually engaged with it. So there is a threshhold there. There is a threshold and they’ve done studies on it. They found that anything longer than a quarter of a second is too long. You lose the temporal connection between the things. -But if you go below like..you know.. half that then the brain
-You’re fine? Yeah your brain doesn’t know. They’re discussing about whether or not a I get to see a tactor I really wanna see a tactor (laughs) Who do I talk to to see a tactor? Do you think..do you think Lathe will let me see-? Ok. -We’re looking at the website for that.
-They’re looking at the website. Because thats one of the rules we were given If its on the website then we can- we can show you. Gotcha. Ok cool. Have you ever met someone that you just click with? You’re on the same wavelength instantly? This is Mark. The other engineers call him “Mr. Glove”. This is the guy that knows all the things The fun part about this conversation is if you watch Mark’s eyes, it’s clear that he just wants to “geek out”, but he can’t, because someone set up this proprietary borderline that he can’t cross. Anyway, this is one of the most enjoyable conversations I’ve had in a really long time. -So that’s Mark.
-Hi So…this is it huh? Yeah, and they’re really pretty simple. As you said they’re pnuematic, as you guessed they’re a bladder. So right here I’m looking at the end -So I’m seeing…how many is that, 12?
-Mhm What do you call- so this is a tactor. So what are these individual dots called? We actually call the individual dots tactors and the whole thing is a panel. So we group a bunch of tactors together on a panel, it makes it easier to route air to them, makes it easier to support them, and we can control each of them individually. Look I know we’re in the weeds here but Mark’s about to
describe a bunch of neat little things about your hands. So if you’ll check out
your hands and do these little activities as he’s describing them, you’ll understand things that you’ve never even thought about before. Like the squish of your skin as you touch a hard object. If you will use your hands to do
them as he’s talking you will understand the sense of touch on a much greater level. So this is the spot where the research stuff that you have to get into
kind of moves away from the traditional electrical mechanical engineering and
kind of moves into how the brain works. It moves into ergonmics, it moves into
human factors, and it moves into psychometrics Psychometrics…I don’t even know
what that means It’s how people process things.
It’s how your brain works for sensation. I don’t know if you’ve ever seen
something called the sensory homunculus it’s a picture of a person scaled relative to how many nerve endings they
have in different parts of their bodies. It’s basically a surface density of your
your tactile receptors. -Is that a good way to say it?
-It’s a very good way of saying it. These panels, we think a lot about the huge hands part of that. They’re incredibly sensitive. And so figuring out how to design a piece of
equipment that is high fidelity and capable enough of tricking those sense
organs is a lot of what our challenge is. -Okay so you’re inflating
this particular tactor here right?
-Right -It’s like a balloon right?
-Mhm If I were to touch this table with my hand right now, okay Touch the table. So it’s not a step function it’s a curve, but it’s not linear either. -Right.
-Because as you press in on
something there’s thickness to my skin and so it starts as a line and then it ramps up and then it gets harder and harder and harder right? -Absolutely.
-Is it an exponential curve? Do you know what that curve is? You don’t have to tell me what it is. You’re hitting on a great question which is what that approach and that that curve of displacement looks like. Remembering that your fingers are sense organs. One of the ways they sense is by stretching, right? They’re deformed. And you actually feel that deformation. So there’s “what is your skin doing?” Which is, it actually gets stiffer as you push on it harder and harder -And that’s part of why we have such good…
-Right. The other part of it is “what’s the
material you’re pushing into?” In some cases, if that material is soft, well that’s gonna deform as well. So you can have slightly different approach. -Yeah
-Like we’re sacks of watery meat stuff and so the relative difference between the
hardness of the object you’re touching is going to affect the attack time -I don’t know what term you
used but I would say attack time-
-Rise time. -Rise time?
-Mhm This is all making sense. There’s also little things like how well does your glove fit? That’s why when I put the glove on, it felt like there was… a physical thimble that goes on the end of my finger is because you wanted a tactor to be aligned with my finger just like that, right? -Right
-Is that because
you have to have something to react against? -Correct
-Is that why you did that?
-Yep When you’re touching something, it’s not just the feedback you feel on your skin, but also the feedback you feel in your joints, your tendons, as you push
against it. So you want to make sure to combine both of those. You’re absolutely right. And I’ve never thought about that at all. If I grab something…if I grab this. And I squeeze it, I can tell how big it is
because of where my tendons are. And I can tell how hard it is because of how hard I’m squeezing it and it’s deflecting my skin between my bones, and all that goes into…it rolls up into one transfer function… …to give you the
answer of what object is in your hand.
-Right. And that’s where things get
complicated. Your world revolves around boundary conditions doesn’t it? Like mathematical boundary conditions. Pretty much yeah. Boundary conditions and contact mechanics. So what is this right here? This tape that I’m seeing? It looks like a tape measure that’s pulling back. That’s the force feedback system. That’s what stops your hand from closing when you grab an object. -It’s a boundary condition.
-Yes. It’s a boundary condition. So instead of pushing on this side of the hand, you’re actually pulling from the other side of the hand, and that’s what stops the hand from moving. Correct And so you have to have some kind of braking mechanism back here… So, can you crush things? Like if you have something virtually can you set a resistance? Like if I were to touch
something hard versus something soft, can you vary that force feedback? (nervous sound) -Yes.
-Okay. Yes. Okay.
(laughter) These guys are like: “everything’s proprietary just wear the glove, stupid boy” I think that one was just a complicated answer. That was a complicated answer? That one is really complicated. So something that matters here is, the entire volume… -There’s a time it takes to do that compression.
-Right. So there’s a lag associated with…
It’s basically the elasticity of the volume right? What’s the word? well there’s a question of “how fast you can push air through to this guy?” and that gives you
how fast you can run that tactor. So it’s this balance between having enough
pressure to activate the system in fast time, granted you’re going through a
small tube. So the effusion rate of the gas is important.
So you’ve got all that figured out. And there’s mechanical lag associated
with the effusion rate, or the head pressure is what it’s called. Yeah. With the velocity of the fluid. The feedback diagram, or the the block diagram of how perfecting this looks it’s gonna look like a plus and a minus
between you two guys right? And include some electrical engineers in there for good measure. We don’t- nobody cares
about electrical engineers let’s be honest (laughter)
Just pretend I didn’t say that. I’m with Jeffery an electrical engineer. So Jeffery I know how the tactors work. Or at least I’ve seen them. I think I figured it out. -You felt them?
-I felt them I make sure we have good information about where the
fingertips and the other parts of the hand are, so that we can turn on tactors
at the right times to the right pressures. Dude that’s awesome. I mean those valves that you’re controlling, can you talk about that at all? I can’t talk about the type of valve we use, but the timing and the hydraulics problem that you mentioned are things we work on pretty frequently. It’s important to get them inflated the right amounts at the right times. We kind of get a pass on how accurate that is
just because of how slow sensation is. -Right
-You know people’s perception of touch especially is not nearly as fast as things like
vision or sound. -Especially thermally. I’m assuming that’s
something you guys are…
-Thats particularly slow. Ed was telling me about the thermal
component to this. Both of these objects are 73 degrees but
that metal feels way colder. Yes when you’re touching these objects you’re not just feeling the temperature of the objects, but you’re feeling the heat transfer between
them and your hand. And that’s because… The reason the metal feels colder is because
it has a higher thermal conductivity than the wood does. -So it’s like sapping more heat out of your hand.
-That’s exactly right yeah. You’re gonna model different types of objects by making more water flow or float a different temperature to pull more heat out of your fingers
that’s basically what you’re doing. Yeah that’s that’s exactly right and it’s important that we don’t just flow water at the right temperature, but we’ve flow it at the temperature
that you makes you think you’re touching the real objects with
different thermal conductivities. Adam showed me another demonstrator where a
dragon flies across the room and breathes fire on my hand which I can feel both thermally and like touch, and then it breathes ice on my hand, which felt cold. Now what’s clear to me is there they’ve got to be
using fluid at this point, because that’s the only way you can
remove that much heat. So I think there’s a tactor for touch
and a pad for thermal. but I think they’re separated
within the 2-point threshold. They wouldn’t tell me.
I wasn’t permitted to look. But one thing that was really interesting, If you switch from hot to cold, it (snaps fingers) happened really fast. So they have to dump the hot
from the lines and get cold in there fast. So there’s a lot going on here
and I want to know mechanically how they do this. This is a heat exchange problem. -This was number two?
-This is number two, you tried number three. Awesome so this is second generation. Ok so it’s just one hand stationary in location. Oh god that makes so much sense. If I were to develop this, I would have started at
moving the hand in space and then add the haptics to that but instead you
started with the haptics We knew people could do motion capture — that’s
been done before — so we wanted to start with the thing that hadn’t been done.
-Clever keep your thumb beside your hand as you go in How are you doing that? I would expect
that that’s fluid in there or that’s actually a solid that I’m touching.
You’re not allowed to tell me? I’m looking at Lathe right now just to see. (offscreen) It’s hot and cold fluid
– Gotcha it’s a chunk of snow it’s a snowball oh
it’s a dragon oh it’s gonna burn my hand I don’t think it likes being poked oh
shit that’s awesome okay that’s good so you guys like Game of Thrones so it’s a
combination of the tactors firing and temperature control. so you haven’t
implemented temperature control into– -Not in the glove, the glove does not have thermal Is that a goal? I don’t think I can talk about that
(Laughter) That’s awsome, so so what we’ve learned is it’s how far
the tactors are fired, the rate at which they they are fired, so it’s not just the rise time.
The timing is important, not just the rise time but the the impact timing is
important the heat transfer with fluid which is something obviously they don’t
want to talk about obviously they’re working on it that’s amazing that all
that stuff actually tricks your brain it does a really good job of tricking your
brain which I did not expect and I did not like VR up at this point seriously
that’s pretty cool we’re living in the future haptic gloves you’re touching
stuff in virtual reality I’m about to tell you about the sponsor which is
23andme 23 pairs of human chromosomes where the name comes from if you don’t
know about 23andme the way it works is you go to 23andme.com/Smarter. I’d appreciate it if you use that link you get this vial you spit in it and you
send it off you can do one of the two kits you can do the health and ancestry
kit or just the ancestry kit but if you have any questions about where this goes
I did a whole video at their lab the cool thing about this though you get
this report online you can learn stuff about your body about your family it’s
it’s amazing but here’s the deal the Food and Drug Administration the FDA
just recently approved 23andmes ability to release new genetic marker
information and I don’t want to go into the specifics of what that is
but it’s stuff you care about and this is new like this is future stuff okay so
go to 23andme.com/Smarter and at least read about the new information that 23andme can release to you because the FDA allowed it it’s cool stuff so I
would appreciate it if you participate in the future with me and go 23andme.com/Smarter
you’re smart people you know how this works: that helps smarter every day.
I hope you love this cuz we’re stinking in the future like right now
and it reminds me of when I was a kid and used to watch Beyond 2000 with my
dad I feel like I’m getting to make that that show now, which is cool. Anyway,
please consider going to 23andme.com/Smarter that’s support Smarter Every Day.
Please consider subscribing, if you enjoyed this video, if not, that’s no big
deal. Thanks for coming along on the ride with me. Anyway, I’m Destin, you’re getting
smarter every day. Have a good one, bye! So I feel like I sufficiently understand at
least a little bit of the system yeah thank you very much! It’s Mark, right?
So I have found the wizard for the glove, it’s Mark.