Can Flock cameras read our lips now? Is
everything we do on the record now?
>> So,
are you ready to do a whole bunch of
crimes?
>> I've never been more ready to do crime
in my entire life.
>> We look like a 1970s
rom-com.
>> We do. We're We're on our first date.
It's going well.
>> Since I joined the I remember seeing
this, my life has really turned around.
I used to be a regular piece of garbage.
>> Let's see it with the audio.
>> That's nuts. Isn't it?
>> Do you know what ALPR stands for?
>> I don't, Brian.
>> Automatic license plate reader. No, you
know this one. You know Flock, right?
>> Ah, yes, heard of that.
>> Okay, so it what what do you know about
Flock?
>> So, they're all over the place. They can
track your license plate and take
pictures and videos of you, so they know
the flow of where people are going.
>> Yeah, from what I understand, the ALPR
is just snap a few photos, but Flock is
rolling out these new PTZ pan-tilt-zoom
cameras with 32x optical zoom, so
however heated the conversation was,
it's about to get more heated.
>> Just because they'll be able to track
people, zoom in, and get very, very
crisp video at a moment's note.
>> Yes, nationwide we got 120,000 Flock
cameras. Here in Austin, we have
240-ish. There's a website, deflock.org,
where people report where they are
because concerned citizens want to know.
I was thinking of 2001: A Space Odyssey
when the HAL 9000 is reading the lips,
and I thought to myself, can Flock
cameras read our lips now? Is everything
we do on the record now?
>> I think absolutely. Let's meet a Flock
camera, and then we'll find out.
>> Okay.
>> Take a look, right here on Lavaca
Street. We got two. You see, there's the
Flock camera. It's fixed and pointing
east, but over its shoulder, you see
that domed guy?
>> Yeah.
>> That's a pan-tilt-zoom security camera.
I I don't think that's a Flock one.
That's probably some city thing, but
that's the question is if that's
upgraded to a super high def
360° 32 x thing.
>> That has all the AI stuff that makes
Flock a unique product.
>> Right. Can it pull a HAL 9000?
>> Ah, and can it read our lips?
>> That, too.
>> Yeah.
>> I That's what I meant with the HAL 9000.
>> cuz he does that and
>> other weird stuff. Yeah.
>> Yeah. What's our test?
>> I don't have 32 x cameras, but we have
from the studio the tail two camera is a
12 x hybrid. So, we'll set that up and
we'll have some conversations.
>> Certainly not crime related.
>> No.
>> About all the crime we commit.
>> Right.
>> Wink. Something in my eye.
>> Yeah, it's the crime.
>> [music]
[music]
>> Out to sea, me hearties.
We'll sleep well tonight, boys.
All right. All right. Tension 10. There
we go.
And let's go. Let's go. Let's go. Let's
go.
Zone five, people.
I'm going to over enunciate.
>> I feel like we're [music] we're going
further out.
>> The mob. Caring is our thing.
>> Yeah. The mob, we care now.
>> Yes.
>> Thanks.
>> Hold on. Hold on.
>> We're talking about Dukakis?
>> Dukakis rode the tank.
>> Yeah.
>> Do you think Michael Rooker will come
back on the show? I hope so.
>> So,
are you ready to do a whole bunch of
crimes?
>> Yeah.
I've never been more ready to do crime
in my entire life.
I feel like crime was something I was
born to do. like breathing or other
ancillary crimes.
>> Justin, I'm beginning to regret wearing
a jacket in 100 degree weather. You have
to.
>> Yay, crime.
>> I'm the best at winning Street Fighter
tournaments.
>> Yep.
>> I make a million dollars without even
earning it.
>> Mhm.
>> I'm the best.
>> I am the best.
>> Is that good? Okay, so here we'll go up
there and next.
>> Pop. Pop.
>> Popsicle.
>> Man, that was work out.
>> You know all the rappers in the top 10?
>> Please allow him to bump the
>> Mhm.
>> Steppin' tall y'all.
>> We'll go up more. Okay, great.
>> Yeah, here we are.
>> Police have the following sketch.
>> And an amateur sketch, that's [music]
right.
>> The whistles go woo.
>> Well, it's been 48 hours.
>> Mhm.
>> It has been a journey.
>> We did close up, medium range, and long
range. And then we did outdoors and
indoors.
>> Right.
>> Now, in my imagination, I really thought
that the latest model would be good
enough for me to say, "Hey, here's a
video with no audio. What is the
transcription?" There we go. So, look at
me, I'm all smug here. This video has no
use for audio. I'm other I'm like blah
blah blah blah blah. And it says here,
"Yeah, I couldn't I couldn't identify
anything."
>> Episode over. [laughter]
Thanks for the ads, suckers.
>> Then I tried another one and it was
like, I think I got the word yeah or
why. Again, at this point, I'm totally
despairing. I'm like, no way, no way, no
way. So, eventually, I start testing,
can you just read lips cuz I'm thinking
to myself, I know that the cameras are
recording at 24 frames per second. maybe
it's just not enough frames to go along
with it. Maybe it's not detailed enough.
I start just recording myself doing a
selfie of it, and it's not able to get
anything. So, eventually I drilled down,
and it turns out that the latest model I
was using doesn't have any kind of video
module. It was just slicing everything
into a contact sheet and looking at the
pictures
>> Got you.
>> couldn't put it together. So, then I
said, "Well, who's working on this?" And
I found two projects. One was called v a
l l r, valor or whatever. I couldn't get
that one to work, but there is something
called open alter ego, and I was able to
get something slightly better, but it
was still spotty, missing context. Just
a few minutes after recording the
selfie, I couldn't tell, looking at it,
exactly what I said. Notice how hard it
is to figure out what we're saying
without any kind of context at all.
>> Yeah, I have no idea.
>> Right.
>> At all what is being said here between
the two of us.
>> Now, we are both
in frame. You can see both of our lips,
but
>> as charitable a setup as we're going to
get, right?
>> Yeah, but that I I'm not sad that I I
can't know that, because that's what lip
readers are for. Like, that is a skill
that is developed.
>> go. Same clip.
>> penguins do you think can be tiptoed in
a pyramid across
the plains?
>> Uh easily an avalanche of elevators.
>> So, in that case, we have so many things
going against us. We're intentionally
doing these very difficult, obtuse
sentences. You are in profile, but this
is the first one that gave me hope. Up
in the distant, upper reaches of the
hotel. And we have two shots. We have
our simulated flock camera, which is a
Obbspot Tail 2, and then we got the GH5.
Do you have any idea what we're saying
on this one.
No.
I do not.
I kind of remember what we were talking
about. That might have been the mobile
leprechaun.
>> That's not anything to do with our lips.
That's just me remembering what we did
when we shot that.
>> And this is the first breakthrough. Take
a look at this. It was able to get a few
key phrases. It got Irish folklore in
Mobile, Alabama. Now, at this point, it
really doesn't have a lot going on and
it's got it's got a bunch of stuff wrong
and there's giant gaps and it it has, I
don't know about this on there.
>> Yeah.
>> But but with a few pieces, I did two
transcription passes. One, say, "What
are the things you're sure of?" And then
I say, "Great. Now, go out and search
around those words and see if you can
piece together the context." This was
the first time that I did it, but
there's a few different moments that I
got 100% right.
>> Wow.
>> Isn't that incredible?
>> That's amazing.
>> And so, that was a little bit of
building out. Like, it it helped that we
were doing a bit that is famous on the
internet. So, there was more context
there. But still,
>> what a lip reader does is they ascribe
context to a thing, right? And so, in
this case here, we'll go on to the next
one. Here's the hotel mid-shot. Do your
best guess.
>> I do not remember anything from this
moment.
>> That's exactly what I had. The amnesia
is real, right?
>> To be fair, that is like the default
assumption. Bri- I talked to Brian.
>> Neither one of us made
>> either of us remember anything that's
happened.
>> That's why it's recorded, so we don't
have to remember
>> anything we recorded [laughter] it.
>> This is just what the robot put
together, so you can make your own
guess. Do you remember this
conversation?
>> Oh, yes. Yeah, oh these are all listing
crimes that we have done.
>> Exactly, right?
>> Yeah. And so now I have a vague thing
and you can tell that there's different
confidence scores and it's filling in
the gaps, but this was the first moment
that I was like, I think I think it
worked.
>> a 10-year-old to mow the lawn and I
don't think it amounted to minimum wage
and I don't think he was licensed,
insured, and bonded.
>> I practiced law in the state of
California
without licensure.
>> the licensure, yeah.
>> a lawyer. And as a lawyer, I recommend
that you don't pay taxes.
>> Good.
>> Yeah.
That's nuts.
>> Isn't it? Okay, so then a step farther
before we get to the outdoor stuff cuz
the outdoor stuff is going to be even
more difficult, right?
>> What you just did was with the current
set of models that are publicly
available, but with this GitHub repo
that is specifically designed for
>> lip reading?
>> I'm an amateur dabbler. This is an
amateur project. Everything is in
development. From what I understand, the
more data you give it, for example, if
it had a whole bunch of videos of Brian
Brushwood talking, it would only get
better and better, right? Again, this is
one afternoon that I spent doing this so
far.
>> Yeah. The outdoor one was really tough
for a few reasons. Number one, obviously
you're going to get a little bit more
atmospheric kind of stuff there, so
maybe glare off the water, anything
that's going to be different than being
indoors. Two, we were in a swan boat
where the neck of the swan kept
periodically would come into the frame
and block our lips.
>> Also, the cameras were on a boat rocking
and once the the face tracking got
blocked by the swan head, it would lose
it and start looping all around. Also,
>> we didn't know on the same way that we
knew indoors now's our time to talk. We
had a pretty clear stop and start. Yes.
We were just kind of babbling
constantly.
Uh so
Rob does have a lot of context on this
and context is king. So in this case,
see if you can make a guess. This is the
most charitable shot of the two of us.
Can you make any guess? I have no idea.
No, I have I know that we look
>> [laughter]
>> we look like a 1970s
rom-com. We we do. We're We're on our
first date. It's going well. We're very
Annie Hall.
>> [laughter]
>> I couldn't like looking at that I
couldn't piece any of that together.
We are very close range though. So yeah.
Of all that we've high-fived right in
front So that's the equivalent of us
walking up to the cops and then loudly
talking right in front of them, right?
And so this is the computer's best
guess.
Wow. Right? Yeah, when you're doing
crimes like this
you got to look good. Wow, that was
exactly what we said. You got to sweat.
So so here now now let's look at it with
the audio. Justin, I'm beginning to
regret wearing a jacket in 100° weather.
You have to. When you're doing crimes
like these, you got to look good. Yeah.
And you got to sweat. Listen, I was
otherwise the crime won't be good. Your
victims will know that you were
half-assing it if the sweat isn't there.
That's what my mentor, Doug Crime, told
me. I was thinking about downloading a
bunch of bit torrents later on today.
Oh, really? Yeah. You wouldn't steal a
car. I was going to Just kidding, we
would.
>> [laughter]
>> Is that Line of Sight Lost? Wow.
Right? That's really difficult. Like
just for the tracking alone
That's jittery. So that was another
piece of it. At first it was just like I
don't know what to do with this. and so
I said, "Well, what would be a process
to get it better?" It's like, "Well, if
you want to do it right, then you'd want
to punch in, you want to crop it, you
want to have one video for one speaker,
one for another, remove all the noise."
And I'm like, "Great, do that." So, we
we cut that all out. Now, keep in mind,
that's only like with one sample size.
Yeah. This was the the biggest eye
opener because the best shot that we
got, the one where it's nice and far,
we're out on the water, it actually
looks like we could be doing a crime
thing. So, there we are out on the
water.
>> You can see both of our lips.
But, it is far, it is pixelated.
We are jabbering, but also we're sliding
into profile.
>> Right.
>> Which makes it even more difficult.
>> Like this begins to look like a
legitimate surveillance shot.
>> I know. I'm letting you do all the work.
>> I'm definitely just drifting [laughter]
on your on your fitness app powers.
>> For a million dollars, I couldn't tell
you what was being said.
>> lot more strenuous than I was expecting.
Uh when Brian said we're going to do a
fun Modern Rogue video, I did not expect
to be uh pumping like uh my my life
depended on it on
>> Because also the the wind came in. In
fact, they warned us at the dock.
They're all like, "A lot of wind."
>> They were the nicest people, but yeah,
man.
Wind's heavy.
You're going to be working out, man.
>> Yeah, that wind's picking up, so it's
going to be a workout.
>> All right, you got it.
>> And then
>> Sweating.
>> paddle out, talk about crime.
>> Yeah, I have no clue what we were doing
out there. That's difficult for a lot of
different reasons. I would be shocked if
we got anything really
>> For this one, my production brain was
like, "Well, this is the best video."
And keep in mind, to be totally fair,
because uh robots will cheat, I made
sure that the robot never encountered
any of the actual audio, right? I That
was really important cuz you never know
what it's going to pinch from. Because
in fact, that's how I got fooled by it
must work because I said I took the clip
of the HAL 9000 lip reading scene and I
said, "Can you read these lips?" And it
said, "Sure can." And it said it all.
But of course,
>> It has transcribing it.
>> Exactly. And that is that is
one thing
>> that's actually happening now in at
least the chat GPT harness, the app that
you can download. There is a just baked
in transcription.
>> Right.
>> So now anything that comes in, it is
running transcription on it.
>> In this case, I loved everything about
that shot and I was really bummed that
it got maybe like five or six words, not
enough for me to remember the context or
anything. But I also know that that
everything is a little bit noisy and I
also know that that the robots have a
tendency to kind of give up and then
say, "I don't know. I tried."
>> Yeah.
>> But in in this case, literally all I
said was, "Could you be more effortful
and see what you could get out of it?"
And what it did was it ran both of these
like 64 times and got a array, a spread
of samples, and then from that
extracted, statistically speaking,
probably the right signal.
>> Wow.
>> And and again, this is like 30 minutes.
Take a look at at this is what it came
back with. And all of a sudden, I
remembered this exact bit. I'm going to
overenunciate and make it very easy to
pin me down on a crime. How about you,
sir?
>> I'm having a great time.
>> And then this is the part that jogged my
memory.
>> Very illegal drugs and or bombs.
Since I joined the I remember saying
this, my life has really turned around.
I used to be a regular piece of garbage.
>> So so now let's see it let's see it
let's see it with the audio.
>> and I'm afraid I'm going to make it very
easy to pin me down on a crime.
>> How about you, sir?
>> I'm having a great time, man.
>> That's good. I'm living the dream.
>> Yes, sir.
>> Yes.
About the very illegal drugs and/or
bombs.
>> Yeah.
Man, I got to say ever since I joined
the mob, my life has really turned
around. I used [laughter] to be a real
piece of garbage. Now I have purpose.
>> Yes.
So you can tell it's sort of making
guesses here and there, but
>> This is my guess as to how the tech
works. To get the lip reading, it's got
to punch in, take the mouth frame by
frame to see how it is moving and then
make the guesses as to what sounds are
normally made when these mouth movements
happen. What
I would imagine the secret sauce that
now brings this level of fidelity,
imperfect though it might be, but we can
get actual positives on it, is the thin
layer of intelligence between all these
decisions. Because otherwise, you would
have a very effective but brittle system
in previous machine learning where
everything would have to be right. In
this version, because it's more robust
and it can just make infinite guesses
and then make guesses based on all the
guesses that it made, the quantity of
intelligence actually yields results.
>> We did a couple of episodes with Perry
Carpenter that everybody should watch.
They're criminally under-viewed. He
talked about kind of the differences of
three different levels. There's
state-grade
You have kind of unlimited resources,
unlimited people, and you could achieve
certain things that would never be
profitable for a business. Then you've
got corporate level stuff where you can
hire a team of programmers, more
resources, but you're going to turn a
profit at the end. And he talks about
what we're doing now is we're
introducing folk-grade software at this
level. So in this case, [music] finally
it's peppering all the way down. So what
that implies to to
is whatever the folks at Flock and so
on, those 32X pan tilt zoom high def
super cameras, if we could do it anyone
could do it.
>> Yeah, I mean especially with the way
that their business model works where
they're more of a platform. [music] That
means that if you're a vendor of Flock
and you were taking that data then you
can do what you want with it and but I
really like that state corporate folk
framework because weirdly the folk is
kind of the same as the state grade. It
is they are both distinct from the
corporate grade because the corporate
grade has to make money. State grade can
make decisions based [music] on what the
state believes is the best and folk
grade can do whatever the hell it wants.
So you means you're going to get a
million strands of spaghetti that are up
against the wall but you were going to
do things that have no bearing in any
kind of profitable world. And so like
stuff like this is
that's real. You placed your prediction
early on say that you were expecting
success. I got to know, did you expect
this level of success?
I did not expect it to pull anything
from the swan boat stuff. I thought it
had a pretty good shot at doing the
indoor stuff because the camera was
[music] fixed, we were largely fixed.
Having it research
what sentences would go together, that
actually helped things fill out.
We will also pause for the [music]
comments. Yes, that would also create a
lot of false positives but the swan boat
stuff
moving camera, moving target, shifting
target that it obscures the mouth. Like
these are all things that
state grade stuff would have problems
with that we got positives on that
especially in the in the the mid to far
ground that's shocking to me. I would
have never guessed that. We saw where it
was challenged and succeeded,
but where was the failure point?
>> The only failure points were it wasn't
working yet. And then I just said, "How
else can we try it?" And then keep
going. Like literally that one prompt
was more effortful attempt, please. And
then it got it good enough. We see the
fringy edges here, but if you're if
you're trying to build a case against
someone on the criminal side or try to
into it consumer actions on the
corporate side.
>> I don't know if any of the things that
we said here that that is shown here
would be legally submittable.
>> No, but as Look, as people who have
watched The Wire, we understand that you
get all of these details.
>> for intelligence gathering, it it is
very very helpful. Because even if
[music] you have guesses as to what
people are talking about when they don't
think that they are being recorded, that
is very very helpful. And what you just
pointed out is something fascinating,
and it is very endemic to this
technology. The difference is not
necessarily
the tech [music] itself. It's the
compute behind it. How long are you
going and how many different ways are
you trying to solve [music] it? That is
often times the difference between
getting to your goal or not getting to
your goal. Like a lot of these frontier
math problems that are falling right
now, it is because the companies want
these records. They want to prove that
their tech is good enough, so they are
willing to put in 7 to 8 million dollars
worth of compute, you know, on the open
market. But it means like for 8 hours, 9
hours, they're running this kind of
stuff. What you did with the most
challenging stuff was exactly that. You
were just like, "Hey,
run this a million times
uh and determine statistically what your
best guess is based on your best
guesses.
>> In terms of what is possible,
I have to imagine if you got one of
these super cameras and it's there,
let's say theoretically providing
security and other things that like a
festival, every conversation, if if I'm
trying to decide where to put my
lemonade stand, I would probably throw
the compute at just give me a transcript
of every single conversation, how many
people are saying they want lemonade and
where.
>> And that's just at the commercial level.
>> It's crazy out there.
>> we're
Flock obviously is in the news now.
This tech is not
proprietary to Flock.
>> Right.
>> Flock is a way that we are talking about
this idea right now. Uh I am here to say
it has broken containment.
>> Yeah, yes.
>> Everybody has a camera. Everybody has
access to this kind of tech and it's
only going to get more available.
>> Yeah. What do you got coming up on
Politics, Politics, Politics?
>> Oh, we're talking about politics.
>> Really?
>> Yeah.
>> Well, I'll be.
>> It's the midterms, so
>> Everybody Everybody go subscribe.
>> Yeah.
>> Bye.
>> Bye.
>> This video is supported in part by
viewers like you at
patreon.com/modernrogue,
where you can get your name in the
credits like George and Jules or Justin
Owen. Or you can watch hundreds, maybe
thousands of hours of unedited footage
from the channel over the past 10 years.
>> 3 4 5 6 7 8
>> The Vikings are in the red zone against
the Packers.
>> Does this count as an unauthorized
broadcast?
>> the play-by-play of the Vikings-Packers
game.
Completion to the 6-yard line by the
Vikings.
>> That'd be great, the best takedown ever.
>> Is foul on the play. Face mask. That
would be first and goal.