USENIX Enigma 2016 – Building a Competitive Hacking Team

Published by Jan Heaney on

USENIX Enigma 2016 – Building a Competitive Hacking Team


[ Applause ] -Thank you. So, when I was a teenager, I wanted to be a hacker. Yeah, I know. But I really loved computers. I taught myself
a few programming languages. I started
dual-booting Gen 2, and I figured,
“Well, being a hacker has got to be
the next step, right?” But I didn’t know
how to get started. I didn’t know any hackers
who could teach me or have any resources
to teach me how to hack, so I did the only thing I could. I went online,
and I tried to — tried to look it up. I found lots of really, really
top-notch resources. They had tutorials. They’d teach me how to hack. I downloaded all the tools
that all the pro hackers use. And I spent, you know,
several months at this. And after a little while,
I realized, well, hopefully what all of you already know, which is, this does not work. And so, I was sitting there. I was like,
“Well, what now? I don’t have anyone else
to turn to. There’s no other way for
me to learn this information,” and I-I gave up,
and that was it. The end. So, several years later, I started studying
computer science at Carnegie Mellon University. In my freshman year,
a friend of mine came to me, and he said, “Hey,
we just started this group on campus called
The Plaid Parliament of Pwning. And this weekend, we’re playing
in a capture-the-flag contest. Do you want to join us?” Now, being
a computer-science student without many friends, of course
I was like, “Oh, yeah, sure. I’ll hang out with you guys. But I have no idea
what capture the flag is.” So, it’s actually
really simple. If you’re familiar
with the playground game, it’s actually the same. The only difference is
now the flags are digital. Well, now
the flags are digital, and you play it on computers. And there’s no running around, and there’s no fields
or anything. It’s all on the Internet. Okay, so, it’s not —
it’s not exactly the same, but it’s still really simple. So, what you have is,
you have a set of flags, and these flags are
on a server somewhere and associated with challenges. A challenge
might be a web server that someone wrote in “C,”
and it has vulnerabilities. It might be a binary
that’s been compiled, and you have
to reverse-engineer it and try to find a serial number, who might be file
that’s been encrypted, and you have to decrypt it. It might be a web application that has SQL injection
or something like that. So, your job —
Well, you go in, you pick a challenge
that you want to work on, and you try to recover the flag. So, maybe you find a buffer
overflow in this web server, and you exploit it,
and you get a flag back. And this flag is worth
some points in this contest. And the flag —
Well, maybe it’s just a file in the file system
or something like that, and you read it, and you just
get this little string. So, I’d signed up for this competition
with my friends. And we were really excited.
We read that the — The top teams
from this competition would actually be flown
to New York City, and we were,
“That sounds pretty cool.” So, came time to play, and we all met on campus in this small office
off in the corner. And we were there 24 hours, working on these competitions,
so all day and all night, we were working
on these things. So, we play this,
and we got first place. We were super thrilled. Now, I know you’re saying,
“Well, Tyler, that’s because of
all those tutorials you read in high school, right?” Eh, maybe not so much. I was definitely the worst
person on the team by far. So, at this point, I was
a little bit scared, right? I was going in saying,
“Well, crap. We’re going to this final round
in New York City, and here I am, a total newb. I don’t want to embarrass
myself or my team, so what am I gonna do?” So, if you’re familiar
with all great stories, the next part
is the training montage. So, I trained for the next month
or so, you know, playing old capture-the-flag
contest challenges, talking to people on my team so I wouldn’t make
a total fool of myself. Now, this seems silly, but this is actually
a really important part of capture-the-flag contests. So, they provide motivation. So, just like Rocky
wasn’t motivated to stop being a lazy bum until he had a fight
with Apollo, hackers can kind of get off
their butts and start working when they’re motivated
by a competition. Now, you may be saying,
“Well, wait a minute. I mean, it feels
like I’m already doing computer security,
and it’s already a competition.” We’ve got
United States versus China or Tavis Ormandy
versus all of antivirus, and this feels like
a competition going on, right? Well, but it’s not
the same, right? These are never-ending battles, and never-ending battles
don’t have a winner and loser, which means there’s
no real incentive for you to try
to go out and win, so it’s not the same level
of motivation. Capture-the-flag contests also
have instant feedback. When you capture a flag, you know right away
that you’ve got something, and this is very different
from something like maybe writing a report, which you would do
in some other competitions because you might write
something, and it’s wrong, and you don’t find out
for a week. In the capture-the-flag
contests, you keep trying
until you get what you want. And this also means that as soon
as you get what you want, you get a little bit
of an adrenaline boost, and this keeps you motivated. There’s also a ranking system. When you’re playing
in these events, you don’t just know
what your own score is. You see a list
of all the teams in order. This means you know
who your worst enemy is. It’s the person right above you. So, you might say, “Well, if I solve
this one more challenge, I’ll move up one little place,”
and that motivates you to just push
a little bit ahead, and you keep doing this
until you’re on top. But it’s not just about teams. It’s also
about personal motivation. So, all these challenges
have point values. So, in my first competition,
I didn’t do very well. Maybe I got 200 points
or something, but I could say to myself,
“Well, next time I’m gonna go in,
and I’ll get 300 points.” And so, in this way, you can set
these very reasonable goals and make personal progress
that you can actually track. So, been training for a month. I went into this competition,
final round in New York with — with the rest of my team. And we won.
We got first place. Again, super exciting,
and you’re probably saying, “Well, you trained
for a month, right? I mean, that’s —
That’s all it takes?” No. Again, no, that’s not
all it takes, sadly. I was still not very good. I’d improved, certainly. But I still wasn’t
where I wanted to be. So, I kept playing.
I kept playing with this team, and we kept playing
throughout the year. So, we played
in several competitions. We didn’t win first place
in all of them, but we did very respectably,
and we were pretty proud. So, at the end
of the school year in May, it came time for DEF CON CTF. So, a lot of you
are probably familiar with DEF CON as the conference, but it’s not just a conference. There’s also a capture-the-flag
competition associated with it, and this has actually
been running every year for the past 20 years. So, in some sense,
this is like the World Series of these capture-the-flag
competitions, so it’s a really big deal to us. So, we’d been doing pretty well,
and we said, “Well, the next step
is try to go in DEF CON.” So we signed up. We went on the online qualifier. And we didn’t make it.
We were the first alternates. We were just —
just barely there, but we didn’t make it. And now, we’re
a competitive group of people, so I know if you have
good foreshadowing abilities, you’d say, “Well,
I know what comes next — another training montage.” And that’s exactly right. So, what are we doing,
though, right? I mean, we can’t
the Rocky strategy of punching frozen meat. That doesn’t really help
with computer security. So, how do we practice? Well, our team practices by participating
in these competitions. Now, that sounds
a little circular and silly, but it’s actually
the best way to do it. So, it’s not just these couple
competitions that I mentioned. There are several of
these competitions every month, and so you can use these
to keep boasting your skills. Every time
you play one of these, you learn a little bit more,
so the more you participate in, the better you do. So, you have to actually
actively participate
in these competitions. A lot of people
will come to me and say, “Well, Tyler, you know,
I signed up for the competitions and clicked around,
and I looked at the problems, and I did this 10 times,
and I’m not any better.” Well, yeah,
that’s not surprising. You have to actually play
the whole competition, and that means
playing from start to end and sometimes even
past the end. A good example of this —
a couple years ago, there was a Russian competition. One of the challenges involved a discrete logarithm
calculation. We saw some program,
wrote something really quickly, and it took about three hours
for it to run. And at the end of the three
hours, it got the flag. The issue is that
was about 10 minutes after the competition ended,
so it didn’t get us any points. Now, it’s easy at this point
to say, “Well, eh, not a big deal.” I mean,
we got the flag, we solved the problem,
wipe our hands clean of it, and move on with our lives, but that’s not how you improve. So, instead, I went back,
I looked at this problem, and I found there was actually
some structure that we hadn’t really looked at that allowed us
to parallelize it. So, I wrote a solver on GPUs,
ended up solving the problem that took three hours
in about 10 minutes. This was a real solution that
would have got us the points, and this is how I learned. The other aspect
of these competitions is you can read write-ups. So, a lot of the teams
that solve these problems will document how they —
how they got the flags. So, you can read through these if you didn’t solve a problem,
and you can say, “Oh, that’s — that’s
what I was missing. That’s how I was
supposed to do it.” Similarly, even if you do solve
a challenge, it turns out there’s lots
of cute tricks and other methods
of solving these things that not everyone
does the same way. So you can read them,
and you can find out new ways and new tricks
and things like that. So, it’s really nice.
You just go online. You can Google CTF write-up,
tens of thousands of results, and read through them. And lastly,
although we try to play in as many competitions
as we can, well, you aren’t gonna play
in every single one, so you can still look at
the problems from the contests you haven’t played. So, we’d been training. And this paid off
really well. We went back to New York. That was great.
Got first place. That was no problem.
We already did that. But then we went
to South Korea. We went to South Korea
and played in a CTF there. We got first place, and we got $20,000,
which was pretty nice. And then we went to Russia. We went to Moscow and played
in a competition there. Again, first place. So, we were doing
pretty well. Our team at this point
had really grown and matured. And it’s important to realize that these CTF teams
are really teams. It’s not a collection
of individuals. And so, this seems like kind
of mushy, feely type of stuff, but it’s actually — There’s
really practical reasons. The first is motivation. I said these competitions
can easily be 24 hours, and so, when you think about it,
sitting in front of a computer for 24 hours isn’t always easy. You might start
checking Facebook or Twitter, or maybe
you want to go to sleep. And that’s understandable,
but if you have someone next to you working, it’s a lot easier
to stay awake and stay motivated and stay focused
on these problems. Also, having a team
allows a diverse set of skills. I mentioned
that not all the challenges are solved in the same way, so having different viewpoints
from different people on your team can really
help give alternate perspectives on these problems. And in a very similar vein,
this allows for specialization. So, for example, I’m
very interested in cryptography. Someone else on my team might be very interested
in Windows exploitation. So I can let them deal with understanding
the latest EMET bypass, and, likewise, they don’t have
to understand the intricacies of a sub-group confinement
attack for Diffie-Hellman. And so we each got to focus
on the things that we’re really interested in, and at the same time,
our team will improve greatly. So, you might be saying,
“Well, so, does this mean
if I want to pick a team, I should have
them list their skills and throw out the ones
that we already have?” That’s not —
That’s not how we do it. It turns that what we found is
that interest and dedication are the most important traits. And this seems
a little silly, but, again, it’s for
very practical reasons. The technical skills
are already there when we’re picking players, and so what we really
are looking for is people that are gonna show up
and continue to show up and have their skills
improve over time. And so,
what we do is, we invite everyone
to come join our team, and the people
that aren’t dedicated, aren’t interested, well,
they just stop showing up. So it’s
a very self-selecting process, and this ends up with us getting
the most dedicated, interested people, and after a couple years,
they turn out to be the best. So, we’d been doing this.
Our team had grown. Now it was time
to qualify for DEF CON. So, this year, things
were a little bit different. They actually
allowed us to qualify through winning certain
well-known contests. So, one of them was ICTF. We won that — easy. So we had already qualified. But then we went
and we also won Codegate, which is another CTF
that was a qualification. And then just to put the nail
in the coffin, we went back, we played
DEF CON qualifier again, and we did well enough
to qualify that way, too. So we made sure that
we really qualified this time. So, but we were getting ready
for DEF CON finals, and it turns out these are —
DEF CON finals are a lot different
from the DEF CON qualifier. So, in CTF nomenclature, there’s kind
of two distinct versions of CTF. One is called Jeopardy. The other’s
called Attack-Defense. Jeopardy is
the most common event, and it’s also usually associated
with online competitions, which is the majority
of what we do. In contrast, Attack-Defense
is more traditional. You actually steal flags
from other teams just like you do
in the playground event. Attack-Defense is also in-person or usually associated
with conferences and things like DEF CON. So, in Jeopardy,
the organizers host all of the challenges. So they create these challenges, and they run themselves
on their own servers, whereas Attack-Defense,
the teams host the challenges. So, the organizers write them, but as a team,
it’s your responsibility to make sure
the challenge is running and make sure
that no one’s hacking into it. So, it’s a lot of different
categories in Jeopardy. In Attack-Defense,
it’s basically just binary exploitation. And the names of these —
well, Jeopardy, a lot of times the scoreboard
or when you pick challenges, it looks a lot
like a Jeopardy board. You just pick a category
that you’re interested in, and then you pick a point value
just like Jeopardy. Attack-Defense — well, things are a little bit
more complicated. So, you don’t just pick that. You have to pick teams,
and then you have to decide which teams to attack
and when and so forth, so it’s a little bit different. So, it’s time for DEF CON
finals. We had qualified,
and we went in. And, well, we did okay,
not great. After a year
of winning competitions, getting seventh place
out of 12 teams didn’t feel so great. We weren’t super thrilled, but we at least
made it to DEF CON, and that was nice. Now, at this point,
our team was about two years old. So, what’s kind of happening? Well, the original founder
of the team, Brian Pak, graduated,
which meant I was in charge. I was no longer the worst person
on the team. I was actually pretty good
at this point, which was nice, and so I took over the team. And the next several years, we continued to grow
and improve. So, the next few years, we not only went back to Russia and South Korea
for these contests. We went to Japan
and Taiwan, as well. And we actually had
to incorporate our team. We formed The Plaid Parliament
of Pwning Incorporated ’cause we were winning so much money
from these contests that we couldn’t handle it
any other way. So, we were pretty happy. I know you’re saying,
“Well, fine, but what about the World Series that you talked about,
DEF CON CTF?” So, the next year, we went back, qualified,
no problem. This was a graph
of the scores over time about an hour
before the competition ended. And this was it
when the competition did end. So, if you notice,
there’s another team that sniped us
at the last minute. So, second place.
It’s okay. You know, second place is pretty good,
I guess, but we weren’t super thrilled, so we went back. Next year, first place —
much better. Not just by a small margin. Actually,
by a large, large gap. Next year, we went back,
and we did the same thing. So, today, well, I guess I’ve actually
become a hacker. But a reasonable question
to ask is, “I see you’ve been
playing these games for the past 6 1/2 years. Does that mean anything
about real security, or are you just playing
these stupid little games?” So, obviously, I think I did
learn a lot about real security. In my view, CTF problems are almost like
distilled real-world problems. So, this means two things. The first is
that they’re more concentrated in the sense
that for the amount of time you put into them,
you get a lot more out of them. But they also are made
of the same stuff. When you do a CTF problem,
you’re using the same skills. You’re thinking
about the same problems that you think about
when you’re solving real-world security challenges. Now, I know you’re gonna say,
“Well, fine, you can say that, but I don’t believe you.” Understandable, so let’s go
over a few examples. So, first one — this was actually
just a couple weeks ago. There was a news report
about a vulnerability, an FFmpeg that let
people read remote files. Someone
on our team saw this, and they said, “Wait a second.” I did two months ago in
a capture-the-flag competition. Before this vulnerability
was reported, they actually had
to independently discover it over the course
of this competition. Fine.
Not exceptional, though. The year before that, there was
a competition called Codegate. One of the intended solutions
for solving a challenge was to actually use
a zero-day and bzip to get memory corruption. Okay, the year
before that, preparing for DEF CON, what a lot of teams
will actually do is find zero-days
and utilities like Wireshark that a lot of teams use. So, they’re finding
these and mentioning that they found
these to attack other teams. Year before that,
I actually wrote a challenge that used an undocumented bug in the Go compiler that allowed for running
untyped safe code in what’s supposed
to be a fairly safe language. Year before that,
there was a web problem. The intended solution
was XPath injection, and the Dutch team
that solved it or one of the Dutch —
one of the teams that solved it, they actually found
a remote code execution bug in the PFP module that affected millions of users
and websites like Yahoo. The year before that,
at DEF CON, there was a team that found
a local privilege escalation bug in FreeBSD. So, yes, there’s probably
some real-world relationships between CTF problems
and security challenges that we actually do. But if that’s the case, then
why are we doing games, right? If I could’ve been doing
real-world work for the past six years,
why wasn’t I making any money and playing these games instead? Well, it was a couple reasons. So, the first is,
CTF problems have graduated difficulty. You wouldn’t go up
to a teenager who said, “I want to learn to be a hacker” and tell them, “Well,
here’s a copy of IDA. Go download Chrome
and have some fun and find some 0-days.” They’re not gonna get anywhere. And similarly, these problems
don’t just start off easy. They actually get very hard. A lot
of the more difficult problems are a lot more complicated and use some things that most people would never see
in their day-to-day work. It’s also, playing
in these competitions is a relatively easy commitment. I mean, you do have to dedicate
24 or 48 hours, but it’s a lot easier
than telling someone, “Well, go quit your job and get
a new job in computer security if that’s what you want
to be doing.” That’s not so easy. There’s also
no barrier of entry. You don’t have to pay anything
to join these competitions. You just sign up online,
and you can start playing. No one makes you take a test
or pass an entrance exam. Again, if you wanted
to do computer security for a living,
it’s not the same thing. You can’t just walk in and say,
“I want to do security” and someone will give you
some interesting work. And lastly,
as we talked about, CTFs help motivate people. So, kind of in short, CTFs work. They don’t just teach you
the technical skills. They also teach you
the creative problem solving that only comes from
doing practical, hands-on work. And so, for training
and computer security, I really don’t think there’s
anything that can be CTFs. So, if you’re like me,
you’re saying, “Damn, I wish I knew about this
when I was a teenager.” Yeah, well,
I thought that, and our team thought that, too. There’s not much we can do,
but maybe there’s something. So, in 2013, our team actually created a competition
specifically for high schoolers to introduce them to security. So, this competition
was fairly successful. We got over 10,000 U.S.
high school students to play. We got several schools
and teachers involved, as well, and it’s actually
a huge success. So, you might be thinking, “Well, I’m not
a teenager anymore. This doesn’t affect me.”
Well, that’s not true. It’s not too late
for you to do this. It’s not just high schoolers or college students
that play in these competitions. It’s professors
and industry professionals that play in these things, too,
and continue to learn things. So, if you’ve never
played in a CTF, I encourage you to go out
and do it. And I think your teenage self
will thank you. And that’s it. Are there any questions? [ Applause ]


2 Comments

Agnus Xendis · April 2, 2016 at 10:23 pm

Good talk.
P.S. You don't need IDA to find bugs in chrome. Chromium is open source πŸ˜‰

Raj Prajapati · June 17, 2019 at 7:03 pm

Awes0me Explanation of CTF.
I Appreciate that πŸ™πŸ™πŸ‘Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *