Hacker Challenges -- Boon or Bane?

In the past year, several businesses have made resources publicly available on the Internet and challenged all comers to find bugs in them or break into them. Incentives offered to those who reported valid break-ins or bugs have ranged from T-shirts to cold cash. Recently, Gene Spafford of Purdue University decried this growing practice in a message circulated widely on the Internet. Cipher has obtained responses from some of the organizations who have sponsored challenges of one sort or another, and circulating them along with that note. We thank Prof. Spafford and the organizations who responded to our request for comments.

A Few Comments on "Hacker Challenges"

by Eugene H. Spafford
COAST Laboratory Director
Purdue University

I note with dismay the increasing number of "hacker challenges" used in marketing security products. I think these are actually harmful to the profession and practice of security, rather than helpful. I believe the harm comes in two ways: (1) the challenges don't serve as any real test of the products, and it denigrates security professionals by suggesting that they should accept them as proof of security; and (2) it helps reinforce the image that there should be some form of reward for hacking through security measures. Neither of these are views we should responsibly seek to promote.

Consider the nature of showing the security of a product. Does a "challenge" meet the goal of testing, which is to increase one's confidence in the correct functioning of the artifact? It really doesn't, for a number of reasons:

Also note that any such challenge also serves to aid potential hackers in their later pursuits:

Furthermore, the whole process sends the wrong message -- that we should build things and then try to break them, or that there is some prestige or glory in breaking systems. That isn't what we need. Instead, we want to promote responsible behavior, using established methods. We need to establish that security is something best done by well-trained professionals, and that hacking into systems is not "job training". (I've argued this point in more detail in "Are Computer Break-Ins Ethical?", Journal of Systems and Software, Jan 1992, 17(1).)

Good security should be carefully designed in and tested using established methods. Tiger teams have a role, but using them (especially ad hoc teams) as a major means of establishing safety is negligent. Security "contests" to demonstrate a system are worse, and should be viewed negatively by potential customers. It should be generally recognized that such contests cannot establish more than cursory confidence in a product, are not a good means of testing, and actually create a climate that may encourage or enable people to try to break the product after it is in use.

If I was a potential customer of any security product, which of the following, somewhat exaggerated approaches would be more likely to convince me that a company had its act together? Which one is the company more likely to be seeking to sell based on smoke and mirrors?

Approach "B" is clearly the one we want to encourage. Approach "A" encourages cycles of "penetrate and patch" and that is what is wrong with most mass-market software available today. However, vendors claim that Approach "A" is what sells more product than Approach "B," in part because it seems to inspire more confidence, and in part because it is cheaper to produce software if they don't use an approach like "B".

If we, as a community and a profession, want better quality and more trustworthy products, we must begin to demonstrate it. The best way is in the marketplace, by showing a willingness to buy based on substance, and not flash. Saying "no" to attempts to sell us products based on "hacker challenges" is one way to do that.


Sameer Parekh, Community ConneXion, (sameer@c2.org URL:http://www.c2.org/):

Most of Gene's points are very valid, and I agree with them. His points are aimed at challenges promoted by a company in order to show that a product is secure. On the other hand, the Community ConneXion challenges are promoted in order to show that a product is *insecure*.

It's easy to prove insecurity, but hard to prove security. The vendor-supported challenges are trying to prove security, which is rather misguided. In proving insecurity though, our challenges are rather simple, as they only require one counter-example to be proven that a system is insecure.

Jon Wiederspan, ComVista (jon@comvista.com URL: http://www.comvista.com/) :

We received a very similar letter from Mr. Spafford when we first began our contest and posted an extensive reply on our site while it was in operation. I will summarize the main points for Cipher readers:

1) Mr. Spafford says that these challenges are a poor way of testing software.
That is true, however it was never our purpose to test the software by running a challenge. The testing has been completed or we would not have been confident enough to place $10,000 on the line. The main purpose of our security challenge was to promote awareness of the existence of security options for Macintosh servers. It was never intended as proof of the security of the system or to replace rigorous testing.

2) Mr. Spafford says that these contests promote hacking.
We disagree with that entirely. By his argument, the Daytona 500 is responsible for people driving too fast on highways. I think there are people who drive as if they are on a race track (one passed me this morning on my way to work) but it is clear that rules on the highway are different from rules on the race track and no court in the land would let a person get away with arguing differently. We clearly stated on our site the limitations of the contest including a warning that we were not condoning similar attacks on systems other than the one provided for the contest.

3) Mr. Spafford says that these contests make it easier to break other systems.
Mr. Spafford is looking in the wrong place. Bulletin boards, newsletters, Web sites and more all exist with information on how to hack into systems. Books have been written on the subject, movies made, and special investigative reports offered on television all on the subject. Writing about what failed on our site will not help hackers significantly. Our site also did not provide free practice to hackers because *none of the attempts worked*. Practice is useless if you do not at some point succeed.

4) Mr. Spafford says that it is wrong to test things by trying to break them.
I don't think he thought about what he was saying there. What is beta testing but an attempt to find where software will break? Stress testing for metal structures? Crash testing cars? It is a fact of life that part of testing a product is to find where it will fail, which means trying actively to break the product in a variety of ways.

In summary, it is our opinion that Mr. Spafford's letter has no bearing on the challenge that we had online. He probably would have been better served by investigating our site more thoroughly before writing the letter.

Jeff Weinstein, Netscape (jsw@netscape.com, URL: http://home.netscape.com/people/jsw)

My quick reaction is that the Netscape Bugs Bounty is not a "hacker challenge". It is a way to reward users for helping to find bugs that get past us. I don't think that we make any claims such as "our product must be secure because no one claimed our hacker prize". We also don't view the bug bounty as a replacement for our own QA efforts, but a supplement to it.

Secure Computing Corporation, sponsors of the Sidewinder challenge reported in Cipher EI#6, declined to comment.