Socially Speaking

The April 5th Waxxi cast with Jimmy Wales is up, and available for your streaming (or downloading) pleasure, here.

At just 32 minutes, this was the shortest cast we’ve done so far, and the first one we haven’t had to cut in two parts. Although the reason for the shortened cast had to do more with Jimmy’s schedule than anything else, it’s a refreshingly nice length.

It’s a fascinating conversation, and while it’s (again, for the first time) just mine and Jimmy’s voices you’ll hear, the interactivity and participation was awesome.
One question, for those of you who prefer text, came from the live chat:

Steve asks: What are the pros and cons of a transparent search algorithm? Are the advantages of having an open book approach to how search is performed worth the risks of some who will use the information to “game” the system?

JW:

Yeah, so that is a really core question. And, in a certain sense, the success or failure of this entire concept hinges on that question: which actually will work better in the long run?

So, there are a couple things that I think are pretty clear. When you have an transparent, open search engine with freely licensed software, when people find that there’s a problem, there’s a potential for people to actually correct it – and actually have oversight into what’s gone wrong and how to fix it – that you really don’t get in a proprietary search engine, unless you hire lots and lots of people.

The political implications are, well, they’re important to me. And I mean political with a small ‘p’, not really talking about government, but talking about the organization of society, and the organization of information in society. I think as citizens and consumers and producers in the world, we should be concerned about secrecy around such a core piece of the infrastructure of the Internet. So that’s one of the major pros.

Now, if you talk to security people – so, people who work in computer security – they’ll always tell you that “security through obscurity is a bad idea.” In other words, if the way you’re keeping something secure is by keeping it secret so people can’t game it, well you’re always subject to people to figuring it out and gaming it without you noticing. You’re subject to that kind of attack all the time.

One of the reasons we trust the encryption algorithms that we use is that they’re published. They’re public, and they’ve been tested by many, many mathematicians and computer programmers. Everybody can throw what they want at it, and try to find a flaw. If you’ve got a secret encryption algorithm, well…you just don’t know: I mean, has it really been tested thoroughly? And so forth.

So, I think the same idea applies to search algorithms. If the only reason it’s good is because it’s secret, well, that never lasts. What you really want is to truly begin to solve the problem in a more systematic way. For that I think the open approach is the best.

On Wednesday, I attended a talk by Wikipedia founder Jimmy Wales (as part of the Free Culture @ NYU series I’ve been blogging about a bit).

It was historic, in that it was the first public talk he’s given on search. Why? Well, a little project that’s been occupying the bulk of his attention – Wikia Search – has become quite public after an interview with the New York Times, and its resulting article.

The thrust of Mr. Wales’ presentation surrounded his vision of Wika Search:

To build a free, democratic, transparent search engine.

After all,

Search is a fundamental part of the infrastructure of the Internet, therefore of society as a whole.

He believes that search should be three things: transparent, participatory, and free.

Referring to the theory of open source, which at the time of its suggestion, was deemed impossible. “It won’t work!”, insisted leading technology companies like (surprise) Microsoft, Sun, IBM, SAP, and Oracle. Enter: redhat, php, Ubuntu, and more. Guess what? It does work. It works because:

It’s a virtuous (not a vicious) cycle.

More involvement –> fundamental improvement –> open code –> everyone can improve it. And so on.

Next comes open (free) content. “What?! That will never, ever work!”, said such establishments as the New York Times, Financial Times, etc. Enter, for one, Wikipedia. You know, one of the world’s most influential brands? ‘Nuff said.

Back to search, and Wikia. Wika was described, as a whole, as:

Every other kind of book, work, or community that people might build.

Think: library (Wikia) vs. encyclopedia (Wikipedia). Wikia Search relies on three fundamental principles:

1. Transparency: all algorithms are published, testable and researchable. [Open]

2. Participatory: bringing the best elements of Wikipedia and trust networks to the problems of search. This brings the power of human participation to the search process, and relies on the social structure: accountability. It, Mr. Wales believes, empowers people to contributre to something that is of very high quality. The community monitors itself, Wikipedia style. In other words, you have a stake in search. [Human-powered]

3. Free: to change the competitive landscape of search; to encourage global innovation; to disallow disambiguation. An example given was that of a serach for Paris Hilton. Search engines of the now – no matter how often you travel and, let’s say, search for and book a room at the Hilton Hotel in Paris, France – still offer up a slew of sites that link to the infamous “celebrity.” Search of the future (human powered, that is) will at the very least be able to as, ‘did you mean the hotel, or the celebrity?’ [Big idea]

Will Wikia Search be a viable threat to Google? You be the judge. (Or, the participant.)

[Originally blogged over here; thought it was repeating, here.]