On the Soapbox

The Case for BBP

Saturday, March 11, 2006
Keywords: Technology, BBP

Background / Introduction

Many years ago, I had my first encounter with online communities when I was the resident Perl guru at Hypermart's community support newsgroup, teaching newcomers how to use Perl for CGI. It was a lot of fun, but Hypermart soon went through a string of purchases--as the company being purchased--and everyone left.

Fast-forward some number of years, and I found myself once again engaged in yet another online community, where I would disassemble and modify firmwares and produce software to automate this process. But instead of a newsgroup that I would read in Outlook Express, I was on the staff of two web-based forums running vBulletin and phpBB (I have since retired from the firmware community).

How times have changed. In the early days of the Internet, online communities meant NNTP (i.e., newsgroups). To clarify, many people (including me) so strongly associate NNTP with Usenet that private NNTP is forgotten. Hypermart's newsgroups were an example of private NNTP: their newsgroup server hosted only their newsgroups and had no Usenet content. In this respect, private NNTP was probably the closest precursor to modern web-based forums: when you visit a web forums site, you only get those forums attached to the corresponding site and not every forum in the world (in contrast to Usenet). Today, private NNTP is all but extinct and web forums dominate.

The problems of NNTP and web forums

Whether or not this de facto death of private NNTP is good is debatable. Let's look at why NNTP failed:

  1. Spam: This problem has two dimensions; the first is the difficulty of controlling spam posts because of the limits of the moderation system and the second is the exposing of everyone's e-mail addresses to harvesters.
  2. The lack of a web interface: Aside from the obvious convenience, not everyone has newsgroup reading software at their disposal, and at public terminals, it is usually not feasible or wise to configure the software. Web interfaces do exist, such as nntp.perl.org, but they are not very robust or useful. In a perfect world where browsers have easy-to-use integrated NNTP clients that can store server and user information on a temporary basis (very much like how FTP is handled by web browsers), such web interfaces would not be necessary, but we do not live in a perfect world.
  3. No server virtualization (i.e., hosting multiple hostnames on a single IP, like Apache's indispensable VirtualHosts)
  4. Lack of features: Granting users privileges, read-first stickies, fancy formatting (mostly because it would allow hyperlinking), editing posts ex post facto, etc.
  5. Relative difficulty of setup: Setting up a web forum is easy: Apache, PHP, and MySQL are included in most Linux distros (and are very easy to find, download, and install for Windows). Setting up phpBB is fairly quick and simple. Most importantly, it's possible to get a high-quality setup without paying a single red cent for the software.

Of course, web forums are not perfect either, and most of these problems are manifested in the interface:

  1. Marking new posts: It's relatively easy in a newsgroup reader to see what posts/threads you have not read. It's easy to manually mark posts are read or unread. Most importantly, newsgroup readers do not suffer from the granuarity problems that web forums suffer of automatically marking threads in a forum read once you've loaded the latest thread listing or every post in a thread as read once you loaded that thread.
  2. Faster and more sensible browsing. Ever get tired of clicking "next page" in a web forum to browse through both lists of threads and the posts in each thread?
  3. Web forums lack the more robust sorting and organization that is possible with proper NNTP clients.
  4. Processor-intensive tasks such as sorting and searching are now handled by the server, which severely crimps scalability.
  5. Threading: Although most web forums are capable of threading the display of threads, most opt for a linear display of threads. Why? It's hard to pick out which are the new posts in a threaded display. With newsgroup software, this is not a problem.
  6. Inconsistent interface across different forums (though with newsgroup readers, the inconsistency lies across different reader software, but in this case, it's the user who ultimately chooses the interface).
  7. Inconsistent data representation (which makes the jobs of search engines nice and fun).
  8. Crossposting can be done only by making separate posts (though enough people frown on crossposting that this is more or less moot).

It would seem that while NNTP had its shortcomings, its replacement by web-based forums has wrought problems of its own. However, the problems associated with web forums are more or less unavoidable, and with the problems that come with having to use a separate client, web forums are necessary.

The best of both worlds

It is my belief that the best solution to these sorts of problems is a new protocol that would allow the use of client software alongside a web-based interface. For example, while people can use Gmail's web interface, they could also access e-mail through Gmail's secure POP3/SMTP interface. While people can read blogs and write blog entries via web interfaces, Atom allows them to read and write blog entries using a client, and the growing popularity of syndication aggregators is indicative of the success of this idea. This is a best-of-both-worlds approach that caters to both the technically savvy and to those who prefer the efficiency robustness of client software while maintaining accessibility for people on the go and for the "Hotmail generation" (i.e., the number of people who have never experienced e-mail outside the webmail context and who are not aware of an Internet outside the web browser).

BBP, the Bulletin Board Protocol

For lack of a better name, I hereby propose the Bulletin Board Protocol. It should do roughly the same things as NNTP while addressing the issues with NNTP that can be addressed. It should also be a protocol that could work well with existing web forum formats (much like how RSS or Atom can easily fit the typical blog model).

The most important element, and the element that would differentiate BBP from the proposals of IETF's nntpext working group would be the two versions of BBP. There should be a "BBP/XML" format that rides atop HTTP. In this respect, BBP/XML would be very similar to Atom or RSS. There are several benefits of doing this:

  1. Application protocols that piggyback on HTTP are unaffected by the increased number network setups that firewall ports left and right as an over-reaction to security concerns.
  2. Secure access could be handled trivially by using HTTPS instead of HTTP.
  3. Building the "server" software would be made easier and installing such software would be made easier because it will just be another application that works alongside Apache or another web server.
  4. Using XML would allow utilization of the large array of XML and SOAP resources out there, which would be nice for the development of client software or anything else that makes use of BBP/XML.

Of course, it is usually the case that there is a tradeoff between performance and ease of development, implementation, and deployment. As such, BBP could also be implemented as a true Internet protocol, which would eliminate the performance overhead of piggybacking on HTTP and allow for optimizations like persistent and stateful connections. A "true" Internet protocol would also qualify BBP for a URL (e.g., bbp://user:pass@example.com/forum/thread/post). Finally, for Internet purists like me, it simply feels good to have a true protocol instead of yet another protocol shoehorned into HTTP. But in this day of 3PR (i.e., Perl, Python, PHP, Ruby), .NET, Java, etc., it seems that people are willing to make this ease-performance tradeoff, and as such, I would expect for BBP/XML to dominate, and "true" BBP support should thus be optional.

Implementation

Since this is a new idea that crossed my mind just last night, I have not ironed out the specifics of this protocol (I'll do that at a later date), but I do have a general idea of how such a protocol may be adopted if this idea ever gains traction. Writing a library of PHP functions would help existing web bulletin software adopt the XML version of this protocol; the goal would be for BBP/XML to be tacked on to existing web bulletin software with the same sort of ease that RSS and Atom were tacked on to blogging software. I suppose a proof-of-concept clients could be implemented by modifying the NNTP portion of Thunderbird or another open-source newsreader. Hopefully, once content is available in this format, there would be a rush of people developing real clients, as was the case with Atom or RSS.

To this end, BBP should be as compatible as possible with the current form and paradigm of web-based forums so that it could be implemented not so much as something new, but rather just a newly appended feature.

Credits

I must credit Mr. Milat for the inspiration from his lament over the the demise of NNTP.