Gigs' Corner: SL Grid 08

by Gigs Taggart, Correspondent

This month begins a new regular feature of The Seventh Sun by contributing correspondent, Gigs Taggart. Gigs is a software engineer who is working with Zero Linden and others on the next generation architecture for the Second Life (SL) grid. This month’s topic is the development of an open grid, what Linden Lab refers to as “SL Grid 08.”

Philip (Rosedale) Linden talks about making SL into the next web. Bigger than the web. The Architecture Working Group (AWG) is putting their money where their mouth is. We have high goals here: 60 million regions with 2 billion users. 100 million people using SL at any one time, 20 thousand avatars per region. Obviously a closed system where Linden Lab (LL) hosts everything can’t be that. That’s the main motivation here.

Clearly the more brain power, the better for LL. This means opening things up on all levels, getting people involved and motivated to write alternate clients, alternate servers, and other services that will be needed to make all this work. It’s got to get a lot bigger if we are going to accomplish our goals. This is all about scaling in the end, radical scaling to the levels Philip and LL wants are going to require a radical strategy, and this is it. At the same time we can’t destroy what we have, so it’s going to be a lot of work to get to there from here. Starting over is not an option. We can’t lose the huge value that exists in SL right now in terms of social networks and content.

The end goal is to have standard protocols for connecting your own grids and servers. When I say standard protocols, I mean standard and open protocols like those on the 2D web today. This would be an open grid based on SL technology. Integration with other virtual worlds could happen once it’s standardized of course, but the primary motivation is to make SL a more open architecture based on open standards that suit the type of things SL needs to scale to a huge level.

Some have asked about Croquet technology in relation to this project. Croquet is another open source virtual world platform being developed at Duke University that follows a radically different architecture than LL. It does not require extensive use of database servers. I’m not sure their design goals are in line with LL’s, however. One of the goals of the SL Grid 08 is that it can incrementally migrate from the existing grid. LL isn’t going to “leave behind” all that has been built so far inside the closed SL platform. It makes sense to migrate gradually, to minimize disruptions in service. The goal of gradual migration goes into all design decisions. It has to be something that the main grid can be slowly transformed into over time. Croquet definitely has its advantages, but it’s such a radically different architecture, that it’s not something that could just be jumped on by LL. That said, once the grid is open, there exists the possibility for portals between various worlds. A good example on the 2D web is the ability to jump to an FTP site in your web browser even though FTP and HTTP are very different protocols.

Existing Second Life related open source projects such as libsecondlife will be useful for having an alternate client for testing the grid protocols. You can think of the SL Viewer that LL makes as comparable to Firefox, and libsecondlife as another viewer like IE or Netscape. It’s good to have more than one client to test with, but libsecondlife itself is only a client for the grid in the end. Anyone with the desire can make their own client or server implementations and as long as they follow the protocol. Then they can integrate their creations the same way that multiple web servers and different types of web clients integrate today.

There are going to be some bits of framework that need to be created and standardized. Just as the web needs to have DNS and TCP and other underlying parts, we will need a framework here. Luckily for us though, the web, and web technologies can be a resource for us. Some of the parts that LL is working on right now (and the community is starting to work on) are the Capabilities URLs (caps), chttp, eventlet and mulib. For collaborative technical documentation on these libraries, see the following:

http://wiki.secondlife.com/wiki/Mulib
http://wiki.secondlife.com/wiki/Chttp
http://wiki.secondlife.com/wiki/Eventlet

These are small technical pieces that will help form the foundation of a new, more scalable and open grid. The new grid is based on web technologies, and a lot of it will run over HTTP. LL likes to call it “SL Grid 08” (optimistically). Some of the new grid might have elements of peer-to-peer technology, but the overall plan right now is to have a mostly normal server-client architecture.

The foundation pieces like chttp and eventlet will be big parts of the new server architecture and are already open source. At the same time, I don’t think LL has any immediate plans for opening their complete server implementation. The caps library is a sort of web service-oriented (SOA) architecture. What I expect will happen is that LL servers will begin “speaking” these standardized and open protocols, and this will make it possible for people to run their own servers, written outside of LL, such as Opensim, or license the LL server from LL.

The opensim project has a history similar to the libsecondlife library. libsecondlife is a client, and opensim is a server. The Opensim library started by reverse engineering the existing protocol—similar to libsecondlife—so right now it speaks the “old protocol.” But, as the new open standard evolves, it will start using the standard the same as LL.

The tentative goal is to accomplish a grid retrofit throughout 2008. LL has people working on the various small libraries I mentioned, with other people moving more and more of the server to capabilities-type web services. Zero Linden and our group are working actively on designing the new architecture and protocol that would allow these new use cases. See http://wiki.secondlife.com/wiki/Architecture_Working_Group

Right now I’m one of the pilot residents invited to work on the AWG. Our immediate goal is to get the low level infrastructure standards and use cases figured out. The initial group included about 10 people with technical expertise from various areas of SL with, along with some people from various large technology corporations that have shown an interest in SL.

There is no intent to move to commercial database servers such as SQL Server and Oracle, because the underlying goal here is to build on entirely open technologies such as MySQL. One good side effect of this is that if the open technologies are lagging, they get pulled forward. Right now the client textures run on a closed, proprietary library, even through there is an open alternative. The open alternative was initially unsuitable, but through LL’s and the community’s actions it is quickly becoming better, almost to the level of the closed one. Pretty soon LL will be replacing the closed one with the open one.

Obviously LL has a business to run and may have to incorporate additional closed technology until a decent open source alternative exists. That helps explain why they might not open their server implementation in the near term. They are testing Havok4 physics engine now, and it’s not open. But, whenever possible, they do want to use open alternatives. There are some open source physics engines, but they are going to need some work to get up to par. One good thing about the Havok4 implementation is that LL has pulled apart the server so it is not so closely tied to Havok. This creates possibilities of open replacements in the future. Physics is pretty tough to keep responsive and yet still get a good level of separation that would allow something like plug and play. It’s not impossible though. Opensim has the option to use several physics engines in a mostly plug and play manner.

I don’t think this will come down to a “space race” (like the U.S. and Soviet Union in the 1960s) between LL and a potential competitor like Google’s “My World.” From what I’ve seen of Google’s plan, assuming the reporting is even accurate, I don’t think it will be a direct competitor to what LL wants to accomplish here. Very few competitors seem to “get it” right. It always seems as if they are missing a piece of the puzzle, particularly in terms of the role of the resident. For example, they might not allow the easy creation of new objects. I think the Sony Playstation Home is like that; it’s a social gathering place, but it doesn’t seem particularly resident created. SL is more like a “wiki,” whereas a lot of competitors seem more like “forums:” SL is about open collaboration, while competitive products seem more controlled and moderated. The closest real competitor to SL is probably HiPiHi, but that’s only because they copied everything SL does verbatim. It’s a Chinese product, and they really did literally copy almost every aspect of SL. The underlying implementation is probably a lot different, but they did redo it feature-by-feature for the most part.

The AWG wiki is the main focus right now. LL is embracing the ideas of collaborative group development expressed in the book Wikinomics. Most of the members of the AWG are not paid participants. All have some interest, be it purely philosophical, to see SL turn into the next generation web, or commercial in that wider hosting opportunities will expand business.

At this point, the AWG is open for participation from whoever is interested in participating. If you are interested, drop by during Zero Linden’s office hours, and read the wiki links. The developers email list can be joined at: https://lists.secondlife.com/cgi-bin/mailman/listinfo/sldev

* * *