Glitches in the Grid

If you're wondering if Second Life seems more buggy than usual, you're not alone. Over the weekend and on Monday, I've heard from a number of people about various things on the Grid going wrong. Yours truly has experienced occasional lag, and sudden viewer crashes, and I've heard a complaint or two about sim crossings. But friends have told me some of their items just haven't been working right. A contest board at a club I frequent had trouble working for the manager. One friends permapetted horse got sick from hunger, and it was supposed to be free of the need to eat the company's food. Another friend was sailing, and even away from the sim borders the boat could barely steer.

So what was going on? On Thursday September 26, Linden Lab would announce in the Grid Status Report that an attempt to allow scripts to run better ended up breaking the ability of some to work with others.

Investigating - We believe we have identified the source of the scripting problems that rolled to the main channel this week (and so are present on the entire grid now). That simulator build had improvements to script scheduling that reduced overhead, allowing more scripts to be run per frame. Unfortunately, it appears that those changes also exacerbated a previously undetected problem with scripts communicating with other scripts.

The most obvious example of the problem concerns the timing of the object_rez event delivered to a script that has requested the creation of a new object. When a script creates a new object, the script that requests the creation gets an object_rez event to indicate that the new object is ready. Normally, the scripts in the new object should be running before the parent script gets the object_rez event. Similarly, scripts get an on_rez event when the object they are in is rezzed; the order in which those on_rez events occur is not defined, but previously all scripts in the object would get those events roughly at the same time. What we've discovered is that in rare circumstances, the on_rez and object_rez events could be delivered before the all scripts in the new object had been started. When that happened, any communication between the scripts would not work as expected.

This out-of-order event delivery was previously so rare that it had never come to our attention, but the script scheduling changes had the side effect of making it more likely that some scripts would get the rez events more quickly, increasing the chance that the events are delivered before the other scripts they expect to communicate with are ready.

We are working now on a fix; we hope to have the fix deployed on a set of test regions on the beta (Aditi) grid for the weekend so that creators can test with it (watch this space). The rolls for next week will revert the main channel to a version prior to the scheduling changes, and the RC channels will get a version with the improved scheduler but modified so that the rez events are never delivered to any script until after all scripts in the rezzed object have been started.

We know that this bug has affected many of you and your enjoyment of Second Life, and are profoundly sorry that it was not found before it was so widely deployed.

 The following day, the Lab announced on the Grid Status Report that there would be rolling restarts on the Grid.

Update - We will be doing a roll to address this issue on Friday, September 27th, at approximately 10:00 PM PDT. We apologize for the disruptive weekend roll and will try to minimize impact. Thank you so much for your patience during this frustrating time.

Early Saturday morning, September 28, Linden Lab would do rolling restarts. In the forums, Mazidox Linden would say:

"Yep, we are rolling the grid right now to revert the most recent update to LSL script improvements. This should prevent further issues with objects communicating. (But if you continue seeing issues with scripts, whether it's communication or something else we want to know about them!)"

On Monday, yesterday, the Lab declared the problem over.

Resolved - The scripting errors which resulted in many scripts, huds, and other items to not function properly should now be resolved.

However, friends of mine are still complaining about problems that are mostly likely due to script errors, "There is something wrong with (Linden Script Language) and (Linden Lab) broke it," one told me. She also told me as far as she knows, the Lab is still working on further fixes. So it looks like there are still glitches on the Grid.

Bixyl Shuftan