April Linden Explains Last Sunday's Outage

If you tried to get on Second Life on Sunday October 27th, then you may have been blocked by issues that persisted all afternoon. On Wednesday, April Linden, the Operations Manager of the Grid, posted in the Tools and Technology blog just what happened in "The Sunday That Was Not a Fun Day."

The trouble started back on Thursday. We had some pretty bad problems talking to key services (packet loss) on one of our Internet links. It didn’t impact everything, but the stuff it hit was pretty important. It started for a few hours on Thursday, but just magically went away on its own. It’s never good when a problem just magically fixes itself because it’s pretty likely it’s gonna happen again. 

The same thing happened again on Friday, but once again it went away on its own before we were able to debug what actually happened. 

We were nervous going into the weekend, and sure enough, it bit us again on Sunday. We started having more Internet communication problems, but this time, it didn’t just go away. 

Now that we had it in a bad state we started to troubleshoot and figured out really fast that it wasn’t our equipment. Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does. 

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

April went on to say that the power outages affecting much of California had nothing to do with the outage, although one of the people working on the issue was, "One of our engineers was working in the dark in a house without power, with his laptop being powered by a long string of extension cords that ran to a generator outside."

We’re really sorry that this past Sunday wasn’t very fun. The weekend before Halloween is a really fun time to be Inworld, and it was a frustrating day all the way around. (I personally love the way our Residents really get into Halloween in a way that’s only possible in Second Life!) Knowing that it wasn't as awesome as it could have been makes me sad, and we’re working to make it better in the future. 

April requested that if anyone suspects an issue that may have been caused by the outage, please let the Support team know.

Source: BBC, Tools and Technology

Bixyl Shuftan

News and Commentary: Linden Lab Offers Apology to Strawberry Singh Following Takedown Demand

By Bixyl Shuftan

Protecting what they consider trademarked images and names has been an issue for companies for decades. For instance, for years the Lego corporation has asked people to call it's building toys "Lego bricks" and not "legos." Some have gotten a reputation for going overboard, such as when Disney once demanded a preschool paint over images of Mickey Mouse and other characters. For Linden Lab, the issue of trademarks usually comes up when one of it's residents makes an item inworld that is trademarked by someone else. But it's own trademarks haven't been an issue for many years, until now.

Strawberry Singh is a noted and award-winning blogger of Second Life, one whom has been doing so for years. On December 13, she announced she was removing one of her tutorial videos, "Introduction to Second Life." The reason, one of the Lindens, Tia Linden, complained to Youtube about it, and Youtube gave Strawberry 48 hours to resolve the issue before taking action on her account. Wondering what was wrong, Strawberry emailed Tia asking what was the issue. Tia soon answered, saying while they appreciated her continued support of Second Life, "we have not given you permission to use Linden Lab's intellectual property in the manner of which you have done." She emailed her again, but did not get an answer by the time the 48 hours was almost up and deleted the video, saying in her blog post, "in the future, I probably won’t be doing any other tutorials like this where I show how to use Second Life because their logos might pop up in the videos." She would tell Hamlet Au when he asked shortly afterwards that she didn't think the Lab cared she was promoting the virtual world.

As Strawberry's blog has a lot of readers, it didn't take long for word to get around, to New World Notes, Modem World, Living Virtually, and presumably other blogs. Hamlet Au would write, "Linden Lab's actions are contrary in spirit if not in letter to its stated policies around medua use of its trademarks." Inara Pey wrote, "the statement that certain images and logos now cannot be use in any capacity. If this is now the case, it is worrying for many of us who routinely blog about Second Life and have used such images and logos. ... Where do we now stand if we are now seeing a shift in position from Linden Lab? Are we now in violation of a new prohibition on image use?" Mona Eberhardt's title of her commentary was more blunt, "Yet Another Stupid D**k Move by Linden Lab." Daniel Voyager has yet to write about the incident, as of the writing of this article, but did end up retweeting a statement from Ebbe Linden (left).

Eventually, the Lab began to realize it had a public relations mess on it's hands. Ebbe Linden ended up defending Tia Linden from an angry Tweet by Mona Eberhardt, "I'm willing to take all kinds of heat, but suggesting that Tia is anything but an incredible team player and professional crosses the line for me. ... We will review policies that make things better and easier for us all though."On Thursday, they issued a public apology on the official Second Life blog.

Recently, the Linden Lab IP team sent a takedown request regarding a YouTube video created by the great Strawberry Singh. She and many others have pointed out that this seems like a mistake, and we agree. We have reversed that takedown request and have reached out directly to Strawberry, but would also like to take this opportunity to publicly apologize to her. 

 

Strawberry, we feel fortunate to have you as a member of our Second Life community (and Sansar as well!), and we are grateful for the public support that your blog, YouTube channel, and other social media activity provides. We’re fans of your work, we are sorry for this misstep, and we hope you will continue sharing your awesome videos.

Following the apology, the Lab explained that the incident came from it's policy of protecting it's trademarked images, and that it would be revised to allow people to write about and video to promote Second Life without worry of hassle, "While we still need policies in place to protect our trademarks, we will apply them as permissively as we can with the goal of encouraging and supporting our community."

Strawberry would tell the Newser, "Aside from a big thank you to all of my readers that gave me so much support yesterday I really don't know what else to say. I was so surprised at the response my post got as my reason for doing the post was just to inform others of what happened so they don't experience the same thing I did. I never expected that people would be so vocal and supportive and do all that they did for me yesterday. I was also pleasantly surprised with the Lab's response too. I was not expecting a response at all so all of it was so encouraging and uplifting and I'm so happy that the end result will be hopefully a better policy for all Second Life bloggers and bloggers in the future."

Linden Lab also mentioned what happened with a parody blog in January 2007 in which the blogger invited the Lab to issue a "cease and desist" letter. But instead Linden Lab's lawyer issued an "uncease and desist" letter, stating, "Linden Lab objects to any implication that it would employ lawyers incapable of distinguishing such obvious parody."

Also at the time, there was at least one instance of someone making modifications of the eye in hand symbol for the purposes of promoting Second Life. In early 2006, Yiffy Yaffle came up with a "Second Life Furry" logo as a way for sims to be identified as having a population of furs. The symbol is seldom seen today, though presumably from lack of interest.

Perhaps the incident was the result of one overzealous employee, but unfortunately Linden Lab has a history of making foul-ups such as this. While the Lab did apologize to Strawberry, that the incident happened in the first place is likely to be seen by many longtime residents as just the latest in a long series of blunders.

Sources: Strawberry Singh, Linden Lab, Washington Post,Twitter

Bixyl Shuftan

April Linden Explains Tuesday's Grid Problems, "We're Sorry About This Outage"

As some of you already know, there were some problems with the Second Life Grid on Tuesday. For a while around Noon and the early afternoon, residents were unable to log in and some were being logged out. At the time, Linden Lab posted an "Unscheduled Maintenance" notice at 11:03 AM stating this, and asking residents to temporarily stop making purchases inworld and buying Lindens.

We are aware that some users are currently being logged out, and all users are currently unable to log in to Second Life. Additionally, please refrain from transacting on the LindeX on in-world as purchases are likely to fail. Some regions are also unavailable at this time, and teleports to other regions may fail currently. We have identified the issue causing this situation and are currently working to resolve it. Please monitor this blog for additional updates.

It was 2:49 PM when they finally gave the all-clear. A few residents I talked to later mentioned the downtime. One would later tell me adjustments she had made at the time while she was going through a number of sets of clothes had been undone, and wondered if what was going on with the Grid had something to do with it.

Several years ago, downtimes happened regularly. With the sometimes self-depreciating humor of the days of Philip Linden, they'd post a picture parodying the monolith and apes scene in 2001: A Space Odyssey, "The Grid Is Down While We Bang On Things." Today, downtimes are much less frequent. And this time, April Linden decided to make a statement the following day explaining what happened.

Shortly after 10:30am, the master node of one of the central databases crashed. This is the same type of crash we’ve experienced before, and we handled it in the same way. We shut down a lot of services (including logins) so we could bring services back up in an orderly manner, and then promptly selected a new master and promoted it up the chain. This took roughly an hour, as it usually does.

A few minutes before 11:30am we started the process of restoring all services to the Grid. When we enabled logins, we did it in our usual method - turning on about half of the servers at once. Normally this works out as a throttle pretty well, but in this case, we were well into a very busy part of the day. Demand to login was very high, and the number of Residents trying to log in at once was more than the new master database node could handle.

Around noon we made the call to close off logins again and allow the system to cool off. While we were waiting for things to settle down we did some digging to try to figure out what was unique about this failure, and what we’ll need to do to prevent it next time.

We tried again at roughly 12:30pm, doing a third of the login hosts at a time, but this too was too much. We had to stop on that attempt and shut down all logins again around 1:00pm.

On our third attempt, which started once the system cooled down again, we took it really slowly, and brought up each login host one at a time. This worked, and everything was back to normal around 2:30pm.

My team is trying to figure out why we had to turn the login servers back on much more slowly than in the past. We’re still not sure. It’s a pretty interesting challenge, and solving hard problems is part of the fun of running Second Life.
Voice services also went down around this time, but for a completely unrelated reason. It was just bad luck and timing.

We did have one bright spot! Our status blog handled the load of thousands of Residents checking it all at once much better. We know it wasn’t perfect, but it showed much improvement over the last central database failure, and we’ll keep getting better 

My team takes the stability of Second Life very seriously, and we’re sorry about this outage. We now have a new challenging problem to solve, and we’re on it.

"We're sorry." Not words that we hear from Linden Lab every day. But this time, it's on the record that they spoke them.

Hat Tip: Inara Pey

Bixyl Shuftan