Unscheduled Maintenance

Yesterday afternoon, people in some SL Discord groups were stating they were having trouble logging in. At 3:15 PM SL time, the Grid Status report stated there was some unscheduled maintenance going on.

We are currently performing unscheduled maintenance. At this time residents may experience trouble logging in, completing transactions, rezzing, and other in-world functions which may be unresponsive or slow. Please follow this blog for updates.

Before long, the issues were resolved, and people were able to log on without trouble.

"Unscheduled Maintenance" Leads to Avatar Glitches Inworld

If you had trouble with your avatar loading properly yesterday, or not loading at all, you weren't alone. In Discord and Facebook, residents were saying their avatars wouldn't fully rezz. As it turned out, Second Life was having "Unscheduled Maintenance" at the time.

At this time we are undergoing unscheduled maintenance that may cause some avatars to appear as a cloud for an extended period of time. We expect this issue to clear up gradually over the next hour or two, and inventory is not affected by this maintenance. Please note that at this time relogging will not resolve an avatar appearing as a cloud and mass relogging may extend the timeline for resolution. We thank you for your patience, and apologize for the inconvenience. Please follow this blog for more updates. 

At 12:14PM SL time, Linden Lab declared the problem resolved.

It was two weeks ago that the Grid had a number of times in which avatars would crash and be unable to login for a while. Sunday the 28th was the worst day, blamed on DDoS attacks, but not the only day.

Image Credit: Brandi Streusel

Bixyl Shuftan

Unscheduled Maintenance Keeps Residents Offline During Clubs' Prime Time

Yesterday evening, about the time when a number of clubs have their events, residents began having trouble logging on. It turned out that Linden Lab was doing a round of "Unscheduled Maintenance" on the Second Life grid.

We are performing an unscheduled maintenance to our login systems. Residents may experience log in issues in world as well as secondlife.com. Please keep an eye on this blog for status updates. 

The trouble, according to their Grid Status Report, had started well before prime time for the clubs, the Linden Lab made the decision to do the maintenance at 5:33 PM SL time. But it would take a couple hours for the Lab to find the problem, and so residents continued have trouble logging on. And if they were on, they were advised against rezzing anything non-copyable or making any money transactions, including Marketplace or club tips.

Eventually, venues around Second Life, including the Furry Fashion Lounge and Club Cutlass, began calling off their events.

Tonight's Lounge event (Rave) is being canceled, due to the second life outages. the grid status is still being monitored, but has not changed.  So it's in our best interest. It will be rescheduled in the near future. For now we will continue monitoring grid status, and it's suggested to refrain from building, scripting, or buying/selling until this is resolved.

Sorry about the Wed Dance at Cutlass, but no one could log in the run the boards.  So I just killed the contest for tonight.  We can try again another night, when LL has it's sh*t together.

Finally at 8:08PM SL time, eight minutes after these and other clubs' events would normally be over, Linden Lab declared the problem fixed and the Grid was back to normal.

Unscheduled Maintenances do not happen often on Second Life these days. In August, one of the Lindens came forward to apologize after the one then. Though with this one coming not long after one in late August, a few residents might wonder what's going on. Still, the outages are less frequent, and shorter, than Second Life's early days when the Grid was closed down for half a day or longer a few days a month, the Lab posting picture in self-depreciating humor of apes hitting a monolith with bones, "The Grid is down while we bang on things." So while what happened was annoying an inconvenient, it wasn't as bad as it used to be.

Bixyl Shuftan

Login Issues Keep Many Residents Offline

 For those trying to get on at around 4-5PM SL time yesterday, they likely saw something like the "login failed" message to the left. Glitches in the gird were once again messing with things, and the Lab had started a round of "unscheduled maintenance." People could still stay online, but those out were staying out.

A number of events were disrupted or canceled. On the official forums, there was a thread with several pages worth of comments about the problem, people complaining about not being able to log in, "I love how the official status page describes an almost complete inability by anyone to log in anywhere as 'Degraded Performance.' " Finally at 5:38, Linden Lab posted they had resolved the login issues, with the announcement about billing issues twenty minutes later.

But not all residents would be able to get back online. As it turned out, Comcast was having some service outages over parts of the country.For customers of the service, that added to their difficulties.

Earlier in Second Life's history, such outages occurred frequently. For hours, no one would be able to log on at all, while the Lindens posted a notice on the website, "The grid is down while we bang on things." More recently, disruption on this scale have been less common. On August 23 last year, residents were unable to log in for a few hours and some were disconnected. This time, April Linden came forward and apologized for the Lab. It was a rare moment the words "we're sorry" were heard from the Lindens.

Bixyl Shuftan

April Linden Explains Tuesday's Grid Problems, "We're Sorry About This Outage"

As some of you already know, there were some problems with the Second Life Grid on Tuesday. For a while around Noon and the early afternoon, residents were unable to log in and some were being logged out. At the time, Linden Lab posted an "Unscheduled Maintenance" notice at 11:03 AM stating this, and asking residents to temporarily stop making purchases inworld and buying Lindens.

We are aware that some users are currently being logged out, and all users are currently unable to log in to Second Life. Additionally, please refrain from transacting on the LindeX on in-world as purchases are likely to fail. Some regions are also unavailable at this time, and teleports to other regions may fail currently. We have identified the issue causing this situation and are currently working to resolve it. Please monitor this blog for additional updates.

It was 2:49 PM when they finally gave the all-clear. A few residents I talked to later mentioned the downtime. One would later tell me adjustments she had made at the time while she was going through a number of sets of clothes had been undone, and wondered if what was going on with the Grid had something to do with it.

Several years ago, downtimes happened regularly. With the sometimes self-depreciating humor of the days of Philip Linden, they'd post a picture parodying the monolith and apes scene in 2001: A Space Odyssey, "The Grid Is Down While We Bang On Things." Today, downtimes are much less frequent. And this time, April Linden decided to make a statement the following day explaining what happened.

Shortly after 10:30am, the master node of one of the central databases crashed. This is the same type of crash we’ve experienced before, and we handled it in the same way. We shut down a lot of services (including logins) so we could bring services back up in an orderly manner, and then promptly selected a new master and promoted it up the chain. This took roughly an hour, as it usually does.

A few minutes before 11:30am we started the process of restoring all services to the Grid. When we enabled logins, we did it in our usual method - turning on about half of the servers at once. Normally this works out as a throttle pretty well, but in this case, we were well into a very busy part of the day. Demand to login was very high, and the number of Residents trying to log in at once was more than the new master database node could handle.

Around noon we made the call to close off logins again and allow the system to cool off. While we were waiting for things to settle down we did some digging to try to figure out what was unique about this failure, and what we’ll need to do to prevent it next time.

We tried again at roughly 12:30pm, doing a third of the login hosts at a time, but this too was too much. We had to stop on that attempt and shut down all logins again around 1:00pm.

On our third attempt, which started once the system cooled down again, we took it really slowly, and brought up each login host one at a time. This worked, and everything was back to normal around 2:30pm.

My team is trying to figure out why we had to turn the login servers back on much more slowly than in the past. We’re still not sure. It’s a pretty interesting challenge, and solving hard problems is part of the fun of running Second Life.
Voice services also went down around this time, but for a completely unrelated reason. It was just bad luck and timing.

We did have one bright spot! Our status blog handled the load of thousands of Residents checking it all at once much better. We know it wasn’t perfect, but it showed much improvement over the last central database failure, and we’ll keep getting better 

My team takes the stability of Second Life very seriously, and we’re sorry about this outage. We now have a new challenging problem to solve, and we’re on it.

"We're sorry." Not words that we hear from Linden Lab every day. But this time, it's on the record that they spoke them.

Hat Tip: Inara Pey

Bixyl Shuftan

Unscheduled Maintenance

Yesterday in the afternoon, things weren't going quite right for some Second Life residents. Some couldn't log in. Others couldn't teleport or ended up in places like the hub in the Adult sim of Arapama. Ending up there, the only way I could escape was to log off and relog at my Home location. So what was going on? A look at the Grid Status revealed a problem.

Unscheduled Maintenance
Posted by Status Desk on October 27th, 2014 at 03:24 pm PDT
[Posted 3:24 PM PDT, 27 October 2014] We are currently undergoing unscheduled maintenance to several systems. Some Residents may experience difficulties with multiple services, including logins, Web dashboards, and searches. Please keep an eye on this blog for details.

Eventually people were able to log in and port as they were before, though for a while, things were frustrating.

A Glitchy Evening

If last night seemed full of glitches, it wasn't just you. A number of people I talked to and chatted with had a number of inventory issues from being cloudy to being unable to change clothes or avatars. As it turned out, the Grid was the subject of some "Unscheduled Maitenance."

[POSTED 2:15 PM PST, 02 January 2014] We are currently performing unscheduled maintenance. During this time, some residents may be temporarily unable to log in, logged off, or transactions may be disrupted. Please refrain from transacting with L$ or rezzing valuable (no-copy) items until the all-clear is given. Please watch this blog for updates.

Although the Lab stated the problem was resolved in a few hours, friends continued to complain of glitches.

Bixyl Shuftan

Grid Hiccup Disrupts Second Life Saturday Night/Sunday Morning

Residents going about Second Life on Saturday night and early Sunday morning found things weren't quite right. Objects wouldn't rezz. Notecards couldn't be made. And some objects when clicked wouldn't work. Furthermore, residents trying to log on were often blocked from doing so. As it turned out, Linden Lab was performing "unscheduled maintenance" on the Grid, as shown by the Grid Status Report.

[Posted 7:35pm PDT, November 3, 2012] We are currently performing an unscheduled inventory server maintenance. During this time, some residents may be logged off and will temporarily be unable to log in for a brief period. This maintenance may also disrupt transactions and logins. Please check back here for updates.

[Posted 11:58pm PDT, 03 November 2012]We are currently undergoing unscheduled maintenance to Second Life. This may cause some Residents to be logged out, or prevent some from being able to log inworld, or even into Secondlife. com Please keep an eye on this blog for updates.

It wasn't until 9AM SL time (12 Noon EST) That the Grid Status Report declared everything had been fixed. Among the events disrupted was the Burn2 Temple Burn, of which the "Australian/Pacific Burn" scheduled at 4AM SL time had been rescheduled for Monday. In the meantime, residents found other places to go to, such as World of Warcraft, Minecraft, or funning around on the SL Universe forums.

It should also be noted that some residents from the area hit by Hurricane Sandy, aka the "Frankenstorm," are still unable to get on Second Life normally due to evacuation, or unreliable power.

Bixyl Shuftan