Technical Difficulties Keep Some Offline, Interfere With Support And Money Transactions

 

 Yesterday, Sunday August 23, some residents had trouble logging on, and if they were on weren't always able to make or see announcements. 

We are currently investigating the outage of our Support services. At this time residents may experience issues accessing their dashboard on Second Life, logging inworld, and the ability to access Live Chat or to submit a Support case. We greatly apologize for this inconvenience. Please follow this blog for any further updates.

Chat did continue to happen to a smaller degree, some residents noticing fewer people than expected showing up at events, and money transactions not going through. 

 

Finally at 2:57 PM SL time, almost three hours after the problem had been announced, Linden Lab stated the situation has been resolved. 

 

Bixyl Shuftan 

 

"Massive" Amazon Server Troubles Take Down Much of Internet

 

 If you've had some trouble online today, you're far from alone. Various Internet services from Snapchat to Netflix to Reddit and were down, and many others had problems because of an Amazon Web Services server outage. The trouble supposedly started today, Monday October 20, at 2:40 AM EST, possibly at one of their largest datacenters in northern Virginia. There were supposedly "over 1,000 businesses impacts, including many banks." The total damage done may be in the billions of dollars. 

 

Second Life was affected as well, the Grid Status report showing "A service provider for Second Life is currently experiencing an outage which is affecting the Destinations page: secondlife.com/destinations/" Search was affected as well. A couple friends told me they had problems getting on. 

 

Supposedly the data engineers have taken care of the trouble, though this is a reminder of how many companies use Amazon servers, and what can happen when something big goes wrong. 

 

Sources, CNN, Techradar 

 

Bixyl Shuftan 

 

Winter Storm Could Mean Outages

 

 

 "The Big Chill" isn't just a movie you can rent these days. This week, a winter storm is making it's way east from the Great Plains across the middle of the country. It starts dropping snow west of the Mississippi on Tuesday, and will continue Wednesday and Thursday as it moves towards the Atlantic. In it's path are places that don't usually see much snow, such as northern Mississippi and Alabama and northeastern North Carolina. There is a risk of power outages, and of course Internet interruptions. So Second Life residents in the area should be prepared for being temporarily offline. 

Source: https://weather.com

 

Bixyl Shuftan

 

Short Outage On Facebook

 

On Tuesday March 5, Facebook experienced a widespread outage. The incident started at 10 AM EST, and lasted for over two hours. The number of users affected was over a half million at it's peak. 

 

A spokesman for Meta, the owners of Facebook, would say, "Earlier today, a technical issue caused people to have difficulty accessing some of our services. We resolved the issue ... for everyone who was impacted." The announcement was made over Twitter. 

Source: Reuters

Power Outage

 

Power went out for yours truly, and tens of thousands of others in my state, due to a storm with high winds. We'll have something in the paper soon.

 

Bixyl Shuftan
 

Internet Trouble

 

For some reason, been having some trouble staying connected to the Internet, loosing my connection twice in two nights and having to spent time with technical support. Things seem fine for the moment as the connection is back once again (as I write this), but things can still go wrong.

Technical Difficulties

 

If you can read this, then somehow I lost my connection to the Internet (or was too sleepy to reset the "deadman's switch").

Hopefully it'll be fixed, soon.

Bixyl Shuftan 

Hurricaine Idalia Threatens American Southeast

 

 

 Just after Hurricane Hilary caused a scare for California, there's another major storm taking aim at Florida, and on course to barrel through Georgia and South & North Carolina: Hurricane Idalia. Projected to strengthen to Category Three before it lands, there's the possibility of it getting still stronger to Cat 4 when it hits the Florida coast this morning. The storm is expected to remain at hurricane strength as it heads through Georgia, and only weakening to tropical storm status in South Carolina, and continue on through southeast North Carolina, being out at sea only very early Friday morning.

 

People in the path of the storm can expect a strong possibility of electrical, and Internet, outage, and some on the outskirts may experience some interruptions as well. So some of us in Second Life may be offline, or notice some friends who are usually on being off.

 

Sources: News Nation, Weather Channel

 

Bixyl Shuftan

 

Back Online

 

I often say the Newser updates 364 days a year, allowing a day for technical difficulties and the unexpected. Well, on Sunday came the unexpected technical difficulties, which a simple rebooting of the Internet equipment didn't solve, and had to call the company. It's fixed now, and we'll soon be posting as before.

Once again, sorry about the crash.

Image Credit: Erik Mondrain
 

Bixyl Shuftan, Editor, SL Newser

Technical Difficulties

 

If you can read this, then somehow I lost my connection to the Internet (or was too sleepy to reset the "deadman's switch").

Hopefully it'll be fixed, soon.

Bixyl Shuftan
  

Linden Lab Explains "February 1st Outage"

 

With the scale of the disruptions on Wednesday February 1st, it was likely Linden Lab would find themselves obligated to their customer base, the residents, to try and explain what happened. It turns out they made a statement about the "Recent February 1st Outage" the following day in the "Tools and Technology" section of the official blog. 

This Wednesday, February 1st, was a rocky one for Second Life, especially for regions on our Release Candidate (RC) channels. As with many large-scale services, we experience service interruptions from time to time, but yesterday's problems were unique and noteworthy enough that they deserve a sincere apology and more thorough explanation. 

 

The Lab stated the main cause was "a bug in object inventory behavior that was introduced by new code" saying the bug, "caused some objects' scripts to enter a non-running state."

RC rolls are regularly scheduled for 7AM Pacific (SLT) every Wednesday. While conducting one of these deployments, we received initial reports of issues relating to script behavior at 8:30AM. We immediately started investigating, and by 9AM it was clear that we needed to stop the roll, because the issue was pervasive and clearly related to the new changes. As we do in these situations, we declared an “Incident” and an Incident Commander took charge. 

By 9:30 we started a rollback, reverting the affected simulators to their previous version. We also evaluated what additional actions we would need to take, as it was unclear that a rollback alone would start the scripts which had stopped. This quickly proved to be the case, and we came up with a new plan - one that would ensure that scripts would perform as expected going forward, but possibly undo the changes our residents had been making since the time we had introduced the bug. Although this meant more downtime, it prevented further content loss, and proved to be the best way to put the grid back in order. The team came up with a quick, clean, and efficient way to achieve this and get everyone back on track. 

By 12PM the decision was made to take this direction. By 1:15PM the code was complete, by 1:40PM we confirmed that it worked. By 2:25PM all regions were brought back up.  

The report in full, signed by Grumpity Linden "and the Second Life team," can be found (here).

"I’m grateful to know that while we may make mistakes, we will not sweep them under the rug, nor look for someone to blame - we will come together and make it right."

EOTB: The Lab Adresses Recent Outages

 

Second Life has seen it's share of glitches over time. But last week there were a number, notably the rolling restarts on Tuesday January 27 that went snafu and resulted in some sims offline for a while. A few days ago in the "Tools and Technology" section of the official blog, the Lab came out and admitted there were multiple problems.

*  *  *  *  *

Well, this is not the way we wanted the past week to go -- multiple outages in the same week! Not fun for anyone. Here’s what happened.

Login and Inventory Issues: Monday, 1/24 - Friday, 1/28
On Monday, January 24, residents’ inventories stopped responding to requests. Some residents were unable to log in as a result. We identified the affected accounts, addressed the problem and restored services to full operation in a little under 3 hours. 

Beginning on Wednesday of this week, we received reports of intermittent inventory issues – failures to rez, unpack and delete inventory items; and intermittent errors upon login.

When the number of reports increased, it became clear that this was an outage. Several Lindens jumped in to diagnose the reports. After some digging, we discovered our back end infrastructure was being overloaded. Once this was resolved, the positive impact was almost immediate. The data now looks good -- on the back end and we are no longer receiving inventory reports.

We’re taking steps, including a deploy late last week, to prevent these issues in the future - and we have already seen progress in making the service more robust.

Weekly Rolling Restarts: Tuesday, 1/25
Every week we restart the simulator servers to keep them running smoothly. We only restart a certain number at the same time, allow them to finish, then start another batch. That’s why we call them “rolling restarts.” But on Tuesday a recent upgrade to the simulators meant that the usual number of simultaneous restarts was too much for the system to handle. The result was load spikes and numerous regions going down. Ultimately more than 12,000 regions were stuck in restart mode, which is 40% of the grid. Not good!

The team came together quickly to bring simulators back up in smaller batches, and then manually fixed blocks of regions. After some trials and monitoring, we found a smaller number of concurrent restarts that worked better. By 1:00 PM all regions were restored and operating normally. We apologize for this lengthy outage!

Advance… Retreat!: Wednesday, 1/26
We had planned a deploy to the production grid that would help us gather information on group chat performance. Unfortunately, the procedure used during testing did not work in production. We figured -- OK, we’ll just roll back. But then the rollback itself had problems!  Residents experienced issues with login, group chat, and presence information. The team was able to isolate the problem and complete the rollback, getting a few more gray hairs in the process. We’re going to take what we’ve learned and do a better deploy that will give us the information to improve group chat in the long run.

Last week was no fun for a lot of people at the Lab. Thank you for your patience. We really don’t like interrupting your enjoyment of Second Life. Here’s hoping we don’t have to come back with another one of these blog posts for quite some time! 

*  *  *  *  *
 
The Lab expressed hope that "we don’t have to come back with another one of these blog posts for quite some time! "
 

A Bad Week For Facebook and Twitch, Windows 11 Launched

It was not the best week for Facebook last week.  On Tuesday October 5, Frances Haugen, a former Facebook manager turned whistleblower, testified before Congress. She stated the company knew it's app Instagram could be hazardous to the mental health of teenage girls, and knew it was not doing enough to combat disinformation on it's social media website that contributed to the worsening political divide in the United States and led to the January 6 riot. While it could have taken measures to protect girls on Instagram and done more about misinformation on Facebook, it had not done so as the company didn't want to cut into it's profits, putting money over the safety of it's customers. Company CEO Mark Zuckerberg would respond with a statement denying her accusations.

But then came more bad news in the form of a six hour outage of both Facebook and Instagram. While for most it was an annoyance, some small businesses have come to rely on Facebook for communication with their customers. Also affected were sets of Oculus VR headgear. The company would eventually offer an apology.  It's estimated that the company lost billions from this outage. Then on Friday, the company suffered another outage and once again Facebook and Instagram were down once more, another blow to it's reputation.

Facebook wasn't the only social media platform to face big trouble. On Wed Oct 6, there was a massive data leak targeting Twitch. Over 100 gigabytes of data was posted online on Wednesday October 6. This includes operational details that the company "fiercely guards," such as how many millions the top streamers made in revenue from broadcasting. "Aside from the salary details, the documents seems to contain the site's source code and even technical details for yet to be released products and platforms," Joe Tidy in an article on BBC News would say, adding if everything was confirmed it would be, "the biggest leak I have ever seen - an entire company's most valuable data cleaned out in one fell swoop." ThreatModeler founder and CEO Archie Agarwal would say of the data breach,"This is as bad as it could possibly be."

While there didn't seem to be any passwords included in the leaked data, it was listed as "part one," hinting there was more to come. Still, it is reccomended that Twitch users change their passwords anyway.

Tuesday October 5 was the release date of Windows 11. Six years ago, Microsoft was saying there would be only new installments of Windows 10. That has changed, and the company has launched a new version that promises to be "clean and fresh and simpler." If you're reluctant to change over to the new system, you'll have plenty of company. The list of machines the company says this version of Windows needs has been described as "ridiculously strict." In a review on PC Gamer, the writers thought Windows 11 had a few nice improvements, but nothing worth upgrading to the new system, and a number of reasons to wait. Officially, PC users have until October 2025 to wait. After that, Microsoft plans no more technical or security updates.

Sources: BBC News, PC Gamer, The Verge   

Bixyl Shuftan
  

800,000 Still Without Power Following Hurricane Ida, 46 Dead In New York As Storm Causes Flooding

 

When Hurricane Ida hit Louisiana on Sunday August 27, it caused a huge amount of damage with over a million losing power in the state and next door Mississippi. But the storm wasn't done yet. It's still powerful remnants moved inland, eventually reaching the northeast US, where it caused major flooding in several states, notably New York City where some were trapped in their basements and drowned. There was video footage of one of the city's subway's choked with floodwaters. The storm has been blamed for 46 deaths, and counting.

Back in Louisiana, power is still out for over 800,000 people, with nine people confirmed dead. Numerous gas stations are described as having long lines of people waiting to get fuel, and some grocery stores are reportedly rationing certain foods and ice as people anticipating gasoline shortages are stocking up to reduce trips by car.

Due to it's level of destruction, "Ida" will be retired as a name for Hurricanes.

Sources: Daily Advent, MSN, Reuters, weather.com,

Bixyl Shuftan
 

Hurricaine Ida Smashes Inland, Knocks Out Power to One Million

 

On Sunday August 29, Hurricane Ida reached Louisiana after quickly strengthening to a Category 4 storm. The result was flooding and damage that brought back memories of Hurricane Katrina in 2005. Over a million people lost electric power, including practically everyone in New Orleans. One or two people have been confirmed dead, though more may be found. But for many, it may be some time before they places they evacuated from are safe to return to, and it may be weeks until some places get full electric power, and Internet, again. 

What's left of Ida is continuing to move northeast, and may create flooding and tornadoes along the way.
 
Sources: Weather Channel, CBS News, AP News 

Bixyl Shuftan
 

Hurricane Ida Srengthens to Cat 4, Heading Into Louisiana

 

Residents in Louisiana and Mississippi are facing lots of rain and power outages as Hurricane Ida is not only making a beeline toward them, it is rapidly gaining in strength along the way. 

Ida has been gaining strength very fast. Currently, it is a Categoty Four storm, with winds next to the eye going 130 miles an hour. The storm will make landfall later today, bringing high winds and possible flooding over much of the Louisiana coast, as well as power and Internet outages. As the storm travels inland, while it will lessen in intensity, the still high winds will continue to knock out power over much of Louisiana and Mississippi.

For more information, check out the weather channel: https://weather.com/ .

Source: Weather Channel

Bixyl Shuftan
 

The Big Chill: Arctic Weather Brings Power Outages

 

It's been a cold week across much of the US. A cold blast of air with temperatures most in the US would associate with Alaska and northern Canada has moved south not just into the northern states but also down south to the southwest and south-central states. Texas in particular has been hit hard as much of the state is without power. But the damage isn't just there and millions across the country have lost electricity, and of course Internet. The power outages aren't just from downed power lines, but blackouts being done to preserve power grids.

Over the news, there are stories of people in southern Texas burning furniture, children's wood blocks, and fenceposts to stay warm and cook.  More than thirty people across the country have died because of the cold, some from carbon monoxide poisoning when they kept the car running in the garage to stay warm. While "warming shelters" were an option in some cities, there was a risk from the Pandemic. In one town, people were advised to boil water before drinking it (which could be a problem without power).

In Second Life, a few residents I usually see haven't been on for a few days. Two from Texas messaged me, saying their power was out and wouldn't likely be getting back on soon, one saying she was told not to expect it to come back on for good until next week. Another ran out of firewood, until hearing where he could get some more and drove through icy roads for over an hour to get it. I've noticed some DJ events in clubs needing a substitute. 

As bad as the situation is now, it won't last forever. Warmer weather should arrive next week, and for most of the affected areas so should the power. And of course their ability to get back online.

Source: Associated Press, weather.com

Image sources: www.weathercentral.com, weather.com

Bixyl Shuftan

"Extremely Active" Hurricaine Season Predicted

The hurricane that turned tropical storm that hit the east coast last week could very well be just the start of some weather troubles for some of us. A report by the National Oceanic and Atmospheric Association (NOAA) forecasted "an 'extremely active' season with 19 to 25 named storms."

The forecast is twice what an average hurricane season has, and includes the nine named storms so far this hurricane season. Part of the reason for the predicted high number of storms is the warmer than normal temperatures in the tropical Atlantic areas.

As the hurricane season only just started, people in places likely to be hit by hurricanes and tropical storms should be prepared for possible power outages (and a disruption of activity online and in Second LIfe, and the possibility of having to evacuate if a particularly severe hurricane heads their way.

Bixyl Shuftan

Stormy Weather

Things were quite breezy in parts of the United States yesterday. And yours truly had his power knocked out for about eight hours. Because of this, some articles will be running late.

Sorry about the trouble.

April Linden Explains Last Sunday's Outage

If you tried to get on Second Life on Sunday October 27th, then you may have been blocked by issues that persisted all afternoon. On Wednesday, April Linden, the Operations Manager of the Grid, posted in the Tools and Technology blog just what happened in "The Sunday That Was Not a Fun Day."

The trouble started back on Thursday. We had some pretty bad problems talking to key services (packet loss) on one of our Internet links. It didn’t impact everything, but the stuff it hit was pretty important. It started for a few hours on Thursday, but just magically went away on its own. It’s never good when a problem just magically fixes itself because it’s pretty likely it’s gonna happen again. 

The same thing happened again on Friday, but once again it went away on its own before we were able to debug what actually happened. 

We were nervous going into the weekend, and sure enough, it bit us again on Sunday. We started having more Internet communication problems, but this time, it didn’t just go away. 

Now that we had it in a bad state we started to troubleshoot and figured out really fast that it wasn’t our equipment. Our stuff was (and still is) working just fine, but we were getting intermittent errors and delays on traffic that was routed through one of our providers. We quickly opened a ticket with the network provider and started engaging with them. That’s never a fun thing to do because these are times when we’re waiting on hold on the phone with a vendor while Second Life isn’t running as well as it usually does. 

After several hours trying to troubleshoot with the vendor, we decided to swing a bigger hammer and adjust our Internet routing. It took a few attempts, but we finally got it, and we were able to route around the problematic network. We’re still trying to troubleshoot with the vendor, but Second Life is back to normal again.

April went on to say that the power outages affecting much of California had nothing to do with the outage, although one of the people working on the issue was, "One of our engineers was working in the dark in a house without power, with his laptop being powered by a long string of extension cords that ran to a generator outside."

We’re really sorry that this past Sunday wasn’t very fun. The weekend before Halloween is a really fun time to be Inworld, and it was a frustrating day all the way around. (I personally love the way our Residents really get into Halloween in a way that’s only possible in Second Life!) Knowing that it wasn't as awesome as it could have been makes me sad, and we’re working to make it better in the future. 

April requested that if anyone suspects an issue that may have been caused by the outage, please let the Support team know.

Source: BBC, Tools and Technology

Bixyl Shuftan