What other grids can learn from OSgrid’s shutdown

The attack that is shutting down OSgrid used a feature that every OpenSim grid has, and other grids need to start protecting themselves now, grid owners told Hypergrid Business.

OSgrid announced on Monday that it will close at midnight on December 31 after a “malicious actor” used inventory archives — IAR files — to flood its servers with junk data. Its asset server grew from 5 terabytes to 17 terabytes in less than a year, according to the announcement.

I contacted OSgrid admins for further details about this malicious actor, but they have not yet responded.

“It has long been known that this attack is possible,” Kitely CEO Ilan Tochner told Hypergrid Business.

And now that it has worked once, others will try it, said Lone Wolf, owner of Wolf Territories Grid.

“Every OpenSimulator grid, and particularly every open or hypergrid-connected one, should assume this technique is now known and widely copied,” Lone Wolf told Hypergrid Business.

The attack didn’t need a security hole, said Milton Teixeira da Silva, owner of Virtual Life Brasil.

“It used a legitimate feature, loading inventory archives, at a scale nobody was watching for,” Teixeira da Silva told Hypergrid Business. “That is a lesson for all of us who run open grids.”

OSgrid’s openness made it a particularly easy target, Lone Wolf said.

“OSgrid’s strength was that anyone could connect their own region server to it,” Lone Wolf said. “That means the grid’s central services, inventory and assets included, have to trust servers run by people they’ve never met.”

Other grid owners want those details, too.

“I personally would like to hear more details about what happened as sharing the issue(s) may help other grids avoid the same fate,” said Terry Ford, owner of DigiWorldz, who is known in-world as Butch Arnold.

Backups, limits, and alarms

Recovering from this kind of attack takes multiple layers of backups, and asset and inventory systems that can automatically remove corrupted data, Tochner said.

“We keep three full system backups dating back months and additional daily backups for each world dating back ten days, enabling us to recover very quickly from attacks or accidents,” Tochner said.

Kitely also backs up active worlds every hour, Tochner added — and just copying a database across multiple servers isn’t enough.

“Grids that only duplicate their database across servers aren’t protected from issues that take time to detect,” he said. “Malicious actors can corrupt existing assets and inventories. If there are no backups of the assets before corruption, the data will be lost by the time the grid admins become aware of the attack.”

Grids also need limits on how much data any one account or region can create, Lone Wolf said.

“Unlimited trust between servers was fine in 2008, but it isn’t anymore,” Lone Wolf said.

And they need alarms, he added.

“A table growing from 3 GB to 139 GB should set off an alarm long before it becomes fatal,” Lone Wolf said.

Grid owners should also scan OAR and IAR files before loading them, Lone Wolf said — especially in the coming weeks, when a lot of content from OSgrid will be moving around. And they should lock down their private ports, so that nobody outside the grid’s own servers can reach them.

Wolf Territories hosts all of its regions itself, runs its own modified version of OpenSim, called WolfSim, and manages its servers with its own platform, WolfStack, which runs a daily security review, Lone Wolf said.

Limits on uploads, alerts when the asset database grows faster than usual, and backups that can be rolled back “cost little and do not take freedom from anyone,” Teixeira da Silva said.

Grid owners should plan for attacks before they happen, Tochner said.

“To grid owners I recommend you stop working on grid improvements until you’ve created a long list of potential risks and mitigation strategies,” Tochner said. “It’s better to create systems and procedures ahead of time than to try to rush to fix things while you’re already suffering from an attack.”

Some small grids are also tightening who can get in. Dorena Verne, owner of Dorena’s World, closed open registration some time ago.

“Anyone who wants to join us has to contact me personally,” Verne said in a post on OpenSimWorld.

And some developers say the fix belongs in the software itself.

“I do not believe OSgrid’s open philosophy is what needs to be called into question,” said Gudule Lapointe, owner of Speculoos World, who is known as Olivier Van Helden in real life. “Sadly, it is the very architecture of OpenSimulator that is ill-suited.”

The next version of Lapointe’s OpenSim Helpers toolkit will move the asset server to the simulator level to prevent the kind of attack OSgrid suffered, and will also automate security certificates and simplify installation, Lapointe told Hypergrid Business.

Did OpenSim’s developers know?

Serra Royale, head of Mobius Grid, said that the vulnerability OSgrid’s attacker used is no secret to OpenSim’s core developers.

“This particular exploit has been known to core for an extended period,” Royale told Hypergrid Business, and community members have tried to get it fixed.

“Throughout my advocacy and that of others, we’ve repeatedly encountered dismissal, with some instances of being told ‘Since no one is aware of this exploit, it’s not a real problem,'” Royale said.

Royale also faulted OSgrid for describing the attack as a philosophical challenge instead of a code vulnerability, saying that “their core philosophy could actually survive with proper code repairs.”

A call for an industry consortium

Myron Curtis, owner of Virtual Worlds Grid, said the attack shows how exposed all virtual worlds are, not just OpenSim grids.

“I believe it exposes just how fragile and vulnerable our virtual worlds are and that includes those that don’t use databases,” Curtis told Hypergrid Business. “The real surprise is that this didn’t happen a lot sooner.”

The industry has long lacked practical standards, and security problems are still widespread, Curtis said.

“The closing of OSgrid highlights the need for a funded Virtual World Consortium that can investigate and create useful solutions to our problems as well as promote our industry,” he said.

So far, the only thing holding the virtual world industry together has been the software platforms everyone uses — and AI is changing that, he added, as more platforms and directions emerge.

“We do need some industry ‘roadmaps’ that offer direction without unduly suppressing creativity,” Curtis said. “It is a lot for the industry to take on, but our people have met such challenges before and we will survive this tragedy.”

Should OSgrid have locked down?

OSgrid’s board said that the only way to stop the attacks would have been to close the grid and become a commercial operation, which its volunteers did not want to do, according to the announcement.

Not everyone agrees there were only two choices.

“I mean, I get it, but I feel like switching to a fake ‘commercial’ model — and just automatically approving known good users — to prevent the junk infusions would be significantly better than shutting down,” Ruddertail said in a post on Reddit. “If they don’t have the energy to do that, fair enough.”

Nick Mit, owner of GBG World, argued for much stricter controls, including a hard cap on inventory size, no IAR loading at all, and active monitoring of all members.

“In my opinion, OSgrid needs stricter management, better oversight, and far fewer freedoms for users,” Mit told Hypergrid Business.

Fabio Bastos, founder of ThinkSim, said that openness and controls can go together.

“I believe that open source and open participation can coexist with storage limits, import controls, monitoring, and accountability for resource usage,” Bastos told Hypergrid Business. But, he added, “it would be unfair to judge, from the outside, the decision of a team reporting such significant burnout.”

Money is part of the problem, too. Storage costs real money, and much of it goes to users who are long gone, Ford said.

“I would make a guess more than 90 percent of our storage costs are due to old users who haven’t logged in in well over a year,” he said.

Free grids are still being paid for by somebody, Ford added. “If you are enjoying OpenSim and aren’t paying something for that enjoyment, just know that somebody else is paying or has paid something on their end for the enjoyment you are experiencing.”

OSgrid itself had announced in September that it would start charging $1 a month for each region beyond a resident’s first, after its hosting bills grew larger than its donations.

But open grids can stay open as long as they watch themselves, da Silva said. “Openness and vigilance can live together.”