SkyGraph

SkyGraph.art

It's the graph from June 2024 when there was only 5 million blueskies. Unfortunately, it's not being updated anymore since I can't afford a supercomputer to collect all the data from 20+ mil users

yes, current load is making it impossible for me to generate datasets on current equipment

I've been somewhat successful in getting official Discover showing me about 80% relevant stuff using "show less"/"show more". 10-20% is still random posts though.

the clustering there still doesn't look flexible enough to make everyone happy, but it's much improved.

hope that helps 🙏

I'm sure skygaze.io is gone for the same reason. they had the funding, but couldn't figure out business model to justify burning this much of electricity

they're gone, unfortunately

our server couldn't handle the load (solvableish) and the size of the dataset (solvable only if you're a startup with good funding, not a pet project)

state at the moment: not enough resources to handle the current load.

possible future solutions:

1) getting more funding (not happening, it's a hobby project, not a startup)
2) other team with more resources will pick up the work since everything is open source

SkyGraph is dead until further notice 😭

Ficsit mug

Good luck pioneers.

Keep that Ficsit property safe.

No working during gaming hours.

Satisfactory 1.0 download in progress

Time's up.

25 minutes left, @samuel.bsky.team

that's 100% what I'll be up to for the next month or two.

so don't ask me 😅

I hate to break the news: Satisfactory 1.0 releases on Tuesday.

You have 2 days to wrap it up, folks.

Why is it so? Well, just a math major gut feeling. later confirmed by my friend who reminded me that even Dijkstra is not streamable, which is an easy graph traversal algorithm compared to Leiden.

When I realize that in order to make SkyGraph map streamable* I'll need to dig out my math PhD work, spend 5 years in a basement finishing it and then publish a paper explaining why it was impossible.

*so that clusters update incrementally in live without needing full dataset recalculation

Obviously, SATA SSD we've been using has hit it's limit.

But if I understand it correctly, IO souldn't be a problem (for now) if using M2 SSDs.

they can't handle the load anymore 😢

Well, this is probably it.

But there's hope.
All the code is here:

github.com/witchmonster...
github.com/witchmonster...
github.com/witchmonster...

Feel free to reach out to @witch.monster if you need any advice on setting it up.

@suboptimal.technology and @redsolver.dev might be of more help here

as @suboptimal.technology is now figuring out how to make the big indexer work (the one for dataset generation), while @redsolver.dev just successfully upgraded infrastructure for his feeds to work with the new firehose throughput

2 SkyGraph @skygraph.art.1d she/her they/them MSI Infinite X (2018 model) gaming PC with Ubuntu installed. Upgraded to 64 GB RAM and added 4TB Samsung EVO SSD www.msi.com/Desktop/infi... 01 SkyGraph @skygraph.art-1d she/her they/them Feeds were running on 8 v. core 16GBram+160GB storage OVH box 2

it's kind of both. dataset needed for generating clusters just grew 10x
firehose throughput, thus memory+cpu demand for indexers grew 4-5x

this is my current setup (that is no longer handling it):

I think exploring possibilities to continue collaboration with people like @redsolver.dev, who already have some funding and infrastructure to rely on is the most sustainable solution for this project

I appreciate it. I'm willing to donate my work (and a bit of money) into this project.

but since I'm not a startup, taking donations is whole another story which is a little bit too much

sorry, after Brazil joined, feeds probably won't go up again. all my indexers are down.

I may offer some limited by PDS solution in the future.

also happy to help and advice with any solution based on SkyGraph open source code if anyone with adequate recources/funding wants to pick up the project

Feeds were running on 8 v. core 16GBram+160GB storage OVH box

MSI Infinite X (2018 model) gaming PC with Ubuntu installed.

Upgraded to 64 GB RAM and added 4TB Samsung EVO SSD

www.msi.com/Desktop/infi...

This actually might be the end of SkyGraph for foreseeable future. I'll need to check how the big indexer (the one I use for updating the dataset) is doing.

Then it's another question whether it would be even possible to construct a dataset this large on current equipment (6 year old PC).

The current SkyFeed infrastructure can't handle the extra load (more than 4x) caused by all the new users. I'm currently setting up two new powerful servers, which should have enough resources to get all feeds back online later today. Thanks for your patience and sorry for the downtime 😊

🥰🫂

My Nebula+ and My Galaxy+ might also recover after recent surge depending on what daily load will look like.

Currently posts in both are 22 hours behind.

By the way, these 6 feeds still work since they're powered by @skyfeed.app infrastructure.

All SkyFeed feeds should now be back online, thanks for your patience! This has been the first and only downtime in the last 4 months.