Mia 🏳️‍⚧️

one day I’ll make something that doesn’t need at least two different database implementations running simultaneously. today is not that day

yeah it's the mass tagging thing again. looks like that account fired off quite a few in the last hour or so pdsls.dev/at://did:plc...

screenshot of a bluesky user counter showing 40 million users

number go up

my kingdom for literally just any media ID in this goddamn file. it’s already over 300MB, you can spare an extra few chars per row for it. I have the playlist ID that was on at the time but that’s not bloody useful.

about ready to throw hands with the utter pillock at apple that decided this was a sensible format for the export.

almost like they don’t want you to use this data for anything…

(I’m not relying on lastfm data exclusively bc my phone and tablet don’t scrobble)

it’s currently three steps:
• import from csv into duckdb
• process, get metadata, store into staging table
• create records

allows me to fudge the metadata which I will 100% need to do and I’d rather do it in datagrip before pushing

This strat will likely break for plays crossing days but that’s not a thing I tend to do, so I can fudge it with lastfm data if that ever happened.

there’s a file with listen history to a daily lvl (kinda hourly but not really*) so I think imma link into that to smooth the data out (and I’ll get track ID that way too - for unknown reasons, activity doesn’t contain any unique media ID)

*a row for song, date + list of hours that song was played.

but only on start events?

truly a cursed file - at some point it just stops including all but one of the timestamps for 24h or so

progressively adding more and more things to the server running parakeet but it seems to be dealing okay.

I would get another but I don’t really want many more atm

everyone else seems to be using svelte so maybe I’ll finally give it a proper shot instead of just using react like I always do.

stewing on a fun little idea off the back of me hopefully dumping years of music history into my PDS tomorrow but I fear I may have to write frontend code again

this should’ve been prefixed with I am not a database engineer but this is what I’ve picked up from too many hours of researching how to get this smaller and faster

but normal columnar stores flat out don’t work for this usecase imo. you need high write speed too but can’t get it because they want large batched writes not many single ones.

Tiger Data’s PG extensions may provide solutions but I haven’t tested them yet (likewise oriole but that broke last time)

tbqh idk - it might be possible to do compression per partition (if you could settle on a good partition - date??) or per page (but I’m not in the weeds enough yet to know the implications of this one). When you start venturing further you get interesting Qs about columnar stores and col compression

something something being locked must feel great for the database

I’m 90% sure you could totally run a relay+appview+cdn of the full (Bluesky) network for significantly under £600 and have it be useable (perf wise).
might even be able to do HA/redundancy for that too.

(I know this isn’t necessarily apples/apples but alas)

a screenshot of Jaeger, a tracing tool. A trace is open and a very tall waterfall is shown (there's a lot of calls to a DB)

pictured: niagara falls

I worked out the seemingly undocumented arcane incantations to get otel+axum working properly and idk why what I did fixed it.

Worked 1st time with tonic after tho, just need to get the trace id passed over.

want to get it plugged in to the DB too but idk how. think I’d need support inside diesel?

I would happily use rust for backend and systems stuff and swift for apps/GUI stuff, that sounds ace tbh.

half the reason I don’t use android is because I refuse to develop for it because I tried it and hate it (I have lost weeks to JNI) but this is getting closer to nice

(yes I tried flutter and RN and I don’t love either)

getting close to the possibility of being able to write android apps without constantly wanting to yeet the laptop out the window, nice.

if/when this can link into UI stuff, we’ll be golden.

it took way too long to get events pushing to jaeger at all - now I (just) need to get all the correct scopes and info recorded.

doing the inter service request linking is going to be an experience, too.

I stand by my comment from a while ago that opentelemetry is painful. I have it half working but not in the way it should and there’s a random warning sometimes. absolutely wonderful stuff.

how much of this is the tracing and Axum integrations? idk.

it’d be good to get the car bug fixed first but I need some test data more than I need *all* test data. would like better metrics first tho*

*I’m hoping jacquard and its zero copy deserialisation might improve consumer perf but want to test that properly

"Oxford commas are a sign you write with ai" I will find such a unique way to rip out your spine that they'll make a movie about it

there's a lot of data that can and should be compressed (and I think you could pull strong ratios out) but it's working out how to do that and not kill the read performance.

It'll be interesting to see if bluesky's kvdb ends up having any large scale compresion on it too