Alnkq

I was also surprised when I discovered this, thinking .duckdb is the native format so it must be faster and more optimized.

Could it be that *.duckdb has to support reasonably fast mutations, while parquet files are intended to be immutable so the engine can afford bolder optimizations?

I considered using that, but Anna's Archive dump is a single compressed block (zstd --list -> 1 row). Like I guess most compressed dumps.

spotify_clean_playlists.sqlite3.zst (34GB, non queryable) ->
spotify_clean_playlists/*.parquet (11GB, queryable)

I wish more dataset releases and dumps were done with Parquet. It's truly excellent, particularly when paired with DuckDB

And where's Bluesky's line between follow farming and good-faith use of starter packs?

I introduced a spam throttler in AppViewLite, so that it can ignore likes, posts, and follows done at non-human speeds.

When it comes to follows, I however then realized that Starter packs exist.
Is there a way of telling the difference?

The SQLite rows are then converted to CLR objects and written to parquet files (one for each table) using DuckDB (via DuckDbSharp)

GitHub - alnkesq/SqliteStreamer: Parses a SQLite database in a streaming way, without requiring random access. Parses a SQLite database in a streaming way, without requiring random access. - alnkesq/SqliteStreamer

Here's the project

So I asked Google Antigravity to create a low-level parser for SQLite, that works in a streaming fashion.

On the first pass, it collects metadata, page indexes, b-tree structure etc.
On the second pass, it collects the row data that it encounters, in whatever order they're physically stored.

But that would perform lots of SSD writes.

And piping it thru zstd was not an option, because SQLite requires random access (which a zstd compressed stream cannot provide)

So I wanted to experiment with Anna's Archive Spotify metadata scrape.

A 34 GB zstd-compressed SQLite database (uncompressed: 116 GB)
A normal person would just decompress it and query it with SQLite...🧵

Dark notifications include:
- User blocked you
- Unfollows
- List additions
- Labels you receive (post or profile)
- Hid your reply
- Detached your quote

If you open your profile menu with a right click (or while holding SHIFT), you can access some hidden features of AppViewLite, including Dark Notifications

It's running on a 10 year old machine (8GB RAM, 1TB SSD), and indexes the full network.

I've been dogfooding AppViewLite as my main appview for almost a year now.

I also use it to follow RSS feeds and various other social networks (it supports multiple protocols)

atproto hot take: moving to a non-bsky pds is not decentralization on its own. you also have to decentralize the apps. if you have your own pds but connect to bluesky's servers to get your bsky.app experience, then they can still unperson you. the only difference is your did and repo is safe.

Now AppViewLite displays a badge when you receive incoming messages.

It will redirect you to the bsky-app messaging page (since chat is not implemented), but at least you don't have to manually check.

This started out as a quick experiments to see if AI would be able to implement a project like this (I never seriously used it before for coding), and I was really impressed.

Mostly done with Google Antigravity, it's the first project I make with AI (and I had no prior QtWidgets experience)

A screenshot of a sample app running on QtWinForms

During the last few weeks I've been working on QtWinForms, a cross-platform reimplementation of System.Windows.Forms running on QtWidgets

github.com/alnkesq/QtWi...
#dotnet #winforms

You can follow Fediverse, RSS, Tumblr, Reddit, Nostr, Telegram, and Twitter accounts.

Such non-bsky likes and follows remain private to your appviewlite profile however, since the official lexicon only supports did:plc: and did:web:

Sam Altman's profile being displayed in AppViewLite

Now you can connect AppViewLite to a Nitter instance and follow Twitter profiles directly from AppViewLite.
#atproto #appview

I don't think the answer to an appview becoming hostile is to effectively use a VPN or proxy (PDS).

It is to switch to a 3rd party appview that better reflects your views and appetite for free speech vs censorship

If you disagree with the overly restrictive policies that an appview applies based on geolocation (because of legal regulations or by voluntary policy), perhaps you will also disagree with their mandatory labels and index filtering

AppViewLite changelog:
- Populate external previews when composing a post
- "Mute this thread"
- Drastically reduce synchronous waits on slice compactations
- Upgraded to .NET 10

I've been running a full appview since last January (minus backfilling) and I'm using it as my main client.

LAN-only however, it's not accessible from the internet.

Probabilistic cache should be increased for best performance. Consider setting APPVIEWLITE_PROBABILISTIC_SET_REL_FOLLOW=1024@4

Now AppViewLite automatically detects when performance is degraded by overly full probabilistic sets, and recommends configuration changes.

AppViewLite changelog:
- Hide this reply
- Reduced memory consumption / GC
- Set language when posting
- Fixed muting of Reddit users
- Fixed ARM support
- Profiler: show network requests
- Fix occasional crashes when scrolling Following
- Preserve page location when switching profiles

In the meantime a workaround is to add some swap space.

My machine has 8 GB of RAM, and I haven't experienced OOMs since adding 8 GB of swap. Memory usage seems to plateau after a while.

I'll leave the current process run for a few more days (current uptime 9 days) and then deploy a version that sets GCLargeObjectHeapCompactionMode to see if it fixes the issue.

That's an issue I'm investigating.

I believe it might be due to Large Object Heap fragmentation rather than an actual leak, since running "dotnet-gcdump collect" makes memory usage go back to normal