NeilMacMullen

Letter of the day (in the Times)

Release v1.3.0 · NeilMacMullen/kusto-loco Core The engine now uses full parallelisation and previous-result caching for much higher performance on large datasets The engine now uses BitArrays to store nullability info, leadig to much-redu...

Pleased to announce Kusto-loco v.1.3.0 is now available. The main focus on this release is improved memory use and multi-threaded querying.

It's now quite possible to deal with datasets of over 50 million records on your local machine without having to upload to ADX.

github.com/NeilMacMulle...

My 53-million row test-file now comes in at just over 4GB - around 75 bytes per row. Since the row contains 2 doubles, a guid , 2 longs, 2 datetimes, a string (with around 2.5M unique 20-char values) and a bool, that's pretty close to optimal without trying to introduce compression.

..paired with BitArrays indicating nullness. This saves a very considerable amount of ram vs the previous approach of boxing all types to nullable objects.

Well, I finally got tired of fighting C# generic support for nullability and wrote a source-generator to apply a search and replace approach to the T type parameter across the types that Kusto-loco supports. The concrete outcome is that I can now store all types as tables of non-null values...

I use both extensively - mostly Code for embedded and random file editing and Studio for C#. I find the MDI in Studio far superior tbh but Code is a wonderful swiss-army knife of extensions.

I often wondered what the world would look like if the most ignorant, dangerous & daft opinions of radio phone-in hosts/callers were permitted to prevail. I never expected to find out…

As someone who's had thjs recently (and the opposite endoscopy!) I can say that narrative around these is massively over-hyped. The worse part of the experience was actually the industrial-strength laxatives required to clean you out before the procedure!

Summarization needs a little work - tjhe geo-hashing only accounts for around 700ms

I've been working on performance for Kusto-loco
This is the engine working on a 54 million row table in under half a second*. Pretty pleased so far :-)

C# #KQL

*Admittedly I have quite a beefy desktop.

If the BBC was doing its job, it would note in its coverage of Reform that 92% of the party's donations come from the fossil fuel industry.

Anyone know how to get rid of this? I've got all the "obvious" copilot features turned off but this seems stubbornly persistent..

c# #csharp #visualstudio

I mean the annoying thing is that it's barely faster than just making the changes by hand...

So I'm reduced to opening the files manually one by one and barking out "AGAIN!" to copilot like an abusive toddler... Loving this future.

I mean what even is the point of this thing? It's happy to do one at a time but useless as an automated assistant...

c# #copilot #visualstudio

kusto-loco v1.2.8 now released.

Pretty pleased to have got arg_max/min in there as well as max_of and min_of

Thanks to @tbui17.bsky.social for some great work behind the scenes.

#kql

I just modified the behaviour of edit-kql to output a value rather than an object if the query returns a single column. I think that makes the behaviour a lot more intuitive for cases such as below where you want to use the path without accessing it as a member.

#kql #powershell

Ah - makes sense. Thanks so much for the advice - will be trying this soon :-)

Thanks - that's very useful info and it's good to know it's possible. I'm a bit confused by the use of the psm1 file - are you saying you load the dlls with add-type and then the cmdlets inside them are magically exposed ?

Neatness for one - it's easier to drag around a single file than a folder full of bits and pieces though to be fair I've solved that one with an installer.

The main reason is that it would be quite nice to use publish-module although maybe that's not even possible for dlls ?

Question for #powershell module authors... is there any convenient way to publish a c# module with a bunch of dlls as a single file without getting deep into the weeds with ILMerge ?

Finally got round to updating Textrude to .net 9. It's a tool which allows for template-based code generation using Scriban scripting and csv/yaml/json input data.

Just search for "textrude" on github..

Well, finally lokqldx has an installer. This is for all of you reluctant to copy a bunch of random dlls across to your machines ;-)

Get it at github.com/NeilMacMulle... #KQL #kustoloco

I thought I'd share some #powershell scripts I find particularly useful. First one is "tab.ps1" which is indispensable if you live in the #windowsterminal. Quickly assign titles and colors to your tab headers... Get it from github.com/NeilMacMulle...

I already use a lot of source-generation for the internal functions so it's a path already well-trodden but that doesn't mean I'm keen to do it again ;-)

..without a lot of refactoring. I'm grudgingly coming to the conclusion that the only way forward is to use some kind of templating/source generation to generate specific variants, e,g MyClass<T> turns into MyClass_Int MyClass_double etc..

You can however use the same interface for int? and string which means there's a lot of generic code in KustoLoco that works across all the types it needs to. But it means the "simple" optimisation of breaking arrays of T? into an array of T and a BitArrray indicating nullness isn't possible....

Arrgh - C# generics and nullability are so frustrating No way to express that variables should be the nullable or non-nullable version of the type parameter.

_Really_ pleased with this PR from @tbui17.bsky.social The completion window now shows icons! Coming to a kusto-loco release soon....

A kind offer but I think the whole process is just needlessly complex and a massive disincentive to open-source authors. :-)

By now you'd hope you could just right-click on a project and say "publish to store". (dotnet tool is a good model and shows that it is actually possible to do )