kabir

Built a *tiny-Mixtral model* (~172M, 8 experts) from scratch
with
- Grouped Query Attention,
- Rolling Buffer KV Cache
- Sparse MoEs
- Rotary Positional Embeddings
Trained it on TinyStories.

github.com/kabir2505/ti...

LLMS Know More than they Show | Notion Key Contributions

Logged the full summary here: www.notion.so/kabir25/LLMS...

• Different errors, different signals: Internal states can even help predict what kind of error a model will make — factual, reasoning, etc.
• Hidden knowledge: Sometimes, models internally know the right answer… but still generate the wrong one externally.

LLMS Know More than they Show | Notion Key Contributions

Takeaways:
• Truth is token-specific: Truthfulness signals are concentrated in certain tokens — probing those can significantly boost error detection.
• Generalization is tough: These probing techniques don’t generalize across datasets, which means LLMs hold multiple fragmented notions of truth.

Read a super interesting paper recently — “LLMs Know More Than They Show”. (openreview.net/forum?id=KRn...) It dives into how large language models actually encode way more truthfulness internally than they let on in their outputs.

which city in Maharashtra?

hello! are you hiring undergrad ml research interns?

implemented the Llama architecture from scratch in pytorch

github.com/kabir2505/De...

let's implement llama today 😋

Not enough ml/dl folks on my feed

hahahaha same

Deep-Learning-History/GANs/WGan at main · kabir2505/Deep-Learning-History Deep learning paper implementations. Contribute to kabir2505/Deep-Learning-History development by creating an account on GitHub.

implemented wgan & wgan-gp in torch
github.com/kabir2505/De...
github.com/kabir2505/De...

onto some more gan models & vaes :)

Spent the day revisiting dropout, so I figured I’d turn it into a blog - kabir25.notion.site/Dropout-16e3...

Implemented *instruction fine-tuning* on a GPT-2 model on a small dataset & mine claimed Robert Frost wrote Pride and Prejudice😅

github.com/kabir2505/pr...

today's agenda..

my notes on the gpt-3 paper: kabir25.notion.site/GPT3-1603fc0...

good morning!!

this is insane, Huge congrats! 👏

If you are into ML theory (RL or not) with a proven track record, and you are interested in an industry research position, PM me. Feel free to spread the word.

today's read :)

NeurIPS FOMO is real 🫠 wish I could teleport..

Deep-Learning-papers/transformers/bert at main · kabir2505/Deep-Learning-papers Deep learning paper implementations. Contribute to kabir2505/Deep-Learning-papers development by creating an account on GitHub.

Built 𝗕𝗘𝗥𝗧 from scratch in pytorch. Took a bit to understand 𝗠̲𝗟̲𝗠̲(Masked Language Modeling) and 𝗡̲𝗦̲𝗣̲ (Next Sentence Prediction) but totally worth the grind.
Code: github.com/kabir2505/De...
Notes: kabir25.notion.site/BERT-1533fc0...

bad recs=zero vibes= productivity tanked!

tackling my first nlp kaggle competition, any suggestions or references?

notes on bert: kabir25.notion.site/BERT-1533fc0...
Still a work in progress..

Diving into BERT today :)