David Chuan-En Lin

Visual transcripts: lecture notes from blackboard-style lecture videos: ACM Transactions on Graphics: Vol 34, No 6 Blackboard-style lecture videos are popular, but learning using existing video player interfaces can be challenging. Viewers cannot consume the lecture material at their own pace, and the content is also difficult to search or skim. For these reasons, ...

• Rubyslippers dl.acm.org/doi/10.1145...
• QuickCut dl.acm.org/doi/10.1145...
• Visual transcripts dl.acm.org/doi/abs/10....

MixT | Proceedings of the 25th annual ACM symposium on User interface software and technology

A few related HCI works I am inspired by
• Video digests dl.acm.org/doi/10.1145...
• MixT dl.acm.org/doi/10.1145...

So, I built this quick tool to "unroll" a video (which is temporal and sequential in nature) into a "flattened" article. The article is segmented into steps and has demonstrative video clips that align with the steps.

I don't want to just read a recipe article because I want to see how specific techniques are performed. For example, kneading pizza dough.

I've been learning new food recipes.

However, it's challenging to watch a cooking video (play/pause/go back/jump forward) while actively cooking at the same time.

Unroll a video.

instructional video → step-by-step animated guide

Interpolate abstract concepts using analogies.

peaceful → dove
aggressive → falcon

Multimodal interpolation with text ↔ image ↔ audio.

Interpolate concepts in latent space

Transforming between modalities could be interesting.

text ↔ image ↔ video

text → image: image generation
image → video: video generation
video → image: highlight detection
image → text: image captioning

🤏 Semantic pinching

What if you can pinch your screen to transform an article 📝 into an emoji 💪 and reverse!

Here is a simple prototype that uses LLM + gestures to transform text between different levels of abstractions:
emoji ↔ word ↔ sentence ↔ paragraph ↔ article

🌐 semanticpinching.vercel.app