Skip to content Skip to sidebar Skip to footer

Tech Things: Inference Time Compute, Deepseek R1, and the Arrival of the Chinese by theahura

When I was in highschool, I took the American Math Competition 12 exam, commonly known as AMC 12. The exam has 25 questions, multiple choice. The questions are challenging, but not impossible. Given enough time, I suspect most of the seniors at my fancy prep school would be able to do quite well on it.

ByHackTechJanuary 24, 20250Comments

News

Understanding how LLM inference works with llama.cpp by multilinear

In this post, we will dive into the internals of Large Language Models (LLMs) to gain a practical understanding of how they work. To aid us in this exploration, we will be using the source code of llama.cpp, a pure c++ implementation of Meta’s LLaMA model. Personally, I have found llama.cpp to be an excellent

ByHackTechNovember 11, 20230Comments

News

LLM Inference Performance Engineering: Best Practices by dkhudia

In this blog post, the MosaicML engineering team shares best practices for how to capitalize on popular open source large language models (LLMs) for production usage. We also provide guidelines for deploying inference services built around these models to help users in their selection of models and deployment hardware. We have worked with multiple PyTorch-based

ByHackTechOctober 13, 20231Comment

News

Fast inference of OpenAI’s Whisper on Rockchip processors by keveman

Useful Transformers is a library for efficient inference of Transformer models. The focus is on low cost, low energy processors to run inference at the edge. The initial implementation is aimed at running OpenAI’s Whisper speech-to-text model efficiently on the RK3588 processors’ based single-board computers. The tiny.en Whisper model runs transcribes speech at 30x real-time

ByHackTechAugust 28, 20231Comment

Sign Up to Our Newsletter