So I have had some time recently to work on understanding the internals of llms and mathematics behind them this has been a very good exercise in terms of getting to know how to structure the llm ops and things to look for while working on deploying systems on scale and monitoring their performance.
Things I have learned so far have been about how drift can cause issues over longterm how bad overfitting can be for real business use cases how the model that looks like perfect can be far from it.
In terms of technical knowledge behind these concepts I am nearly done with chapter 3 from the llm from scratch by Sebastian Raschka which has provided amazing insights on the process that llms go through from the transformer architecture to causal and multiheaded attention mechanisms basically how llms go from tokenization and encoding to calculating attention scores, attention weights and developin attention mechanisms.
Looking forward to completing the book and going through detailed internals of mathematics behind the llms.