Large Language Models in Production Round-table Conversation

MLOps.community

LLM in Production Round Table with Demetrios Brinkmann, Diego Oppenheimer, David Hershey, Hannes Hapke, James Richards, and Rebecca Qian. // Abstract Using LLM in production. That's right. Hype or here to stay? The conversation answers some of the questions that have been asked by our community members like; performance & cost of production, the difference in architectures, Reliability issues, and a bunch of random tangents. We have some heavy hitters for this event! // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links LLM in Production survey: https://docs.google.com/forms/d/e/1FAIpQLSerEryK4xHEZTq0hSu-sVmBHilOzaT71BfCQgXe_uIRgIah-g/viewform Virtual LLMs in Production Conference registration: https://home.mlops.community/public/events/llms-in-production-conference-2023-04-13 Chinchilla papers: https://paperswithcode.com/method/chinchilla, https://arxiv.org/abs/2203.15556 --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Diego on LinkedIn: https://www.linkedin.com/in/diego/ Connect with David on LinkedIn: https://www.linkedin.com/in/david-hershey-458ab081/ Connect with Hannes on LinkedIn: https://www.linkedin.com/in/hanneshapke/ Connect with James on LinkedIn: https://www.linkedin.com/in/james-richards-4baa73a7/ Connect with Rebecca on LinkedIn: https://www.linkedin.com/in/rebeccaqian/ Timestamps: [00:00] Round table success to Virtual LLM in Production Conference on April 13th! [00:18] Register for the Virtual LLM in Production Conference now! [00:44] LLM in Production survey [01:40] Lightning round of introduction of speakers [04:34] Large Language Models definition [09:17] What do we consider large? [10:35] Thought process in use cases production [14:30] LLM open source huge movements [16:50] Problems qualifications [19:25] Production use cases frameworks directions [25:25] Open-source language models tokenizer [26:25] Language models democratization [29:25] Three categories for LLMs in Production [31:22] Latency at 2 levels [33:27] Defining production [34:57] Hitting the latency problems [38:20] Fundamental latency barrier [40:39] Latency use case requirement [44:25] Costs and the use cases [48:12] Product management involvement in costing [49:38] LLMs Hallucination definition [52:05] Building deterministic systems trust [55:21] Wrap up