LLMs in Production 101: Navigating the Challenges

Large Language Models (LLMs) and Generative AI have taken industries across the world by a storm. These powerful tech tools are changing everything, from how chips are designed to diverse fields like finance, healthcare, and just about everything else. LLMs are now accessible beyond giant tech companies or research institutions, thanks to the likes of OpenAI and Meta that have democratized genAI, helping bridge the gap between technical expertise and everyday users.  

With such advancements and progress happening every now and then in GenAI and businesses increasingly embracing LLMs for a plethora of applications, understanding their production scenario and addressing the challenges encountered when scaling these models from proof-of-concept (POC) to full production becomes very important.

Building a POC for applications using LLMs is now more accessible than ever, thanks to advanced and accessible LLMs, managed LLMOps services, genAI application development frameworks, and pre-built templates, that provide valuable resources for POC development.

• Open-source and highly performant LLM families, like Llama3, Mistral, Google Gemma, etc.

• SDKs and frameworks like Ollama, vLLM,  LangChain, LangFlow, LlamaIndex, etc.

  • State-of-the-art proprietary LLM solutions such as OpenAI GPT, Anthropic Claude and Google Gemini.

• HuggingFace’s extensive libraries of pre-trained models and public datasets.

• Cloud-based compute to fine-tune, run and develop on top of the LLMs.

Yet, only a small fraction of these projects progressed from the prototype stage to full production, impacting the real world significantly. 

A key challenge in this transition is establishing app trustworthiness: determining whether an app is reliable, accurate, and fair enough for widespread use.

This series of articles will delve into the specific challenges faced when scaling LLMs to production, highlighting the critical need for robust solutions that ensure the safety, reliability, and fairness of AI applications. In the further articles of the "LLMs in Production series", we’ll address each of these challenges in depth.

Complexities in LLM Production

Hallucinations

source: https://profile.caotouchan.tech/the-mirage-of-knowledge-hallucination-in-llms-2b5b79bda467

Hallucinations have become a topic of discussion in LLM productions and a possible barrier for businesses trying to incorporate LLMs in production.

LLMs hallucination can be false or misleading information generated by the model, grammatically accurate yet factually inaccurate, or a unique answer which is different from the prompt information.

For some use cases, it is a feature (if you are doing fictional content generation) and for others (practically any industry that requires utmost factual correctness– healthcare, fintech, insurtech, legal, etc.) a major drawback.

LLM hallucinations result from a blend of factors, encompassing discrepancies between the source and references within the training data, the manipulation of misleading prompts, dependence on fragmented or conflicting datasets, overfitting issues, and the model's inclination to speculate based on patterns rather than factual precision. A comprehensive grasp of these underlying causes is vital for effectively managing hallucinations and upholding the credibility and dependability of outputs generated by LLMs.

We have a blog discussing hallucinations in detail, and the potential solutions.

Scalability 

As we transition from POCs to production, scaling becomes an important part in improving efficiency and handling deployment needs. 

A significant challenge in scaling LLMs involves effectively handling extensive volumes of data. These models rely on large datasets for optimal performance. However, as these datasets expand we face certain issues like

• Managing enormous amounts of data at production.

• Increasing storage expenses.

• Chances of compromised data security 

When scaling large language models, there is an equal need to address significant infrastructure complexity as well. When your user-base goes from 100 users to 100,000, each and every individual component– hardware, software, and algorithms– should scale to efficiently leverage LLMs for extracting insights from extensive contextual datasets. 

Here are some of the challenges experienced developers encounter when deploying LLMs at scale in production:

• Distributed compute for training.

• Handling workload spikes via horizontal scaling for inference deployments.

• Employing algorithmic optimizations (parallelism, attention mechanisms, etc.) for enhanced compute efficiency and performance.

• A global compute shortage and expensive compute, resulting in quota issues and rate-limiting, often affecting service level agreements (SLAs).

Accuracy and Factual Correctness

Even though LLMs are good at recognizing patterns and writing text that sounds human, they often have trouble staying accurate over time and when dealing with different domains.

Your LLMs don’t know how to say “No, I don’t have this information”. Language models are probabilistic and somewhat stochastic in nature. If you make a query, and the model’s training set does not contain that specific information, instead of a denial of response, a LLM defaults to generating the “best possible answer”, often hallucinating a response that seems coherent, but lacks factuality.

Factual in-correctness is primarily observed in the following cases:

• While working with general purpose models (pre-trained foundational models that haven’t been fine-tuned on your domain-specific data).

• An environment where the knowledge-base is dynamic, and updates rapidly/regularly.

To respond to your queries with acceptable accuracy, the model needs to know your data. 

Imagine a scenario where a model was fine-tuned on the financial data of the top 200 companies of the world, with a knowledge cut-off of January 2024. In March, one of the 200 companies made the headlines for a major scam, and the stock price tanks beyond recovery. Now, if you ask the model whether it is a good stock to invest in in April 2024, the fact that the LLM is not aware of the latest market updates can result in the model recommending the stock purchase as a good decision, which can lead to catastrophic results.

Hence, ensuring factual correctness is one of the crucial requirements for LLMs in production, especially for business-critical applications.

Connecting the LLM to an external data source– database, search engine, etc., a technique called Retrieval Augmented Generation (RAG), provides a solution to this issue. Strategic prompt engineering techniques like COT Prompting, Few Shot Prompting are some ways to tackle this issue, which we will discuss further in the series.

Model and Data Security

LLMs are changing the way we use technology, but they come with big concerns about keeping data safe. Models learn from huge amounts of data gathered from many public (and arguably private) data sources. Companies like Microsoft, OpenAI, Meta and Google have extensive access to your private data and digital footprint. This raises serious concerns regarding keeping information secure.

Additionally, because of the sheer parameter count and the stochastic nature, with the correct prompting these LLMs can sometimes accidentally reveal details from the data they were trained on, or from the external data sources they are connected to, resulting in sensitive data leaks.

The technique generally used here is known as an LLM injection attack– a security vulnerability where malicious input is crafted to manipulate the outputs of an LLM, leading it to generate unintended or harmful responses.

Some examples and potentials risks include: 

• Unintentional exposure of private financial information from a firm’s private financial documents.

• Personal identifiable information (PII) leaks.

• Unbalanced datasets in LLM training or LLM injections, resulting in unfair outcomes or discrimination, eventually leading to class action lawsuits.

  …and many more.

Cost and Optimization

Creating business solutions using Generative AI, especially with proprietary solutions such as GPT-4, Claude3, and similar LLMs, seems relatively easier than operation on an open-source stack. The primary reasons are the following:

• Models like GPT-4 have been pre-trained on a huge corpus of high quality data, resulting in better accuracy and quality of generations.

• You can access and develop using simple SDKs– easy to build POCs.

• Proprietary LLMs are accessible via managed services– automated LLMOps, iterative feature and performance updates– offloading these operational overheads from the developers.

Naturally, developers tend to prefer these services as their first preference, especially for building application POCs. However, these managed, proprietary LLM services do not scale very well in production, owing to the following challenges:

• Cost: When your prompts demand extensive context and necessitate a multi-step chain-of-thought sequence to get your desirable outcomes, the tokens (and thus expenses) accumulate rapidly! As your user base grows, cost viability soon becomes an issue.

• Rate-Limiting: Rate-limiting on the API calls you can make to the model can severely hinder scaling your application.

Cost of LLM API’s like GPT4(8K context) is $0.03/1K tokens (input cost) and $0.06/1K tokens (output cost) and we know how the cost in LLM applications can quickly compound once they are in production.

We have a detailed blog on this, discussing the different options available today for scalable LLM compute in production. Do check it out.

Conclusion 

In conclusion, deploying Large Language Models presents significant opportunities, alongside challenges. Solutions such as fine-tuning, data augmentation, and integration with external tools and knowledge sources (RAG) can address issues like hallucinations and factual correctness. Addressing cost requires complex algorithmic and infrastructure optimizations. Model and data security requires adoption and continuous updation of security measures at both the software and hardware-level.

Despite challenges, with billions of parameters and remarkable language comprehension, LLMs continue to evolve rapidly. Proactive strategies and leveraging AI advancements are key to realizing the full potential of these transformative models in real-world applications.

We will discuss potential solutions revolving around these challenges in production, in the next articles of the series.