A carregar...
Publicação
Evaluating the Feasibility of Local LLMs in Privacy-Sensitive Environments: A Comparative Performance Framework
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 1.1 MB | Adobe PDF |
Autores
Orientador(es)
Resumo(s)
In recent years, large language models (LLMs) have made great advancements in natural
language processing (NLP) and other artificial intelligence (AI) applications and have been used
in many industries such as healthcare, finance and law, which work with sensitive, confidential
and proprietary data. However, most state-of-the-art LLMs are cloud-based, raising concerns
about data privacy and security in these sectors. A safe alternative to this could be the
implementation of local LLMs, which offer a promising solution by enabling organizations to
keep control of their data whilst benefiting from the same AI capabilities. This study evaluates
the feasibility of implementing these models in such industries through development of a
framework to benchmark LLMs capabilities in diverse NLP tasks. The two most researched
local LLMs and the two most researched cloud-based LLMs are tested in this benchmark, to
assess the local models’ capabilities, compared to state-of-the-art models. After analyzing the
results, we concluded that, despite being competitive in some of the benchmarks, the local
LLMs weren’t consistent through the entire evaluation process and consistently
underperformed compared to the cloud-based models. These suggest that local models,
although not as accurate or versatile as the proprietary LLMs, could be viable for lightweight
NLP workloads where latency, cost-efficiency, or privacy are primary concerns. Tasks involving
simple fact retrieval or comprehension can often be handled by these models, but they can
be relied on for complex and high-stakes applications.
Descrição
Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Data Science
Palavras-chave
Local LLMs Cloud-based LLMs NLP Framework Benchmark Llama 3.3 Claude 3.7 Sonnet GPT-4.1 Mistral Small SDG 4 - Quality education SDG 9 - Industry, innovation and infrastructure