A carregar...
Publicação
NOVA IMS Chatbot: Developing a RAG Chatbot for NOVA Information Management School
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 2.53 MB | Adobe PDF |
Autores
Orientador(es)
Resumo(s)
Universities face growing volumes of repetitive student inquiries related to courses,
admissions, and procedures. While human support remains essential for complex cases,
scalable chatbot systems can streamline routine communication and improve student
experience. Although Large Language Models have advanced conversational agents, their
static knowledge and hallucination risks limit their reliability in domains requiring accurate,
up-to-date, and context specific information, such as academic environments. Retrieval
Augmented Generation offers a promising solution by grounding responses in relevant
documents. This study systematically tests Retrieval Augmented Generation configurations,
including sparse (TF-IDF, BM25), dense (BAAI/bge) and hybrid retrieval methods, reranking,
and metadata incorporation. We evaluated generation quality using lightweight LLaMA 3.2
models (1B and 3B), tested across baseline, naïve RAG, and zero-shot prompt engineered
variants. Performance was measured through retrieval metrics (Precision@k, Recall@k,
MRR@k), semantic similarity (BERTScore, SAS, SemScore), and human evaluations. Results
show that hybrid retrieval with reranking and metadata yields the highest precision, while
prompt optimization significantly enhances generation quality. Lightweight models
demonstrated strong generation performance, suggesting that effective domain specific
systems can be achieved with smaller computational requirements. The findings provide a
practical framework for developing an efficient chatbot in educational institutions with limited
computational resources. Deeper investigation into the effectiveness of smaller models
warrants further exploration into the relationship between model architecture, retrieval
system design, and performance outcomes.
Descrição
Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Data Science
Palavras-chave
Chatbot Retrieval-Augmented-Generation Llama-3.2 Academic Artificial Intelligence SDG 4 - Quality education SDG 9 - Industry, innovation and infrastructure