Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement
maio 20, 2026 § Deixe um comentário
- Marcelo Tibau UNIRIO
- Rafael Tavares da Silva UNIRIO
- Sean Wolfgand Matsui Siqueira UNIRIO
- Bernardo Pereira Nunes Australian National University
DOI: https://doi.org/10.5753/sbsi.2026.248557
Resumo
Research Context: Searching as Learning (SaL) frames web search as a process where users construct and refine knowledge. However, measuring knowledge gain in natural search environments remains a methodological challenge. Scientific and/or Practical Problem: Traditional behavioral proxies (e.g., dwell time, clicks) scale well but fail to capture conceptual change, while pre/post-tests provide richer insights but are intrusive. This gap limits the development of search systems that can evaluate and promote learning. Proposed Solution and/or Analysis: This study advances a computational measure based on entropy reduction and semantic similarity, and novelly operationalizes it through a browser plug-in that enables real-time measurement in natural search environments, extending prior formalizations and prototype-based validations of the DKG metric. Related IS Theory: The study draws on Shannon’s Information Theory and Information Processing Theory in IS to conceptualize knowledge gain as uncertainty reduction supported by socio-technical processes. Research Method: An experiment combined three structured search tasks, pre/post-tests, and Concurrent Think-Aloud protocols. Quantitative measures (Transfer of Learning scores, also known as ToL, and values from the proposed metric) were triangulated with qualitative coding using OISS and ESKiP frameworks. Summary of Results: Statistical analysis showed a moderate positive correlation between ToL and the proposed metric (r = 0.62, p < 0.01). Bland–Altman analysis revealed systematic differences in scale, with ToL showing higher values, yet relative patterns were consistent. Transcripts emphasized how strategies such as query specialization, evaluation of sources, and persistence in reformulation aligned with higher values. Contributions and Impact to IS area: The study contributes a validated computational metric and artifacts for measuring knowledge gain in real search environments. It reinforces the sociotechnical view of IS by linking human strategies, processes, and technological advantages, and points to adaptive search systems that could measure and promote learning.
TIBAU, Marcelo; SILVA, Rafael Tavares da; SIQUEIRA, Sean Wolfgand Matsui; NUNES, Bernardo Pereira. Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 22. , 2026, Vitória/ES. Anais […]. Porto Alegre: Sociedade Brasileira de Computação, 2026 . p. 478-496. ISSN 3086-4836. DOI: https://doi.org/10.5753/sbsi.2026.248557.
Artificial Indifference
dezembro 12, 2024 § Deixe um comentário
In the previous essay (The Artificial Other), we explored how the risks associated with artificial intelligence often mirror elements of human hubris, much like Timothy Treadwell’s ill-fated immersion into the wild, as depicted in Werner Herzog’s Grizzly Man. Treadwell’s story is one of passionate yet misguided engagement with Alaskan grizzly bears — a world governed by the harsh and indifferent logic of nature. His overconfidence in his ability to connect with these creatures on his own terms ultimately led to tragedy. It serves as a poignant reminder that nature, as majestic as it may be, operates without regard for care, justice, or morality. It is neither good nor evil; it simply exists. This unyielding indifference, captured so vividly by Herzog, underscores a deeper and more unsettling existential truth: humanity’s inherent vulnerability to forces beyond its control.
LLMs: The Wall Is Now a Mirror
dezembro 6, 2024 § Deixe um comentário
From The Information — Dec 5th 2024
Back in November, I wrote about how Large Language Models (LLMs) seem to be hitting a wall. My piece, “LLMs Are Hitting the Wall: What’s Next?”, explored the challenges of scaling these models and the growing realization that brute force and larger datasets aren’t enough to push them closer to true intelligence. I argued that while LLMs excel in pattern recognition and syntactic fluency, their lack of deeper reasoning and genuine understanding exposes critical limitations.
LLMs Progresso Algorítmico – Parte 2
julho 8, 2024 § Deixe um comentário
Segundo vídeo sobre o progresso algorítmico dos LLMs. Aqui conversamos sobre o que esperar do futuro dos LLMs.
Material adicional:
Sistemas de pensamento: https://www.uiux.pt/2021/04/01/how-we-think-and-make-decisions/
Tree of Thoughts: https://arxiv.org/abs/2305.10601
AlphaGo: https://www.zdnet.com/article/deepmind-alphago-zero-learns-on-its-own-without-meatbag-intervention/
Diplomacy: https://arxiv.org/abs/2210.05492
Self-improvement looping (Imagination-Searching-Criticizing): https://www.linkedin.com/pulse/toward-self-improvement-llms-via-imagination-vlad-bogolin-cnzje/
PIT reward model: https://hackernoon.com/ai-self-improvement-how-pit-revolutionizes-llm-enhancement
Prediction Assignment – Practical Machine Learning
novembro 11, 2016 § Deixe um comentário
To those whom are eager to know more about Machine Learning and how it goes in a real life work, I share a paper I wrote with analysis, codes and algorithms of a Machine Learning Prediction Assignment. I wrote the codes in R, which is a statistical programming language. I also would like to thank PUC-Rio for providing the dataset that I worked.
Executive Summary
Using devices such as Jawbone Up, Nike FuelBand, and Fitbit it is now possible to collect a large amount of data about personal activity relatively inexpensively. These type of devices are part of the quantified self movement – a group of enthusiasts who take measurements about themselves regularly to improve their health, to find patterns in their behavior, or because they are tech geeks. One thing that people regularly do is quantify how much of a particular activity they do, but they rarely quantify how well they do it. In this project, your goal will be to use data from accelerometers on the belt, forearm, arm, and dumbell of 6 participants. They were asked to perform barbell lifts correctly and incorrectly in 5 different ways.
Data source
The data for this project came from the Human Activity Recognition study, conducted by Pontifícia Universidade Católica – Rio de Janeiro.
Ugulino, W.; Cardador, D.; Vega, K.; Velloso, E.; Milidiu, R.; Fuks, H. Wearable Computing: Accelerometers’ Data Classification of Body Postures and Movements. Proceedings of 21st Brazilian Symposium on Artificial Intelligence. Advances in Artificial Intelligence – SBIA 2012. In: Lecture Notes in Computer Science. , pp. 52-61. Curitiba, PR: Springer Berlin / Heidelberg, 2012. ISBN 978-3-642-34458-9. DOI: 10.1007/978-3-642-34459-6_6.
The paper
It can be accessed at:
Marcelo Tibau 