Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement
maio 20, 2026 § Deixe um comentário
- Marcelo Tibau UNIRIO
- Rafael Tavares da Silva UNIRIO
- Sean Wolfgand Matsui Siqueira UNIRIO
- Bernardo Pereira Nunes Australian National University
DOI: https://doi.org/10.5753/sbsi.2026.248557
Resumo
Research Context: Searching as Learning (SaL) frames web search as a process where users construct and refine knowledge. However, measuring knowledge gain in natural search environments remains a methodological challenge. Scientific and/or Practical Problem: Traditional behavioral proxies (e.g., dwell time, clicks) scale well but fail to capture conceptual change, while pre/post-tests provide richer insights but are intrusive. This gap limits the development of search systems that can evaluate and promote learning. Proposed Solution and/or Analysis: This study advances a computational measure based on entropy reduction and semantic similarity, and novelly operationalizes it through a browser plug-in that enables real-time measurement in natural search environments, extending prior formalizations and prototype-based validations of the DKG metric. Related IS Theory: The study draws on Shannon’s Information Theory and Information Processing Theory in IS to conceptualize knowledge gain as uncertainty reduction supported by socio-technical processes. Research Method: An experiment combined three structured search tasks, pre/post-tests, and Concurrent Think-Aloud protocols. Quantitative measures (Transfer of Learning scores, also known as ToL, and values from the proposed metric) were triangulated with qualitative coding using OISS and ESKiP frameworks. Summary of Results: Statistical analysis showed a moderate positive correlation between ToL and the proposed metric (r = 0.62, p < 0.01). Bland–Altman analysis revealed systematic differences in scale, with ToL showing higher values, yet relative patterns were consistent. Transcripts emphasized how strategies such as query specialization, evaluation of sources, and persistence in reformulation aligned with higher values. Contributions and Impact to IS area: The study contributes a validated computational metric and artifacts for measuring knowledge gain in real search environments. It reinforces the sociotechnical view of IS by linking human strategies, processes, and technological advantages, and points to adaptive search systems that could measure and promote learning.
TIBAU, Marcelo; SILVA, Rafael Tavares da; SIQUEIRA, Sean Wolfgand Matsui; NUNES, Bernardo Pereira. Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 22. , 2026, Vitória/ES. Anais […]. Porto Alegre: Sociedade Brasileira de Computação, 2026 . p. 478-496. ISSN 3086-4836. DOI: https://doi.org/10.5753/sbsi.2026.248557.
Implementing Knowledge Gain Measurement in Real Search Environments
maio 20, 2026 § Deixe um comentário
- Rafael Tavares da Silva UNIRIO
- Sean Wolfgand Matsui Siqueira UNIRIO
- Marcelo Tibau UNIRIO
DOI: https://doi.org/10.5753/sbsi_estendido.2026.249057
Resumo
The operationalization of learning metrics in real search environments remains an open challenge in the Searching as Learning (SaL) paradigm. While behavioral proxies offer scalability, they capture conceptual change only indirectly; structured assessments provide more direct evidence but often compromise ecological validity. The Degree of Knowledge Gain (DKG) metric addresses this tension by combining Shannon entropy with semantic similarity between queries and clicked documents to model the progressive reduction of uncertainty during search. This paper reports on two technological artifacts developed to embed DKG computation into real-world search workflows, within the scope of the CNPq project 3C-BPA: Comportamento de busca, Complexidade da informação e pensamento Crítico na Busca como um Processo de Aprendizagem. A standalone search engine prototype established the feasibility of real-time DKG computation but exposed limitations in ecological validity and operational sustainability. These were addressed by a Chrome browser extension that estimates the metric unobtrusively while users interact with their preferred search engines. To assess the extension’s applicability, an experiment was conducted combining preand post-tests with the Concurrent Think-Aloud (CTA) protocol and automated interaction logging. Preliminary results indicate that DKG is sensitive to variation in search strategy use as participants who engaged in systematic query reformulation and multi-source evaluation achieved stronger knowledge gains, while those exhibiting disorientation and limited cognitive regulation showed more modest outcomes. Beyond its empirical contributions, the study illustrates how undergraduate research participation can play a substantive role in advancing the development and application of formal learning metrics in information science.
SILVA, Rafael Tavares da; SIQUEIRA, Sean Wolfgand Matsui; TIBAU, Marcelo. Implementing Knowledge Gain Measurement in Real Search Environments. In: CONCURSO DE TESES, DISSERTAÇÕES E TCCS EM SI – IC/IT – SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 22. , 2026, Vitória/ES. Anais […]. Porto Alegre: Sociedade Brasileira de Computação, 2026 . p. 61-70. DOI: https://doi.org/10.5753/sbsi_estendido.2026.249057.
Marcelo Tibau 