Algorithms for the adaptive assessment of procedural knowledge and skills.

Procedural knowledge space theory (PKST) was recently proposed by Stefanutti (British Journal of Mathematical and Statistical Psychology, 72(2) 185-218, 2019) for the assessment of human problem-solving skills. In PKST, the problem space formally represents how a family of problems can be solved and the knowledge space represents the skills required for solving those problems. The Markov solution process model (MSPM) by Stefanutti et al. (Journal of Mathematical Psychology, 103, 102552, 2021) provides a probabilistic framework for modeling the solution process of a task, via PKST. In this article, three adaptive procedures for the assessment of problem-solving skills are proposed that are based on the MSPM. Beside execution correctness, they also consider the sequence of moves observed in the solution of a problem with the aim of increasing efficiency and accuracy of assessments. The three procedures differ from one another in the assumption underlying the solution process, named pre-planning, interim-planning, and mixed-planning. In two simulation studies, the three adaptive procedures have been compared to one another and to the continuous Markov procedure (CMP) by Doignon and Falmagne (1988a). The last one accounts for dichotomous correct/wrong answers only. Results show that all the MSP-based adaptive procedures outperform the CMP in both accuracy and efficiency. These results have been obtained in the framework of the Tower of London test but the procedures can also be applied to all psychological and neuropsychological tests that have a problem space. Thus, the adaptive procedures presented in this paper pave the way to the adaptive assessment in the area of neuropsychological tests.

Search and Evaluation of Coevolving Problem and Solution Spaces in a Complex Healthcare Design Science Research Project.

This research employs design ethnography to study the design process of a design science research (DSR) project conducted over eight years. The DSR project focuses on chronic wounds and how Information Technology (IT) might support the management of those wounds. Since this is a new and complex problem not previously addressed by IT, it requires an exploration and discovery process. As such, we found that traditional DSR methodologies were not well-suited to guiding the design process. Instead we discovered that focusing on search, and in particular, the co-evolution of the problem and solution spaces, provides a much better focus for managing the DSR design process. The presentation of our findings from the ethnographic study includes a new representation for capturing the co-evolving problem/solution spaces, an illustration of the search process and co-evolving problem/solution spaces using the DSR project we studied, the need for changes in the purpose of DSR evaluation activities when using a search-focused design process, and how our proposed process extends and augments current DSR methodologies. Studying the DSR design process generates the knowledge that research project managers need for managing and guiding a DSR project, and contributes to our knowledge of the design process for research-oriented projects. Managerial Relevance Statement: From a managerial perspective, studying the design process provides the knowledge that research project managers need for managing and guiding DSR projects. In particular, research project managers can guide the search process by understanding when and why to explore different search spaces, to expand the solutions investigated, and to focus on promising solutions and to evaluate them. Overall, this research contributes to our knowledge of design and the design process, especially for highly research-oriented problems and solutions.

Investigating information-processing performance of different command team structures in the NATO Problem Space.

The structure of command teams is a significant factor on their communications and ability to process, and act upon, information. The NATO Problem Space was used in this study to represent three of the main dimensions in the battle-space environment: familiarity, rate of change, and strength of information position. Results show that the five common team structures (chain, Y, circle, wheel and all-connected) did not generally perform as predicted in team literature. Findings suggest that under dynamic and highly variable conditions, high levels of synchronisation and trust should be present. On the other hand, synchronisation and trust are less important in hierarchical, highly centralised structures, because team members are more willing to accept the authority of a single leader and this tight control ensures that these teams can perform well as long as the Problem Space is familiar, information is explicit and the environment does not change. Practitioner Summary: Some types of team structures are better suited to particular constraints of the battle-space than others. This research has shown that the much touted all-connected structure is often the worst performing structure and that the traditional hierarchy of command and control has much merit in the digital information age.

SYFSA: a framework for systematic yet flexible systems analysis.

Although technological or organizational systems that enforce systematic procedures and best practices can lead to improvements in quality, these systems must also be designed to allow users to adapt to the inherent uncertainty, complexity, and variations in healthcare. We present a framework, called Systematic Yet Flexible Systems Analysis (SYFSA) that supports the design and analysis of Systematic Yet Flexible (SYF) systems (whether organizational or technical) by formally considering the tradeoffs between systematicity and flexibility. SYFSA is based on analyzing a task using three related problem spaces: the idealized space, the natural space, and the system space. The idealized space represents the best practice-how the task is to be accomplished under ideal conditions. The natural space captures the task actions and constraints on how the task is currently done. The system space specifies how the task is done in a redesigned system, including how it may deviate from the idealized space, and how the system supports or enforces task constraints. The goal of the framework is to support the design of systems that allow graceful degradation from the idealized space to the natural space. We demonstrate the application of SYFSA for the analysis of a simplified central line insertion task. We also describe several information-theoretic measures of flexibility that can be used to compare alternative designs, and to measure how efficiently a system supports a given task, the relative cognitive workload, and learnability.

The Referential Problem Space revisited: An ecological hypothesis of the evolutionary and developmental origins of pointing.

Pointing by great apes poses a significant challenge to contemporary theories about the evolutionary and developmental foundations of cognitive development, because pointing has long been viewed by theoreticians as an evolved, human-unique developmental stepping-stone to linguistic reference. Although reports of pointing by great apes have existed in the scientific literature for over a century, only in recent decades has it become clear that ape pointing is definitely an intentionally communicative signal, by the same criteria we adjudge human pointing to be intentionally communicative. Theoretical responses to this changed empirical landscape have generally taken the approach of asserting, without any direct evidence (indeed, in the absence of any possibility of direct evidence), that pointing by humans is psychologically distinct from and more cognitively complex than the pointing of apes. It is commonplace in the contemporary literature to appeal to imaginary, species-unique causal factors to account for human pointing, rendering a large body of contemporary theoretical work untestable with scientific methods: scientific arguments require the public availability of core theoretical entities. In this paper, I will analyze the circumstances of pointing by apes and humans and develop an alternative theoretical model of pointing that does not rely upon non-physical constructs. According to the view espoused, here, pointing develops as a solution to a particular kind of developmental problem, characterized by (a) a developing capacity for tool use, (b) barriers to direct action, and (c) a history of caregiver responsiveness. Pointing by both apes and humans is explicable without invoking imaginary, mental causes. This article is categorized under: Cognitive Biology > Evolutionary Roots of Cognition Cognitive Biology > Cognitive Development Psychology > Comparative Psychology.

On the assessment of procedural knowledge: From problem spaces to knowledge spaces.

By generalizing and completing the work initiated by Stefanutti and Albert (2003, Journal of Universal Computer Science, 9, 1455), this article provides the mathematical foundations of a theoretical approach whose primary goal is to construct a bridge between problem solving, as initially conceived by Newell and Simon (1972, Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.), and knowledge assessment (Doignon and Falmagne, 1985, International Journal of Man-Machine Studies, 23, 175; Doignon and Falmagne, 1999, Knowledge spaces. Berlin, Germany: Springer-Verlag.; Falmagne et al., 2013, Knowledge spaces: Applications in education. New York, NY: Springer-Verlag; Falmagne and Doignon, 2011, Learning spaces: Interdisciplinary applied mathematics. Berlin, Germany: Springer-Verlag.). It is shown that the collection of all possible knowledge states for a given problem space is a learning space. An algorithm for deriving a learning space from a problem space is illustrated. As an example, the algorithm is used to derive the learning space of a neuropsychological test whose problem space is well known: the Tower of London (TOL; Shallice, 1982, Philosophical Transactions of the Royal Society of London B: Biological Sciences, 298, 199). The derived learning space could then be used for adaptively assessing individual planning skills with the TOL.

Problem Space Matters: Evaluation of a German Enrichment Program for Gifted Children.

We studied the development of cognitive abilities related to intelligence and creativity (N = 48, 6-10 years old), using a longitudinal design (over one school year), in order to evaluate an Enrichment Program for gifted primary school children initiated by the government of the German federal state of Rhineland-Palatinate (Entdeckertag Rheinland Pfalz, Germany; ET; Day of Discoverers). A group of German primary school children (N = 24), identified earlier as intellectually gifted and selected to join the ET program was compared to a gender-, class- and IQ- matched group of control children that did not participate in this program. All participants performed the Standard Progressive Matrices (SPM) test, which measures intelligence in well-defined problem space; the Creative Reasoning Task (CRT), which measures intelligence in ill-defined problem space; and the test of creative thinking-drawing production (TCT-DP), which measures creativity, also in ill-defined problem space. Results revealed that problem space matters: the ET program is effective only for the improvement of intelligence operating in well-defined problem space. An effect was found for intelligence as measured by SPM only, but neither for intelligence operating in ill-defined problem space (CRT) nor for creativity (TCT-DP). This suggests that, depending on the type of problem spaces presented, different cognitive abilities are elicited in the same child. Therefore, enrichment programs for gifted, but also for children attending traditional schools, should provide opportunities to develop cognitive abilities related to intelligence, operating in both well- and ill-defined problem spaces, and to creativity in a parallel, using an interactive approach.

Intelligence and Creativity in Problem Solving: The Importance of Test Features in Cognition Research.

This paper discusses the importance of three features of psychometric tests for cognition research: construct definition, problem space, and knowledge domain. Definition of constructs, e.g., intelligence or creativity, forms the theoretical basis for test construction. Problem space, being well or ill-defined, is determined by the cognitive abilities considered to belong to the constructs, e.g., convergent thinking to intelligence, divergent thinking to creativity. Knowledge domain and the possibilities it offers cognition are reflected in test results. We argue that (a) comparing results of tests with different problem spaces is more informative when cognition operates in both tests on an identical knowledge domain, and (b) intertwining of abilities related to both constructs can only be expected in tests developed to instigate such a process. Test features should guarantee that abilities can contribute to self-generated and goal-directed processes bringing forth solutions that are both new and applicable. We propose and discuss a test example that was developed to address these issues.