Policy Learning for Actively Labeled Sample Selection on Lumbar Semi-supervised Classification.

Large labeled data bring significant performance improvement, but acquiring labeled medical data is particularly challenging due to the laborious, time-consuming, and medically qualified annotation. Semi-supervised learning has been employed to leverage unlabeled data. However, the quality and quantity of annotated data have a great influence on the performance of the semi-supervised model. Selecting informative samples through active learning is crucial and could improve model performance. Therefore, we propose a unified semi-supervised active learning architecture (RL-based SSAL) that alternately trains a semi-supervised network and performs active sample selection. Semi-supervised model is first well trained for sample selection, and selected label-required samples are annotated and added to the previously labeled dataset for subsequent semi-supervised model training. To learn to select the most informative samples, we adopt a policy learning-based approach that treats sample selection as a decision-making process. A novel reward function based on the product of predictive confidence and uncertainty is designed, aiming to select samples with high confidence and uncertainty. Comparisons with a semi-supervised baseline on collected lumbar disc herniation dataset demonstrate the effectiveness of the proposed RL-based SSAL, achieving over 3% promotion across different amounts of labeled data. Comparisons with other active learning methods and ablation studies reveal the superiority of proposed policy learning based on active sample selection and reward function. Our model trained with only 200 labeled data achieves an accuracy of 89.32% which is comparable to the performance achieved with the entire labeled dataset, demonstrating its significant advantage.

Effect-Invariant Mechanisms for Policy Generalization.

Policy learning is an important component of many real-world learning systems. A major challenge in policy learning is how to adapt efficiently to unseen environments or tasks. Recently, it has been suggested to exploit invariant conditional distributions to learn models that generalize better to unseen environments. However, assuming invariance of entire conditional distributions (which we call full invariance) may be too strong of an assumption in practice. In this paper, we introduce a relaxation of full invariance called effect-invariance (e-invariance for short) and prove that it is sufficient, under suitable assumptions, for zero-shot policy generalization. We also discuss an extension that exploits e-invariance when we have a small sample from the test environment, enabling few-shot policy generalization. Our work does not assume an underlying causal graph or that the data are generated by a structural causal model; instead, we develop testing procedures to test e-invariance directly from data. We present empirical results using simulated data and a mobile health intervention dataset to demonstrate the effectiveness of our approach.

Evidence of policy learning in emergency declarations as communication tools in Australia.

Emergency declarations are important legal tools for the state to protect itself and its citizens during times of crisis. Such declarations permit the exercise of extraordinary powers to address an emergency or disaster. They present an opportunity to explore policy learning in crises, through the ability to examine emergency declaration instruments and the detail of post-emergency inquiries and reviews. This paper briefly assesses Australian law that provides for emergency declarations and places it in the context of theories of policy learning and change. Two case studies reveal evidence of policy learning in emergency declaration practice in Australia. There is an emerging practice of using declarations primarily or purely as tools to communicate the seriousness of an emergency. This policy learning has occurred both within and between jurisdictions, including the federal government. This paper also probes opportunities for future research on policy learning and emergency legislation, especially in relation to the COVID-19 pandemic.

From Local Action to Global Policy: A Comparative Policy Content Analysis of National Policies to Address Musculoskeletal Health to Inform Global Policy Development.

BACKGROUND: Global policy to guide action on musculoskeletal (MSK) health is in a nascent phase. Lagging behind other non-communicable diseases (NCDs) there is currently little global policy to assist governments to develop national approaches to MSK health. Considering the importance of comparison and learning for global policy development, we aimed to perform a comparative analysis of national MSK policies to identify areas of innovation and draw common themes and principles that could guide MSK health policy. METHODS: Multi-modal search strategy incorporating a systematic online search targeted at the 30 most populated nations; a call to networked experts; a specified question in a related eDelphi questionnaire; and snowballing methods. Extracted data were organised using an a priori framework adapted from the World Health Organization (WHO) Building Blocks and further inductive coding. Subsequently, texts were open coded and thematically analysed to derive specific sub-themes and principles underlying texts within each theme, serving as abstracted, transferable concepts for future global policy. RESULTS: The search yielded 165 documents with 41 retained after removal of duplicates and exclusions. Only three documents were comprehensive national strategies addressing MSK health. The most common conditions addressed in the documents were pain (non-cancer), low back pain, occupational health, inflammatory conditions, and osteoarthritis. Across eight categories, we derived 47 sub-themes with transferable principles that could guide global policy for: service delivery; workforce; medicines and technologies; financing; data and information systems; leadership and governance; citizens, consumers and communities; and research and innovation. CONCLUSION: There are few examples of national strategic policy to address MSK health; however, many countries are moving towards this by documenting the burden of disease and developing policies for MSK services. This review found a breadth of principles that can add to this existing work and may be adopted to develop comprehensive system-wide MSK health approaches at national and global levels.

Learning-based near-optimal tracking control for industrial processes with slow and fast modes.

This paper devotes to solving the optimal tracking control (OTC) problem of singular perturbation systems in industrial processes under the framework of reinforcement learning (RL) technology. The encountered challenges include the different time scales in system operations and an unknown slow process. The immeasurability of slow process states especially increases the difficulty of finding the optimal tracking controller. To overcome these challenges, a novel off-policy ridge RL method is developed after decomposing the singular perturbed systems using the singular perturbation (SP) theory and replacing unmeasured states using important mathematical manipulations. Theoretical analysis of approximate equivalence of the sum of solutions of subproblems to the solution of the OTC problem is presented. Finally, a mixed separation thickening process (MSTP) and a numerical example are used to verify the effectiveness.

COVID-19 and the second exams fiasco across the UK: four nations trying to avoid immediate policy failure.

In 2021, the UK and devolved governments tried to avoid the school exams fiasco of 2020. Their immediate marker of success was to prevent a similar U-turn on their COVID-19 school exams replacement policies. They still cancelled the traditional exam format, and sought teacher assessments to determine their grades, but this time without using an algorithm to standardise the results. The outcomes produced some concerns about inequity, since the unequal exam results are similar to those experienced in 2020. However, we did not witness the same sense of acute political crisis. We explain these developments by explaining this year's 'windows of opportunity' overseen by four separate governments, in which the definition of the problem, feasibility of each solution, and motive of policymakers to select one, connects strongly to the previous U-turn. A policy solution that had been rejected during the first window became a lifeline during the second and a likely choice during the third. This action solved an immediate crisis despite exacerbating the problem that ministers had previously sought to avoid ('grade inflation'). It produced another year of stark education inequity, but also ensured that inequity went from part of an acute political crisis to its usual status as a chronic low-attention policy problem.

Lesson-drawing in tobacco control: A qualitative study of stakeholder perceptions in five North-Western European countries.

INTRODUCTION: The theory of 'family of nations' posits that countries draw policy lessons predominantly from similar countries. Lesson-drawing in tobacco control has, however, been primarily studied in the 'English-speaking' family. We examined in five diverse North-Western European countries whether the government engages in lesson-drawing regarding best practices in tobacco control, which countries they look at for guidance, and why these were chosen as a reference. METHODS: Perceptions of 29 policy participants from civil society and government were assessed by means of interviews conducted in Belgium, Finland, Germany, Ireland, and the Netherlands. Relevant excerpts were grouped according to country and a bottom-up thematic analysis was performed. RESULTS: The tobacco control instruments described by the policy participants were tobacco marketing bans (display ban and plain packaging) and smoke-free policies. German interviewees stated that the German federal government is not inclined to engage in foreign lesson-drawing. All other governments were perceived to look at Australia for lessons because of its global leadership in tobacco control. At the same time however, lessons from Australia were easily dismissed because it is an 'island' and far away. Irish interviewees observed their government to primarily look at other English-speaking countries. Governments in Belgium, Finland and the Netherlands were observed to primarily look at nearby European countries for lessons. CONCLUSIONS: Countries in North-Western Europe seem to draw policy lessons based on proximity and similarity to other countries concerning marketing bans and smoke-free policies. Proponents of tobacco control may use these findings to facilitate effective lesson-drawing in their countries.

Policy communities, devolution and policy transfer: The case of alcohol pricing in Wales.

This study investigates how processes of horizontal policy transfer can unfold in the context of devolution, examining the development of legislation on minimum unit pricing (MUP) in Wales, following on from Scotland's earlier policy decision. The study draws on a range of sources, including primary documents, media coverage, and interviews with policy participants. Our analysis identifies the importance of the specific character of Welsh political institutions, particularly the emphasis given to participation and consultation in policymaking. In the case of MUP, we document a process of policy-oriented learning, where policymakers made a concerted effort to draw on an assortment of expertise and experiences, including but not limited to the Scottish model. We also find that the Welsh public health policy community was well placed to support the framing of MUP and to address limitations in policy capacity. The findings hold implications for future studies of learning, devolution, and alcohol policy more generally.

The networked micro-decision context: a new lens on transformative urban governance.

Recent large-scale societal disruptions, from the COVID-19 pandemic to intensifying wildfires and weather events, reveal the importance of transforming governance systems so they can address complex, transboundary, and rapidly evolving crises. Yet current knowledge of the decision-making dynamics that yield transformative governance remains scant. Studies typically focus on the aggregate outputs of government decisions, while overlooking their micro-level underpinnings. This is a key oversight because drivers of policy change, such as learning or competition, are prosecuted by people rather than organizations. We respond to this knowledge gap by introducing a new analytical lens for understanding policymaking, aimed at uncovering how characteristics of decision-makers and the structure of their relationships affect their likelihood of effectuating transformative policy responses. This perspective emphasizes the need for a more dynamic and relational view on urban governance in the context of transformation.

Policy learning and change during crisis: COVID-19 policy responses across six states.

Whereas policy change is often characterized as a gradual and incremental process, effective crisis response necessitates that organizations adapt to evolving problems in near real time. Nowhere is this dynamic more evident than in the case of COVID-19, which forced subnational governments to constantly adjust and recalibrate public health and disease mitigation measures in the face of changing patterns of viral transmission and the emergence of new information. This study assesses (a) the extent to which subnational policies changed over the course of the pandemic; (b) whether these changes are emblematic of policy learning; and (c) the drivers of these changes, namely changing political and public health conditions. Using a novel dataset analyzing each policy's content, including its timing of enactment, substantive focus, stringency, and similar variables, results indicate the pandemic response varied significantly across states. The states examined were responsive to both changing public health and political conditions. This study identifies patterns of preemptive policy learning, which denotes learning in anticipation of an emerging hazard. In doing so, the study provides important insights into the dynamics of policy learning and change during disaster.

Mientras que el cambio de política a menudo se caracteriza como un proceso gradual e incremental, la respuesta efectiva a la crisis requiere que las organizaciones se adapten a los problemas en evolución casi en tiempo real. En ninguna parte esta dinámica es más evidente que en el caso de COVID­19, que obligó a los gobiernos subnacionales a ajustar y recalibrar constantemente las medidas de salud pública y mitigación de enfermedades ante los patrones cambiantes de transmisión viral y la aparición de nueva información. Este estudio evalúa (a) la medida en que las políticas subnacionales cambiaron en el transcurso de la pandemia; (b) si estos cambios son emblemáticos del aprendizaje de políticas; y (c) los impulsores de estos cambios, a saber, las cambiantes condiciones políticas y de salud pública. Usando un nuevo conjunto de datos que analiza el contenido de cada política, incluido el momento de la promulgación, el enfoque sustantivo, el rigor y variables similares, los resultados indican que la respuesta a la pandemia varió significativamente entre los estados. Los estados examinados respondieron a cambios tanto en la salud pública como en las condiciones políticas. Este estudio identifica patrones de aprendizaje de políticas preventivas, lo que denota aprendizaje en previsión de un peligro emergente. Al hacerlo, el estudio proporciona información importante sobre la dinámica del aprendizaje y el cambio de políticas durante un desastre.

Policy by Pilot? Learning From Demonstration Projects for Integrated Care Comment on "Integration or Fragmentation of Health Care? Examining Policies and Politics in a Belgian Case Study".

Analysis of policy implementation for chronic disease in Belgium highlights the difficulties of launching experiments for integrated care in a health system with fragmented governance. It also entreats us to consider the inherent challenges of piloting integrated care for chronic disease. Sociomedical characteristics of chronic disease -political, social, and economic aspects of improving outcomes - pose distinct problems for pilot projects, particularly because addressing health inequity requires collaboration across health and social sectors and a long-term, life-course perspective on health. Drawing on recent US experience with demonstration projects for health service delivery reform and on chronic disease research, I discuss constraints of and lessons from pilot projects. The policy learning from pilots lies beyond their technical evaluative yield. Pilot projects can evince political and social challenges to achieving integrated chronic disease care, and can illuminate overlooked perspectives, such as those of community-based organizations (CBOs), thereby potentially extending the terms of policy debate.

Hardware Efficient Direct Policy Imitation Learning for Robotic Navigation in Resource-Constrained Settings.

Direct policy learning (DPL) is a widely used approach in imitation learning for time-efficient and effective convergence when training mobile robots. However, using DPL in real-world applications is not sufficiently explored due to the inherent challenges of mobilizing direct human expertise and the difficulty of measuring comparative performance. Furthermore, autonomous systems are often resource-constrained, thereby limiting the potential application and implementation of highly effective deep learning models. In this work, we present a lightweight DPL-based approach to train mobile robots in navigational tasks. We integrated a safety policy alongside the navigational policy to safeguard the robot and the environment. The approach was evaluated in simulations and real-world settings and compared with recent work in this space. The results of these experiments and the efficient transfer from simulations to real-world settings demonstrate that our approach has improved performance compared to its hardware-intensive counterparts. We show that using the proposed methodology, the training agent achieves closer performance to the expert within the first 15 training iterations in simulation and real-world settings.

Seeking policy solutions in a complex system: experimentalist governance in China's healthcare reform.

The proliferation of "wicked" policy problems in complex systems requires an experimental approach of problem-solving. Experimentalist governance offers a conducive framework through which to seek policy solutions amidst high levels of complexity in a multilevel governance structure. This study conceptualizes four distinctive experimental modalities based on varying levels of technical complexity and interest complexity, both of which represent salient constraints for policy reforms in a complex system, the health sector in particular. Trail-blazing pilots, crowdsourcing pilots, managed pilots, and road-testing pilots are all associated with distinct mechanisms of experimentation in a multilevel governance structure. Through four illustrative cases from China's massive experimental program of public hospital reform, this study demonstrates how experimentalist governance seeks policy solutions in the health sector. Should governance arrangements, policy capacity, pragmatism, and informational devices become aligned in a conducive way, experimentalist governance can play an instrumental role in seeking solutions for difficult problems in a complex policy system. A governance structure capable of policy learning and adaptive management is the key.

Learning multi-agent cooperation.

Advances in reinforcement learning (RL) have resulted in recent breakthroughs in the application of artificial intelligence (AI) across many different domains. An emerging landscape of development environments is making powerful RL techniques more accessible for a growing community of researchers. However, most existing frameworks do not directly address the problem of learning in complex operating environments, such as dense urban settings or defense-related scenarios, that incorporate distributed, heterogeneous teams of agents. To help enable AI research for this important class of applications, we introduce the AI Arena: a scalable framework with flexible abstractions for associating agents with policies and policies with learning algorithms. Our results highlight the strengths of our approach, illustrate the importance of curriculum design, and measure the impact of multi-agent learning paradigms on the emergence of cooperation.

In trust we trust: The impact of trust in government on excess mortality during the COVID-19 pandemic.

The COVID-19 pandemic has brought forward myriad challenges to public policy, central of which is understanding the different contextual factors that can influence the effectiveness of policy responses across different systems. In this article, we explore how trust in government can influence the ability of COVID-19 policy responses to curb excess mortality during the pandemic. Our findings indicate that stringent policy responses play a central role in curbing excess mortality. They also indicate that such relationship is not only influenced by systematic and structural factors, but also by citizens' trust in government. We leverage our findings to propose a set of recommendations for policymakers on how to enhance crisis policymaking and strengthen the designs of the widely used underlying policy learning processes.

Data-driven Interpretable Policy Construction for Personalized Mobile Health.

To promote healthy behaviors, many mobile health applications provide message-based interventions, such as tips, motivational messages, or suggestions for healthy activities. Ideally, the intervention policies should be carefully designed so that users obtain the benefits without being overwhelmed by overly frequent messages. As part of the HeartSteps physical-activity intervention, users receive messages intended to disrupt sedentary behavior. HeartSteps uses an algorithm to uniformly spread out the daily message budget over time, but does not attempt to maximize treatment effects. This limitation motivates constructing a policy to optimize the message delivery decisions for more effective treatments. Moreover, the learned policy needs to be interpretable to enable behavioral scientists to examine it and to inform future theorizing. We address this problem by learning an effective and interpretable policy that reduces sedentary behavior. We propose Optimal Policy Trees + (OPT+), an innovative batch off-policy learning method, that combines a personalized threshold learning and an extension of Optimal Policy Trees under a budget-constrained setting. We implement and test the method using data collected in HeartSteps V2/V3. Computational results demonstrate a significant reduction in sedentary behavior with a lower delivery budget. OPT+ produces a highly interpretable and stable output decision tree thus enabling theoretical insights to guide future research.

Benchmarking for accountability on obesity prevention: evaluation of the Healthy Food Environment Policy Index (Food-EPI) in Australia (2016-2020).

OBJECTIVE: Despite broad agreement on the need for comprehensive policy action to improve the healthiness of food environments, implementation of recommended policies has been slow and fragmented. Benchmarking is increasingly being used to strengthen accountability for action. However, there have been few evaluations of benchmarking and accountability initiatives to understand their contribution to policy change. This study aimed to evaluate the impact of the Healthy Food Environment Policy Index (Food-EPI) Australia initiative (2016-2020) that assessed Australian governments on their progress in implementing recommended policies for improving food environments. DESIGN: A convergent mixed methods approach was employed incorporating data from online surveys (conducted in 2017 and 2020) and in-depth semi-structured interviews (conducted in 2020). Data were analysed against a pre-defined logic model. SETTING: Australia. PARTICIPANTS: Interviews: twenty stakeholders (sixteen government, four non-government). Online surveys: fifty-three non-government stakeholders (52 % response rate) in 2017; thirty-four non-government stakeholders (36 % response rate) in 2020. RESULTS: The Food-EPI process involved extensive engagement with government officials and the broader public health community across Australia. Food-EPI Australia was found to support policy processes, including as a tool to increase knowledge of good practice, as a process for collaboration and as an authoritative reference to support policy decisions and advocacy strategies. CONCLUSIONS: Key stakeholders involved in the Food-EPI Australia process viewed it as a valuable initiative that should be repeated to maximise its value as an accountability mechanism. The highly collaborative nature of the initiative was seen as a key strength that could inform design of other benchmarking processes.

The COVID-19 exams fiasco across the UK: four nations and two windows of opportunity.

All four UK and devolved governments performed a 'U-turn' on their COVID-19 school exams replacement policies. After cancelling exams, they sought teacher estimates on their grades, but supported an algorithm to standardise the results. When the results produced a public outcry over unfair consequences, they initially defended their decision but reverted quickly to teacher assessment. We explain these developments by comparing two 'windows of opportunity' overseen by four separate governments, in which the definition of the problem, feasibility of each solution, and motive of policymakers to select one over the other lurched dramatically within a week of the exams results. These experiences highlight the confluence of events and choices and the timing and order of choice. A policy solution that had been rejected during the first window, and would have been criticised heavily if chosen first, became a lifeline during the second. As such, while it is important to understand why the standardisation process went so wrong, we focus on why the policymaking process went so wrong. Supplementary Information: The online version contains supplementary material available at 10.1057/s41293-021-00162-y.

Learning parametric policies and transition probability models of markov decision processes from data.

We consider the problem of estimating the policy and transition probability model of a Markov Decision Process from data (state, action, next state tuples). The transition probability and policy are assumed to be parametric functions of a sparse set of features associated with the tuples. We propose two regularized maximum likelihood estimation algorithms for learning the transition probability model and policy, respectively. An upper bound is established on the regret, which is the difference between the average reward of the estimated policy under the estimated transition probabilities and that of the original unknown policy under the true (unknown) transition probabilities. We provide a sample complexity result showing that we can achieve a low regret with a relatively small amount of training samples. We illustrate the theoretical results with a healthcare example and a robot navigation experiment.

Model free adaptive control of the under-actuated robot manipulator with the chaotic dynamics.

Development of practical control approaches for the under-actuated chaotic systems such as the robot manipulators are challenging due to the unpredictable character of the chaotic dynamics, and the inevitable real-time application properties like delays, saturations, and uncertainties In this paper, we propose a model free digital adaptive control approach, which considers the time delay of the control signal, actuator saturation, and non-parametric uncertainties, for an under-actuated manipulator. We also develop a chaos control to learn the unbiased and smooth digital control policy inside the chaotic regions of the continuous time under-actuated manipulator. We perform real-time experiments in a dynamic environment with the proposed digital adaptive control. Then we compare the results of the learning and control with and without chaos control. We observe that the proposed model free adaptive control approach can accurately learn both the long-term predictor and unbiased control policy even in the chaotic regions of the under-actuated robot manipulator.