Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification.

Large vision-language models, such as Contrastive Vision-Language Pre-training (CLIP), pre-trained on large-scale image-text datasets, have demonstrated robust zero-shot transfer capabilities across various downstream tasks. To further enhance the few-shot recognition performance of CLIP, Tip-Adapter augments the CLIP model with an adapter that incorporates a key-value cache model constructed from the few-shot training set. This approach enables training-free adaptation and has shown significant improvements in few-shot recognition, especially with additional fine-tuning. However, the size of the adapter increases in proportion to the number of training samples, making it difficult to deploy in practical applications. In this paper, we propose a novel CLIP adaptation method, named Proto-Adapter, which employs a single-layer adapter of constant size regardless of the amount of training data and even outperforms Tip-Adapter. Proto-Adapter constructs the adapter's weights based on prototype representations for each class. By aggregating the features of the training samples, it successfully reduces the size of the adapter without compromising performance. Moreover, the performance of the model can be further enhanced by fine-tuning the adapter's weights using a distance margin penalty, which imposes additional inter-class discrepancy to the output logits. We posit that this training scheme allows us to obtain a model with a discriminative decision boundary even when trained with a limited amount of data. We demonstrate the effectiveness of the proposed method through extensive experiments of few-shot classification on diverse datasets.

Secrets of Event-based Optical Flow, Depth and Ego-motion Estimation by Contrast Maximization.

Event cameras respond to scene dynamics and provide signals naturally suitable for motion estimation with advantages, such as high dynamic range. The emerging field of event-based vision motivates a revisit of fundamental computer vision tasks related to motion, such as optical flow and depth estimation. However, state-of-the-art event-based optical flow methods tend to originate in frame-based deep-learning methods, which require several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate dense optical flow, depth, and ego-motion from events alone. The proposed method sensibly models the space-time properties of event data and tackles the event alignment problem. It designs the objective function to prevent overfitting, deals better with occlusions, and improves convergence using a multi-scale approach. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark and is competitive on the DSEC benchmark. Moreover, it allows us to simultaneously estimate dense depth and ego-motion, exposes the limitations of current flow benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Along with various downstream applications shown, we hope the proposed method becomes a cornerstone on event-based motion-related tasks. Code is available at https://github.com/tub-rip/event_based_optical_flow.

Synthetic Document Images with Diverse Shadows for Deep Shadow Removal Networks.

Shadow removal for document images is an essential task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large, diverse dataset for document shadow removal takes time and effort. Thus, only small real datasets are available. Graphic renderers have been used to synthesize shadows to create relatively large datasets. However, the limited number of unique documents and the limited lighting environments adversely affect the network performance. This paper presents a large-scale, diverse dataset called the Synthetic Document with Diverse Shadows (SynDocDS) dataset. The SynDocDS comprises rendered images with diverse shadows augmented by a physics-based illumination model, which can be utilized to obtain a more robust and high-performance deep shadow removal network. In this paper, we further propose a Dual Shadow Fusion Network (DSFN). Unlike natural images, document images often have constant background colors requiring a high understanding of global color features for training a deep shadow removal network. The DSFN has a high global color comprehension and understanding of shadow regions and merges shadow attentions and features efficiently. We conduct experiments on three publicly available datasets, the OSR, Kligler's, and Jung's datasets, to validate our proposed method's effectiveness. In comparison to training on existing synthetic datasets, our model training on the SynDocDS dataset achieves an enhancement in the PSNR and SSIM, increasing them from 23.00 dB to 25.70 dB and 0.959 to 0.971 on average. In addition, the experiments demonstrated that our DSFN clearly outperformed other networks across multiple metrics, including the PSNR, the SSIM, and its impact on OCR performance.

Event-Based Background-Oriented Schlieren.

Schlieren imaging is an optical technique to observe the flow of transparent media, such as air or water, without any particle seeding. However, conventional frame-based techniques require both high spatial and temporal resolution cameras, which impose bright illumination and expensive computation limitations. Event cameras offer potential advantages (high dynamic range, high temporal resolution, and data efficiency) to overcome such limitations due to their bio-inspired sensing principle. This article presents a novel technique for perceiving air convection using events and frames by providing the first theoretical analysis that connects event data and schlieren. We formulate the problem as a variational optimization one combining the linearized event generation model with a physically-motivated parameterization that estimates the temporal derivative of the air density. The experiments with accurately aligned frame- and event camera data reveal that the proposed method enables event cameras to obtain on par results with existing frame-based optical flow techniques. Moreover, the proposed method works under dark conditions where frame-based schlieren fails, and also enables slow-motion analysis by leveraging the event camera's advantages. Our work pioneers and opens a new stack of event camera applications, as we publish the source code as well as the first schlieren dataset with high-quality frame and event data.

Action Quality Assessment Model Using Specialists' Gaze Location and Kinematics Data-Focusing on Evaluating Figure Skating Jumps.

Action quality assessment (AQA) tasks in computer vision evaluate action quality in videos, and they can be applied to sports for performance evaluation. A typical example of AQA is predicting the final score from a video that captures an entire figure skating program. However, no previous studies have predicted individual jump scores, which are of great interest to competitors because of the high weight of competition. Despite the presence of unnecessary information in figure skating videos, human specialists can focus and reduce information when they evaluate jumps. In this study, we clarified the eye movements of figure skating judges and skaters while evaluating jumps and proposed a prediction model for jump performance that utilized specialists' gaze location to reduce information. Kinematic features obtained from the tracking system were input into the model in addition to videos to improve accuracy. The results showed that skaters focused more on the face, whereas judges focused on the lower extremities. These gaze locations were applied to the model, which demonstrated the highest accuracy when utilizing both specialists' gaze locations. The model outperformed human predictions and the baseline model (RMSE:0.775), suggesting a combination of human specialist knowledge and machine capabilities could yield higher accuracy.

Association of Psychological Readiness to Return to Sports With Subjective Level of Return at 12 Months After ACL Reconstruction.

Background: Return-to-sports (RTS) rates after anterior cruciate ligament (ACL) reconstruction (ACLR) differ according to the level at which patients return. It is unclear whether the level of RTS is affected by psychological readiness to return. Purpose: To examine the association between psychological readiness to RTS and subjective RTS level 12 months after ACLR. Study Design: Case-control study; Level of evidence, 3. Methods: A total of 47 patients who underwent unilateral primary ACLR surgery were enrolled. Assessments at 6 and 12 months postoperatively consisted of knee strength testing (isokinetic quadriceps and hamstring strength), the International Knee Documentation Committee Subjective Knee Evaluation Form (IKDC-SKF), and the Anterior Cruciate Ligament-Return to Sport after Injury (ACL-RSI) scale to measure psychological readiness to RTS. Patients were assigned to 1 of 3 subgroups based on their subjective assessment of RTS level at 12 months postoperatively: RTS at or above preinjury level (RTS≥Pre; n = 19), RTS below preinjury level (RTS

Boosting Semantic Segmentation by Conditioning the Backbone with Semantic Boundaries.

In this paper, we propose the Semantic-Boundary-Conditioned Backbone (SBCB) framework, an effective approach to enhancing semantic segmentation performance, particularly around mask boundaries, while maintaining compatibility with various segmentation architectures. Our objective is to improve existing models by leveraging semantic boundary information as an auxiliary task. The SBCB framework incorporates a complementary semantic boundary detection (SBD) task with a multi-task learning approach. It enhances the segmentation backbone without introducing additional parameters during inference or relying on independent post-processing modules. The SBD head utilizes multi-scale features from the backbone, learning low-level features in early stages and understanding high-level semantics in later stages. This complements common semantic segmentation architectures, where features from later stages are used for classification. Extensive evaluations using popular segmentation heads and backbones demonstrate the effectiveness of the SBCB. It leads to an average improvement of 1.2% in IoU and a 2.6% gain in the boundary F-score on the Cityscapes dataset. The SBCB framework also improves over- and under-segmentation characteristics. Furthermore, the SBCB adapts well to customized backbones and emerging vision transformer models, consistently achieving superior performance. In summary, the SBCB framework significantly boosts segmentation performance, especially around boundaries, without introducing complexity to the models. Leveraging the SBD task as an auxiliary objective, our approach demonstrates consistent improvements on various benchmarks, confirming its potential for advancing the field of semantic segmentation.

Outdoor Vision-and-Language Navigation Needs Object-Level Alignment.

In the field of embodied AI, vision-and-language navigation (VLN) is a crucial and challenging multi-modal task. Specifically, outdoor VLN involves an agent navigating within a graph-based environment, while simultaneously interpreting information from real-world urban environments and natural language instructions. Existing outdoor VLN models predict actions using a combination of panorama and instruction features. However, these methods may cause the agent to struggle to understand complicated outdoor environments and ignore the details in the environments to fail to navigate. Human navigation often involves the use of specific objects as reference landmarks when navigating to unfamiliar places, providing a more rational and efficient approach to navigation. Inspired by this natural human behavior, we propose an object-level alignment module (OAlM), which guides the agent to focus more on object tokens mentioned in the instructions and recognize these landmarks during navigation. By treating these landmarks as sub-goals, our method effectively decomposes a long-range path into a series of shorter paths, ultimately improving the agent's overall performance. In addition to enabling better object recognition and alignment, our proposed OAlM also fosters a more robust and adaptable agent capable of navigating complex environments. This adaptability is particularly crucial for real-world applications where environmental conditions can be unpredictable and varied. Experimental results show our OAlM is a more object-focused model, and our approach outperforms all metrics on a challenging outdoor VLN Touchdown dataset, exceeding the baseline by 3.19% on task completion (TC). These results highlight the potential of leveraging object-level information in the form of sub-goals to improve navigation performance in embodied AI systems, paving the way for more advanced and efficient outdoor navigation.

Reactive Azlactone Intermediate Drives Fungal Secondary Metabolite Cross-Pathway Generation.

Pathogenic fungi of Aspergillus section Fumigati are known to produce various secondary metabolites. A reported isolation of a compound with an atypical carbon skeleton called fumimycin from A. fumisynnematus prompted us to examine a related fungus, A. lentulus, for production of similar products. Here we report the isolation of fumimycin and a related new racemic compound we named lentofuranine. Detailed analyses revealed that both compounds were assembled by a nonenzymatic condensation of a polyketide intermediate from the terrein biosynthetic pathway and a highly reactive azlactone intermediate produced by an unrelated nonribosomal peptide synthetase carrying a terminal condensation-like domain. While highly reactive azlactone is commonly used in chemical synthesis, its production by a conventional non-metalloenzyme and employment as a biosynthetic pathway intermediate is unprecedented. The observed unusual carbon skeleton formation is likely due to the reactivity of azlactone. Our finding provides another example of a chemical principle being aptly exploited by a biological system.

Improvements in asymmetry in knee flexion motion during landing are associated with the postoperative period and quadriceps strength after anterior cruciate ligament reconstruction.

This study investigated the relationship between quadriceps strength and knee kinematics during a drop vertical jump (DVJ) at 6, 9 and 12 months after anterior cruciate ligament reconstruction (ACLR) in 9 male and 22 female athletes (16.6 ± 2.1 years old). Isokinetic quadriceps strength was measured by a dynamometer (Biodex System 3). Knee flexion excursion was assessed using two-dimensional analysis. Knee flexion excursion at 6 months was significantly smaller in the involved limb than in the uninvolved limb independent of quadriceps strength (56.7° ± 9.3°, 63.4° ± 11.4°, P < 0.001). At 9 months, only the low quadriceps strength group demonstrated a similar interlimb difference (57.2°± 12.3°, 63.3° ± 10.5°, P < 0.001). At 12 months, there was no significant interlimb difference in knee flexion excursion regardless of quadriceps strength. These findings indicate that restoration in symmetrical knee flexion excursion during a DVJ requires rehabilitation as well as quadriceps strength.

Psychological readiness at 9 months after anterior cruciate ligament reconstruction -which factors affect?

OBJECTIVE: To investigate the association of psychological readiness at 9 months after anterior cruciate ligament reconstruction (ACLR) with knee strength and range-of-motion (ROM) at 3 and 9 months postoperatively. DESIGN: Retrospective cohort study. SETTING: Private orthopedic hospital. PARTICIPANTS: Seventy-eight patients after ACLR. MAIN OUTCOMES MEASURES: The participants completed the anterior cruciate ligament-return to sport after injury (ACL-RSI) scale as an assessment of psychological readiness at 9 months after ACLR. Isometric and isokinetic quadriceps and hamstring strength were measured by a dynamometer at 3 and 9 months postoperatively. Knee extension and flexion ROM were measured in 5° increments with a standard goniometer at 3 and 9 months postoperatively. RESULTS: Univariate regression analysis showed that age, limb symmetry index (LSI) of quadriceps strength at 3 and 9 months, and knee extension ROM deficit (>5°) at 3 and 9 months were associated with ACL-RSI scores at 9 months (P < 0.05). Age and LSI of quadriceps strength at 3 months remained significant predictors of ACL-RSI scores at 9 months in multiple regression analysis (R2 = 0.20, P < 0.001). CONCLUSIONS: Early improvements in quadriceps strength after ACLR may have a positive impact on psychological readiness at 9 months postoperatively.

Rate of Torque Development in the Quadriceps after Anterior Cruciate Ligament Reconstruction with Hamstring Tendon Autografts in Young Female Athletes.

This study aims to compare the limb symmetry index (LSI) of the rate of torque development (RTD) of the quadriceps with that of the peak torque after anterior cruciate ligament reconstruction (ACLR) using semitendinosus and gracilis tendon (STG) autografts and to investigate the associations of the LSI of torque parameters with patient-reported knee function. The participants included 23 female athletes after ACLR with STG grafts. Isometric quadriceps tests were performed using an isokinetic dynamometer. The peak torque, RTD100 (0 to 100 ms) and RTD200 (100 to 200 ms) were determined using torque-time curves. Comparisons of the LSI of torque parameters was performed by ANOVA. Univariate regression analysis was used to examine the relationship between the LSI of torque parameters and the patient-reported knee function. The LSIs of the peak torque and RTD200 were significantly lower than that of the RTD100 (p = 0.049, p = 0.039, respectively). Regression analysis showed that the LSI of the peak torque was associated with the patient-reported knee function (R2 = 0.40, p = 0.001). It would be useful to evaluate the peak torque in young female athletes under the age of 18 and at 8-10 months after ACLR with STG grafts.

Detection of Osteoarthritis from Multimodal Hand Data.

Osteoarthritis (OA) describes a degenerative joint disorder that is prevalent among older people and typically results in swollen and inflamed joints. The aim of this paper is to develop a method using images, videos and thermal data of 100 patients taken at Keio University Hospital to detect OA in hands. By using hand pose estimation on the video data, joint angles can be calculated and subsequently transformed into feature vectors. For the thermal and RGB images, hand keypoint detectors were trained to identify and crop the appropriate joints within the images. The resulting extracted features are combined and further trained on Support Vector Machines and Convolutional Neural Networks to obtain the final binary classification for each joint. While the proposed method generally shows favorable accuracy and F1-scores on the Proximal (PIP) and Distal Interphalangeal (DIP) joints, the performance on the Metacarpophalangeal (MCP) joints is limited by the low occurrence of affected joints in the dataset. We further compare the different modalities and found that, apart from the combined approach, using video data provides the best results. Clinical Relevance- The proposed method shows promising first results for the usage of visual and thermal data in combination with machine learning in order to detect OA.

Event Collapse in Contrast Maximization Frameworks.

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space-time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.

Non-Deep Active Learning for Deep Neural Networks.

One way to improve annotation efficiency is active learning. The goal of active learning is to select images from many unlabeled images, where labeling will improve the accuracy of the machine learning model the most. To select the most informative unlabeled images, conventional methods use deep neural networks with a large number of computation nodes and long computation time, but we propose a non-deep neural network method that does not require any additional training for unlabeled image selection. The proposed method trains a task model on labeled images, and then the model predicts unlabeled images. Based on this prediction, an uncertainty indicator is generated for each unlabeled image. Images with a high uncertainty index are considered to have a high information content, and are selected for annotation. Our proposed method is based on a very simple and powerful idea: select samples near the decision boundary of the model. Experimental results on multiple datasets show that the proposed method achieves higher accuracy than conventional active learning methods on multiple tasks and up to 14 times faster execution time from 1.2 × 106 s to 8.3 × 104 s. The proposed method outperforms the current SoTA method by 1% accuracy on CIFAR-10.

Efficient Transformer-Based Compressed Video Modeling via Informative Patch Selection.

Recently, Transformer-based video recognition models have achieved state-of-the-art results on major video recognition benchmarks. However, their high inference cost significantly limits research speed and practical use. In video compression, methods considering small motions and residuals that are less informative and assigning short code lengths to them (e.g., MPEG4) have successfully reduced the redundancy of videos. Inspired by this idea, we propose Informative Patch Selection (IPS), which efficiently reduces the inference cost by excluding redundant patches from the input of the Transformer-based video model. The redundancy of each patch is calculated from motions and residuals obtained while decoding a compressed video. The proposed method is simple and effective in that it can dynamically reduce the inference cost depending on the input without any policy model or additional loss term. Extensive experiments on action recognition demonstrated that our method could significantly improve the trade-off between the accuracy and inference cost of the Transformer-based video model. Although the method does not require any policy model or additional loss term, its performance approaches that of existing methods that do require them.

Determinant analysis and developing evaluation indicators of grade of execution score of double axel jump in figure skating.

A figure skating jump score is determined by the sum of the base value based on the difficulty and grade of execution (GOE) that indicates the performance quality. Therefore, performing a high-quality jump to obtain a high GOE is essential to win a competition. However, the relationship between the GOE and kinematic parameters remains unclear. We analysed the horizontal distance, vertical height, and landing speed of double axel jumps in the Ladies' Short Program at the 2019 World Championships. The highest GOE group had significantly larger horizontal distances than the middle and lower groups, while the landing speed and vertical height were not significantly different. A principal component regression analysis was conducted to clarify the contrast between the three variables affecting the GOE. The results showed that greater horizontal distance and landing speed compared to vertical height (component 1) and greater horizontal distance compared to landing speed (component 3) contributed to higher GOE. We divided skaters into four clusters using these two components and provided general GOE acquisition strategies for each cluster. Finally, to apply our results to the industry, we proposed two new evaluation indicators which are highly correlated with the two components and easy to interpret.

Torque-angle curve of the knee flexors in athletes with a prior history of hamstring strain.

OBJECTIVES: To investigate the knee flexor torque-angle curve after hamstring strain injury using different muscle action types and angular velocities. DESIGN: Cross-sectional. SETTING: Controlled laboratory. PARTICIPANTS: Thirteen collegiate athletes injured hamstring strain (21.0 ± 0.8 years; 173.9 ± 6.5 cm; 70.1 ± 10.5 kg). MAIN OUTCOME MEASURES: Concentric and eccentric knee flexor torque was measured at 60 & 300°/sec. Peak torque and average torque every 10° were determined from torque-angle curve and injured side was compared with non-injured side. RESULTS: No significant differences were found in the concentric muscle actions. However, the eccentric peak torque was significantly lower on the injured side at 60°/sec (p = 0.048) and at 300°/sec (p = 0.002). The average eccentric torque was significantly lower on the injured side at 60°/sec from 10° to 20° of knee flexion (p = 0.012-0.018) and at 300°/sec from 10° to 60° of knee flexion (p = 0.005-0.049). CONCLUSION: The knee flexor torque-angle curve changes with eccentric muscle action after hamstring injury. Eccentric torque declines were close to full knee extension at 60°/sec and a wide range of knee flexion at 300°/sec. The assessment and rehabilitation of eccentric hamstring strength may be important to consider the effect of the angular velocity after hamstring strain injury.

Dynamic postural control correlates with activities of daily living and quality of life in patients with knee osteoarthritis.

BACKGROUND: Knee osteoarthritis (OA) negatively affects dynamic postural control, which is a basic function that individuals use to perform activities of daily living (ADL). The purpose of this study was to investigate the associations of center of pressure (COP) control during the transition from double-leg to single-leg standing with subjective assessments of ADL and quality of life (QOL) in patients with knee OA. METHODS: Thirty-six patients (29 females) with moderate-to-severe knee OA participated. Dynamic postural control was evaluated during the transition from double-leg to single-leg standing. Each patient stood on a force plate, lifted the less affected limb as fast as possible, and maintained single-leg standing with the more affected limb. The COP movements corresponding to anticipatory postural adjustment (APA) and transitional phases were assessed. The maximum displacement and peak velocity of the COP movements in the medial-lateral direction were calculated. The Knee Injury and Osteoarthritis Outcome Score (KOOS) was used for the subjective assessment of ADL and QOL. Pearson's product correlation analysis was performed to investigate the associations of COP movements in the APA and transitional phases with KOOS-ADL and KOOS-QOL. RESULTS: In the APA phase, the maximum COP displacement was significantly correlated with KOOS-ADL (r = -0.353, P = 0.035) and KOOS-QOL (r = -0.379, P = 0.023). In the transitional phase, the maximum COP displacement and peak COP velocity were significantly correlated with KOOS-ADL (maximum displacement: r = 0.352, P = 0.035; peak velocity: r = 0.438, P = 0.008) and with KOOS-QOL (maximum displacement: r = 0.357, P = 0.032; peak velocity: r = 0.343, P = 0.040). CONCLUSIONS: The present study showed that smaller COP movements in the APA phase and smaller and slower COP movements in the transitional phase correlated with poorer ADL and QOL conditions in patients with knee OA. These findings suggest that poor dynamic postural control is associated with poor ADL and QOL conditions in patients with moderate-to-severe medial knee OA. Conservative treatment for patients with knee OA may need to focus on dynamic postural control during the transition from double-leg to single-leg standing.

Optical Flow Estimation by Matching Time Surface with Event-Based Cameras.

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.