new

Get trending papers in your email inbox!

Subscribe

Daily Papers

byAK and the research community

Jul 20

High-Fidelity Digital Twin Dataset Generation for Inverter-Based Microgrids Under Multi-Scenario Disturbances

Public power-system datasets often lack electromagnetic transient (EMT) waveforms, inverter control dynamics, and diverse disturbance coverage, which limits their usefulness for training surrogate models and studying cyber-physical behavior in inverter-based microgrids. This paper presents a high-fidelity digital twin dataset generated from a MATLAB/Simulink EMT model of a low-voltage AC microgrid with ten inverter-based distributed generators. The dataset records synchronized three-phase PCC voltages and currents, per-DG active power, reactive power, and frequency, together with embedded scenario labels, producing 38 aligned channels sampled at Δt = 2~μs over T = 1~s (N = 500{,}001 samples) per scenario. Eleven operating and disturbance scenarios are included: normal operation, load step, voltage sag (temporary three-phase fault), load ramp, frequency ramp, DG trip, tie-line trip, reactive power step, single-line-to-ground faults, measurement noise injection, and communication delay. To ensure numerical stability without altering sequence length, invalid samples (NaN, Inf, and extreme outliers) are repaired using linear interpolation. Each scenario is further validated using system-level evidence from mean frequency, PCC voltage magnitude, total active power, voltage unbalance, and zero-sequence current to confirm physical observability and correct timing. The resulting dataset provides a consistent, labeled EMT benchmark for surrogate modeling, disturbance classification, robustness testing under noise and delay, and cyber-physical resilience analysis in inverter-dominated microgrids. The dataset and processing scripts will be released upon acceptance

  • 3 authors
·
Mar 9

Repeating Nuclear Transients as Candidate Electromagnetic Counterparts of LISA Extreme Mass Ratio Inspirals

Extreme-mass-ratio inspirals (EMRIs) are one of the primary targets for the recently adopted millihertz gravitational-wave (GW) observatory LISA. Some previous studies have argued that a fraction of all EMRIs form in matter-rich environments, and can potentially explain the dozens of soft X-ray band (sim 10^{-1} rm keV), low-frequency (sim 0.1 mHz) periodic phenomena known as quasi-periodic eruptions (QPEs) and quasi-periodic oscillations (QPOs). Here, using a representative EMRI population retrofitted with cutoffs on LISA-band SNRs and luminosity distances to account for the sensitivity of current instruments, we estimate the mean frequency band in which QPEs and QPOs originating from detectable LISA EMRIs may be emitting an X-ray signal ``today'' (i.e., in 2024) to be 0.46 pm 0.22 mHz. We also model the well-known QPO source, RE J1034+396, which falls in this frequency band, as an EMRI assuming its primary black hole mass to be 10^6-10^7 M_odot. Through a prior-predictive analysis, we estimate the orbiting compact object's mass to be 46^{+ 10}_{-40} M_odot and the source's LISA-band SNR as approx 14, highlighting it as a candidate multi-messenger EMRI target. We also highlight the role of current and near-future X-ray and UV observatories in enabling multi-messenger observations of EMRIs in conjunction with LISA, and conclude with a discussion of caveats of the current analysis, such as the exclusion of eccentricity and inclination from the model, and the measurability of sub-solar mass compact object EMRIs.

  • 5 authors
·
Apr 1, 2024

Cross-Frequency Collaborative Training Network and Dataset for Semi-supervised First Molar Root Canal Segmentation

Root canal (RC) treatment is a highly delicate and technically complex procedure in clinical practice, heavily influenced by the clinicians' experience and subjective judgment. Deep learning has made significant advancements in the field of computer-aided diagnosis (CAD) because it can provide more objective and accurate diagnostic results. However, its application in RC treatment is still relatively rare, mainly due to the lack of public datasets in this field. To address this issue, in this paper, we established a First Molar Root Canal segmentation dataset called FMRC-2025. Additionally, to alleviate the workload of manual annotation for dentists and fully leverage the unlabeled data, we designed a Cross-Frequency Collaborative training semi-supervised learning (SSL) Network called CFC-Net. It consists of two components: (1) Cross-Frequency Collaborative Mean Teacher (CFC-MT), which introduces two specialized students (SS) and one comprehensive teacher (CT) for collaborative multi-frequency training. The CT and SS are trained on different frequency components while fully integrating multi-frequency knowledge through cross and full frequency consistency supervisions. (2) Uncertainty-guided Cross-Frequency Mix (UCF-Mix) mechanism enables the network to generate high-confidence pseudo-labels while learning to integrate multi-frequency information and maintaining the structural integrity of the targets. Extensive experiments on FMRC-2025 and three public dental datasets demonstrate that CFC-MT is effective for RC segmentation and can also exhibit strong generalizability on other dental segmentation tasks, outperforming state-of-the-art SSL medical image segmentation methods. Codes and dataset will be released.

  • 6 authors
·
Apr 16, 2025

RADIANCE: Radio-Frequency Adversarial Deep-learning Inference for Automated Network Coverage Estimation

Radio-frequency coverage maps (RF maps) are extensively utilized in wireless networks for capacity planning, placement of access points and base stations, localization, and coverage estimation. Conducting site surveys to obtain RF maps is labor-intensive and sometimes not feasible. In this paper, we propose radio-frequency adversarial deep-learning inference for automated network coverage estimation (RADIANCE), a generative adversarial network (GAN) based approach for synthesizing RF maps in indoor scenarios. RADIANCE utilizes a semantic map, a high-level representation of the indoor environment to encode spatial relationships and attributes of objects within the environment and guide the RF map generation process. We introduce a new gradient-based loss function that computes the magnitude and direction of change in received signal strength (RSS) values from a point within the environment. RADIANCE incorporates this loss function along with the antenna pattern to capture signal propagation within a given indoor configuration and generate new patterns under new configuration, antenna (beam) pattern, and center frequency. Extensive simulations are conducted to compare RADIANCE with ray-tracing simulations of RF maps. Our results show that RADIANCE achieves a mean average error (MAE) of 0.09, root-mean-squared error (RMSE) of 0.29, peak signal-to-noise ratio (PSNR) of 10.78, and multi-scale structural similarity index (MS-SSIM) of 0.80.

  • 3 authors
·
Aug 21, 2023

Solving High Frequency and Multi-Scale PDEs with Gaussian Processes

Machine learning based solvers have garnered much attention in physical simulation and scientific computing, with a prominent example, physics-informed neural networks (PINNs). However, PINNs often struggle to solve high-frequency and multi-scale PDEs, which can be due to spectral bias during neural network training. To address this problem, we resort to the Gaussian process (GP) framework. To flexibly capture the dominant frequencies, we model the power spectrum of the PDE solution with a student t mixture or Gaussian mixture. We apply the inverse Fourier transform to obtain the covariance function (by Wiener-Khinchin theorem). The covariance derived from the Gaussian mixture spectrum corresponds to the known spectral mixture kernel. Next, we estimate the mixture weights in the log domain, which we show is equivalent to placing a Jeffreys prior. It automatically induces sparsity, prunes excessive frequencies, and adjusts the remaining toward the ground truth. Third, to enable efficient and scalable computation on massive collocation points, which are critical to capture high frequencies, we place the collocation points on a grid, and multiply our covariance function at each input dimension. We use the GP conditional mean to predict the solution and its derivatives so as to fit the boundary condition and the equation itself. As a result, we can derive a Kronecker product structure in the covariance matrix. We use Kronecker product properties and multilinear algebra to promote computational efficiency and scalability, without low-rank approximations. We show the advantage of our method in systematic experiments. The code is released at https://github.com/xuangu-fang/Gaussian-Process-Slover-for-High-Freq-PDE.

  • 6 authors
·
Nov 8, 2023

Beyond the Mean: Limit Theory and Tests for Infinite-Mean Autoregressive Conditional Durations

Integrated autoregressive conditional duration (ACD) models serve as natural counterparts to the well-known integrated GARCH models used for financial returns. However, despite their resemblance, asymptotic theory for ACD is challenging and also not complete, in particular for integrated ACD. Central challenges arise from the facts that (i) integrated ACD processes imply durations with infinite expectation, and (ii) even in the non-integrated case, conventional asymptotic approaches break down due to the randomness in the number of durations within a fixed observation period. Addressing these challenges, we provide here unified asymptotic theory for the (quasi-) maximum likelihood estimator for ACD models; a unified theory which includes integrated ACD models. Based on the new results, we also provide a novel framework for hypothesis testing in duration models, enabling inference on a key empirical question: whether durations possess a finite or infinite expectation. We apply our results to high-frequency cryptocurrency ETF trading data. Motivated by parameter estimates near the integrated ACD boundary, we assess whether durations between trades in these markets have finite expectation, an assumption often made implicitly in the literature on point process models. Our empirical findings indicate infinite-mean durations for all the five cryptocurrencies examined, with the integrated ACD hypothesis rejected -- against alternatives with tail index less than one -- for four out of the five cryptocurrencies considered.

  • 4 authors
·
May 9, 2025

FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement

Convolutional recurrent networks (CRN) integrating a convolutional encoder-decoder (CED) structure and a recurrent structure have achieved promising performance for monaural speech enhancement. However, feature representation across frequency context is highly constrained due to limited receptive fields in the convolutions of CED. In this paper, we propose a convolutional recurrent encoder-decoder (CRED) structure to boost feature representation along the frequency axis. The CRED applies frequency recurrence on 3D convolutional feature maps along the frequency axis following each convolution, therefore, it is capable of catching long-range frequency correlations and enhancing feature representations of speech inputs. The proposed frequency recurrence is realized efficiently using a feedforward sequential memory network (FSMN). Besides the CRED, we insert two stacked FSMN layers between the encoder and the decoder to model further temporal dynamics. We name the proposed framework as Frequency Recurrent CRN (FRCRN). We design FRCRN to predict complex Ideal Ratio Mask (cIRM) in complex-valued domain and optimize FRCRN using both time-frequency-domain and time-domain losses. Our proposed approach achieved state-of-the-art performance on wideband benchmark datasets and achieved 2nd place for the real-time fullband track in terms of Mean Opinion Score (MOS) and Word Accuracy (WAcc) in the ICASSP 2022 Deep Noise Suppression (DNS) challenge (https://github.com/alibabasglab/FRCRN).

  • 4 authors
·
Jun 15, 2022

Rethinking Brain Tumor Segmentation from the Frequency Domain Perspective

Precise segmentation of brain tumors, particularly contrast-enhancing regions visible in post-contrast MRI (areas highlighted by contrast agent injection), is crucial for accurate clinical diagnosis and treatment planning but remains challenging. However, current methods exhibit notable performance degradation in segmenting these enhancing brain tumor areas, largely due to insufficient consideration of MRI-specific tumor features such as complex textures and directional variations. To address this, we propose the Harmonized Frequency Fusion Network (HFF-Net), which rethinks brain tumor segmentation from a frequency-domain perspective. To comprehensively characterize tumor regions, we develop a Frequency Domain Decomposition (FDD) module that separates MRI images into low-frequency components, capturing smooth tumor contours and high-frequency components, highlighting detailed textures and directional edges. To further enhance sensitivity to tumor boundaries, we introduce an Adaptive Laplacian Convolution (ALC) module that adaptively emphasizes critical high-frequency details using dynamically updated convolution kernels. To effectively fuse tumor features across multiple scales, we design a Frequency Domain Cross-Attention (FDCA) integrating semantic, positional, and slice-specific information. We further validate and interpret frequency-domain improvements through visualization, theoretical reasoning, and experimental analyses. Extensive experiments on four public datasets demonstrate that HFF-Net achieves an average relative improvement of 4.48\% (ranging from 2.39\% to 7.72\%) in the mean Dice scores across the three major subregions, and an average relative improvement of 7.33% (ranging from 5.96% to 8.64%) in the segmentation of contrast-enhancing tumor regions, while maintaining favorable computational efficiency and clinical applicability. Code: https://github.com/VinyehShaw/HFF.

  • 8 authors
·
Jun 11, 2025

Comparison of Unsupervised Metrics for Evaluating Judicial Decision Extraction

The rapid advancement of artificial intelligence in legal natural language processing demands scalable methods for evaluating text extraction from judicial decisions. This study evaluates 16 unsupervised metrics, including novel formulations, to assess the quality of extracting seven semantic blocks from 1,000 anonymized Russian judicial decisions, validated against 7,168 expert reviews on a 1--5 Likert scale. These metrics, spanning document-based, semantic, structural, pseudo-ground truth, and legal-specific categories, operate without pre-annotated ground truth. Bootstrapped correlations, Lin's concordance correlation coefficient (CCC), and mean absolute error (MAE) reveal that Term Frequency Coherence (Pearson r = 0.540, Lin CCC = 0.512, MAE = 0.127) and Coverage Ratio/Block Completeness (Pearson r = 0.513, Lin CCC = 0.443, MAE = 0.139) best align with expert ratings, while Legal Term Density (Pearson r = -0.479, Lin CCC = -0.079, MAE = 0.394) show strong negative correlations. The LLM Evaluation Score (mean = 0.849, Pearson r = 0.382, Lin CCC = 0.325, MAE = 0.197) showed moderate alignment, but its performance, using gpt-4.1-mini via g4f, suggests limited specialization for legal textse. These findings highlight that unsupervised metrics, including LLM-based approaches, enable scalable screening but, with moderate correlations and low CCC values, cannot fully replace human judgment in high-stakes legal contexts. This work advances legal NLP by providing annotation-free evaluation tools, with implications for judicial analytics and ethical AI deployment.

  • 5 authors
·
Oct 2, 2025

Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning

Recent progress in Large Language Model (LLM) reasoning is increasingly driven by the refinement of post-training loss functions and alignment strategies. However, standard Reinforcement Learning (RL) paradigms like Group Relative Policy Optimization (GRPO) remain constrained by static uniformity: uniform prompt sampling and a fixed number of rollouts per prompt. For heterogeneous, heavy-tailed reasoning data, this creates structural inefficiencies that waste compute on already-solved patterns while under-training the long tail of hard problems. To address this, we propose Multi-Adversary Group Distributionally Robust Optimization (GDRO), an optimization-first framework that moves beyond uniform reasoning models by dynamically adapting the training distribution. We introduce an Online Difficulty Classifier that partitions prompts into dynamic pass@k difficulty groups. We then propose two independent GDRO games for post-training: (1) Prompt-GDRO, which employs an EMA-debiased multiplicative-weights bandit sampler to target the intensive difficulty margin and upweight persistently hard groups without frequency bias; and (2) Rollout-GDRO, which uses a shadow-price controller to reallocate rollouts across groups, maximizing gradient variance reduction on hard tasks under a fixed mean budget (compute-neutral). We provide no-regret guarantees for both controllers and additionally a variance-proxy analysis motivating a square-root optimal rollout allocation for Rollout-GDRO. We validate our framework on the DAPO 14.1k dataset using Qwen3-Base models. Prompt-GDRO and Rollout-GDRO achieve average relative gains of +10.6% and +10.1%, respectively, in pass@8 accuracy across 1.7B, 4B, and 8B scales compared to the GRPO baseline. Qualitative analysis shows an emergent curriculum: the adversaries shift resources to the evolving reasoning frontier, enhancing the reasoning model's performance.

tencent Tencent
·
Jan 27 2

Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation

Sentence embeddings are a foundational component for semantic search, clustering, classification, and retrieval-augmented generation. This paper presents embeddingmagibu-200m, a Turkish-focused sentence embedding model that produces 768-dimensional L2-normalized vectors and supports an 8,192-token context window, far exceeding the 512-token limit of earlier BERT-based Turkish encoders. Instead of full pretraining, an efficient three-stage adaptation pipeline is introduced: (1) construct a Turkish-optimized multilingual tokenizer with a 131,072 vocabulary by pruning redundant tokens from the teacher's vocabulary and incorporating multilingual tokens via frequency analysis on a 40-language corpus, (2) clone a teacher embedding model while preserving transformer backbone weights and initializing a compatible embedding table for the new vocabulary via mean-composition token mapping, and (3) perform offline embedding distillation from precomputed teacher vectors using a cosine similarity objective over a balanced 40-language Wikipedia corpus. The resulting student model contains approximately 200M parameters and trains in roughly four hours on a single GPU by avoiding online teacher inference during training, at a total cost of 5-20. Empirically, Pearson/Spearman correlations of 77.55%/77.45% are obtained on STSbTR, surpassing the 300M-parameter teacher model (73.84%/72.92%). On TR-MTEB (26 tasks), a mean score of 63.9% is achieved (7th out of 26 models), providing a competitive cost-quality trade-off with 33% fewer parameters than the teacher. To facilitate reproducibility and downstream use, all artifacts are released including model weights, tokenizer files, precomputed embedding datasets, and open-source cloning and distillation tooling.

magibu magibu
·
May 27 2

When to Personalize Household Object Search: A Rigidity-Gated Hybrid Policy

Service robots searching for household objects rely on spatial priors to reduce search cost, yet object locations can vary with resident traits. Collecting longitudinal, trait-specific in-home trajectories is invasive and hard to scale. We study when personalization helps and propose PerSim, a rigidity-gated hybrid policy that combines a trait-conditioned prior with a population-frequency baseline, personalizing only when placement behavior is variable. To scale resident-conditioned dynamics, we employ a human-calibrated simulation pipeline to generate and validate object-placement transitions in diverse home layouts, and train a predictor that injects continuous Big Five vectors to output room-level priors and within-room co-occurrence cues. In a unified human study (N=200), dual-layer validation shows that (i) synthetic transitions are judged behaviorally plausible (mean 3.85/5, p < 1e-6), and (ii) in a blinded A/B comparison, personalization is favored primarily for low-rigidity objects (p=0.005), while the population-frequency baseline remains strong for universally placed items, yielding a decision rule for when to personalize. In an offline objective test, we observe a small but significant improvement on unseen continuous trait vectors over nearest discrete configuration matching (p=0.035), supporting interpolation in five-dimensional trait space. Finally, in a home digital twin we show that PerSim reduces expected search cost by combining room visitation effort with within-room cue checking, demonstrating end-to-end gains beyond isolated prediction metrics.

  • 6 authors
·
Jul 1

Compound Estimation for Binomials

Many applications involve estimating the mean of multiple binomial outcomes as a common problem -- assessing intergenerational mobility of census tracts, estimating prevalence of infectious diseases across countries, and measuring click-through rates for different demographic groups. The most standard approach is to report the plain average of each outcome. Despite simplicity, the estimates are noisy when the sample sizes or mean parameters are small. In contrast, the Empirical Bayes (EB) methods are able to boost the average accuracy by borrowing information across tasks. Nevertheless, the EB methods require a Bayesian model where the parameters are sampled from a prior distribution which, unlike the commonly-studied Gaussian case, is unidentified due to discreteness of binomial measurements. Even if the prior distribution is known, the computation is difficult when the sample sizes are heterogeneous as there is no simple joint conjugate prior for the sample size and mean parameter. In this paper, we consider the compound decision framework which treats the sample size and mean parameters as fixed quantities. We develop an approximate Stein's Unbiased Risk Estimator (SURE) for the average mean squared error given any class of estimators. For a class of machine learning-assisted linear shrinkage estimators, we establish asymptotic optimality, regret bounds, and valid inference. Unlike existing work, we work with the binomials directly without resorting to Gaussian approximations. This allows us to work with small sample sizes and/or mean parameters in both one-sample and two-sample settings. We demonstrate our approach using three datasets on firm discrimination, education outcomes, and innovation rates.

  • 2 authors
·
Dec 30, 2025

First Light and Reionisation Epoch Simulations (FLARES) X: Environmental Galaxy Bias and Survey Variance at High Redshift

Upcoming deep galaxy surveys with JWST will probe galaxy evolution during the epoch of reionisation (EoR, 5leq zleq10) over relatively compact areas (e.g. sim 300\,arcmin^2 for the JADES GTO survey). It is therefore imperative that we understand the degree of survey variance, to evaluate how representative the galaxy populations in these studies will be. We use the First Light And Reionisation Epoch Simulations (FLARES) to measure the galaxy bias of various tracers over an unprecedentedly large range in overdensity for a hydrodynamic simulation, and use these relations to assess the impact of bias and clustering on survey variance in the EoR. Star formation is highly biased relative to the underlying dark matter distribution, with the mean ratio of the stellar to dark matter density varying by a factor of 100 between regions of low and high matter overdensity (smoothed on a scale of 14,h^{-1}cMpc). This is reflected in the galaxy distribution --the most massive galaxies are found solely in regions of high overdensity. As a consequence of the above, galaxies in the EoR are highly clustered, which can lead to large variance in survey number counts. For mean number counts Nlesssim 100 (1000), in a unit redshift slice of angular area 300\,arcmin^2 (1.4\,deg^2), the 2-sigma range in N is roughly a factor of four (two). We present relations between the expected variance and survey area for different survey geometries; these relations will be of use to observers wishing to understand the impact of survey variance on their results.

  • 8 authors
·
Jan 23, 2023

Questioning the Survey Responses of Large Language Models

As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models with varying scientific motivations. In this work, we examine what we can learn from a model's survey responses on the basis of the well-established American Community Survey (ACS) by the U.S. Census Bureau. Evaluating more than a dozen different models, varying in size from a few hundred million to ten billion parameters, hundreds of thousands of times each on questions from the ACS, we systematically establish two dominant patterns. First, smaller models have a significant position and labeling bias, for example, towards survey responses labeled with the letter "A". This A-bias diminishes, albeit slowly, as model size increases. Second, when adjusting for this labeling bias through randomized answer ordering, models still do not trend toward US population statistics or those of any cognizable population. Rather, models across the board trend toward uniformly random aggregate statistics over survey responses. This pattern is robust to various different ways of prompting the model, including what is the de-facto standard. Our findings demonstrate that aggregate statistics of a language model's survey responses lack the signals found in human populations. This absence of statistical signal cautions about the use of survey responses from large language models at present time.

  • 3 authors
·
Jun 13, 2023

Usage Bibliometrics as a Tool to Measure Research Activity

Measures for research activity and impact have become an integral ingredient in the assessment of a wide range of entities (individual researchers, organizations, instruments, regions, disciplines). Traditional bibliometric indicators, like publication and citation based indicators, provide an essential part of this picture, but cannot describe the complete picture. Since reading scholarly publications is an essential part of the research life cycle, it is only natural to introduce measures for this activity in attempts to quantify the efficiency, productivity and impact of an entity. Citations and reads are significantly different signals, so taken together, they provide a more complete picture of research activity. Most scholarly publications are now accessed online, making the study of reads and their patterns possible. Click-stream logs allow us to follow information access by the entire research community, real-time. Publication and citation datasets just reflect activity by authors. In addition, download statistics will help us identify publications with significant impact, but which do not attract many citations. Click-stream signals are arguably more complex than, say, citation signals. For one, they are a superposition of different classes of readers. Systematic downloads by crawlers also contaminate the signal, as does browsing behavior. We discuss the complexities associated with clickstream data and how, with proper filtering, statistically significant relations and conclusions can be inferred from download statistics. We describe how download statistics can be used to describe research activity at different levels of aggregation, ranging from organizations to countries. These statistics show a correlation with socio-economic indicators. A comparison will be made with traditional bibliometric indicators. We will argue that astronomy is representative of more general trends.

  • 2 authors
·
Jun 7, 2017

First systematic study reporting the changes in eclipse cut-off frequency for pulsar J1544+4937

We present results from a long-term monitoring of frequency dependent eclipses of the radio emission from PSR J1544+4937 which is a ``black widow spider'' millisecond pulsar (MSP) in a compact binary system. The majority of such systems often exhibit relatively long duration radio eclipses caused by ablated material from their companion stars. With the wide spectral bandwidth of upgraded Giant Metrewave Radio Telescope (uGMRT), we present first systematic study of temporal variation of eclipse cut-off frequency. With decade-long monitoring of 39 eclipses for PSR J1544+4937, we notice significant changes in the observed cut-off frequency ranging from 343 pm 7 MHz to > 740 MHz. We also monitored changes in eclipse cut-off frequency on timescales of tens of days and observed a maximum change of ge 315 MHz between observations that were separated by 22 days. In addition, we observed a change of sim 47 MHz in eclipse cut-off frequency between adjacent orbits, i.e. on timescales of sim 2.9 hours. We infer that such changes in the eclipse cut-off frequency depict an eclipse environment for the PSR J1544+4937 system that is dynamically evolving, where, along with the change in electron density, the magnetic field could also be varying. We also report a significant correlation between the eclipse cut-off frequency and the mass loss rate of the companion. This study provides the first direct evidence of mass loss rate affecting the frequency dependent eclipsing in a spider MSP.

  • 6 authors
·
Nov 3, 2023

Examining the Impact of Income Inequality and Gender on School Completion in Malaysia: A Machine Learning Approach Utilizing Malaysia's Public Sector Open Data

This study examines the relationship between income inequality, gender, and school completion rates in Malaysia using machine learning techniques. The dataset utilized is from the Malaysia's Public Sector Open Data Portal, covering the period 2016-2022. The analysis employs various machine learning techniques, including K-means clustering, ARIMA modeling, Random Forest regression, and Prophet for time series forecasting. These models are used to identify patterns, trends, and anomalies in the data, and to predict future school completion rates. Key findings reveal significant disparities in school completion rates across states, genders, and income levels. The analysis also identifies clusters of states with similar completion rates, suggesting potential regional factors influencing educational outcomes. Furthermore, time series forecasting models accurately predict future completion rates, highlighting the importance of ongoing monitoring and intervention strategies. The study concludes with recommendations for policymakers and educators to address the observed disparities and improve school completion rates in Malaysia. These recommendations include targeted interventions for specific states and demographic groups, investment in early childhood education, and addressing the impact of income inequality on educational opportunities. The findings of this study contribute to the understanding of the factors influencing school completion in Malaysia and provide valuable insights for policymakers and educators to develop effective strategies to improve educational outcomes.

  • 1 authors
·
Jan 30, 2025

A Large-Scale Dataset of Search Interests Related to Disease X Originating from Different Geographic Regions

The World Health Organization added Disease X to their shortlist of blueprint priority diseases to represent a hypothetical, unknown pathogen that could cause a future epidemic. During different virus outbreaks of the past, such as COVID-19, Influenza, Lyme Disease, and Zika virus, researchers from various disciplines utilized Google Trends to mine multimodal components of web behavior to study, investigate, and analyze the global awareness, preparedness, and response associated with these respective virus outbreaks. As the world prepares for Disease X, a dataset on web behavior related to Disease X would be crucial to contribute towards the timely advancement of research in this field. Furthermore, none of the prior works in this field have focused on the development of a dataset to compile relevant web behavior data, which would help to prepare for Disease X. To address these research challenges, this work presents a dataset of web behavior related to Disease X, which emerged from different geographic regions of the world, between February 2018 and August 2023. Specifically, this dataset presents the search interests related to Disease X from 94 geographic regions. The dataset was developed by collecting data using Google Trends. The relevant search interests for all these regions for each month in this time range are available in this dataset. This paper also discusses the compliance of this dataset with the FAIR principles of scientific data management. Finally, an analysis of this dataset is presented to uphold the applicability, relevance, and usefulness of this dataset for the investigation of different research questions in the interrelated fields of Big Data, Data Mining, Healthcare, Epidemiology, and Data Analysis with a specific focus on Disease X.

  • 5 authors
·
Dec 19, 2023

"All of Me": Mining Users' Attributes from their Public Spotify Playlists

In the age of digital music streaming, playlists on platforms like Spotify have become an integral part of individuals' musical experiences. People create and publicly share their own playlists to express their musical tastes, promote the discovery of their favorite artists, and foster social connections. These publicly accessible playlists transcend the boundaries of mere musical preferences: they serve as sources of rich insights into users' attributes and identities. For example, the musical preferences of elderly individuals may lean more towards Frank Sinatra, while Billie Eilish remains a favored choice among teenagers. These playlists thus become windows into the diverse and evolving facets of one's musical identity. In this work, we investigate the relationship between Spotify users' attributes and their public playlists. In particular, we focus on identifying recurring musical characteristics associated with users' individual attributes, such as demographics, habits, or personality traits. To this end, we conducted an online survey involving 739 Spotify users, yielding a dataset of 10,286 publicly shared playlists encompassing over 200,000 unique songs and 55,000 artists. Through extensive statistical analyses, we first assess a deep connection between a user's Spotify playlists and their real-life attributes. For instance, we found individuals high in openness often create playlists featuring a diverse array of artists, while female users prefer Pop and K-pop music genres. Building upon these observed associations, we create accurate predictive models for users' attributes, presenting a novel DeepSet application that outperforms baselines in most of these users' attributes.

  • 4 authors
·
Jan 25, 2024

Evolution of the Accretion Disk and Corona During the Outburst of the Neutron Star Transient MAXI J1807+132

Low-mass X-ray binaries with a neutron star as the primary object show a complex array of phenomenology during outbursts. The observed variability in X-ray emission primarily arises from changes in the innermost regions of the accretion disk, neutron star surface, and corona. In this work, we present the results of a comprehensive X-ray spectral and timing analysis of the neutron star transient MAXI J1807+132 during its 2023 outburst using data from the NICER observatory. The outburst is marked by a very rapid rise in the count rate by about a factor of 20 in a day. The source undergoes full state transitions and displays hysteresis effect in the hardness and rms intensity diagrams. Spectral analysis with a three-component model is consistent with disk truncation during the hard states and reaching the last stable orbit during the intermediate and soft states. We discuss the different values of the last stable radius in the context of possible distance of the source and magnetic field strength. The characteristic frequencies throughout the hard and intermediate states are found to be strongly correlated with the inner radius of the disk. Together with the spectral and fast variability properties, we attempt to trace the evolution of the size of the corona along the outburst. Following the main outburst, the source undergoes a high amplitude reflare wherein it shows a complex behavior with relatively high variability (10 %), but low hardness.

  • 7 authors
·
Dec 11, 2024

Astrometric Effects of a Stochastic Gravitational Wave Background

A stochastic gravitational wave background causes the apparent positions of distant sources to fluctuate, with angular deflections of order the characteristic strain amplitude of the gravitational waves. These fluctuations may be detectable with high precision astrometry, as first suggested by Braginsky et al. in 1990. Several researchers have made order of magnitude estimates of the upper limits obtainable on the gravitational wave spectrum \Omega_gw(f), at frequencies of order f ~ 1 yr^-1, both for the future space-based optical interferometry missions GAIA and SIM, and for VLBI interferometry in radio wavelengths with the SKA. For GAIA, tracking N ~ 10^6 quasars over a time of T ~ 1 yr with an angular accuracy of \Delta \theta ~ 10 \mu as would yield a sensitivity level of \Omega_gw ~ (\Delta \theta)^2/(N T^2 H_0^2) ~ 10^-6, which would be comparable with pulsar timing. In this paper we take a first step toward firming up these estimates by computing in detail the statistical properties of the angular deflections caused by a stochastic background. We compute analytically the two point correlation function of the deflections on the sphere, and the spectrum as a function of frequency and angular scale. The fluctuations are concentrated at low frequencies (for a scale invariant stochastic background), and at large angular scales, starting with the quadrupole. The magnetic-type and electric-type pieces of the fluctuations have equal amounts of power.

  • 2 authors
·
Sep 21, 2010

Probing X-ray Timing and Spectral Variability in the Blazar PKS 2155-304 Over a Decade of XMM-Newton Observations

Blazars, a class of active galactic nuclei (AGN) powered by supermassive black holes, are known for their remarkable variability across multiple timescales and wavelengths. With advancements in both ground- and space-based telescopes, our understanding of AGN central engines has significantly improved. However, the mechanisms driving this variability remain elusive, and continue to fascinate both theorists and observers alike. The primary objective of this study is to constrain the X-ray variability properties of the TeV blazar PKS 2155-304. We conduct a comprehensive X-ray spectral and timing analysis, focusing on both long-term and intra-day variability. This analysis uses data from 22 epochs of XMM-Newton EPIC-pn observations, collected over 15 years (2000-2014). To investigate the variability of the source, we applied both timing and spectral analyses. For the timing analysis, we estimated fractional variability, variability amplitude, minimum variability timescales, flux distribution, and power spectral density (PSD). In the spectral analysis, we fitted the X-ray spectra using power-law, log-parabola, and broken power-law (BPL) models to determine the best-fitting parameters. Additionally, we studied the hardness ratio (HR). We observed moderate intra-day variability in most of the light curves. Seven out of the twenty-two observations showed a clear bimodal flux distribution, indicating the presence of two distinct flux states. Our analysis revealed a variable power-law PSD slope. Most HR plots did not show significant variation with flux, except for one observation (OBSID 0124930501), where HR increased with flux (Count/s). The fitted X-ray spectra favored the BPL model for the majority of observations. The findings of this work shed light on the intraday variability of blazars, providing insights into the non-thermal jet processes that drive the observed flux variations.

  • 8 authors
·
Oct 2, 2024

European Pulsar Timing Array Limits On An Isotropic Stochastic Gravitational-Wave Background

We present new limits on an isotropic stochastic gravitational-wave background (GWB) using a six pulsar dataset spanning 18 yr of observations from the 2015 European Pulsar Timing Array data release. Performing a Bayesian analysis, we fit simultaneously for the intrinsic noise parameters for each pulsar, along with common correlated signals including clock, and Solar System ephemeris errors, obtaining a robust 95% upper limit on the dimensionless strain amplitude A of the background of A<3.0times 10^{-15} at a reference frequency of 1yr^{-1} and a spectral index of 13/3, corresponding to a background from inspiralling super-massive black hole binaries, constraining the GW energy density to Omega_gw(f)h^2 < 1.1times10^{-9} at 2.8 nHz. We also present limits on the correlated power spectrum at a series of discrete frequencies, and show that our sensitivity to a fiducial isotropic GWB is highest at a frequency of sim 5times10^{-9}~Hz. Finally we discuss the implications of our analysis for the astrophysics of supermassive black hole binaries, and present 95% upper limits on the string tension, Gmu/c^2, characterising a background produced by a cosmic string network for a set of possible scenarios, and for a stochastic relic GWB. For a Nambu-Goto field theory cosmic string network, we set a limit Gmu/c^2<1.3times10^{-7}, identical to that set by the {\it Planck} Collaboration, when combining {\it Planck} and high-ell Cosmic Microwave Background data from other experiments. For a stochastic relic background we set a limit of Omega^relic_gw(f)h^2<1.2 times10^{-9}, a factor of 9 improvement over the most stringent limits previously set by a pulsar timing array.

  • 36 authors
·
Apr 14, 2015

GMRT observation of neutral atomic hydrogen gas in the COSMOS field at z sim 0.37

We present the results of HI spectral stacking analysis of Giant Metrewave Radio Telescope (GMRT) observations targeting the COSMOS field. The GMRT data cube contains 474 field galaxies with redshifts known from the zCOSMOS-bright 10k catalogue. Spectra for the galaxies are co-added and the stacked spectrum allows us to make a sim 3σ measurement of the average HI mass. Using this average HI mass along with the integral optical B-band luminosity of the galaxies and the luminosity density of the COSMOS field, a volume normalisation is applied to obtain the cosmic HI mass density (Ω_{rm HI}). We find a cosmic HI mass density of Ω_{rm HI} = (0.42 pm 0.16) times 10^{-3} at z sim 0.37, which is the highest-redshift measurement of Ω_{rm HI} ever made using HI spectral stacking. The value we obtained for Ω_{rm HI} at z sim 0.37 is consistent with that measured from large blind 21-cm surveys at z = 0 as well as measurements from other HI stacking experiments at lower redshifts. Our measurement in conjunction with earlier measurements indicates that there has been no significant evolution of HI gas abundance over the last 4 Gyr. A weighted mean of Ω_{rm HI} from all 21-cm measurements at redshifts z lesssim 0.4 gives Ω_{rm HI} = (0.35 pm 0.01) times 10^{-3}. The Ω_{rm HI} measured (from HI 21-cm emission measurements) at z lesssim 0.4 is however approximately half that measured from Damped Lyman-α Absorption (DLA) systems at z gtrsim 2. Deeper surveys with existing and upcoming instruments will be critical to understand the evolution of Ω_{rm HI} in the redshift range intermediate between z sim 0.4 and the range probed by DLA observations.

  • 5 authors
·
May 6, 2016

From Chaos to Synchrony in Recurrent Excitatory-Inhibitory Networks with Target-Specific Inhibition

Biological neural networks can operate in qualitatively distinct dynamical regimes, and transitions between these regimes are thought to underlie changes in computation and behavior. The seminal work of Sompolinsky, Crisanti, and Sommers (SCS) showed that random recurrent networks undergo a transition from quiescence to asynchronous chaos, establishing a paradigmatic link between random connectivity, dynamical instability, and internally generated fluctuations in neural circuits. Here, we extend this framework to two-population firing-rate networks with segregated excitatory and inhibitory neurons and target-specific inhibitory couplings that break excitation--inhibition balance. Using dynamical mean-field theory, we derive self-consistent equations for the macroscopic mean activities and autocorrelations, together with stability criteria distinguishing mean-driven and fluctuation-driven instabilities. We show that target-specific inhibition organizes the phase diagram into three qualitative classes: inhibition-dominated or strictly balanced networks display only quiescent activity and asynchronous chaos; excitation-dominated networks display persistent activity together with either synchronous chaos with non-vanishing mean activity or coherent oscillations, depending on the stability-matrix eigenvalues. Crucially, coherent oscillations do not coexist with chaotic fluctuations around the periodic mean trajectory; rather, their onset suppresses the chaotic component, reminiscent of input-induced suppression of chaos. These results generalize SCS theory to recurrent networks with explicit excitatory--inhibitory structure and identify target-specific inhibition as a key control parameter for large-scale neural dynamics.

  • 4 authors
·
May 13

A study of a deterministic model for meningitis epidemic

A compartmental deterministic model that allows (1) immunity from two stages of infection and carriage, and (2) disease induced death, is used in studying the dynamics of meningitis epidemic process in a closed population. It allows for difference in the transmission rate of infection to a susceptible by a carrier and an infective. It is generalized to allow a proportion ({\phi}) of those susceptibles infected to progress directly to infectives in stage I. Both models are used in this study. The threshold conditions for the spread of carrier and infectives in stage I are derived for the two models. Sensitivity analysis is performed on the reproductive number derived from the next generation matrix. The case-carrier ratio profile for various parameters and threshold values are shown. So also are the graphs of the total number ever infected as influenced by {\epsilon} and {\phi}. The infection transmission rate (eta), the odds in favor of a carrier, over an infective, in transmitting an infection to a susceptible ({\epsilon}) and the carrier conversion rate ({\phi}) to an infective in stage I, are identified as key parameters that should be subject of attention for any control intervention strategy. The case-carrier ratio profiles provide evidence of a critical case-carrier ratio attained before the number of reported cases grows to an epidemic level. They also provide visual evidence of epidemiological context, in this case, epidemic incidence (in later part of dry season) and endemic incidence (during rainy season). Results from total proportion ever infected suggest that the model, in which {\phi}=0 obtained, can adequately represent, in essence, the generalized model for this study.

  • 2 authors
·
Mar 31, 2023

KIC 4150611: A quadruply eclipsing heptuple star system with a g-mode period-spacing pattern Asteroseismic modelling of the g-mode period-spacing pattern

In this work, we aim to estimate the stellar parameters of the primary (Aa) by performing asteroseismic analysis on its period-spacing pattern. We use the C-3PO neural network to perform asteroseismic modelling of the g-mode period-spacing pattern of Aa, discussing the interplay of this information with external constraints from spectroscopy (T_{rm eff} and log(g)) and eclipse modelling (R). To estimate the level of uncertainty due to different frequency extraction and pattern identification processes, we consider four different variations on the period-spacing patterns. To better understand the correlations between and the uncertainty structure of our parameter estimates, we also employed a classical, parameter-based MCMC grid search on four different stellar grids. The best-fitting, externally constrained model to the period-spacing pattern arrives at estimates of the stellar properties for Aa of: M=1.51 pm 0.05 M_odot, X_c =0.43 pm 0.04, R=1.66 pm 0.1 R_odot, f_{rm ov}=0.010, Omega_c=1.58 pm 0.01 d^{-1} with rigid rotation to within the measurement errors, log(T_{rm eff})=3.856 pm 0.008 dex, log(g)=4.18 pm 0.04 dex, and log(L)=0.809 pm 0.005 dex, which agree well with previous measurements from eclipse modelling, spectroscopy, and the Gaia DR3 luminosity. We find that the near-core properties of the best-fitting asteroseismic models are consistent with external constraints from eclipse modelling and spectroscopy. Aa appears to be a typical example of a gamma Dor star, fitting well within existing populations. We find that Aa is quasi-rigidly rotating to within the uncertainties, and note that the asteroseismic age estimate for Aa (1100 pm 100 Myr) is considerably older than the young (35 Myr) age implied by previous isochrone fits to the B binary in the literature. Our MCMC parameter-based grid-search agrees well with our pattern-modelling approach.

  • 10 authors
·
Nov 27, 2024

Utilizing localized fast radio bursts to constrain their progenitors and the expansion history of the Universe

Fast radio bursts (FRBs) are increasingly being used for cosmological applications such as measuring the Hubble constant and baryon abundance. The increasing number of localized FRBs and precise measurement of dispersion measure (DM) make them a suitable probe for such an approach. We use a sample of 110 localized FRBs as well as a small sub-sample of 24 FRBs with scattering timescale measurements or limits. We infer the Hubble constant (H_0) and the DM distribution of the host galaxies simultaneously by fitting our model to the FRB DM measurements. With current data, our results are in agreement with both high and low redshift measurements of H_0, obtained using Cosmic Microwave Background (CMB) and Type Ia supernovae data respectively. We project that with about 200 localized FRBs, we would be in a position to distinguish between the two scenarios at 4sigma confidence. In addition, the host DM is expected to be related to star formation in the host galaxy and the stellar age of the progenitors. We show that young progenitors with an age of less than 1 Myr are consistent with our inferred distribution of host DM at 95 percent confidence. These young sources may be associated with long scatter broadening times and large DM from their source environments. Indeed, we find that scatter broadening times of FRBs are inconsistent with the Milky Way ISM, but at the same time, do not appear to be strongly correlated with the FRBs' redshift or with the SFR or stellar mass of their host galaxies. This suggests that scattering is dominated by the immediate environment of the sources.

  • 2 authors
·
Mar 11, 2025

Crowdsourcing Dermatology Images with Google Search Ads: Creating a Real-World Skin Condition Dataset

Background: Health datasets from clinical sources do not reflect the breadth and diversity of disease in the real world, impacting research, medical education, and artificial intelligence (AI) tool development. Dermatology is a suitable area to develop and test a new and scalable method to create representative health datasets. Methods: We used Google Search advertisements to invite contributions to an open access dataset of images of dermatology conditions, demographic and symptom information. With informed contributor consent, we describe and release this dataset containing 10,408 images from 5,033 contributions from internet users in the United States over 8 months starting March 2023. The dataset includes dermatologist condition labels as well as estimated Fitzpatrick Skin Type (eFST) and Monk Skin Tone (eMST) labels for the images. Results: We received a median of 22 submissions/day (IQR 14-30). Female (66.72%) and younger (52% < age 40) contributors had a higher representation in the dataset compared to the US population, and 32.6% of contributors reported a non-White racial or ethnic identity. Over 97.5% of contributions were genuine images of skin conditions. Dermatologist confidence in assigning a differential diagnosis increased with the number of available variables, and showed a weaker correlation with image sharpness (Spearman's P values <0.001 and 0.01 respectively). Most contributions were short-duration (54% with onset < 7 days ago ) and 89% were allergic, infectious, or inflammatory conditions. eFST and eMST distributions reflected the geographical origin of the dataset. The dataset is available at github.com/google-research-datasets/scin . Conclusion: Search ads are effective at crowdsourcing images of health conditions. The SCIN dataset bridges important gaps in the availability of representative images of common skin conditions.

  • 20 authors
·
Feb 28, 2024

Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulations

Understanding the performance of machine learning (ML) models across diverse data distributions is critically important for reliable applications. Despite recent empirical studies positing a near-perfect linear correlation between in-distribution (ID) and out-of-distribution (OOD) accuracies, we empirically demonstrate that this correlation is more nuanced under subpopulation shifts. Through rigorous experimentation and analysis across a variety of datasets, models, and training epochs, we demonstrate that OOD performance often has a nonlinear correlation with ID performance in subpopulation shifts. Our findings, which contrast previous studies that have posited a linear correlation in model performance during distribution shifts, reveal a "moon shape" correlation (parabolic uptrend curve) between the test performance on the majority subpopulation and the minority subpopulation. This non-trivial nonlinear correlation holds across model architectures, hyperparameters, training durations, and the imbalance between subpopulations. Furthermore, we found that the nonlinearity of this "moon shape" is causally influenced by the degree of spurious correlations in the training data. Our controlled experiments show that stronger spurious correlation in the training data creates more nonlinear performance correlation. We provide complementary experimental and theoretical analyses for this phenomenon, and discuss its implications for ML reliability and fairness. Our work highlights the importance of understanding the nonlinear effects of model improvement on performance in different subpopulations, and has the potential to inform the development of more equitable and responsible machine learning models.

  • 5 authors
·
May 4, 2023

The NANOGrav Nine-year Data Set: Limits on the Isotropic Stochastic Gravitational Wave Background

We compute upper limits on the nanohertz-frequency isotropic stochastic gravitational wave background (GWB) using the 9-year data release from the North American Nanohertz Observatory for Gravitational Waves (NANOGrav) collaboration. We set upper limits for a GWB from supermassive black hole binaries under power law, broken power law, and free spectral coefficient GW spectrum models. We place a 95\% upper limit on the strain amplitude (at a frequency of yr^{-1}) in the power law model of A_{rm gw} < 1.5times 10^{-15}. For a broken power law model, we place priors on the strain amplitude derived from simulations of Sesana (2013) and McWilliams et al. (2014). We find that the data favor a broken power law to a pure power law with odds ratios of 22 and 2.2 to one for the McWilliams and Sesana prior models, respectively. The McWilliams model is essentially ruled out by the data, and the Sesana model is in tension with the data under the assumption of a pure power law. Using the broken power-law analysis we construct posterior distributions on environmental factors that drive the binary to the GW-driven regime including the stellar mass density for stellar-scattering, mass accretion rate for circumbinary disk interaction, and orbital eccentricity for eccentric binaries, marking the first time that the shape of the GWB spectrum has been used to make astrophysical inferences. We then place the most stringent limits so far on the energy density of relic GWs, Omega_gw(f),h^2 < 4.2 times 10^{-10}, yielding a limit on the Hubble parameter during inflation of H_*=1.6times10^{-2}~m_{Pl}, where m_{Pl} is the Planck mass. Our limit on the cosmic string GWB, Omega_gw(f), h^2 < 2.2 times 10^{-10}, translates to a conservative limit of Gmu<3.3times 10^{-8} - a factor of 4 better than the joint Planck and high-l CMB data from other experiments.

  • 48 authors
·
Aug 12, 2015

DESI Bright Galaxy Survey: Final Target Selection, Design, and Validation

Over the next five years, the Dark Energy Spectroscopic Instrument (DESI) will use 10 spectrographs with 5000 fibers on the 4m Mayall Telescope at Kitt Peak National Observatory to conduct the first Stage-IV dark energy galaxy survey. At z < 0.6, the DESI Bright Galaxy Survey (BGS) will produce the most detailed map of the Universe during the dark energy dominated epoch with redshifts of >10 million galaxies over 14,000 deg^2. In this work, we present and validate the final BGS target selection and survey design. From the Legacy Surveys, BGS will target a r < 19.5 magnitude-limited sample (BGS Bright); a fainter 19.5 < r < 20.175 sample, color-selected to have high redshift efficiency (BGS Faint); and a smaller low-z quasar sample. BGS will observe these targets using exposure times, scaled to achieve uniform completeness, and visit each point on the footprint three times. We use observations from the Survey Validation programs conducted prior to the main survey along with realistic simulations to show that BGS can complete its strategy and make optimal use of `bright' time. We demonstrate that BGS targets have stellar contamination <1% and that their densities do not depend strongly on imaging properties. We also confirm that BGS Bright will achieve >80% fiber assignment efficiency. Finally, we show that BGS Bright and Faint will achieve >95% redshift success rates with no significant dependence on observing conditions. BGS meets the requirements for an extensive range of scientific applications. BGS will yield the most precise Baryon Acoustic Oscillations and Redshift-Space Distortions measurements at z < 0.4. It also presents opportunities to exploit new methods that require highly complete and dense galaxy samples (e.g. N-point statistics, multi-tracers). BGS further provides a powerful tool to study galaxy populations and the relations between galaxies and dark matter.

  • 60 authors
·
Aug 16, 2022

Properties of tensorial free cumulants

In the past two years, several points of view have been proposed to address the question of the generalization of the theory of free probability to random tensors with different invariances, and it is unclear at this point whether they lead to the same notions of tensorial free cumulants and freeness. One way to approach this problem, developed by Collins, Gurau and the second named author for local unitary invariant random tensors, relies on finite size quantities involving averages over the invariance group, and whose asymptotics naturally possess the properties expected for tensorial generalizations of free cumulants of arbitrary orders. At this point, this approach has only been carried out for certain distributions, and for a subset of the moments that define such theories, and a more systematic and exhaustive study is lacking. This is the program initiated in this paper: we link this approach to the one proposed by Nechita and Park; extend a number of their results as well as those of the aforementioned paper to arbitrary orders of fluctuations, thereby generalizing higher order free cumulants; push further the study of distributions with larger invariance groups; detail the link with the asymptotics of the free-energies of the tensor HCIZ and BGW integrals; and provide formulae for tensorial free cumulants of products of tensors. Another important question is that of the definition of concrete distributions whose tensorial free-cumulants take non-trivial values. We compute the tensorial free cumulants for Gaussian random tensors with non-trivial covariances, and show that they provide such examples.

  • 2 authors
·
May 2

Heterogeneous synaptic motifs bridge microscale structure and macroscale nonlinear dynamics

Recent breakthroughs in synaptic-resolution network connectomics have revealed that brain circuits feature fine-scale structural connectivity, such as pairs of correlated synaptic couplings known as second-order motifs. Large-scale recordings of neuronal activity in networks containing nonlinear neurons reveal macroscopic heterogeneous population dynamics throughout the brain. These findings rekindle the inquiry into this intriguing question: Can microscale synaptic structures contribute to macroscopic heterogeneous dynamics and computations in ways that canonical brain circuit models cannot? To answer this question, we create random RNNs with various cell types, nonlinear non-negative neural responses, and arbitrary marginal and second-order correlated synaptic statistics. We derive mean-field low-rank equations for P-population networks in which the pre- and postsynaptic neuronal population identities determine the synaptic and motif strengths. Our framework requires 2P latent dynamic variables with P variables describing mean population activity and P variables capturing within-population variability. Theoretical and simulational results demonstrate that chain motifs induce correlations in synaptic variability, enabling microscopic fluctuations to be integrated and influence mesoscopic mean population dynamics. We apply this framework to reverse engineer network connectivity that recapitulates the heterogeneous activity across the population in the mouse primary visual cortex. By bridging the gap between synaptic organization and nonlinear heterogeneous population dynamics, our results offer a principled approach and testable predictions regarding the relationship between fine-scale connectivity, heterogeneous dynamics, and functional computations.

  • 4 authors
·
Jun 25

Fast and Eager k-Medoids Clustering: O(k) Runtime Improvement of the PAM, CLARA, and CLARANS Algorithms

Clustering non-Euclidean data is difficult, and one of the most used algorithms besides hierarchical clustering is the popular algorithm Partitioning Around Medoids (PAM), also simply referred to as k-medoids clustering. In Euclidean geometry the mean-as used in k-means-is a good estimator for the cluster center, but this does not exist for arbitrary dissimilarities. PAM uses the medoid instead, the object with the smallest dissimilarity to all others in the cluster. This notion of centrality can be used with any (dis-)similarity, and thus is of high relevance to many domains and applications. A key issue with PAM is its high run time cost. We propose modifications to the PAM algorithm that achieve an O(k)-fold speedup in the second ("SWAP") phase of the algorithm, but will still find the same results as the original PAM algorithm. If we relax the choice of swaps performed (while retaining comparable quality), we can further accelerate the algorithm by eagerly performing additional swaps in each iteration. With the substantially faster SWAP, we can now explore faster initialization strategies, because (i) the classic ("BUILD") initialization now becomes the bottleneck, and (ii) our swap is fast enough to compensate for worse starting conditions. We also show how the CLARA and CLARANS algorithms benefit from the proposed modifications. While we do not study the parallelization of our approach in this work, it can easily be combined with earlier approaches to use PAM and CLARA on big data (some of which use PAM as a subroutine, hence can immediately benefit from these improvements), where the performance with high k becomes increasingly important. In experiments on real data with k=100,200, we observed a 458x respectively 1191x speedup compared to the original PAM SWAP algorithm, making PAM applicable to larger data sets, and in particular to higher k.

  • 2 authors
·
Aug 12, 2020

A Labelled Dataset for Sentiment Analysis of Videos on YouTube, TikTok, and Other Sources about the 2024 Outbreak of Measles

The work of this paper presents a dataset that contains the data of 4011 videos about the ongoing outbreak of measles published on 264 websites on the internet between January 1, 2024, and May 31, 2024. The dataset is available at https://dx.doi.org/10.21227/40s8-xf63. These websites primarily include YouTube and TikTok, which account for 48.6% and 15.2% of the videos, respectively. The remainder of the websites include Instagram and Facebook as well as the websites of various global and local news organizations. For each of these videos, the URL of the video, title of the post, description of the post, and the date of publication of the video are presented as separate attributes in the dataset. After developing this dataset, sentiment analysis (using VADER), subjectivity analysis (using TextBlob), and fine-grain sentiment analysis (using DistilRoBERTa-base) of the video titles and video descriptions were performed. This included classifying each video title and video description into (i) one of the sentiment classes i.e. positive, negative, or neutral, (ii) one of the subjectivity classes i.e. highly opinionated, neutral opinionated, or least opinionated, and (iii) one of the fine-grain sentiment classes i.e. fear, surprise, joy, sadness, anger, disgust, or neutral. These results are presented as separate attributes in the dataset for the training and testing of machine learning algorithms for performing sentiment analysis or subjectivity analysis in this field as well as for other applications. Finally, this paper also presents a list of open research questions that may be investigated using this dataset.

  • 7 authors
·
Jun 11, 2024

Predicting the Flu from Instagram

Conventional surveillance systems for monitoring infectious diseases, such as influenza, face challenges due to shortage of skilled healthcare professionals, remoteness of communities and absence of communication infrastructures. Internet-based approaches for surveillance are appealing logistically as well as economically. Search engine queries and Twitter have been the primarily used data sources in such approaches. The aim of this study is to assess the predictive power of an alternative data source, Instagram. By using 317 weeks of publicly available data from Instagram, we trained several machine learning algorithms to both nowcast and forecast the number of official influenza-like illness incidents in Finland where population-wide official statistics about the weekly incidents are available. In addition to date and hashtag count features of online posts, we were able to utilize also the visual content of the posted images with the help of deep convolutional neural networks. Our best nowcasting model reached a mean absolute error of 11.33 incidents per week and a correlation coefficient of 0.963 on the test data. Forecasting models for predicting 1 week and 2 weeks ahead showed statistical significance as well by reaching correlation coefficients of 0.903 and 0.862, respectively. This study demonstrates how social media and in particular, digital photographs shared in them, can be a valuable source of information for the field of infodemiology.

  • 2 authors
·
Nov 27, 2018

First Light And Reionisation Epoch Simulations (FLARES) XVI: Size Evolution of Massive Dusty Galaxies at Cosmic Dawn from UV to IR

We use the First Light And Reionisation Epoch Simulations (FLARES) to study the evolution of the rest-frame ultraviolet (UV) and far-infrared (FIR) sizes for a statistical sample of massive (gtrsim10^{9}M_{odot}) high redshift galaxies (z in [5,10]). Galaxies are post-processed using the SKIRT radiative transfer code, to self-consistently obtain the full spectral energy distribution and surface brightness distribution. We create mock observations of the galaxies for the Near Infrared Camera (NIRCam) to study the rest-frame UV 1500 xC5 morphology. We also generate mock rest-frame FIR (50 mum) photometry and mock ALMA (158 mum) (0.01"-0.03" and approx0.3" angular resolution) observations to study the dust-continuum. We find the effect of dust on observed sizes reduces with increasing wavelength from the UV to optical (sim0.6 times the UV at 0.4mum), with no evolution in FIR sizes. Observed sizes vary within 0.4-1.2 times the intrinsic sizes at different signal to noise ratios (SNR = 5-20) across redshifts. The effect of PSF and noise makes bright structures prominent, whereas fainter regions blend with noise, leading to an underestimation (factor of 0.4-0.8) of sizes at SNR=5. At SNR=15-20, the underestimation reduces (factor of 0.6-0.9) at z=5-8 but due to PSF, at z=9-10, bright cores are dominant, resulting in an overestimation (factor of 1.0-1.2). For ALMA, low resolution sizes are effected by noise which acts as extended emission. The size evolution in UV broadly agrees with current observational samples and other simulations. This work is one of the first to analyse the panchromatic sizes of a statistically significant sample of simulated high-redshift galaxies, complementing a growing body of research highlighting the importance of conducting an equivalent comparison between observed galaxies and their simulated counterparts in the early Universe.

  • 12 authors
·
Aug 20, 2024