<?xml version="1.0" encoding="UTF-8"?>
 <rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" xmlns:syn="http://purl.org/rss/1.0/modules/syndication/" xmlns:admin="http://webns.net/mvcb/">
  <channel rdf:about="http://pinboard.in">
    <title>Pinboard (cshalizi)</title>
    <link>https://pinboard.in/u:cshalizi/public/</link>
    <description>recent bookmarks from cshalizi</description>
    <items>
      <rdf:Seq>	<rdf:li rdf:resource="https://doi.org/10.1108/FTCIT-09-2025-0149"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/math/0504472"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/cond-mat/0102181"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2405.00126"/>
	<rdf:li rdf:resource="https://link.springer.com/article/10.1007/s10472-024-09943-9"/>
	<rdf:li rdf:resource="https://www.nowpublishers.com/article/Details/MAL-112"/>
	<rdf:li rdf:resource="https://www.nowpublishers.com/article/Details/CIT-107"/>
	<rdf:li rdf:resource="https://www.nowpublishers.com/article/Details/CIT-142"/>
	<rdf:li rdf:resource="https://openreview.net/forum?id=UvmDCdSPDOW"/>
	<rdf:li rdf:resource="https://www.degruyter.com/document/doi/10.1515/9780691256740/html"/>
	<rdf:li rdf:resource="https://philpapers.org/rec/KINBCC"/>
	<rdf:li rdf:resource="https://journals.aps.org/prxlife/abstract/10.1103/PRXLife.2.033009"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2410.06224"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2401.17377"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2401.17505"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2306.04050"/>
	<rdf:li rdf:resource="https://www.journals.uchicago.edu/doi/10.1086/661645"/>
	<rdf:li rdf:resource="https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.010401"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2402.07999"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2207.12382"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2305.11042"/>
	<rdf:li rdf:resource="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.108.014304"/>
	<rdf:li rdf:resource="https://ieeexplore.ieee.org/document/9737725"/>
	<rdf:li rdf:resource="https://ieeexplore.ieee.org/abstract/document/1101710?casa_token=fmWVrk6dbi0AAAAA:TXo6urBwf2bD2CP9jPDGqaPGj03Zg50-inwRk3gb-z3C6RC8fZ3UcNaCJAWFU3nclfhj6uE"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2305.02960"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2302.05380"/>
	<rdf:li rdf:resource="https://journals.aps.org/pra/abstract/10.1103/PhysRevA.33.1134"/>
	<rdf:li rdf:resource="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.62.7508"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/1102.0250"/>
	<rdf:li rdf:resource="https://dml.cz/bitstream/handle/10338.dmlcz/135833/Kybernetika_44-2008-1_5.pdf"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2302.09780"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2303.05369"/>
	<rdf:li rdf:resource="https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2212.13556"/>
	<rdf:li rdf:resource="https://www-degruyter-com.cmu.idm.oclc.org/document/doi/10.1515/9781478023630/html#overview"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2202.00666"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2202.04985"/>
	<rdf:li rdf:resource="https://jmlr.org/papers/v23/20-644.html"/>
	<rdf:li rdf:resource="https://www.biorxiv.org/content/10.1101/2022.07.02.498577v1"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2206.02340"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2203.08089"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2206.04640"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2206.02072"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2206.02025"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2205.15227"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2112.14721"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2203.01916"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2206.02765"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2108.03099"/>
	<rdf:li rdf:resource="https://aip.scitation.org/doi/10.1063/5.0063384"/>
	<rdf:li rdf:resource="https://www.jmlr.org/papers/v23/20-644.html"/>
	<rdf:li rdf:resource="https://philpapers.org/rec/JAVHAC"/>
	<rdf:li rdf:resource="https://www.journals.uchicago.edu/doi/abs/10.1086/718416"/>
	<rdf:li rdf:resource="https://press.uchicago.edu/ucp/books/book/chicago/I/bo113686763"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/1508.01167"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2110.01584"/>
	<rdf:li rdf:resource="https://www.nature.com/articles/s41598-020-73380-x"/>
	<rdf:li rdf:resource="https://marketurbanism.com/2021/07/05/is-diversity-segregation/"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2102.05802"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2102.09584"/>
	<rdf:li rdf:resource="https://publikationen.sulb.uni-saarland.de/bitstream/20.500.11880/31480/1/dissertation-marx.pdf"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2107.03789"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2106.06511"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2106.01645"/>
	<rdf:li rdf:resource="https://jmlr.org/papers/v22/20-867.html"/>
	<rdf:li rdf:resource="http://www.isiweb.ee.ethz.ch/papers/arch/mass-1990-1.pdf"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/nlin/0001042"/>
	<rdf:li rdf:resource="https://ieeexplore.ieee.org/abstract/document/1091610?casa_token=7_0gluYlO2gAAAAA:PYxZ1NlWfA5CZSUG89-Q_sjSrNYlYJNtc4FwlrSJFVdfag2cDPhcEzaP_IYq8MZcnfRjRb0"/>
	<rdf:li rdf:resource="https://arxiv.org/abs/2105.11940"/>
	<rdf:li rdf:resource="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.103.052408"/>
      </rdf:Seq>
    </items>
  </channel><item rdf:about="https://doi.org/10.1108/FTCIT-09-2025-0149">
    <title>Volume 23 Issue 3-4 | Foundations and Trends in Communications and Information Theory | Emerald Publishing</title>
    <dc:date>2026-05-19T15:46:33+00:00</dc:date>
    <link>https://doi.org/10.1108/FTCIT-09-2025-0149</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Online learning is a foundational paradigm underlying applications from recommendation systems to the continual learning of modern AI models. Yet much of its theory centers on either fully adversarial or purely stochastic settings. However, real-world environments typically fall between these extremes, making classical models inadequate for describing practical behavior. This monograph develops a unified perspective for analyzing online learning under more nuanced and realistic environments. The authors approach the problem through the lens of universality from information theory and extend tools such as the Shtarkov sum, covering numbers and packing arguments to the online setting, revealing deeper structural connections between these two fields. Building on this viewpoint, they characterize minimax regret for logarithmic and Lipschitz losses, analyze expected regret under i.i.d. and more general stochastic processes and study hybrid adversarial–stochastic scenarios. The authors further develop constructive algorithms that achieve near-optimal regret guarantees, yielding a coherent and fine-grained information-theoretic framework of online universal learning."]]></description>
<dc:subject>to_read information_theory learning_theory learning_under_dependence in_NB low-regret_learning online_learning</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:eb2f88390f74/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_under_dependence"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:low-regret_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:online_learning"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/math/0504472">
    <title>[math/0504472] Szemerédi's regularity lemma revisited</title>
    <dc:date>2026-04-08T02:30:02+00:00</dc:date>
    <link>https://arxiv.org/abs/math/0504472</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Szemerédi's regularity lemma is a basic tool in graph theory, and also plays an important role in additive combinatorics, most notably in proving Szemerédi's theorem on arithmetic progressions . In this note we revisit this lemma from the perspective of probability theory and information theory instead of graph theory, and observe a variant of this lemma which introduces a new parameter F. This stronger version of the regularity lemma was iterated in a recent paper of the author to reprove the analogous regularity lemma for hypergraphs."

--- Re last tag, I ought to try to find time to think about this as a form of (approximate) statistical sufficiency, and/or the information bottleneck.]]></description>
<dc:subject>have_read tao.terence graph_theory information_theory probability coarse_graining sufficiency in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:b71042d4baec/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:tao.terence"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:graph_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:probability"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:coarse_graining"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:sufficiency"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/cond-mat/0102181">
    <title>[cond-mat/0102181] Regularities Unseen, Randomness Observed: Levels of Entropy Convergence</title>
    <dc:date>2026-03-02T18:38:10+00:00</dc:date>
    <link>https://arxiv.org/abs/cond-mat/0102181</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to stochastic and deterministic processes by using successive derivatives of the Shannon entropy growth curve. This leads, in turn, to natural measures of apparent memory stored in a source and the amounts of information that must be extracted from observations of a source in order for it to be optimally predicted and for an observer to synchronize to it. One consequence of ignoring these structural properties is that the missed regularities are converted to apparent randomness. We demonstrate that this problem arises particularly for small data sets; e.g., in settings where one has access only to short measurement sequences."]]></description>
<dc:subject>in_NB information_theory computational_mechanics prediction have_read heard_the_talk heard_the_talk_a_quarter_century_ago_in_fact kith_and_kin crutchfield.james_p. feldman.david_p. probability</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:90869a348890/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:computational_mechanics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:prediction"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:heard_the_talk"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:heard_the_talk_a_quarter_century_ago_in_fact"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:kith_and_kin"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:crutchfield.james_p."/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:feldman.david_p."/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:probability"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2405.00126">
    <title>[2405.00126] A variational approach to sampling in diffusion processes</title>
    <dc:date>2025-04-23T16:00:56+00:00</dc:date>
    <link>https://arxiv.org/abs/2405.00126</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We revisit the work of Mitter and Newton on an information-theoretic interpretation of Bayes' formula through the Gibbs variational principle. This formulation allowed them to pose nonlinear estimation for diffusion processes as a problem in stochastic optimal control, so that the posterior density of the signal given the observation path could be sampled by adding a drift to the signal process. We show that this control-theoretic approach to sampling provides a common mechanism underlying several distinct problems involving diffusion processes, specifically importance sampling using Feynman-Kac averages, time reversal, and Schrödinger bridges."]]></description>
<dc:subject>to_read stochastic_processes stochastic_differential_equations control_theory_and_control_engineering information_theory raginsky.maxim via:mraginsky to_teach:statistics_and_generative_ai in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:bb9efdd86c90/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_differential_equations"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:control_theory_and_control_engineering"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:raginsky.maxim"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:mraginsky"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:statistics_and_generative_ai"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://link.springer.com/article/10.1007/s10472-024-09943-9">
    <title>Deep data density estimation through Donsker-Varadhan representation | Annals of Mathematics and Artificial Intelligence</title>
    <dc:date>2025-03-29T14:34:34+00:00</dc:date>
    <link>https://link.springer.com/article/10.1007/s10472-024-09943-9</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Estimating the data density is one of the challenging problem topics in the deep learning society. In this paper, we present a simple yet effective methodology for estimating the data density using the Donsker-Varadhan variational lower bound on the KL divergence and the modeling based on the deep neural network. We demonstrate that the optimal critic function associated with the Donsker-Varadhan representation on the KL divergence between the data and the uniform distribution can estimate the data density. Also, we present the deep neural network-based modeling and its stochastic learning procedure. The experimental results and possible applications of the proposed method demonstrate that it is competitive with the previous methods for data density estimation and has a lot of possibilities for various applications."

--- Not sure what to make of this, from the abstract and a quick scan.  Maybe worth going over in my copious spare time?]]></description>
<dc:subject>density_estimation information_theory optimization neural_networks in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:f7f2b2cd009b/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:density_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:optimization"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:neural_networks"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.nowpublishers.com/article/Details/MAL-112">
    <title>now publishers - Generalization Bounds: Perspectives from Information Theory and PAC-Bayes</title>
    <dc:date>2025-03-20T13:04:53+00:00</dc:date>
    <link>https://www.nowpublishers.com/article/Details/MAL-112</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["A fundamental question in theoretical machine learning is generalization. Over the past decades, the PAC-Bayesian approach has been established as a flexible framework to address the generalization capabilities of machine learning algorithms and design new ones. Recently, it has garnered increased interest due to its potential applicability for a variety of learning algorithms, including deep neural networks. In parallel, an information-theoretic view of generalization has developed, wherein the relation between generalization and various information measures has been established. This framework is intimately connected to the PAC-Bayesian approach, and a number of results have been independently discovered in both strands.
"In this monograph, we highlight this strong connection and present a unified treatment of PAC-Bayesian and information- theoretic generalization bounds. We present techniques and results that the two perspectives have in common, and discuss the approaches and interpretations that differ. In particular, we demonstrate how many proofs in the area share a modular structure, through which the underlying ideas can be intuited. We pay special attention to the conditional mutual information (CMI) framework, analytical studies of the information complexity of learning algorithms, and the application of the proposed methods to deep learning. This monograph is intended to provide a comprehensive introduction to information-theoretic generalization bounds and their connection to PAC-Bayes, serving as a foundation from which the most recent developments are accessible. It is aimed broadly towards researchers with an interest in generalization and theoretical machine learning."]]></description>
<dc:subject>to:NB downloaded books:noted to_read learning_theory information_theory raginsky.maxim to_teach:childs_garden_of_statistical_learning_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:72659330abf1/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:downloaded"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:noted"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:raginsky.maxim"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:childs_garden_of_statistical_learning_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.nowpublishers.com/article/Details/CIT-107">
    <title>now publishers - Universal Features for High-Dimensional Learning and Inference</title>
    <dc:date>2025-03-20T13:03:17+00:00</dc:date>
    <link>https://www.nowpublishers.com/article/Details/CIT-107</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["This monograph develops unifying perspectives on the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, natural notions of universality are introduced, and a local equivalence among them is established. The analysis is naturally expressed via information geometry, which provides both conceptual and computational insights. The development reveals the complementary roles of the singular value decomposition, Hirschfeld-Gebelein-Rényi maximal correlation, the canonical correlation and principle component analyses of Hotelling and Pearson, Tishby’s information bottleneck, Wyner’s and Gács-Körner common information, Ky Fan k-norms, and Breiman and Friedman’s alternating conditional expectations algorithm. Among other uses, the framework facilitates understanding and optimizing aspects of learning systems, including multinomial logistic (softmax) regression and neural network architecture, matrix factorization methods for collaborative filtering and other applications, rank-constrained multivariate linear regression, and forms of semi-supervised learning."]]></description>
<dc:subject>to:NB information_theory pattern_discovery information_bottleneck principal_components low-dimensional_summaries information_geometry collaborative_filtering</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:62fffa70f391/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:pattern_discovery"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_bottleneck"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:principal_components"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:low-dimensional_summaries"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_geometry"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:collaborative_filtering"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.nowpublishers.com/article/Details/CIT-142">
    <title>now publishers - A Toolbox for Refined Information-Theoretic Analyses</title>
    <dc:date>2025-03-20T13:02:28+00:00</dc:date>
    <link>https://www.nowpublishers.com/article/Details/CIT-142</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["This monograph offers a toolbox of mathematical techniques that have been effective and widely applicable in informationtheoretic analyses. The first tool is a generalization of the method of types to Gaussian settings, and then to general exponential families. The second tool is Laplace and saddlepoint integration, which allow to refine the results of the method of types, and can obtain various precise asymptotic results. The third is the type class enumeration method, a principled method to evaluate the exact random-coding exponent of coded systems, which results in the best known exponent in various problems. The fourth is a subset of tools aimed at evaluating the expectation of non-linear functions of random variables, either via integral representations, by a refinement of Jensen’s inequality via change-of-measure, by complementing Jensen’s inequality with a reversed inequality, or by a class of generalized Jensen’s inequalities that are applicable for functions beyond convex/concave. Various examples of all these tools are provided throughout the monograph."

--- At 184 pp., it earns the last tag.]]></description>
<dc:subject>to:NB downloaded to_read information_theory books:noted</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ae8a1f9bfa89/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:downloaded"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:noted"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://openreview.net/forum?id=UvmDCdSPDOW">
    <title>Information-Theoretic Diffusion | OpenReview</title>
    <dc:date>2025-03-10T14:39:50+00:00</dc:date>
    <link>https://openreview.net/forum?id=UvmDCdSPDOW</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation.  We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with Minimum Mean Square Error regression, the so-called I-MMSE relations. We generalize the I-MMSE relations to \emph{exactly} relate the data distribution to an optimal denoising regression problem, leading to an elegant refinement of existing diffusion bounds.  This new insight leads to several improvements for probability distribution estimation, including a theoretical justification for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-specific generative models used in variational methods."]]></description>
<dc:subject>density_estimation information_theory neural_networks ver_steeg.greg tab_closure generative_diffusion_models in_NB have_read to_teach:statistics_and_generative_ai</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:9889cc2216d2/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:density_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:neural_networks"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:ver_steeg.greg"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:tab_closure"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:generative_diffusion_models"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:statistics_and_generative_ai"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.degruyter.com/document/doi/10.1515/9780691256740/html">
    <title>The Data Economy</title>
    <dc:date>2025-03-02T14:51:28+00:00</dc:date>
    <link>https://www.degruyter.com/document/doi/10.1515/9780691256740/html</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The most valuable firms in the global economy are valued largely for their data. Amazon, Apple, Google, and others have proven the competitive advantage of a good data set. And yet despite the growing importance of data as a strategic asset, modern economic theory neglects its role. In this book, Isaac Baley and Laura Veldkamp draw on a range of theoretical frameworks at the research frontier in macroeconomics and finance to model and measure data economies. Starting from the premise that data is digitized information that facilitates prediction and reduces uncertainty, Baley and Veldkamp uncover the ways that firm-level data choices resonate throughout the broader macroeconomic and financial landscapes.
"With The Data Economy, Baley and Veldkamp put forward a broad research agenda with a formal yet accessible approach, offering an analysis of the data economy and its welfare effects that will be of interest to practitioners, researchers, and graduate students. The tools presented, many of them information-related methods from macroeconomics and finance, are theoretical but introduced with careful attention to how they can inform or enable measurement. Applications include assessing the economic worth of data and unraveling its influence on the structure of production, inflation, and pricing dynamics; firm and investor behavior; advertising; market power; and asset pricing. Baley and Veldkamp bring readers to the cutting edge of this novel research area, equipping them to formulate their own theoretical advances and policy analysis."]]></description>
<dc:subject>to:NB economics macroeconomics finance econometrics information_theory books:noted</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:c7f09031cfe2/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:economics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:macroeconomics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:finance"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:econometrics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:noted"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://philpapers.org/rec/KINBCC">
    <title>David Kinney &amp; Tania Lombrozo, Building Compressed Causal Models of the World - PhilPapers</title>
    <dc:date>2024-12-11T19:55:08+00:00</dc:date>
    <link>https://philpapers.org/rec/KINBCC</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["A given causal system can be represented in a variety of ways. How do agents determine which variables to include in their causal representations, and at what level of granularity? Using techniques from Bayesian networks, information theory, and decision theory, we develop a formal theory according to which causal representations reflect a trade-off between compression and informativeness, where the optimal trade-off depends on the decision-theoretic value of information for a given agent in a given context. This theory predicts that, all else being equal, agents prefer causal models that are as compressed as possible. When compression is associated with information loss, however, all else is not equal, and our theory predicts that agents will favor compressed models only when the information they sacrifice is not informative with respect to the agent’s anticipated decisions. We then show, across six studies reported here (N=2,364) and one study reported in the supplemental materials (N=182), that participants’ preferences over causal models are in keeping with the predictions of our theory. Our theory offers a unification of different dimensions of causal evaluation identified within the philosophy of science (proportionality and stability), and contributes to a more general picture of human cognition according to which the capacity to create compressed (causal) representations plays a central role"]]></description>
<dc:subject>to:NB cognitive_science graphical_models causality information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ebc6286ac206/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cognitive_science"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:graphical_models"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:causality"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/prxlife/abstract/10.1103/PRXLife.2.033009">
    <title>PRX Life 2, 033009 (2024) - Intrinsic Motivation in Dynamical Control Systems</title>
    <dc:date>2024-12-11T19:49:20+00:00</dc:date>
    <link>https://journals.aps.org/prxlife/abstract/10.1103/PRXLife.2.033009</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Biological systems often choose actions without an explicit reward signal, a phenomenon known as intrinsic motivation. The computational principles underlying this behavior remain poorly understood. In this study, we investigate an information-theoretic approach to intrinsic motivation, based on maximizing an agent's empowerment (the mutual information between its past actions and future states). We show that this approach generalizes previous attempts to formalize intrinsic motivation, and we provide a computationally efficient algorithm for computing the necessary quantities. We test our approach on several benchmark control problems, and we explain its success in guiding intrinsically motivated behaviors by relating our information-theoretic control function to fundamental properties of the dynamical system representing the combined agent-environment system. This opens the door for designing practical artificial, intrinsically motivated controllers and for linking animal behaviors to their dynamical properties."]]></description>
<dc:subject>to:NB information_theory nemenman.ilya tishby.naftali polani.daniel freedom_as_self-control</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:a53dde351dfb/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:nemenman.ilya"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:tishby.naftali"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:polani.daniel"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:freedom_as_self-control"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2410.06224">
    <title>[2410.06224] The Fast Möbius Transform: An algebraic approach to information decomposition</title>
    <dc:date>2024-12-11T19:41:27+00:00</dc:date>
    <link>https://arxiv.org/abs/2410.06224</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The partial information decomposition (PID) and its extension integrated information decomposition (ΦID) are promising frameworks to investigate information phenomena involving multiple variables. An important limitation of these approaches is the high computational cost involved in their calculation. Here we leverage fundamental algebraic properties of these decompositions to enable a computationally-efficient method to estimate them, which we call the fast Möbius transform. Our approach is based on a novel formula for estimating the Möbius function that circumvents important computational bottlenecks. We showcase the capabilities of this approach by presenting two analyses that would be unfeasible without this method: decomposing the information that neural activity at different frequency bands yield about the brain's macroscopic functional organisation, and identifying distinctive dynamical properties of the interactions between multiple voices in baroque music. Overall, our proposed approach illuminates the value of algebraic facets of information decomposition and opens the way to a wide range of future analyses."]]></description>
<dc:subject>to:NB information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:2f828f3a7111/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2401.17377">
    <title>[2401.17377] Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens</title>
    <dc:date>2024-07-17T15:10:18+00:00</dc:date>
    <link>https://arxiv.org/abs/2401.17377</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Are n-gram language models still relevant in this era of neural large language models (LLMs)? Our answer is yes, and we showcase their values in both text analysis and improving neural LLMs. This was done by modernizing n-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens. This is the largest n-gram LM ever built. Second, existing n-gram LMs use small n which hinders their performance; we instead allow n to be arbitrarily large, by introducing a new ∞-gram LM with backoff. Instead of pre-computing n-gram count tables (which would be very expensive), we develop an engine named infini-gram -- powered by suffix arrays -- that can compute ∞-gram (as well as n-gram with arbitrary n) probabilities with millisecond-level latency. The ∞-gram framework and infini-gram engine enable us to conduct many novel and interesting analyses of human-written and machine-generated text: we find that the ∞-gram LM has fairly high accuracy for next-token prediction (47%), and can complement neural LLMs to greatly reduce their perplexity. When analyzing machine-generated text, we also observe irregularities in the machine--∞-gram agreement level with respect to the suffix length, which indicates deficiencies in neural LLM pretraining and the positional embeddings of Transformers."]]></description>
<dc:subject>in_NB information_theory variable-length_markov_models_aka_context_trees large_language_models_(so_called) straight_into_my_veins have_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:068875cb896c/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:variable-length_markov_models_aka_context_trees"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:large_language_models_(so_called)"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:straight_into_my_veins"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2401.17505">
    <title>[2401.17505] Arrows of Time for Large Language Models</title>
    <dc:date>2024-07-17T15:08:29+00:00</dc:date>
    <link>https://arxiv.org/abs/2401.17505</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality, addressing a question first raised in (Shannon, 1951). For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one. This difference is at the same time subtle and very consistent across various modalities (language, model size, training time, ...). Theoretically, this is surprising: from an information-theoretic point of view, there should be no such difference. We provide a theoretical framework to explain how such an asymmetry can appear from sparsity and computational complexity considerations, and outline a number of perspectives opened by our results."]]></description>
<dc:subject>large_language_models_(so_called) information_theory in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:51cafbb6d2fb/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:large_language_models_(so_called)"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2306.04050">
    <title>[2306.04050] LLMZip: Lossless Text Compression using Large Language Models</title>
    <dc:date>2024-07-17T15:01:25+00:00</dc:date>
    <link>https://arxiv.org/abs/2306.04050</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h."]]></description>
<dc:subject>in_NB large_language_models_(so_called) information_theory entropy_estimation to_read re:large_language_models_in_statistical_perspective</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:f400f83de902/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:large_language_models_(so_called)"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:re:large_language_models_in_statistical_perspective"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.journals.uchicago.edu/doi/10.1086/661645">
    <title>From Information Theory to French Theory: Jakobson, Lévi-Strauss, and the Cybernetic Apparatus | Critical Inquiry: Vol 38, No 1</title>
    <dc:date>2024-06-15T20:40:18+00:00</dc:date>
    <link>https://www.journals.uchicago.edu/doi/10.1086/661645</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA[--- I feel like this was expanded into a book?]]></description>
<dc:subject>to_read cybernetics structuralism information_theory via:??? in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:a39bfe5d6b24/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cybernetics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:structuralism"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:???"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.010401">
    <title>Phys. Rev. Lett. 128, 010401 (2022) - Eavesdropping on the Decohering Environment: Quantum Darwinism, Amplification, and the Origin of Objective Classical Reality</title>
    <dc:date>2024-05-14T15:15:23+00:00</dc:date>
    <link>https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.128.010401</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA[“How much information about a system S can one extract from a fragment F of the environment E that decohered it?” is the central question of Quantum Darwinism. To date, most answers relied on the quantum mutual information of SF, or on the Holevo bound on the channel capacity of F to communicate the classical information encoded in S. These are reasonable upper bounds on what is really needed but much harder to calculate—the accessible information in the fragment F about S. We consider a model based on imperfect c-not gates where all the above can be computed, and discuss its implications for the emergence of objective classical reality. We find that all relevant quantities, such as the quantum mutual information as well as various bounds on the accessible information exhibit similar behavior. In the regime relevant for the emergence of objective classical reality this includes scaling independent of the quality of the imperfect c-not gates or the size of E, and even nearly independent of the initial state of S."

--- Ungated: [http://arxiv.org/abs/2107.00035]]]></description>
<dc:subject>information_theory quantum_mechanics decoherence via:? in_NB zurek.w.h.</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:71c3fdd0e944/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:quantum_mechanics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:decoherence"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:?"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:zurek.w.h."/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2402.07999">
    <title>[2402.07999] NetInfoF Framework: Measuring and Exploiting Network Usable Information</title>
    <dc:date>2024-03-12T01:33:46+00:00</dc:date>
    <link>https://arxiv.org/abs/2402.07999</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Given a node-attributed graph, and a graph task (link prediction or node classification), can we tell if a graph neural network (GNN) will perform well? More specifically, do the graph structure and the node features carry enough usable information for the task? Our goals are (1) to develop a fast tool to measure how much information is in the graph structure and in the node features, and (2) to exploit the information to solve the task, if there is enough. We propose NetInfoF, a framework including NetInfoF_Probe and NetInfoF_Act, for the measurement and the exploitation of network usable information (NUI), respectively. Given a graph data, NetInfoF_Probe measures NUI without any model training, and NetInfoF_Act solves link prediction and node classification, while two modules share the same backbone. In summary, NetInfoF has following notable advantages: (a) General, handling both link prediction and node classification; (b) Principled, with theoretical guarantee and closed-form solution; (c) Effective, thanks to the proposed adjustment to node similarity; (d) Scalable, scaling linearly with the input size. In our carefully designed synthetic datasets, NetInfoF correctly identifies the ground truth of NUI and is the only method being robust to all graph scenarios. Applied on real-world datasets, NetInfoF wins in 11 out of 12 times on link prediction compared to general GNN baselines."]]></description>
<dc:subject>to:NB network_data_analysis classifiers entropy_estimation information_theory faloutsos.christos</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:d6e3be34531e/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:network_data_analysis"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:classifiers"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:faloutsos.christos"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2207.12382">
    <title>[2207.12382] On Confidence Sequences for Bounded Random Processes via Universal Gambling Strategies</title>
    <dc:date>2023-12-08T14:36:17+00:00</dc:date>
    <link>https://arxiv.org/abs/2207.12382</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["This paper considers the problem of constructing a confidence sequence for bounded random processes. Building upon the gambling approach pioneered by Hendriks (2018) and Jun and Orabona (2019) and following the recent work of Waudby-Smith and Ramdas (2020) and Orabona and Jun (2021), this paper revisits the idea of Cover (1991)'s universal portfolio in constructing confidence sequences and demonstrates new properties, based on a natural \emph{two-horse race} perspective on the gambling approach. The main result of this paper is a new algorithm based on a mixture of lower bounds, which closely approximates the performance of Cover's universal portfolio with only constant per-round time complexity. A higher-order generalization of a lower bound in (Fan et al, 2015), which is invoked in the proposed algorithm, may be of independent interest."]]></description>
<dc:subject>to:NB prediction confidence_sets universal_prediction information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:01c7adc068b9/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:prediction"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:confidence_sets"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:universal_prediction"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2305.11042">
    <title>[2305.11042] A unified framework for information-theoretic generalization bounds</title>
    <dc:date>2023-09-29T15:59:52+00:00</dc:date>
    <link>https://arxiv.org/abs/2305.11042</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["This paper presents a general methodology for deriving information-theoretic generalization bounds for learning algorithms. The main technical tool is a probabilistic decorrelation lemma based on a change of measure and a relaxation of Young's inequality in Lψp Orlicz spaces. Using the decorrelation lemma in combination with other techniques, such as symmetrization, couplings, and chaining in the space of probability measures, we obtain new upper bounds on the generalization error, both in expectation and in high probability, and recover as special cases many of the existing generalization bounds, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. In addition, the Fernique-Talagrand upper bound on the expected supremum of a subgaussian process emerges as a special case."]]></description>
<dc:subject>in_NB learning_theory information_theory raginsky.maxim</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:89c7fa1175e6/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:raginsky.maxim"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.108.014304">
    <title>Phys. Rev. E 108, 014304 (2023) - Dynamical independence: Discovering emergent macroscopic processes in complex dynamical systems</title>
    <dc:date>2023-07-24T01:19:02+00:00</dc:date>
    <link>https://journals.aps.org/pre/abstract/10.1103/PhysRevE.108.014304</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We introduce a notion of emergence for macroscopic variables associated with highly multivariate microscopic dynamical processes. Dynamical independence instantiates the intuition of an emergent macroscopic process as one possessing the characteristics of a dynamical system “in its own right,” with its own dynamical laws distinct from those of the underlying microscopic dynamics. We quantify (departure from) dynamical independence by a transformation-invariant Shannon information-based measure of dynamical dependence. We emphasize the data-driven discovery of dynamically independent macroscopic variables, and introduce the idea of a multiscale “emergence portrait” for complex systems. We show how dynamical dependence may be computed explicitly for linear systems in both time and frequency domains, facilitating discovery of emergent phenomena across spatiotemporal scales, and outline application of the linear operationalization to inference of emergence portraits for neural systems from neurophysiological time-series data. We discuss dynamical independence for discrete- and continuous-time deterministic dynamics, with potential application to Hamiltonian mechanics and classical complex systems such as flocking and cellular automata."

--- As rvenkat says, the lack of reference to Crutchfield et al. is striking (even if I am among the alii: [https://arxiv.org/abs/cond-mat/0303625].)  On the one hand: sic transit gloria mundi, etc., etc.  On the other hand: oh come _on_.
--- The limiting case of their dynamical independence would be when the coarse-grained variable follows a deterministic process of its own.  (There are then very general reasons to expect an H theorem a la Boltzmann: [http://arxiv.org/abs/cond-mat/0508089].)  Otherwise, it would seem very hard for to avoid some leakage of information from the microscale to the macro.  For an extreme example, let X=the continuous logistic map, say with r=4 and Y=the binary sequence that's 0 if X is =< 1/2 and 1 otherwise.  (This is the "generating" partition.)  The latter, symbolic-dynamical sequence is in fact a perfect model of IID coin-tossing (a Bernoulli(0.5) stochastic process), so conditioning on the past of Y gives no information about its future, but conditioning on X gives perfect information about the future of Y.  If conditioning on X seems like cheating, say X'=the discrete symbol sequence we get by dividing [0,1] into pre-pre-pre-... pre-images of the cells of the Y partition.  X' is discrete, but depending on how much we refined the generating partition, it lets us look arbitrarily far into the future of Y.  (We'd still get a lot of information from X'' which just divides [0,1] into many equal-length intervals.)  Now to be quite fair there are places where they acknowledge that "dynamical independence" will generally be imperfect, etc.
--- As for treating everything as a linear-and-Gaussian process, I realize the authors have gotten away with publishing that advice for decades at this point, but it was always dumb, and I think if you pressed them they'd admit it.]]></description>
<dc:subject>complexity_measures information_theory via:rvenkat emergence macro_from_micro transfer_entropy in_NB have_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ebde8153c608/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:complexity_measures"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:rvenkat"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:emergence"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:macro_from_micro"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://ieeexplore.ieee.org/document/9737725">
    <title>Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation | IEEE Journals &amp; Magazine | IEEE Xplore</title>
    <dc:date>2023-06-28T16:21:16+00:00</dc:date>
    <link>https://ieeexplore.ieee.org/document/9737725</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in function spaces. However, a systematic and practical approach to improving the tightness of such variational formulas, and accordingly accelerate statistical learning and estimation from data, is currently lacking. Here we develop such a methodology for building new, tighter variational representations of divergences. Our approach relies on improved objective functionals constructed via an auxiliary optimization problem. Furthermore, the calculation of the functional Hessian of objective functionals unveils the local curvature differences around the common optimal variational solution; this quantifies and orders the tightness gains between different variational representations. Finally, numerical simulations utilizing neural network optimization demonstrate that tighter representations can result in significantly faster learning and more accurate estimation of divergences in both synthetic and real datasets (of more than 1000 dimensions), often accelerated by nearly an order of magnitude."]]></description>
<dc:subject>information_theory entropy_estimation in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:083cce09b1e2/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://ieeexplore.ieee.org/abstract/document/1101710?casa_token=fmWVrk6dbi0AAAAA:TXo6urBwf2bD2CP9jPDGqaPGj03Zg50-inwRk3gb-z3C6RC8fZ3UcNaCJAWFU3nclfhj6uE">
    <title>Teams, signaling, and information theory | IEEE Journals &amp; Magazine | IEEE Xplore</title>
    <dc:date>2023-06-08T21:50:43+00:00</dc:date>
    <link>https://ieeexplore.ieee.org/abstract/document/1101710?casa_token=fmWVrk6dbi0AAAAA:TXo6urBwf2bD2CP9jPDGqaPGj03Zg50-inwRk3gb-z3C6RC8fZ3UcNaCJAWFU3nclfhj6uE</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The purpose of this paper is to unify results from three separate and, at least superficially, unrelated subject matters, namely, team decision theory, market signaling in economics, and the classical Shannon information theory."

--- Ungated copy: http://people.eecs.berkeley.edu/~wong/wong_pubs/wong56.pdf]]></description>
<dc:subject>in_NB information_theory economics via:mraginsky collective_cognition have_read control_theory_and_control_engineering distributed_systems</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:cabb49a09007/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:economics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:mraginsky"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:collective_cognition"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:control_theory_and_control_engineering"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:distributed_systems"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2305.02960">
    <title>[2305.02960] Majorizing Measures, Codes, and Information</title>
    <dc:date>2023-05-06T22:44:46+00:00</dc:date>
    <link>https://arxiv.org/abs/2305.02960</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The majorizing measure theorem of Fernique and Talagrand is a fundamental result in the theory of random processes. It relates the boundedness of random processes indexed by elements of a metric space to complexity measures arising from certain multiscale combinatorial structures, such as packing and covering trees. This paper builds on the ideas first outlined in a little-noticed preprint of Andreas Maurer to present an information-theoretic perspective on the majorizing measure theorem, according to which the boundedness of random processes is phrased in terms of the existence of efficient variable-length codes for the elements of the indexing metric space."]]></description>
<dc:subject>to_read empirical_processes information_theory raginsky.maxim in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:1d743b81e124/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:empirical_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:raginsky.maxim"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2302.05380">
    <title>[2302.05380] On the Interventional Kullback-Leibler Divergence</title>
    <dc:date>2023-05-02T21:15:08+00:00</dc:date>
    <link>https://arxiv.org/abs/2302.05380</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Modern machine learning approaches excel in static settings where a large amount of i.i.d. training data are available for a given task. In a dynamic environment, though, an intelligent agent needs to be able to transfer knowledge and re-use learned components across domains. It has been argued that this may be possible through causal models, aiming to mirror the modularity of the real world in terms of independent causal mechanisms. However, the true causal structure underlying a given set of data is generally not identifiable, so it is desirable to have means to quantify differences between models (e.g., between the ground truth and an estimate), on both the observational and interventional level.
"In the present work, we introduce the Interventional Kullback-Leibler (IKL) divergence to quantify both structural and distributional differences between models based on a finite set of multi-environment distributions generated by interventions from the ground truth. Since we generally cannot quantify all differences between causal models for every finite set of interventional distributions, we propose a sufficient condition on the intervention targets to identify subsets of observed variables on which the models provably agree or disagree."]]></description>
<dc:subject>in_NB causal_inference information_theory via:rvenkat scholkopf.bernhard</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:c5364517c43b/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:causal_inference"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:rvenkat"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:scholkopf.bernhard"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/pra/abstract/10.1103/PhysRevA.33.1134">
    <title>Phys. Rev. A 33, 1134 (1986) - Independent coordinates for strange attractors from mutual information</title>
    <dc:date>2023-04-24T21:53:56+00:00</dc:date>
    <link>https://journals.aps.org/pra/abstract/10.1103/PhysRevA.33.1134</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The mutual information I is examined for a model dynamical system and for chaotic data from an experiment on the Belousov-Zhabotinskii reaction. An N logN algorithm for calculating I is presented. As proposed by Shaw, a minimum in I is found to be a good criterion for the choice of time delay in phase-portrait reconstruction from time-series data. This criterion is shown to be far superior to choosing a zero of the autocorrelation function."]]></description>
<dc:subject>have_read state-space_reconstruction dynamical_systems time_series information_theory fraser.andrew_m. swinney.harry_l. have_taught cleaning_out_the_filing_cabinet_for_the_first_time_since_2005 in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:7c8c3564ea61/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:state-space_reconstruction"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:dynamical_systems"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:time_series"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:fraser.andrew_m."/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:swinney.harry_l."/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_taught"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cleaning_out_the_filing_cabinet_for_the_first_time_since_2005"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.62.7508">
    <title>Phys. Rev. E 62, 7508 (2000) - Measuring statistical dependence and coupling of subsystems</title>
    <dc:date>2023-04-24T21:35:17+00:00</dc:date>
    <link>https://journals.aps.org/pre/abstract/10.1103/PhysRevE.62.7508</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We investigate recently proposed measures for the statistical dependence of systems with complex dynamical behavior. We consider appropriate model systems, to ensure that influences of individual properties of the systems are excluded. We demonstrate that it is indeed possible to obtain nontrivial directional information, but we also argue that the interpretation of this information is difficult."]]></description>
<dc:subject>to:NB to_reread directed_information_and_transfer_entropy information_theory cleaning_out_the_filing_cabinet_for_the_first_time_since_2005 have_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ecfc45894742/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_reread"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:directed_information_and_transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cleaning_out_the_filing_cabinet_for_the_first_time_since_2005"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/1102.0250">
    <title>[1102.0250] Information-Theoretic Viewpoints on Optimal Causal Coding-Decoding Problems</title>
    <dc:date>2023-04-22T02:05:50+00:00</dc:date>
    <link>https://arxiv.org/abs/1102.0250</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["In this paper we consider an interacting two-agent sequential decision-making problem consisting of a Markov source process, a causal encoder with feedback, and a causal decoder. Motivated by a desire to foster links between control and information theory, we augment the standard formulation by considering general alphabets and a cost function operating on current and previous symbols. Using dynamic programming, we provide a structural result whereby an optimal scheme exists that operates on appropriate sufficient statistics. We emphasize an example where the decoder alphabet lies in a space of beliefs on the source alphabet, and the additive cost function is a log likelihood ratio pertaining to sequential information gain. We also consider the inverse optimal control problem, where a fixed encoder/decoder pair satisfying statistical conditions is shown to be optimal for some cost function, using probabilistic matching. We provide examples of the applicability of this framework to communication with feedback, hidden Markov models and the nonlinear filter, decentralized control, brain-machine interfaces, and queuing theory."]]></description>
<dc:subject>to:NB control_theory_and_control_engineering information_theory via:mraginsky coleman.todd_p.</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:a63bd2599b56/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:control_theory_and_control_engineering"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:mraginsky"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:coleman.todd_p."/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://dml.cz/bitstream/handle/10338.dmlcz/135833/Kybernetika_44-2008-1_5.pdf">
    <title>Stochastic Control Optimal in the Kullback Sense</title>
    <dc:date>2023-04-22T01:19:34+00:00</dc:date>
    <link>https://dml.cz/bitstream/handle/10338.dmlcz/135833/Kybernetika_44-2008-1_5.pdf</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The paper solves the problem of minimization of the Kullback divergence between a partially known and a completely known probability distribution. It considers two probability
distributions of a random vector (u1, x1, . . . , uT , xT ) on a sample space of 2T dimensions.
One of the distributions is known, the other is known only partially. Namely, only the
conditional probability distributions of xτ given u1, x1, . . . , uτ−1, xτ−1, uτ are known for
τ = 1, . . . , T. Our objective is to determine the remaining conditional probability distributions of uτ given u1, x1, . . . , uτ−1, xτ−1 such that the Kullback divergence of the partially
known distribution with respect to the completely known distribution is minimal. Explicit
solution of this problem has been found previously for Markovian systems in Karn´y [6].
The general solution is given in this paper."]]></description>
<dc:subject>to:NB control_theory_and_control_engineering information_theory via:mraginsky</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:0ab3127abfee/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:control_theory_and_control_engineering"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:mraginsky"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2302.09780">
    <title>[2302.09780] Compressing Tabular Data via Latent Variable Estimation</title>
    <dc:date>2023-03-18T13:59:36+00:00</dc:date>
    <link>https://arxiv.org/abs/2302.09780</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Data used for analytics and machine learning often take the form of tables with categorical entries. We introduce a family of lossless compression algorithms for such data that proceed in four steps: (i) Estimate latent variables associated to rows and columns; (ii) Partition the table in blocks according to the row/column latents; (iii) Apply a sequential (e.g. Lempel-Ziv) coder to each of the blocks; (iv) Append a compressed encoding of the latents.
"We evaluate it on several benchmark datasets, and study optimal compression in a probabilistic model for that tabular data, whereby latent values are independent and table entries are conditionally independent given the latent values. We prove that the model has a well defined entropy rate and satisfies an asymptotic equipartition property. We also prove that classical compression schemes such as Lempel-Ziv and finite-state encoders do not achieve this rate. On the other hand, the latent estimation strategy outlined above achieves the optimal rate."]]></description>
<dc:subject>to:NB information_theory montanari.andrea</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ca96e7d369cd/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:montanari.andrea"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2303.05369">
    <title>[2303.05369] Data-dependent Generalization Bounds via Variable-Size Compressibility</title>
    <dc:date>2023-03-18T13:51:32+00:00</dc:date>
    <link>https://arxiv.org/abs/2303.05369</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["In this paper, we establish novel data-dependent upper bounds on the generalization error through the lens of a "variable-size compressibility" framework that we introduce newly here. In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data. This is shown to yield bounds that depend on the empirical measure of the given input data at hand, rather than its unknown distribution. Our new generalization bounds that we establish are tail bounds, tail bounds on the expectation, and in-expectations bounds. Moreover, it is shown that our framework also allows to derive general bounds on any function of the input data and output hypothesis random variables. In particular, these general bounds are shown to subsume and possibly improve over several existing PAC-Bayes and data-dependent intrinsic dimension-based bounds that are recovered as special cases, thus unveiling a unifying character of our approach. For instance, a new data-dependent intrinsic dimension based bounds is established, which connects the generalization error to the optimization trajectories and reveals various interesting connections with rate-distortion dimension of process, Rényi information dimension of process, and metric mean dimension."]]></description>
<dc:subject>information_theory learning_theory to_read in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:09b32ab25cd5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web">
    <title>ChatGPT Is a Blurry JPEG of the Web | The New Yorker</title>
    <dc:date>2023-02-13T14:59:17+00:00</dc:date>
    <link>https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA[Where the last tag means "ask AEO for her hard-copy so I can actually read it some night"

--- ETA after reading: this is really great.]]></description>
<dc:subject>information_theory large_language_models_(so_called) chiang.ted natural_language_processing have_read tracked_down_references in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:feb4fe7ee7ac/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:large_language_models_(so_called)"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:chiang.ted"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:natural_language_processing"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:tracked_down_references"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2212.13556">
    <title>[2212.13556] Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization</title>
    <dc:date>2023-01-18T03:11:36+00:00</dc:date>
    <link>https://arxiv.org/abs/2212.13556</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["To date, no "information-theoretic" frameworks for reasoning about generalization error have been shown to establish minimax rates for gradient descent in the setting of stochastic convex optimization. In this work, we consider the prospect of establishing such rates via several existing information-theoretic frameworks: input-output mutual information bounds, conditional mutual information bounds and variants, PAC-Bayes bounds, and recent conditional variants thereof. We prove that none of these bounds are able to establish minimax rates. We then consider a common tactic employed in studying gradient methods, whereby the final iterate is corrupted by Gaussian noise, producing a noisy "surrogate" algorithm. We prove that minimax rates cannot be established via the analysis of such surrogates. Our results suggest that new ideas are required to analyze gradient descent using information-theoretic techniques."]]></description>
<dc:subject>to:NB optimization information_theory learning_theory minimax</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:c922f5bb9181/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:optimization"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:minimax"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www-degruyter-com.cmu.idm.oclc.org/document/doi/10.1515/9781478023630/html#overview">
    <title>Code: From Information Theory to French Theory</title>
    <dc:date>2023-01-17T05:10:01+00:00</dc:date>
    <link>https://www-degruyter-com.cmu.idm.oclc.org/document/doi/10.1515/9781478023630/html#overview</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Bernard Dionysius Geoghegan traces the shared intellectual and political history of computer scientists, cyberneticists, anthropologists, linguists, and theorists across the humanities as they developed a communication and computational-based theory that grasped culture and society in terms of codes."]]></description>
<dc:subject>in_NB books:noted history_of_ideas information_theory cybernetics structuralism the_french_disease</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:75e147a7d9e2/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:noted"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:history_of_ideas"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cybernetics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:structuralism"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:the_french_disease"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2202.00666">
    <title>[2202.00666] Locally Typical Sampling</title>
    <dc:date>2023-01-17T02:27:30+00:00</dc:date>
    <link>https://arxiv.org/abs/2202.00666</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions."]]></description>
<dc:subject>natural_language_processing stochastic_processes to_read information_theory in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:236279d2f53c/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:natural_language_processing"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2202.04985">
    <title>[2202.04985] Generalization Bounds via Convex Analysis</title>
    <dc:date>2022-08-25T16:12:32+00:00</dc:date>
    <link>https://arxiv.org/abs/2202.04985</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Since the celebrated works of Russo and Zou (2016,2019) and Xu and Raginsky (2017), it has been well known that the generalization error of supervised learning algorithms can be bounded in terms of the mutual information between their input and the output, given that the loss of any fixed hypothesis has a subgaussian tail. In this work, we generalize this result beyond the standard choice of Shannon's mutual information to measure the dependence between the input and the output. Our main result shows that it is indeed possible to replace the mutual information by any strongly convex function of the joint input-output distribution, with the subgaussianity condition on the losses replaced by a bound on an appropriately chosen norm capturing the geometry of the dependence measure. This allows us to derive a range of generalization bounds that are either entirely new or strengthen previously known ones. Examples include bounds stated in terms of p-norm divergences and the Wasserstein-2 distance, which are respectively applicable for heavy-tailed loss distributions and highly smooth loss functions. Our analysis is entirely based on elementary tools from convex analysis by tracking the growth of a potential function associated with the dependence measure and the loss function."

--- Last tag is almost certainly too ambitious...]]></description>
<dc:subject>to:NB learning_theory information_theory convexity to_teach:childs_garden_of_statistical_learning_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:113aa8fd4ed1/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:convexity"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:childs_garden_of_statistical_learning_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://jmlr.org/papers/v23/20-644.html">
    <title>Data-Derived Weak Universal Consistency</title>
    <dc:date>2022-07-19T13:56:11+00:00</dc:date>
    <link>https://jmlr.org/papers/v23/20-644.html</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Many current applications in data science need rich model classes to adequately represent the statistics that may be driving the observations. Such rich model classes may be too complex to admit uniformly consistent estimators. In such cases, it is conventional to settle for estimators with guarantees on convergence rate where the performance can be bounded in a model-dependent way, i.e. pointwise consistent estimators. But this viewpoint has the practical drawback that estimator performance is a function of the unknown model within the model class that is being estimated. Even if an estimator is consistent, how well it is doing at any given time may not be clear, no matter what the sample size of the observations. In these cases, a line of analysis favors sample dependent guarantees. We explore this framework by studying rich model classes that may only admit pointwise consistency guarantees, yet enough information about the unknown model driving the observations needed to gauge estimator accuracy can be inferred from the sample at hand. In this paper we obtain a novel characterization of lossless compression problems over a countable alphabet in the data-derived framework in terms of what we term deceptive distributions. We also show that the ability to estimate the redundancy of compressing memoryless sources is equivalent to learning the underlying single-letter marginal in a data-derived fashion. We expect that the methodology underlying such characterizations in a data-derived estimation framework will be broadly applicable to a wide range of estimation problems, enabling a more systematic approach to data-derived guarantees."

--- Last tag is contingent on reading it and liking it, of course.  
]]></description>
<dc:subject>in_NB learning_theory statistics information_theory to_read to_teach:childs_garden_of_statistical_learning_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:7ba350217b53/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:statistics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:childs_garden_of_statistical_learning_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.biorxiv.org/content/10.1101/2022.07.02.498577v1">
    <title>The cost of information acquisition by natural selection | bioRxiv</title>
    <dc:date>2022-07-19T13:31:40+00:00</dc:date>
    <link>https://www.biorxiv.org/content/10.1101/2022.07.02.498577v1</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Natural selection enriches genotypes that are well-adapted to their environment. Over successive generations, these changes to the frequencies of types accumulate information about the selective conditions. Thus, we can think of selection as an algorithm by which populations acquire information about their environment. Kimura (1961) pointed out that every bit of information that the population gains this way comes with a minimum cost in terms of unrealized fitness (substitution load). Due to the gradual nature of selection and ongoing mismatch of types with the environment, a population that is still gaining information about the environment has lower mean fitness than a counter-factual population that already has this information. This has been an influential insight, but here we find that experimental evolution of Escherichia coli with mutations in a RNA polymerase gene (rpoB) violates Kimura’s basic theory. To overcome the restrictive assumptions of Kimura’s substitution load and develop a more robust measure for the cost of selection, we turn to ideas from computational learning theory. We reframe the ‘learning problem’ faced by an evolving population as a population versus environment (PvE) game, which can be applied to settings beyond Kimura’s theory – such as stochastic environments, frequency-dependent selection, and arbitrary environmental change. We show that the learning theoretic concept of ‘regret’ measures relative lineage fitness and rigorously captures the efficiency of selection as a learning process. This lets us establish general bounds on the cost of information acquisition by natural selection. We empirically validate these bounds in our experimental system, showing that computational learning theory can account for the observations that violate Kimura’s theory. Finally, we note that natural selection is a highly effective learning process in that selection is an asymptotically optimal algorithm for the problem faced by evolving populations, and no other algorithm can consistently outperform selection in general. Our results highlight the centrality of information to natural selection and the value of computational learning theory as a perspective on evolutionary biology."

--- Huh, I guess Haldane's measure of selection _is_ like a log-probability-loss.]]></description>
<dc:subject>to:NB to_read information_theory evolutionary_biology low-regret_learning bergstrom.carl_t. re:do-institutions-evolve</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:db39fe1ff713/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:evolutionary_biology"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:low-regret_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:bergstrom.carl_t."/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:re:do-institutions-evolve"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2206.02340">
    <title>[2206.02340] Minimizing the Expected Posterior Entropy Yields Optimal Summary Statistics</title>
    <dc:date>2022-07-02T14:15:13+00:00</dc:date>
    <link>https://arxiv.org/abs/2206.02340</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Extracting low-dimensional summary statistics from large datasets is essential for efficient (likelihood-free) inference. We propose obtaining summary statistics by minimizing the expected posterior entropy (EPE) under the prior predictive distribution of the model. We show that minimizing the EPE is equivalent to learning a conditional density estimator for the posterior as well as other information-theoretic approaches. Further summary extraction methods (including minimizing the L2 Bayes risk, maximizing the Fisher information, and model selection approaches) are special or limiting cases of EPE minimization. We demonstrate that the approach yields high fidelity summary statistics by applying it to both a synthetic benchmark as well as a population genetics problem. We not only offer concrete recommendations for practitioners but also provide a unifying perspective for obtaining informative summary statistics."]]></description>
<dc:subject>to:NB information_theory approximate_bayesian_computation</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:6fda8722f915/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:approximate_bayesian_computation"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2203.08089">
    <title>[2203.08089] On Suspicious Coincidences and Pointwise Mutual Information</title>
    <dc:date>2022-06-15T18:56:22+00:00</dc:date>
    <link>https://arxiv.org/abs/2203.08089</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Barlow (1985) hypothesized that the co-occurrence of two events A and B is "suspicious" if P(A,B)≫P(A)P(B). We first review classical measures of association for 2×2 contingency tables, including Yule's Y (Yule, 1912), which depends only on the odds ratio λ, and is independent of the marginal probabilities of the table. We then discuss the mutual information (MI) and pointwise mutual information (PMI), which depend on the ratio P(A,B)/P(A)P(B), as measures of association. We show that, once the effect of the marginals is removed, MI and PMI behave similarly to Y as functions of λ. The pointwise mutual information is used extensively in some research communities for flagging suspicious coincidences, but it is important to bear in mind the sensitivity of the PMI to the marginals, with increased scores for sparser events."]]></description>
<dc:subject>to:NB likelihood hypothesis_testing information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:1a4b2498b6a5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:likelihood"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:hypothesis_testing"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2206.04640">
    <title>[2206.04640] Regret Bounds for Information-Directed Reinforcement Learning</title>
    <dc:date>2022-06-10T14:13:21+00:00</dc:date>
    <link>https://arxiv.org/abs/2206.04640</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Information-directed sampling (IDS) has revealed its potential as a data-efficient algorithm for reinforcement learning (RL). However, theoretical understanding of IDS for Markov Decision Processes (MDPs) is still limited. We develop novel information-theoretic tools to bound the information ratio and cumulative information gain about the learning target. Our theoretical results shed light on the importance of choosing the learning target such that the practitioners can balance the computation and regret bounds. As a consequence, we derive prior-free Bayesian regret bounds for vanilla-IDS which learns the whole environment under tabular finite-horizon MDPs. In addition, we propose a computationally-efficient regularized-IDS that maximizes an additive form rather than the ratio form and show that it enjoys the same regret bound as vanilla-IDS. With the aid of rate-distortion theory, we improve the regret bound by learning a surrogate, less informative environment. Furthermore, we extend our analysis to linear MDPs and prove similar regret bounds for Thompson sampling as a by-product."]]></description>
<dc:subject>to:NB reinforcement_learning low-regret_learning information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:b9eba2acd75f/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:reinforcement_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:low-regret_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2206.02072">
    <title>[2206.02072] Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning</title>
    <dc:date>2022-06-09T08:28:12+00:00</dc:date>
    <link>https://arxiv.org/abs/2206.02072</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment, whose underlying dynamics likely exceed the agent's capacity for representation. In this work, we consider the scenario where agent limitations may entirely preclude identifying an exactly value-equivalent model, immediately giving rise to a trade-off between identifying a model that is simple enough to learn while only incurring bounded sub-optimality. To address this problem, we introduce an algorithm that, using rate-distortion theory, iteratively computes an approximately-value-equivalent, lossy compression of the environment which an agent may feasibly target in lieu of the true model. We prove an information-theoretic, Bayesian regret bound for our algorithm that holds for any finite-horizon, episodic sequential decision-making problem. Crucially, our regret bound can be expressed in one of two possible forms, providing a performance guarantee for finding either the simplest model that achieves a desired sub-optimality gap or, alternatively, the best model given a limit on agent capacity."]]></description>
<dc:subject>to:NB reinforcement_learning information_theory van_roy.benjamin misspecification</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:6d4a69c92fca/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:reinforcement_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:van_roy.benjamin"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:misspecification"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2206.02025">
    <title>[2206.02025] Between Rate-Distortion Theory &amp; Value Equivalence in Model-Based Reinforcement Learning</title>
    <dc:date>2022-06-09T08:27:21+00:00</dc:date>
    <link>https://arxiv.org/abs/2206.02025</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment. In this work, we entertain an extreme scenario wherein some combination of immense environment complexity and limited agent capacity entirely precludes identifying an exactly value-equivalent model. In light of this, we embrace a notion of approximate value equivalence and introduce an algorithm for incrementally synthesizing simple and useful approximations of the environment from which an agent might still recover near-optimal behavior. Crucially, we recognize the information-theoretic nature of this lossy environment compression problem and use the appropriate tools of rate-distortion theory to make mathematically precise how value equivalence can lend tractability to otherwise intractable sequential decision-making problems."]]></description>
<dc:subject>to:NB reinforcement_learning information_theory misspecification van_roy.benjamin</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:b5af81210f25/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:reinforcement_learning"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:misspecification"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:van_roy.benjamin"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2205.15227">
    <title>[2205.15227] Geometrical approach to excess/housekeeping entropy production in discrete systems</title>
    <dc:date>2022-06-09T08:26:41+00:00</dc:date>
    <link>https://arxiv.org/abs/2205.15227</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We propose a geometrical excess/housekeeping decomposition of the entropy production for discrete systems such as Markov jump processes and chemical reaction networks. Unlike the Hatano-Sasa approach, our decomposition is always well defined, including in chemical systems that do not obey complex balance and may not have a stable steady state. We provide refinements of previously known thermodynamic uncertainty relations, including short- and finite-time relations in a Markov jump process, and a short-time relation in chemical reaction networks. In addition, our housekeeping term can be divided into cyclic contributions, thus generalizing Schnakenberg's cyclic decomposition of entropy production beyond steady states. Finally, we extend optimal transport theory by providing a thermodynamic speed limit for the L2-Wasserstein distance that holds in the absence of detailed balance or even a stable steady state."]]></description>
<dc:subject>to:NB statistical_mechanics stochastic_processes information_theory non-equilibrium</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:dc284accc571/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:statistical_mechanics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:non-equilibrium"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2112.14721">
    <title>[2112.14721] Decomposing the local arrow of time in interacting systems</title>
    <dc:date>2022-06-09T08:22:44+00:00</dc:date>
    <link>https://arxiv.org/abs/2112.14721</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We show that the evidence for a local arrow of time, which is equivalent to the entropy production in thermodynamic systems, can be decomposed. In a system with many degrees of freedom, there is a term that arises from the irreversible dynamics of the individual variables, and then a series of non--negative terms contributed by correlations among pairs, triplets, and higher--order combinations of variables. We illustrate this decomposition on simple models of noisy logical computations, and then apply it to the analysis of patterns of neural activity in the retina as it responds to complex dynamic visual scenes. We find that neural activity breaks detailed balance even when the visual inputs do not, and that this irreversibility arises primarily from interactions between pairs of neurons."

--- I presume this is the PRL.]]></description>
<dc:subject>to:NB information_theory entropy stochastic_processes non-equilibrium bialek.william</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:3abdcc7647cd/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:non-equilibrium"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:bialek.william"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2203.01916">
    <title>[2203.01916] Emergence of local irreversibility in complex interacting systems</title>
    <dc:date>2022-06-09T08:21:40+00:00</dc:date>
    <link>https://arxiv.org/abs/2203.01916</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Living systems are fundamentally irreversible, breaking detailed balance and establishing an arrow of time. But how does the evident arrow of time for a whole system arise from the interactions among its multiple elements? We show that the local evidence for the arrow of time, which is the entropy production for thermodynamic systems, can be decomposed. First, it can be split into two components: an independent term reflecting the dynamics of individual elements and an interaction term driven by the dependencies among elements. Adapting tools from non--equilibrium physics, we further decompose the interaction term into contributions from pairs of elements, triplets, and higher--order terms. We illustrate our methods on models of cellular sensing and logical computations, as well as on patterns of neural activity in the retina as it responds to visual inputs. We find that neural activity can define the arrow of time even when the visual inputs do not, and that the dominant contribution to this breaking of detailed balance comes from interactions among pairs of neurons."

--- I presume this is the PRE.]]></description>
<dc:subject>to:NB information_theory entropy stochastic_processes bialek.william non-equilibrium</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:eadd1135d83f/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:bialek.william"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:non-equilibrium"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2206.02765">
    <title>[2206.02765] Communication-constrained hypothesis testing: Optimality, robustness, and reverse data processing inequalities</title>
    <dc:date>2022-06-07T14:18:20+00:00</dc:date>
    <link>https://arxiv.org/abs/2206.02765</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We study hypothesis testing under communication constraints, where each sample is quantized before being revealed to a statistician. Without communication constraints, it is well known that the sample complexity of simple binary hypothesis testing is characterized by the Hellinger distance between the distributions. We show that the sample complexity of simple binary hypothesis testing under communication constraints is at most a logarithmic factor larger than in the unconstrained setting and this bound is tight. We develop a polynomial-time algorithm that achieves the aforementioned sample complexity. Our framework extends to robust hypothesis testing, where the distributions are corrupted in the total variation distance. Our proofs rely on a new reverse data processing inequality and a reverse Markov inequality, which may be of independent interest. For simple M-ary hypothesis testing, the sample complexity in the absence of communication constraints has a logarithmic dependence on M. We show that communication constraints can cause an exponential blow-up leading to Ω(M) sample complexity even for adaptive algorithms."]]></description>
<dc:subject>to:NB hypothesis_testing information_theory distributed_systems via:mraginsky</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:d72c10f5aae3/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:hypothesis_testing"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:distributed_systems"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:mraginsky"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2108.03099">
    <title>[2108.03099] Causal Inference Theory with Information Dependency Models</title>
    <dc:date>2022-06-06T12:55:51+00:00</dc:date>
    <link>https://arxiv.org/abs/2108.03099</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Inferring the potential consequences of an unobserved event is a fundamental scientific question. To this end, Pearl's celebrated do-calculus provides a set of inference rules to derive an interventional probability from an observational one. In this framework, the primitive causal relations are encoded as functional dependencies in a Structural Causal Model (SCM), which are generally mapped into a Directed Acyclic Graph (DAG) in the absence of cycles. In this paper, by contrast, we capture causality without reference to graphs or functional dependencies, but with information fields and Witsenhausen's intrinsic model. The three rules of do-calculus reduce to a unique sufficient condition for conditional independence, the topological separation, which presents interesting theoretical and practical advantages over the d-separation. With this unique rule, we can deal with systems that cannot be represented with DAGs, for instance systems with cycles and/or 'spurious' edges. We treat an example that cannot be handled-to the extent of our knowledge-with the tools of the current literature. We also explain why, in the presence of cycles, the theory of causal inference might require different tools, depending on whether the random variables are discrete or continuous."

--- The absence of a citation to Raginsky (2011) [https://arxiv.org/abs/1110.0718] is distinctly suspicious.]]></description>
<dc:subject>to:NB causality graphical_models information_theory color_me_skeptical</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:ffb707aff322/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:causality"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:graphical_models"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:color_me_skeptical"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://aip.scitation.org/doi/10.1063/5.0063384">
    <title>Integrated information as a common signature of dynamical and information-processing complexity: Chaos: An Interdisciplinary Journal of Nonlinear Science: Vol 32, No 1</title>
    <dc:date>2022-04-13T02:24:37+00:00</dc:date>
    <link>https://aip.scitation.org/doi/10.1063/5.0063384</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The apparent dichotomy between information-processing and dynamical approaches to complexity science forces researchers to choose between two diverging sets of tools and explanations, creating conflict and often hindering scientific progress. Nonetheless, given the shared theoretical goals between both approaches, it is reasonable to conjecture the existence of underlying common signatures that capture interesting behavior in both dynamical and information-processing systems. Here, we argue that a pragmatic use of integrated information theory (IIT), originally conceived in theoretical neuroscience, can provide a potential unifying framework to study complexity in general multivariate systems. By leveraging metrics put forward by the integrated information decomposition framework, our results reveal that integrated information can effectively capture surprisingly heterogeneous signatures of complexity—including metastability and criticality in networks of coupled oscillators as well as distributed computation and emergent stable particles in cellular automata—without relying on idiosyncratic, ad hoc criteria. These results show how an agnostic use of IIT can provide important steps toward bridging the gap between informational and dynamical approaches to complex systems.
"Originally conceived within theoretical neuroscience, integrated information theory (IIT) has been rarely used in other fields—such as complex systems or non-linear dynamics—despite the great value it has to offer. In this article, we inspect the basics of IIT, dissociating it from its contentious claims about the nature of consciousness. Relieved of this philosophical burden, IIT presents itself as an appealing formal framework to study complexity in biological or artificial systems, applicable in a wide range of domains. To illustrate this, we present an exploration of integrated information in complex systems and relate it to other notions of complexity commonly used in systems such as coupled oscillators and cellular automata. Through these applications, we advocate for IIT as a valuable framework capable of revealing common threads between diverging branches of complexity science."

--- On a quick skim, a lower & distorted form of what (e.g.) JPC and co have been doing since the 1990s.  Last tag applies.
]]></description>
<dc:subject>to:NB information_theory complexity_measures color_me_skeptical</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:e3b4a49d658c/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:complexity_measures"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:color_me_skeptical"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.jmlr.org/papers/v23/20-644.html">
    <title>Data-Derived Weak Universal Consistency</title>
    <dc:date>2022-03-27T15:54:29+00:00</dc:date>
    <link>https://www.jmlr.org/papers/v23/20-644.html</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Many current applications in data science need rich model classes to adequately represent the statistics that may be driving the observations. Such rich model classes may be too complex to admit uniformly consistent estimators. In such cases, it is conventional to settle for estimators with guarantees on convergence rate where the performance can be bounded in a model-dependent way, i.e. pointwise consistent estimators. But this viewpoint has the practical drawback that estimator performance is a function of the unknown model within the model class that is being estimated. Even if an estimator is consistent, how well it is doing at any given time may not be clear, no matter what the sample size of the observations. In these cases, a line of analysis favors sample dependent guarantees. We explore this framework by studying rich model classes that may only admit pointwise consistency guarantees, yet enough information about the unknown model driving the observations needed to gauge estimator accuracy can be inferred from the sample at hand. In this paper we obtain a novel characterization of lossless compression problems over a countable alphabet in the data-derived framework in terms of what we term deceptive distributions. We also show that the ability to estimate the redundancy of compressing memoryless sources is equivalent to learning the underlying single-letter marginal in a data-derived fashion. We expect that the methodology underlying such characterizations in a data-derived estimation framework will be broadly applicable to a wide range of estimation problems, enabling a more systematic approach to data-derived guarantees."

]]></description>
<dc:subject>to:NB learning_theory information_theory to_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:c0172f9b465c/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://philpapers.org/rec/JAVHAC">
    <title>Anta Javier, Historical and Conceptual Foundations of Information Physics - PhilPapers</title>
    <dc:date>2022-01-19T14:53:25+00:00</dc:date>
    <link>https://philpapers.org/rec/JAVHAC</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The main objective of this dissertation is to philosophically assess how the use of informational concepts in the field of classical thermostatistical physics has historically evolved from the late 1940s to the present day. I will first analyze in depth the main notions that form the conceptual basis on which 'informational physics' historically unfolded, encompassing (i) different entropy, probability and information notions, (ii) their multiple interpretative variations, and (iii) the formal, numerical and semantic-interpretative relationships among them. In the following, I will assess the history of informational thermophysics during the second half of the twentieth century. Firstly, I analyse the intellectual factors that gave rise to this current in the late forties (i.e., popularization of Shannon's theory, interest in a naturalized epistemology of science, etc.), then study its consolidation in the Brillouinian and Jaynesian programs, and finally claim how Carnap (1977) and his disciples tried to criticize this tendency within the scientific community. Then, I evaluate how informational physics became a predominant intellectual current in the scientific community in the nineties, made possible by the convergence of Jaynesianism and Brillouinism in proposals such as that of Tribus and McIrvine (1971) or Bekenstein (1973) and the application of algorithmic information theory into the thermophysical domain. As a sign of its radicality at this historical stage, I explore the main proposals to include information as part of our physical reality, such as Wheeler’s (1990), Stonier’s (1990) or Landauer’s (1991), detailing the main philosophical arguments (e.g., Timpson, 2013; Lombardi et al. 2016a) against those inflationary attitudes towards information. Following this historical assessment, I systematically analyze whether the descriptive exploitation of informational concepts has historically contributed to providing us with knowledge of thermophysical reality via (i) explaining thermal processes such as equilibrium approximation, (ii) advantageously predicting thermal phenomena, or (iii) enabling understanding of thermal property such as thermodynamic entropy. I argue that these epistemic shortcomings would make it impossible to draw ontological conclusions in a justified way about the physical nature of information. In conclusion, I will argue that the historical exploitation of informational concepts has not contributed significantly to the epistemic progress of thermophysics. This would lead to characterize informational proposals as 'degenerate science' (à la Lakatos 1978a) regarding classical thermostatistical physics or as theoretically underdeveloped regarding the study of the cognitive dynamics of scientists in this physical domain."]]></description>
<dc:subject>philosophy_of_science information_theory entropy physics_of_information thermodynamics in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:9580bf8064ea/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:philosophy_of_science"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:physics_of_information"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:thermodynamics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.journals.uchicago.edu/doi/abs/10.1086/718416">
    <title>A Philosopher against the Bandwagon. Carnap and the Informationalization of Thermal Physics | HOPOS: The Journal of the International Society for the History of Philosophy of Science: Vol 0, No ja</title>
    <dc:date>2022-01-19T14:47:05+00:00</dc:date>
    <link>https://www.journals.uchicago.edu/doi/abs/10.1086/718416</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA[--- Library only subscribes to the archives going up to 2015. :(
]]></description>
<dc:subject>to_read carnap.rudolf entropy information_theory thermodynamics physics_of_information re:backwards_arrow_of_time in_NB</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:49fb54d404b5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:carnap.rudolf"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:thermodynamics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:physics_of_information"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:re:backwards_arrow_of_time"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://press.uchicago.edu/ucp/books/book/chicago/I/bo113686763">
    <title>Information and Experimental Knowledge, Mattingly</title>
    <dc:date>2022-01-08T21:09:12+00:00</dc:date>
    <link>https://press.uchicago.edu/ucp/books/book/chicago/I/bo113686763</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["An ambitious new model of experimentation that will reorient our understanding of the key features of experimental practice.
"What is experimental knowledge, and how do we get it? While there is general agreement that experiment is a crucial source of scientific knowledge, how experiment generates that knowledge is far more contentious. In this book, philosopher of science James Mattingly explains how experiments function. Specifically, he discusses what it is about experimental practice that transforms observations of what may be very localized, particular, isolated systems into what may be global, general, integrated empirical knowledge. Mattingly argues that the purpose of experimentation is the same as the purpose of any other knowledge-generating enterprise—to change the state of information of the knower. This trivial-seeming point has a non-trivial consequence: to understand a knowledge-generating enterprise, we should follow the flow of information. Therefore, the account of experimental knowledge Mattingly provides is based on understanding how information flows in experiments: what facilitates that flow, what hinders it, and what characteristics allow it to flow from system to system, into the heads of researchers, and finally into our store of scientific knowledge."]]></description>
<dc:subject>to:NB books:noted philosophy_of_science experiments information_theory books:in_library downloaded</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:3151da38adf5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:noted"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:philosophy_of_science"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:experiments"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:books:in_library"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:downloaded"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/1508.01167">
    <title>[1508.01167] The Divergence Index: A Decomposable Measure of Segregation and Inequality</title>
    <dc:date>2021-10-20T14:58:42+00:00</dc:date>
    <link>https://arxiv.org/abs/1508.01167</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Decomposition analysis is a critical tool for understanding the social and spatial dimensions of inequality, segregation, and diversity. In this paper, I propose a new measure - the Divergence Index - to address the need for a decomposable measure of segregation. Although the Information Theory Index has been used to decompose segregation within and between communities, I argue that it measures relative diversity not segregation. I demonstrate the importance of this conceptual distinction with two empirical analyses: I decompose segregation and relative homogeneity in the Detroit metropolitan area, and I analyze the relationship between the indexes in the 100 largest U.S. cities. I show that it is problematic to interpret the Information Theory Index as a measure of segregation, especially when analyzing local-level results or any decomposition of overall results. Segregation and diversity are important aspects of residential differentiation, and it is critical that we study each concept as the structure and stratification of the U.S. population becomes more complex."]]></description>
<dc:subject>to:NB have_read information_theory segregation social_measurement to_teach:statistics_of_inequality_and_discrimination</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:441c6a09d64b/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:segregation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:social_measurement"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:statistics_of_inequality_and_discrimination"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2110.01584">
    <title>[2110.01584] Information-theoretic generalization bounds for black-box learning algorithms</title>
    <dc:date>2021-10-11T02:10:53+00:00</dc:date>
    <link>https://arxiv.org/abs/2110.01584</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning."

--- Very excited!

--- ETA after reading: This is a lovely paper which both makes a lot of sense at the conceptual level _and_ gives decent, calculable bounds for realistic situations.  My last to_teach tag is a bit aspirational, since I'd need to do a lot of the information-theoretic background in that class, but honestly I'm now kind of tempted to add that, in order to get to this point!]]></description>
<dc:subject>information_theory learning_theory raginsky.maxim ver_steeg.greg galstyan.aram have_read to_teach:childs_garden_of_statistical_learning_theory in_NB to:blog</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:e9ad543352fe/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:raginsky.maxim"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:ver_steeg.greg"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:galstyan.aram"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:childs_garden_of_statistical_learning_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:blog"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://www.nature.com/articles/s41598-020-73380-x">
    <title>Sources of predictive information in dynamical neural networks | Scientific Reports</title>
    <dc:date>2021-07-26T14:39:03+00:00</dc:date>
    <link>https://www.nature.com/articles/s41598-020-73380-x</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Behavior involves the ongoing interaction between an organism and its environment. One of the prevailing theories of adaptive behavior is that organisms are constantly making predictions about their future environmental stimuli. However, how they acquire that predictive information is still poorly understood. Two complementary mechanisms have been proposed: predictions are generated from an agent’s internal model of the world or predictions are extracted directly from the environmental stimulus. In this work, we demonstrate that predictive information, measured using bivariate mutual information, cannot distinguish between these two kinds of systems. Furthermore, we show that predictive information cannot distinguish between organisms that are adapted to their environments and random dynamical systems exposed to the same environment. To understand the role of predictive information in adaptive behavior, we need to be able to identify where it is generated. To do this, we decompose information transfer across the different components of the organism-environment system and track the flow of information in the system over time. To validate the proposed framework, we examined it on a set of computational models of idealized agent-environment systems. Analysis of the systems revealed three key insights. First, predictive information, when sourced from the environment, can be reflected in any agent irrespective of its ability to perform a task. Second, predictive information, when sourced from the nervous system, requires special dynamics acquired during the process of adapting to the environment. Third, the magnitude of predictive information in a system can be different for the same task if the environmental structure changes."]]></description>
<dc:subject>to:NB prediction information_theory color_me_skeptical via:vaguery</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:d6fce5786969/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:prediction"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:color_me_skeptical"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:vaguery"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://marketurbanism.com/2021/07/05/is-diversity-segregation/">
    <title>Is Diversity &quot;Segregation&quot;? - Market Urbanism</title>
    <dc:date>2021-07-14T15:30:24+00:00</dc:date>
    <link>https://marketurbanism.com/2021/07/05/is-diversity-segregation/</link>
    <dc:creator>cshalizi</dc:creator><dc:subject>information_theory cities social_measurement racism to_teach:statistics_of_inequality_and_discrimination track_down_references have_read segregation</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:6ce7c57c8e0b/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:cities"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:social_measurement"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:racism"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_teach:statistics_of_inequality_and_discrimination"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:track_down_references"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:segregation"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2102.05802">
    <title>[2102.05802] Fisher Information and Mutual Information Constraints</title>
    <dc:date>2021-07-12T14:50:43+00:00</dc:date>
    <link>https://arxiv.org/abs/2102.05802</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We consider the processing of statistical samples X∼Pθ by a channel p(y|x), and characterize how the statistical information from the samples for estimating the parameter θ∈ℝd can scale with the mutual information or capacity of the channel. We show that if the statistical model has a sub-Gaussian score function, then the trace of the Fisher information matrix for estimating θ from Y can scale at most linearly with the mutual information between X and Y. We apply this result to obtain minimax lower bounds in distributed statistical estimation problems, and obtain a tight preconstant for Gaussian mean estimation. We then show how our Fisher information bound can also imply mutual information or Jensen-Shannon divergence based distributed strong data processing inequalities."]]></description>
<dc:subject>to:NB information_theory estimation minimax re:HEAS statistics</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:172027cea48f/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:minimax"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:re:HEAS"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:statistics"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2102.09584">
    <title>[2102.09584] Entropy under disintegrations</title>
    <dc:date>2021-07-12T14:49:23+00:00</dc:date>
    <link>https://arxiv.org/abs/2102.09584</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We consider the differential entropy of probability measures absolutely continuous with respect to a given σ-finite reference measure on an arbitrary measurable space. We state the asymptotic equipartition property in this general case; the result is part of the folklore but our presentation is to some extent novel. Then we study a general framework under which such entropies satisfy a chain rule: disintegrations of measures. We give an asymptotic interpretation for conditional entropies in this case. Finally, we apply our result to Haar measures in canonical relation."]]></description>
<dc:subject>to:NB information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:da13eb843581/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://publikationen.sulb.uni-saarland.de/bitstream/20.500.11880/31480/1/dissertation-marx.pdf">
    <title>Information-Theoretic Causal Discovery</title>
    <dc:date>2021-07-11T16:48:01+00:00</dc:date>
    <link>https://publikationen.sulb.uni-saarland.de/bitstream/20.500.11880/31480/1/dissertation-marx.pdf</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["It is well-known that correlation does not equal causation, but how can we infer causal relations from data? Causal discovery tries to answer precisely this question by rigorously analyzing under which assumptions it is feasible to infer causal networks from passively collected, so-called observational data. Particularly, causal discovery aims to infer a directed graph among a set of observed random variables under assumptions which are as realistic as possible.
"A key assumption in causal discovery is faithfulness. That is, we assume that separations in the true graph imply independencies in the distribution and vice versa. If faithfulness holds and we have access to a perfect independence oracle, traditional causal discovery approaches can infer the Markov equivalence class of the true causal graph—i.e., infer the correct undirected network and even some of the edge directions. In a real-world setting, faithfulness may be violated, however, and neither do we have access to such an independence oracle. Beyond that, we are interested in inferring the complete DAG structure and not just the Markov equivalence class. To circumvent or at least alleviate these limitations, we take an information-theoretic approach.
"In the first part of this thesis, we consider violations of faithfulness that can be induced by exclusive or relations or cancelling paths, and develop a weaker faithfulness assumption, called 2-adjacency faithfulness, to detect some of these mechanisms. Further, we analyze under which conditions it is possible to infer the correct DAG structure even if such violations occur.
"In the second part, we focus on independence testing via conditional mutual information (CMI). CMI is an information-theoretic measure of dependence based on Shannon entropy. We first suggest estimating CMI for discrete variables via normalized maximum likelihood instead of the plug-in maximum likelihood estimator that tends to overestimate dependencies. On top of that, we show that CMI can be consistently estimated for discrete-continuous mixture random variables by simply discretizing the continuous parts of each variable.
"Last, we consider the problem of distinguishing the two Markov equivalent graphs X → Y and Y → X, which is a necessary step towards discovering all edge directions. To solve this problem, it is inevitable to make assumptions about the generating mechanism. We build upon the idea which states that the cause is algorithmically independent of its mechanism. We propose two methods to approximate this postulate via the Minimum Description Length (MDL) principle: one for univariate numeric data and one for multivariate mixed-type data. Finally, we combine insights from our MDL-based approach and regression-based methods with strong guarantees and show we can identify cause and effect via L0-regularized regression."]]></description>
<dc:subject>to:NB causal_discovery information_theory entropy_estimation statistics</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:b23d42380320/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:causal_discovery"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:entropy_estimation"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:statistics"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2107.03789">
    <title>[2107.03789] How to measure things</title>
    <dc:date>2021-07-09T14:29:29+00:00</dc:date>
    <link>https://arxiv.org/abs/2107.03789</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["In classical information theory, a causal relationship between two random variables is typically modelled by assuming that, for every possible state of one of the variables, there exists a particular distribution of states of the second variable. Let us call these two variables the causal and caused variables, respectively. We assume that both of these random variables are continuous and one-dimensional. Carrying out independent transformations on the causal and caused variable creates two new random variables. Here, we consider transformations that are differentiable and strictly increasing. We call these increasing transformations. If, for example, the mass of an object is a caused variable, a logarithmic transformation could be applied to produce a new caused variable. Any causal relationship (as defined here) is associated with a channel capacity, which is the maximum rate that information could be sent if the causal relationship was used as a signalling system. Channel capacity is unaffected when the variables are changed by use of increasing transformations. For any causal relationship we show that there is always a way to transform the caused variable such that the entropy associated with the caused variable is independent of the value of the causal variable. Furthermore, the resulting universal entropy has an absolute value that is equal to the channel capacity associated with the causal relationship. This observation may be useful in statistical applications, and it implies that, for any causal relationship, there is a `natural' way to transform a continuous caused variable. With additional constraints on the causal relationship, we show that a natural transformation of both variables can be found such that the transformed system behaves like a good measuring device, with the expected value of the caused variable being approximately equal to the value of the causal variable."]]></description>
<dc:subject>to:NB information_theory measurement color_me_skeptical</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:da5e6b8fd589/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:measurement"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:color_me_skeptical"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2106.06511">
    <title>[2106.06511] Dynamical independence: discovering emergent macroscopic processes in complex dynamical systems</title>
    <dc:date>2021-06-18T16:55:43+00:00</dc:date>
    <link>https://arxiv.org/abs/2106.06511</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["We introduce a notion of emergence for coarse-grained macroscopic variables associated with highly-multivariate microscopic dynamical processes, in the context of a coupled dynamical environment. Dynamical independence instantiates the intuition of an emergent macroscopic process as one possessing the characteristics of a dynamical system "in its own right", with its own dynamical laws distinct from those of the underlying microscopic dynamics. We quantify (departure from) dynamical independence by a transformation-invariant Shannon information-based measure of dynamical dependence. We emphasise the data-driven discovery of dynamically-independent macroscopic variables, and introduce the idea of a multiscale "emergence portrait" for complex systems. We show how dynamical dependence may be computed explicitly for linear systems via state-space modelling, in both time and frequency domains, facilitating discovery of emergent phenomena at all spatiotemporal scales. We discuss application of the state-space operationalisation to inference of the emergence portrait for neural systems from neurophysiological time-series data. We also examine dynamical independence for discrete- and continuous-time deterministic dynamics, with potential application to Hamiltonian mechanics and classical complex systems such as flocking and cellular automata."

--- *ahem* https://arxiv.org/abs/cond-mat/0303625 *ahem*]]></description>
<dc:subject>to:NB to_read emergence macro_from_micro information_theory via:cris_moore abstraction</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:4229df5da19b/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:emergence"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:macro_from_micro"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:via:cris_moore"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:abstraction"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2106.01645">
    <title>[2106.01645] Rényi Divergence in General Hidden Markov Models</title>
    <dc:date>2021-06-07T03:55:45+00:00</dc:date>
    <link>https://arxiv.org/abs/2106.01645</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["In this paper, we examine the existence of the Rényi divergence between two time invariant general hidden Markov models with arbitrary positive initial distributions. By making use of a Markov chain representation of the probability distribution for the general hidden Markov model and eigenvalue for the associated Markovian operator, we obtain, under some regularity conditions, convergence of the Rényi divergence. By using this device, we also characterize the Rényi divergence, and obtain the Kullback-Leibler divergence as {\alpha} \rightarrow 1 of the Rényi divergence. Several examples, including the classical finite state hidden Markov models, Markov switching models, and recurrent neural networks, are given for illustration. Moreover, we develop a non-Monte Carlo method that computes the Rényi divergence of two-state Markov switching models via the underlying invariant probability measure, which is characterized by the Fredholm integral equation."]]></description>
<dc:subject>to:NB stochastic_processes markov_models state-space_models information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:acea15a6abd1/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:stochastic_processes"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:markov_models"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:state-space_models"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://jmlr.org/papers/v22/20-867.html">
    <title>Optimal Bounds between f-Divergences and Integral Probability Metrics</title>
    <dc:date>2021-06-07T02:50:08+00:00</dc:date>
    <link>https://jmlr.org/papers/v22/20-867.html</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["The families of ff-divergences (e.g. the Kullback-Leibler divergence) and Integral Probability Metrics (e.g. total variation distance or maximum mean discrepancies) are widely used to quantify the similarity between probability distributions. In this work, we systematically study the relationship between these two families from the perspective of convex duality. Starting from a tight variational representation of the ff-divergence, we derive a generalization of the moment-generating function, which we show exactly characterizes the best lower bound of the ff-divergence as a function of a given IPM. Using this characterization, we obtain new bounds while also recovering in a unified manner well-known results, such as Hoeffding's lemma, Pinsker's inequality and its extension to subgaussian functions, and the Hammersley-Chapman-Robbins bound. This characterization also allows us to prove new results on topological properties of the divergence which may be of independent interest."]]></description>
<dc:subject>to:NB information_theory probability</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:730147524cb5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:probability"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="http://www.isiweb.ee.ethz.ch/papers/arch/mass-1990-1.pdf">
    <title>Causality, Feedback and Directed Information (Massey, 1990)</title>
    <dc:date>2021-06-01T20:59:38+00:00</dc:date>
    <link>http://www.isiweb.ee.ethz.ch/papers/arch/mass-1990-1.pdf</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA[It is shown that the "usual definition" of a discrete memoryless channel
(DMC) in fact prohibits the use of feedback. The difficulty stems from the
confusion of causality and statistical dependence. An adequate definition of a
DMC is given, as well as a definition of using a channel without feedback. A
definition, closely based on an old idea of Marko, is given for the directed
information flowing from one sequence to another. This directed information
is used to give a simple proof of the well-known fact that the use of feedback
cannot increase the capacity of a DMC. It is shown that, when feedback is
present, directed information is a more useful quantity than the traditional
mutual information."]]></description>
<dc:subject>in_NB directed_information_and_transfer_entropy information_theory have_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:e70009f7024c/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:directed_information_and_transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/nlin/0001042">
    <title>[nlin/0001042] Measuring Information Transfer</title>
    <dc:date>2021-06-01T20:58:42+00:00</dc:date>
    <link>https://arxiv.org/abs/nlin/0001042</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["An information theoretic measure is derived that quantifies the statistical coherence between systems evolving in time. The standard time delayed mutual information fails to distinguish information that is actually exchanged from shared information due to common history and input signals. In our new approach, these influences are excluded by appropriate conditioning of transition probabilities. The resulting transfer entropy is able to distinguish driving and responding elements and to detect asymmetry in the coupling of subsystems."]]></description>
<dc:subject>in_NB have_read directed_information_and_transfer_entropy schreiber.thomas information_theory</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:9ec80722a0e0/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:directed_information_and_transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:schreiber.thomas"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://ieeexplore.ieee.org/abstract/document/1091610?casa_token=7_0gluYlO2gAAAAA:PYxZ1NlWfA5CZSUG89-Q_sjSrNYlYJNtc4FwlrSJFVdfag2cDPhcEzaP_IYq8MZcnfRjRb0">
    <title>The Bidirectional Communication Theory - A Generalization of Information Theory (Marko, 1973)</title>
    <dc:date>2021-06-01T20:57:20+00:00</dc:date>
    <link>https://ieeexplore.ieee.org/abstract/document/1091610?casa_token=7_0gluYlO2gAAAAA:PYxZ1NlWfA5CZSUG89-Q_sjSrNYlYJNtc4FwlrSJFVdfag2cDPhcEzaP_IYq8MZcnfRjRb0</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["A generalization of information theory is presented with the aim of distinguishing the direction of information flow for mutually coupled statistical systems. The bidirectional communication theory refers to two systems. Two directed transinformations are defined which are a measure of the statistical coupling between the systems. Their sum equals Shannon's transinformation. An information flow diagram explains the relation between the directed transinformations and the entropies of the sources. An extension to a group of such systems has also been proposed. The theory is able to describe the informational relationships between living beings and other multivariate complex systems as encountered in economy. An application example referring to group behavior with monkeys is given."

--- Cited by Massey's 1990 directed information paper.  Marko's "directed transinformation" is in fact almost exactly the same as Schreiber's (2000) "transfer entropy".  The application is... uncompelling on first reading.]]></description>
<dc:subject>in_NB directed_information_and_transfer_entropy information_theory have_read</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:c824d75e33e5/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:in_NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:directed_information_and_transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:have_read"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://arxiv.org/abs/2105.11940">
    <title>[2105.11940] Response and flux of information in extended non-equilibrium dynamics</title>
    <dc:date>2021-05-26T15:53:50+00:00</dc:date>
    <link>https://arxiv.org/abs/2105.11940</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["It is well known that entropy production is a proxy to the detection of non-equilibrium, i.e. of the absence of detailed balance; however, due to the global character of this quantity, its knowledge does not allow to identify spatial currents or fluxes of information among specific elements of the system under study. In this respect, much more insight can be gained by studying transfer entropy and response, which allow quantifying the relative influence of parts of the system and the asymmetry of the fluxes. In order to understand the relation between the above-mentioned quantities, we investigate spatially asymmetric extended systems. First, we consider a simplified linear stochastic model, which can be studied analytically; then, we include nonlinear terms in the dynamics. Extensive numerical investigation shows the relation between entropy production and the above-introduced degrees of asymmetry. Finally, we apply our approach to the highly nontrivial dynamics generated by the Lorenz '96 model for Earth oceanic circulation."

--- I need notebooks on information flow, and on directed information and transfer entropy.]]></description>
<dc:subject>to:NB information_theory non-equilibrium statistical_mechanics directed_information_and_transfer_entropy vulpiani.angelo information_flow</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:23117255ee93/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:non-equilibrium"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:statistical_mechanics"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:directed_information_and_transfer_entropy"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:vulpiani.angelo"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_flow"/>
</rdf:Bag></taxo:topics>
</item>
<item rdf:about="https://journals.aps.org/pre/abstract/10.1103/PhysRevE.103.052408">
    <title>Phys. Rev. E 103, 052408 (2021) - Mechanisms underlying vaccination protocols that may optimally elicit broadly neutralizing antibodies against highly mutable pathogens</title>
    <dc:date>2021-05-18T20:00:30+00:00</dc:date>
    <link>https://journals.aps.org/pre/abstract/10.1103/PhysRevE.103.052408</link>
    <dc:creator>cshalizi</dc:creator><description><![CDATA["Effective prophylactic vaccines usually induce the immune system to generate potent antibodies that can bind to an antigen and thus prevent it from infecting host cells. B cells produce antibodies by a Darwinian evolutionary process called affinity maturation (AM). During AM, the B cell population evolves in response to the antigen to produce antibodies that bind specifically and strongly to the antigen. Highly mutable pathogens pose a major challenge to the development of effective vaccines because antibodies that are effective against one strain of the virus may not protect against a mutant strain. Antibodies that can protect against diverse strains of a mutable pathogen have high “breadth” and are called broadly neutralizing antibodies (bnAbs). In spite of extensive studies, an effective vaccination strategy that can generate bnAbs in humans does not exist for any highly mutable pathogen. Here we study a minimal model to explore the mechanisms underlying how the selection forces imposed by antigens can be optimally chosen to guide AM to maximize the evolution of bnAbs. For logistical reasons, only a finite number of antigens can be administered in a finite number of vaccinations; that is, guiding the nonequilibrium dynamics of AM to produce bnAbs must be accomplished nonadiabatically. The time-varying Kullback-Leibler divergence (KLD) between the existing B cell population distribution and the fitness landscape imposed by antigens is a quantitative metric of the thermodynamic force acting on B cells. If this force is too small, adaptation is minimal. If the force is too large, contrary to expectations, adaptation is not faster; rather, the B cell population is extinguished for reasons that we describe. We define the conditions necessary for the force to be set optimally such that the flux of B cells from low to high breadth states is maximized. Even in this case we show why the dynamics of AM prevent perfect adaptation. If two shots of vaccination are allowed, the optimal protocol is characterized by a relatively low optimal KLD during the first shot that appropriately increases the diversity of the B cell population so that the surviving B cells have a high chance of evolving into bnAbs upon subsequently increasing the KLD during the second shot. Phylogenetic tree analysis further reveals the evolutionary pathways that lead to bnAbs. The connections between the mechanisms revealed by our analyses and recent simulation studies of bnAb evolution, the problem of generalist versus specialist evolution, and learning theory are discussed."]]></description>
<dc:subject>to:NB evolutionary_biology immunology information_theory re:fitness_sampling</dc:subject>
<dc:source>https://pinboard.in/</dc:source>
<dc:identifier>https://pinboard.in/u:cshalizi/b:3939038e9753/</dc:identifier>
<taxo:topics><rdf:Bag>	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:to:NB"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:evolutionary_biology"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:immunology"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:information_theory"/>
	<rdf:li rdf:resource="https://pinboard.in/u:cshalizi/t:re:fitness_sampling"/>
</rdf:Bag></taxo:topics>
</item>
</rdf:RDF>