导语
建模和推断是大多数科学领域的核心，尤其是对不断发展的复杂系统。关键是，我们所拥有的信息往往是不确定的和不充分的，从而导致欠定推理问题（有无穷多解的问题）；多种推理、模型和理论都与我们所拥有的关于此类系统的信息相一致。信息论（特别是最大信息熵原理）提供了一种处理这种复杂性的方法。在过去几十年里，它已经被应用于许多学科内部和跨学科的许多问题。
这篇文章回顾了最大熵原理的历史发展，概述了这个理论的许多应用及其对复杂系统的拓展，并更详细地讨论了最近在构建基于这个方法的综合理论方面的一些进展。文章还讨论了在信息论推理领域的前沿工作：应用于具有时变约束的复杂动力学系统，如高度扰动的生态系统或快速变化的经济系统。本文发表于《美国国家科学院院刊》（PNAS），作者为圣塔菲研究所研究员。
Amos Golan，John Hart | 作者
刘志航 | 译者
刘培源 | 审校
邓一雪 | 编辑
地址：https://pattern.swarma.org/article/227
本文首发于“集智斑图”网站，欢迎扫码登录集智斑图，获取完整学习路径。
论文题目： Information theory: A foundation for complexity science 论文链接： https://www.pnas.org/doi/10.1073/pnas.2119089119
1. 简要的历史视角
2. 关于概率解释的旁注
3. 最近的应用：概述
4. 用于发展理论的最大熵原理
5. 生态学理论的例子
6. 经济理论的例子
7. 基于最大熵原理的批评和失败
8. 最大熵原理方法对动力学系统的扩展
9. 开放性问题
10. 小结与结论
1. 简要的历史视角
1. 简要的历史视角
2. 关于概率解释的旁注
2. 关于概率解释的旁注
3. 最近的应用：概述
3. 最近的应用：概述
4. 用于发展理论的最大熵原理
4. 用于发展理论的最大熵原理
5. 生态学理论的例子
5. 生态学理论的例子
图2. METE 的数学结构。经验上可检验的指标，如物种的丰度分布和个体的代谢率分布、物种—区域和地方性—区域关系以及能量等价原则，都来自于对该理论中两个基本分布的特定数学运算：生态结构函数 R 和空间分布 Π，而这两个分布又是在文本中规定的约束条件下用最大熵原理推导出来的。改编自参考文献[53]。
6. 经济理论的例子
6. 经济理论的例子
7. 基于最大熵原理的批评和失败
7. 基于最大熵原理的批评和失败
8. 最大熵原理方法对动力学系统的扩展
8. 最大熵原理方法对动力学系统的扩展
9. 开放性问题
9. 开放性问题
10. 小结与结论
10. 小结与结论
参考文献
1 J. Skilling, Data analysis: The maximum entropy method. Nature 309, 748–749 (1984).
2 E. T. Jaynes, Information theory and statistical mechanics. Phys. Rev. 106, 620–630 (1957).
3 J. Bernoulli, Ars Conjectandi (Thurneysen Brothers, 1713).
4 T. Simpson, A letter to the Right Honourable George Earl of Macclesfield, President of the Royal Society, on the advantage of taking the mean of a number of observations, in practical astronomy. Philos. Trans. R. Soc. Lond. 49, 82–93 (1755).
5 T. Bayes, An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. 53, 37–418 (1764).
6 A. de Moivre, The Doctrine of Chances, or a Method of Calculating the Probabilities of Events in Play (Pearson, 1718).
7 P. S. Laplace, Théorie Analytique des Probabilitis (Gauthier-Villars, ed. 3, 1886).
8 C. E. Shannon, A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423 (1948).
9 E. T. Jaynes, On the rationale of maximum-entropy method. Proc. IEEE 70, 939–952 (1982).
10 R. D. Levine, An information theoretical approach to inversion problems. J. Phys. Math. Gen. 13, 91–108 (1980).
11 R. D. Levine, M. Tribus, The Maximum Entropy Formalism (MIT Press, 1979).
12 Y. Tikochinsky, N. Z. Tishby, R. D. Levine, Alternative approach to maximum-entropy inference. Phys. Rev. A 30, 2638–2644 (1984).
13 J. Skilling, “The axioms of maximum entropy” in Maximum-Entropy and Bayesian Methods in Science and Engineering, G. J. Erickson, C. R. Smith, Eds. (Springer Netherlands, Dordrecht, the Netherlands, 1988), pp. 173–187.
14 K. M. Hanson, R. N. Silver, Eds., Maximum Entropy and Bayesian Methods (Fundamental Theories of Physics, Springer Netherlands, Dordrecht, the Netherlands, 1996), vol. 79.
15 A. Golan, G. G. Judge, D. Miller, Maximum Entropy Econometrics: Robust Estimation with Limited Data (John Wiley & Sons, 1996).
16 A. Golan, Foundations of Info-Metrics: Modeling, Inference, and Imperfect Information (Oxford University Press, 2018).
17 J. Harte, Maximum Entropy and Ecology: A Theory of Abundance, Distribution and Energetics (Oxford University Press, 2011).
18 J. Harte, K. Umemura, M. Brush, DynaMETE: A hybrid 最大熵原理-plus-mechanism theory of dynamic macroecology. Ecol. Lett. 24, 935–949 (2021).
19 J. E. Shore, R. W. Johnson, Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans. Inf. Theory 26, 26–37 (1980).
20 I. Csiszar, Why least squares and maximum entropy? An aximomatic approach to inference for linear inverse problem. Ann. Stat. 19, 2032–2066 (1991).
21 G. B. Arfken, H. J. Weber, Mathematical Methods for Physicists (Elsevier, 2005).
22 J. M. Keynes, A Treatise on Probability (Macmillan, 1921).
23 F. P. Ramsey, “Truth and probability (1926)” in The Foundations of Mathematics and Other Logical Essays, R. B. Braithwaite, Ed. (K. Paul, Trench, Trubner & Co., 1931), pp. 156–198.
24 S. F. Gull, G. J. Daniell, Image reconstruction from incomplete and noisy data. Nature 272, 686–690 (1978).
25 S. F. Gull, T. J. Newton, Maximum entropy tomography. Appl. Opt. 25, 156–160 (1986).
26 R. Kikuchi, B. H. Soffer, Maximum entropy image restoration. I. The entropy expression. J. Opt. Soc. Am. 67, 1656–1665 (1977).
27 J. P. Burg, Maximum Entropy Spectral Analysis (Stanford University, 1975).
28 A. van den Bos, Alternative interpretation of maximum entropy spectral analysis (Corresp.). IEEE Trans. Inf. Theory 17, 493–494 (1971).
29 D. L. Donoho, I. M. Johnstone, J. C. Hoch, A. S. Stern, Maximum entropy and the nearly black object. J. R. Stat. Soc. B 54, 41–81 (1992).
30 A. Zellner, “Bayesian methods and entropy in economics and econometrics” in Maximum Entropy and Bayesian Methods: Laramie, Wyoming, 1990, W. T. Grandy, L. H. Schick, Eds. (Fundamental Theories of Physics, Springer Netherlands, Dordrecht, the Netherlands, 1991), vol. 43, pp. 17–31.
31 T. M. Cover, J. A. Thomas, Elements of Information Theory (Wiley-Interscience, ed. 2, 2006).
32 A. McCallum, D. Freitag, F. C. N. Pereira, “Maximum entropy Markov models for information extraction and segmentation” in Proceedings of the Seventeenth International Conference on Machine Learning (Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000), pp. 591–598.
33 A. Golan, G. Judge, S. Robinson, Recovering information from incomplete or partial multisectoral economic data. Rev. Econ. Stat. 76, 541 (1994).
34 A. Golan, S. J. Vogel, Estimation of non-stationary social accounting matrix coefficients with supply-side information. Econ. Syst. Res. 12, 447–471 (2000).
35 S. J. Phillips, R. P. Anderson, R. E. Schapire, Maximum entropy modeling of species geographic distributions. Ecol. Modell. 190, 231–259 (2006).
36 B. Haegeman, R. S. Etienne, Entropy maximization and the spatial distribution of species. Am. Nat. 175, E74–E90 (2010).
37 K. Nigam, J. Lafferty, A. Mccallum, “Using maximum entropy for text classification” in IJCAI Workshop on Machine Learning for Information Filtering (1999), pp. 61–67.
38 T. Mora, A. M. Walczak, W. Bialek, C. G. Callan Jr., Maximum entropy models for antibody diversity. Proc. Natl. Acad. Sci. U.S.A. 107, 5405–5410 (2010).
39 P. J. Steinbach, R. Ionescu, C. R. Matthews, Analysis of kinetics using a hybrid maximum-entropy/nonlinear-least-squares method: Application to protein folding. Biophys. J. 82, 2244–2255 (2002).
40 M. Weigt, R. A. White, H. Szurmant, J. A. Hoch, T. Hwa, Identification of direct residue contacts in protein-protein interaction by message passing. Proc. Natl. Acad. Sci. U.S.A. 106, 67–72 (2009).
41 F. Remacle, N. Kravchenko-Balasha, A. Levitzki, R. D. Levine, Information-theoretic analysis of phenotype changes in early stages of carcinogenesis. Proc. Natl. Acad. Sci. U.S.A. 107, 10324–10329 (2010).
42 S. Zadran, F. Remacle, R. Levine, Surprisal analysis of Glioblastoma Multiform (GBM) microRNA dynamics unveils tumor specific phenotype. PLoS One 9, e108171 (2014).
43 W. Bialek et al., Statistical mechanics for natural flocks of birds. Proc. Natl. Acad. Sci. U.S.A. 109, 4786–4791 (2012).
44 Y. Shemesh et al., High-order social interactions in groups of mice. eLife 2, e00759 (2013).
45 R. J. Williams, Simple 最大熵原理models explain food web degree distributions. Theor. Ecol. 3, 45–52 (2010).
46 R. J. Williams, Biology, methodology or chance? The degree distributions of bipartite ecological networks. PLoS One 6, e17645 (2011).
47 Y. Roudi, S. Nirenberg, P. E. Latham, Pairwise maximum entropy models for studying large biological systems: When they can work and when they can’t. PLoS Comput. Biol. 5, e1000380 (2009).
48 N. Friedman, Inferring cellular networks using probabilistic graphical models. Science 303, 799–805 (2004).
49 T. R. Lezon, J. R. Banavar, M. Cieplak, A. Maritan, N. V. Fedoroff, Using the principle of entropy maximization to infer genetic interaction networks from gene expression patterns. Proc. Natl. Acad. Sci. U.S.A. 103, 19033–19038 (2006).
50 E. Schneidman, M. J. Berry II, R. Segev, W. Bialek, Weak pairwise correlations imply strongly correlated network states in a neural population. Nature 440, 1007–1012 (2006).
51 L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, W. Bialek, Collective behavior of place and non-place neurons in the hippocampal network. Neuron 96, 1178–1191.e4 (2017).
52 H. Karloff, K. E. Shirley, Maximum Entropy Summary Trees in Computer Graphics Forum 32, No. 3 (Wiley Online Library, 2013), pp. 71–80.
53 J. Harte, E. A. Newman, Maximum information entropy: A foundation for ecological theory. Trends Ecol. Evol. 29, 384–389 (2014).
54 A. Golan, A Discrete Stochastic Model of Economic Production and A Model of Fluctuations in Production—Theory and Empirical Evidence (University of California, Berkeley, CA, 1988).
55 D. K. Foley, A statistical equilibrium theory of markets. J. Econ. Theory 62, 321–345 (1994).
56 A. Golan, J. Bono, Identifying strategies and beliefs without rationality assumptions. https://doi.org/10.2139/ssrn.1611216 (18 May 2010).
57 P. D. Grünwald, A. P. Dawid, Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. Ann. Stat. 32, 1367–1433 (2004).
58 H. Reiss, A. D. Hammerich, E. W. Montroll, Thermodynamic treatment of nonphysical systems: Formalism and an example (single-lane traffic). J. Stat. Phys. 42, 647–687 (1986).
59 S. Pressé, K. Ghosh, J. Lee, K. A. Dill, Principles of maximum entropy and maximum caliber in statistical physics. Rev. Mod. Phys. 85, 1115–1141 (2013).
60 H. Haken, Ed., Complex Systems—Operational Approaches in Neurobiology, Physics, and Computers (Springer, Berlin, Germany, 1985).
61 P. D. Dixit et al., Perspective: Maximum caliber is a general variational principle for dynamical systems. J. Chem. Phys. 148, 010901 (2018).
62 Z. F. Brotzakis, M. Vendruscolo, P. G. Bolhuis, A method of incorporating rate constants as kinetic constraints in molecular dynamics simulations. Proc. Natl. Acad. Sci. U.S.A. 118, e2012433118 (2021).
63 R. D. Levine, Information theory approach to molecular reaction dynamics. Annu. Rev. Phys. Chem. 29, 59–92 (1978).
64 R. D. Levine, Molecular Reaction Dynamics (Cambridge University Press, 2005).
65 A. Caticha, The entropic dynamics approach to quantum mechanics. Entropy (Basel) 21, 943 (2019).
66 S. Ipek, A. Caticha, The entropic dynamics of quantum scalar fields coupled to gravity. Symmetry (Basel) 12, 1324 (2020).
67 G. B. West, J. H. Brown, B. J. Enquist, A general model for ontogenetic growth. Nature 413, 628–631 (2001).
68 J. Harte, T. Zillio, E. Conlisk, A. B. Smith, Maximum entropy and the state-variable approach to macroecology. Ecology 89, 2700–2711 (2008).
69 J. Harte, A. B. Smith, D. Storch, Biodiversity scales from plots to biomes with a universal species-area curve. Ecol. Lett. 12, 789–797 (2009).
70 E. P. White, K. M. Thibault, X. Xiao, Characterizing species abundance distributions across taxa and ecosystems using a simple maximum entropy model. Ecology 93, 1772–1778 (2012).
71 X. Xiao, D. J. McGlinn, E. P. White, A strong test of the maximum entropy theory of ecology. Am. Nat. 185, E70–E80 (2015).
72 E. A. Newman et al., Disturbance macroecology: A comparative study of community structure metrics in a high‐severity disturbance regime. Ecosphere 11, e03022 (2020).
73 J. Harte, A. Rominger, W. Zhang, Integrating macroecological metrics and community taxonomic structure. Ecol. Lett. 18, 1068–1077 (2015).
74 A. Golan, A multivariable stochastic theory of size distribution of firms with empirical evidence. Adv. Econom. 10, 1–46 (1994).
75 A. Caticha, A. Golan, An entropic framework for modeling economies. Physica A 408, 149–163 (2014).
76 R. D. McKelvey, T. R. Palfrey, Quantal response equilibria for normal form games. Games Econ. Behav. 10, 6–38 (1995).
77 A. Golan, G. Judge, J. Perloff, A generalized maximum entropy approach to recovering information from multinomial response data. J. Am. Stat. Assoc. 91, 841–853 (1996).
78 E. Scharfenaker, D. Foley, Quantal response statistical equilibrium in economic interactions: Theory and estimation. Entropy (Basel) 19, 444 (2017).
79 R. A. Kempton, L. R. Taylor, Log-series and log-normal parameters as diversity discriminants for the Lepidoptera. J. Anim. Ecol. 43, 381–399 (1974).
80 S. Carey, J. Harte, R. del Moral, Effect of community assembly and primary succession on the species-area relationship in disturbed ecosystems. Ecography 29, 866–872 (2006).
81 S. R. Supp, X. Xiao, S. K. M. Ernest, E. P. White, An experimental test of the response of macroecological patterns to altered species interactions. Ecology 93, 2505–2511 (2012).
82 J. Franzman et al., Shifting macroecological patterns and static theory failure in a stressed alpine plant community. Ecosphere 12, e03548 (2021).
83 N. Goldenfeld, C. Woese, Life is physics: Evolution as a collective phenomenon far from equilibrium. Annu. Rev. Condens. Matter Phys. 2, 375–399 (2011).
（参考文献可上下滑动查看）
集智俱乐部因果涌现社区
跨尺度、跨层次的涌现是复杂系统研究的关键问题，生命起源和意识起源这两座仰之弥高的大山是其代表。而因果涌现理论、机器学习重整化技术、自指动力学等近年来新兴的理论与工具，有望破解复杂系统的涌现规律。同时，新兴的因果表示学习、量子因果等领域也将为因果涌现研究注入新鲜血液。
因果涌现读书会启动：连接因果、涌现与自指——跨尺度动力学与因果规律的探索
因果与复杂系统特刊征稿
因果与复杂系统特刊征稿
期刊：Entropy (ISSN 1099-4300)
特刊主题：因果关系与复杂系统（Causality and Complex Systems）
征稿截止日期：2023年4月25日
特刊链接：
https://www.mdpi.com/journal/entropy/special_issues/causality_complex_systems
推荐阅读