https://pinboard.in/u:sechilds/public/ recent bookmarks from sechilds 2020-11-21T12:16:34+00:00 https://twitter.com/polesasunder/status/1329898864485216258 sechilds statistics https://pinboard.in/u:sechilds/b:d8501beaaf71/ 2017-11-23T23:30:13+00:00 https://massivesci.com/articles/science-truth-error-replication-crisis/ sechilds statistics https://pinboard.in/u:sechilds/b:908c9a657e4f/ 2017-11-04T12:40:27+00:00 https://k-d-w.org/node/101 sechilds python statistics https://pinboard.in/u:sechilds/b:db2ca9d005b4/ 2017-09-08T11:42:43+00:00 https://twitter.com/chris_m_neill/status/905525042066444290 sechilds funny statistics economics https://pinboard.in/u:sechilds/b:6d01feb16f2e/ 2017-08-02T01:22:43+00:00 https://osf.io/preprints/psyarxiv/mky9j/?_ga=2.29887741.370827084.1500902659-399963933.1500902659 sechilds statistics https://pinboard.in/u:sechilds/b:26a2666646e7/ 2017-08-01T11:30:37+00:00 https://www.vox.com/science-and-health/2017/7/31/16021654/p-values-statistical-significance-redefine-0005 sechilds statistics science https://pinboard.in/u:sechilds/b:c7ec2d1681b6/ 2017-08-01T11:30:36+00:00 https://twitter.com/voxdotcom/status/892052309290741761 sechilds statistics science https://pinboard.in/u:sechilds/b:433296dc38dc/ 2017-07-30T13:43:26+00:00 https://medium.com/towards-data-science/how-to-calculate-statistical-power-for-your-meta-analysis-e108ee586ae8 sechilds statistics data:science https://pinboard.in/u:sechilds/b:b6f741a31970/ 2017-07-29T10:37:17+00:00 https://twitter.com/freakonometrics/status/891081923996594177 sechilds statistics https://pinboard.in/u:sechilds/b:fa011ba4246c/ 2017-07-27T11:02:56+00:00 https://twitter.com/quantombone/status/890356426173841408 sechilds machine_learning statistics https://pinboard.in/u:sechilds/b:61b8bba2c62a/ 2017-07-24T21:52:25+00:00 http://flowingdata.com/2017/07/24/an-interactive-to-explain-histograms-for-normal-people/ sechilds statistics data:visualization https://pinboard.in/u:sechilds/b:5da1f1d15814/ 2017-07-23T12:17:33+00:00 https://lectures.quantecon.org/py/ols.html sechilds Python statistics regression @followup PyData https://pinboard.in/u:sechilds/b:4cfda1b34839/ 2017-07-16T13:02:10+00:00 https://twitter.com/errorinn/status/886306209397178370 sechilds funny survey statistics https://pinboard.in/u:sechilds/b:ec1171b0007a/ 2017-06-29T11:24:55+00:00 https://elitedatascience.com/bias-variance-tradeoff sechilds verheard after class: “doesn’t the Bias-Variance Tradeoff sound like the name of a treaty from a history documentary?” > > Ok, that’s fair… but it’s also one of the most important concepts to understand for supervised machine learning and predictive modeling. > > Unfortunately, because it’s often taught through dense math formulas, it’s earned a tough reputation. > > But as you’ll see in this guide, it’s not that bad. In fact, the Bias-Variance Tradeoff has simple, practical implications around model complexity, over-fitting, and under-fitting.]]> data machine_learning statistics infographic @followup https://pinboard.in/ https://pinboard.in/u:sechilds/b:39d505a5fd0b/ 2017-06-27T11:24:21+00:00 http://rpsychologist.com/d3/NHST/ sechilds Type I and Type II errors, β, α, p-values, power and effect sizes – the ritual of null hypothesis significance testing contains many strange concepts. > > Much has been said about significance testing – most of it negative. Methodologists constantly point out that researchers misinterpret p-values. Some say that it is at best a meaningless exercise and at worst an impediment to scientific discoveries. Consequently, I believe it is extremely important that students and researchers correctly interpret statistical tests. This visualization is meant as an aid for students when they are learning about statistical hypothesis testing. The visualization is based on a one-sample Z-test. You can vary the sample size, power, signifance level and effect size using the sliders to see how the sampling distributions change.]]> data:visualization statistics R https://pinboard.in/ https://pinboard.in/u:sechilds/b:f31650e5a0ef/ 2017-06-27T11:24:02+00:00 http://rpsychologist.com/d3/correlation/ sechilds Correlation is one of the most widely used tools in statistics. The correlation coefficient summarizes the association between two variables. In this visualization I show a scatter plot of two variables with a given correlation. The variables are samples from the standard normal distribution, which are then transformed to have a given correlation by using Cholesky decomposition. By moving the slider you will see how the shape of the data changes as the association becomes stronger or weaker. You can also look at the Venn diagram to see the amount of shared variance between the variables. It is also possible drag the data points to see how the correlation is influenced by outliers.]]> data:visualization data statistics R https://pinboard.in/ https://pinboard.in/u:sechilds/b:41e9d1856bcd/ 2017-06-26T17:05:27+00:00 http://rpsychologist.com/d3/tdist/ sechilds 30). However, for small samples the difference is important. You might recall that the t-distribution is used when the population variance is unknown. Simply put, estimating the variance from the sample leads to greater uncertainty and a more spread out distribution, as can be seen by the t-distributions heavier tails. This interactive visualization lets you explore how the t-distribution approaches the normal distribution as the degrees of freedom increase. It also shows the absolute and relative error when the normal approximation is used.]]> R statistics https://pinboard.in/u:sechilds/b:569b4e0479cb/ 2017-06-21T18:10:41+00:00 http://rpsychologist.com/d3/cohend/ sechilds R statistics data:visualization https://pinboard.in/ https://pinboard.in/u:sechilds/b:1c4adebbc019/ 2017-06-16T12:29:34+00:00 http://xcelab.net/rm/statistical-rethinking/ sechilds statistics statistics:bayesian book R https://pinboard.in/u:sechilds/b:893a8ffc88ba/ 2017-05-20T16:11:23+00:00 http://www.cookbook-r.com/Statistical_analysis/t-test/ sechilds You want to test whether two samples are drawn from populations with different means, or test whether one sample is drawn from a population with a mean different from some theoretical mean.]]> statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:466a865c84ac/ 2017-05-13T13:50:14+00:00 http://link.springer.com/article/10.1007%2Fs10618-012-0300-z sechilds Outlier detection research has been seeing many new algorithms every year that often appear to be only slightly different from existing methods along with some experiments that show them to “clearly outperform” the others. However, few approaches come along with a clear analysis of existing methods and a solid theoretical differentiation. Here, we provide a formalized method of analysis to allow for a theoretical comparison and generalization of many existing methods. Our unified view improves understanding of the shared properties and of the differences of outlier detection models. By abstracting the notion of locality from the classic distance-based notion, our framework facilitates the construction of abstract methods for many special data types that are usually handled with specialized algorithms. In particular, spatial neighborhood can be seen as a special case of locality. Here we therefore compare and generalize approaches to spatial outlier detection in a detailed manner. We also discuss temporal data like video streams, or graph data such as community networks. Since we reproduce results of specialized approaches with our general framework, and even improve upon them, our framework provides reasonable baselines to evaluate the true merits of specialized approaches. At the same time, seeing spatial outlier detection as a special case of local outlier detection, opens up new potentials for analysis and advancement of methods.]]> statistics data https://pinboard.in/ https://pinboard.in/u:sechilds/b:7f5ec68d804d/ 2017-05-13T13:44:41+00:00 https://www.datascience.com/blog/intro-to-anomaly-detection-learn-data-science-tutorials sechilds This overview is intended for beginners in the fields of data science and machine learning. Almost no formal professional experience is needed to follow along, but the reader should have some basic knowledge of calculus (specifically integrals), the programming language Python, functional programming, and machine learning.]]> python data statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:788c05a0516d/ 2017-05-13T12:21:16+00:00 https://docs.google.com/presentation/d/1g6X0vo_wFLxxGXNKRnsz69Mheb_m9d9W98Ck-bsAj90/edit?pref=2&pli=1#slide=id.g84985ef7c_2_75 sechilds statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:dce4aabe58f7/ 2017-05-12T00:23:54+00:00 https://fivethirtyeight.com/features/trump-noncitizen-voters/#calculator sechildssurvey politics:us:trump statistics https://pinboard.in/u:sechilds/b:2d9f082a2b5b/ 2017-05-11T23:21:52+00:00 http://daniellakens.blogspot.ca/2017/05/how-power-analysis-implicitly-reveals.html sechilds When designing a study, you need to justify the sample size you aim to collect. If one of your goals is to observe a p-values lower than the alpha level you decided upon (e.g., 0.05), one justification for the sample size can be a power analysis. A power analysis tells you the probability of observing a statistically significant effect, based on a specific sample size, alpha level, and true effect size. At our department, people who use power as a sample size justification need to aim for 90% power if they want to get money from the department to collect data. > > A power analysis is performed based on the effect size you expect to observe. When you expect an effect with a Cohen’s d of 0.5 in an independent two-tailed t-test, and you use an alpha level of 0.05, you will have 90% power with 86 participants in each group. What this means, is that only 10% of the distribution of effects sizes you can expect when d = 0.5 and n = 86 falls below the critical value required to get a p < 0.05 in an independent t-test. > > In the figure below, the power analysis is visualized by plotting the distribution of Cohen’s d given 86 participants per group when the true effect size is 0 (or the null-hypothesis is true), and when d = 0.5. The blue area is the Type 2 error rate (the probability of not finding p < α, when there is a true effect).]]> data statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:c1978dc975c6/ 2017-05-11T18:42:14+00:00 http://journals.sagepub.com/doi/full/10.1177/0956797611417632 sechilds statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:4b89cb05dc7d/ 2017-05-11T18:38:16+00:00 http://andrewgelman.com/2017/05/10/p-hacking-intention-cheat-effect/ sechilds statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:1f8db8c07eaa/ 2017-05-11T17:19:13+00:00 https://twitter.com/i/web/status/862718749727653889 sechilds statistics survey politics:us:trump https://twitter.com/ https://pinboard.in/u:sechilds/b:8aac34069de3/ 2017-05-11T15:44:16+00:00 http://benalexkeen.com/principle-component-analysis-in-python/ sechilds Python statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:a57ebdf250f4/ 2017-04-24T16:36:07+00:00 https://twitter.com/chrisalbon/status/856547309701652480/photo/1 sechilds statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:edf237edcc6f/ 2017-03-31T14:53:42+00:00 http://notstatschat.tumblr.com/post/158976786141/prerequisites sechilds statistics @followup https://pinboard.in/u:sechilds/b:eb4fb527b934/ 2017-03-28T02:10:31+00:00 http://www.marsja.se/how-to-python-descriptives-statistics-numpy/ sechildspython statistics https://pinboard.in/u:sechilds/b:729c12f21145/ 2017-02-27T23:54:36+00:00 https://crdcn.org/article/job-opportunity-analyst-a-rdc sechilds statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:ab98fa73349c/ 2017-02-11T14:46:43+00:00 http://andrewgelman.com/2017/02/11/measurement-error-replication-crisis/ sechilds RW: In your article, you “caution against the fallacy of assuming that that which does not kill statistical significance makes it stronger.” What do you mean by that? > > AG: We blogged about the “What does not kill my statistical significance makes it stronger” fallacy here. As anyone who’s designed a study and gathered data can tell you, getting statistical significance is difficult. And we also know that noisy data and small sample sizes make statistical significance even harder to attain. So if you do get statistical significance under such inauspicious conditions, it’s tempting to think of this as even stronger evidence that you’ve found something real. This reasoning is erroneous, however. Statistically speaking, a statistical significant result obtained under highly noisy conditions is more likely to be an overestimate and can even be in the wrong direction. In short: a finding from a low-noise study can be informative, while the finding at the same significance level from a high-noise study is likely to be little more than . . . noise.]]> statistics data https://pinboard.in/ https://pinboard.in/u:sechilds/b:4cb938a15fb6/ 2017-02-08T13:37:53+00:00 http://debrouwere.org/2017/02/01/unlearning-descriptive-statistics/ sechildsstatistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:dd7a24b0b080/ 2017-01-03T20:53:47+00:00 http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off sechilds science statistics https://pinboard.in/u:sechilds/b:69fc477422ce/ 2016-12-05T19:43:27+00:00 http://robjhyndman.com/hyndsight/tscv/ sechildsstatistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:7d92e857c8a6/ 2016-10-31T12:41:11+00:00 https://www.linkedin.com/pulse/common-probability-distributions-data-scientists-crib-diego sechilds statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:e27d76a3465f/ 2016-07-29T00:43:10+00:00 https://twitter.com/rasbt/status/758825158127738881/photo/1 sechilds statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:c91cdbbe056d/ 2016-01-19T19:44:08+00:00 https://twitter.com/DiegoKuonen/status/689533822669570048/photo/1 sechilds #DataScience #Education ]]> Statistics DataScience Education https://twitter.com/ https://pinboard.in/u:sechilds/b:bbe0c58e37bf/ 2016-01-19T19:44:08+00:00 http://onlinelibrary.wiley.com/doi/10.1111/test.12083/abstract sechilds #DataScience #Education ]]> Statistics DataScience Education https://twitter.com/ https://pinboard.in/u:sechilds/b:6383e5f2d0a8/ 2015-08-12T13:55:24+00:00 http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/ sechilds visualization @followup statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:b30ae493a736/ 2015-04-18T09:08:39+00:00 https://twitter.com/DiegoKuonen/status/589354843468365824/photo/1 sechilds Free access: > #Statistics ]]> Statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:b778c9f9c791/ 2015-04-18T09:08:39+00:00 http://mkweb.bcgsc.ca/pointsofsignificance/ sechilds Free access: > #Statistics ]]> Statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:01c93d759d04/ 2015-04-18T09:08:39+00:00 http://www.nature.com/collections/qghhqm/pointsofsignificance sechilds Free access: > #Statistics ]]> Statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:8dbab577602b/ 2015-01-03T14:51:33+00:00 http://www.statschat.org.nz/2015/01/03/cancer-isnt-just-bad-luck/ sechilds statistics health:cancer https://instapaper.com/ https://pinboard.in/u:sechilds/b:4918ba1d8d0c/ 2015-01-02T21:24:04+00:00 http://www.theguardian.com/science/grrlscientist/2015/jan/02/bad-luck-bad-journalism-and-cancer-rates sechilds statistics health:cancer https://instapaper.com/ https://pinboard.in/u:sechilds/b:f03525b2f139/ 2014-01-03T16:02:08+00:00 http://www.refsmmat.com/statistics/index.html sechilds If you’re a practicing scientist, you probably use statistics to analyze your data. From basic t tests and standard error calculations to Cox proportional hazards models and geospatial kriging systems, we rely on statistics to give answers to scientific problems. > > This is unfortunate, because most of us don’t know how to do statistics. > > Statistics Done Wrong is a guide to the most popular statistical errors and slip-ups committed by scientists every day, in the lab and in peer-reviewed journals. Many of the errors are prevalent in vast swathes of the published literature, casting doubt on the findings of thousands of papers. Statistics Done Wrong assumes no prior knowledge of statistics, so you can read it before your first statistics course or after thirty years of scientific practice. > > Dive in: the whole guide is available online!]]> data statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:4d28884d7ced/ 2013-12-02T11:12:18+00:00 http://statistical-research.com/probabilities-and-p-values/ sechildsP-values seem to be the bane of a statistician’s existence. I’ve seen situations where entire narratives are written without p-values and only provide the effects. It can also be used as a data reduction tool but ultimately it reduces the world into a binary system: yes/no, accept/reject. Not only that but the binary threshold is determined on a roughly subjective criterion. It reminds me of Jon Stewart’s recent discussion “good thing/bad thing“. Taking the most complex of issues and reducing them down to two outcomes. >Below is a simple graph that shows how p-values don’t tell the whole story. Sometimes, data is reduced so much that solid decisions are difficult to make. The graph on the left shows a situation where there are identical p-values but very different effects. The graph on the right shows where the p-values are very different, and one is quite low, but the effects are the same.]]> statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:3b52d64fde36/ 2013-11-29T15:55:22+00:00 http://fivebooks.com/interviews/andrew-gelman-on-statistics sechildsStatistics is what people think math is. Statistics is about patterns and that’s what people think math is about. The difference is that in math, you have to get very complicated before you get to interesting patterns. The math that we can all easily do – things like circles and triangles and squares – doesn’t really describe reality that much. Mandelbrot, when he wrote about fractals and talked about the general idea of self-similar processes, made it clear that if you want to describe nature, or social reality, you need very complicated mathematical constructions. The math that we can all understand from high school is just not going to be enough to capture the interesting features of real world patterns. Statistics, however, can capture a lot more patterns at a less technical level, because statistics, unlike mathematics, is all about uncertainty and variation.]]> statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:9ac1591e34fc/ 2013-11-15T22:19:49+00:00 http://andrewgelman.com/2013/11/14/statistics-least-important-part-data-science/ sechildsStatistics is important—don’t get me wrong—statistics helps us correct biases from nonrandom samples (and helps us reduce the bias at the sampling stage), statistics helps us estimate causal effects from observational data (and helps us collect data so that causal inference can be performed more directly), statistics helps us regularize so that we’re not overwhelmed by noise (that’s one of my favorite topics!), statistics helps us fit models, statistics helps us visualize data and models and patterns. Statistics can do all sorts of things. I love statistics! But it’s not the most important part of data science, or even close.]]> data:analysis statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:d9ccff8094c7/ 2013-11-15T21:58:57+00:00 http://simplystatistics.org/2013/11/15/whats-the-future-of-inference/ sechildsJan said that inference as an activity belongs in the substantive field that raised the problem.  Statisticians should not do inference.  Statisticians might, he said, design tools to help specialists have an easier time doing inference. But the inferential act itself requires intimate substantive knowledge, and so the statistician can assist, but not do. >Ultimately, I can see how statisticians would want to stay away from the inference business. That part is hard, it's controversial, it involves messy details about sampling, and opens one up to criticism. And statisticians love to criticize other people. Why would anyone want to get mixed up with that? This is why machine learning is so attractive--it's all about prediction and in-sample learning.]]> statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:0a14463321d3/ 2013-02-28T15:30:11+00:00 http://andrewgelman.com/2013/02/26/18052/ sechilds data:machine_learning statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:32930db4dfb7/ 2012-12-24T23:28:35+00:00 http://errorstatistics.com/2012/12/24/13-well-worn-criticisms-of-significance-tests-and-how-to-avoid-them/ sechildsstatistics statistics:philosophy econometrics https://instapaper.com/ https://pinboard.in/u:sechilds/b:6cdf16d92981/ 2012-11-08T11:03:03+00:00 http://andrewgelman.com/2012/11/that-last-satisfaction-at-the-end-of-the-career/ sechilds statistics crime:fraud https://pinboard.in/ https://pinboard.in/u:sechilds/b:4569e1c73ba3/ 2012-10-31T23:32:53+00:00 http://www.econometricsbysimulation.com/2012/10/regression-analysis-ols.html sechildsstatistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:059bab03024b/ 2012-10-23T10:05:23+00:00 http://news.ycombinator.com/item?id=4685928 sechilds statistics startup https://pinboard.in/ https://pinboard.in/u:sechilds/b:f68b06302012/ 2012-08-31T14:35:54+00:00 http://www.econometricsbysimulation.com/2012/08/cronbachs-alpha.html sechilds statistics econ https://instapaper.com/ https://pinboard.in/u:sechilds/b:f68ce5e37835/ 2012-07-14T16:21:05+00:00 https://normaldeviate.wordpress.com/2012/07/14/modern-two-sample-tests/ sechilds statistics https://pinboard.in/ https://pinboard.in/u:sechilds/b:25af4c81a7f6/ 2012-07-10T19:39:08+00:00 http://blogs.reuters.com/felix-salmon/2012/07/10/how-economists-get-tripped-up-by-statistics/ sechildseconometrics statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:772d59856094/ 2012-07-09T20:26:39+00:00 http://davegiles.blogspot.com/2012/07/decline-and-fall-of-power-curve.html sechilds econometrics statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:be56d4449d8e/ 2012-07-09T18:06:30+00:00 http://www.mostlyharmlesseconometrics.com/2012/07/probit-better-than-lpm/ sechildsstatistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:810173956d00/ 2012-07-05T11:30:20+00:00 http://errorstatistics.com/2012/07/04/comment-on-falsification/ sechilds statistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:211c625fbf5e/ 2012-06-22T12:54:24+00:00 http://normaldeviate.wordpress.com/2012/06/12/statistics-versus-machine-learning-5-2/ sechildsstatistics data:machine_learning https://instapaper.com/ https://pinboard.in/u:sechilds/b:83943f8db9d5/ 2012-06-22T12:54:09+00:00 http://normaldeviate.wordpress.com/2012/06/18/48/ sechildsstatistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:8c764eb5782f/ 2012-06-22T12:38:56+00:00 https://normaldeviate.wordpress.com/2012/06/21/90/ sechildsstatistics https://instapaper.com/ https://pinboard.in/u:sechilds/b:b2e888aa6ee0/ 2012-06-15T22:34:21+00:00 http://stats.stackexchange.com/questions/30059/basic-easy-rules-for-statistics/30062?atw=1#30062 sechilds statistics https://twitter.com/ https://pinboard.in/u:sechilds/b:138570f1a5de/ 2012-05-27T10:19:19+00:00 http://diffuseprior.wordpress.com/2012/04/23/probitlogit-marginal-effects-in-r-2/ sechildsstatistics R via:phnk https://pinboard.in/u:sechilds/b:5a8f1024f52e/ 2012-05-19T14:04:40+00:00 http://andrewgelman.com/2012/05/15049/ sechildsstatistics cross_validation https://instapaper.com/ https://pinboard.in/u:sechilds/b:754e764735b6/ 2012-05-17T23:14:51+00:00 http://stats.stackexchange.com/questions/28683/bootstrap-vs-other-simulated-data-methods?atw=1 sechilds statistics statistics:bootstrap econometrics:simulation https://twitter.com/ https://pinboard.in/u:sechilds/b:a4c78d7a4b58/