https://pinboard.in/u:jm/public/ recent bookmarks from jm 2016-02-09T12:55:23+00:00 http://danluu.com/perf-tracing/ jmThe cause [of some mystery widespread 250ms hangs] was kernel throttling of the CPU for processes that went beyond their usage quota. To enforce the quota, the kernel puts all of the relevant threads to sleep until the next multiple of a quarter second. When the quarter-second hand of the clock rolls around, it wakes up all the threads, and if those threads are still using too much CPU, the threads get put back to sleep for another quarter second. The phase change out of this mode happens when, by happenstance, there aren’t too many requests in a quarter second interval and the kernel stops throttling the threads. After finding the cause, an engineer found that this was happening on 25% of disk servers at Google, for an average of half an hour a day, with periods of high latency as long as 23 hours. This had been happening for three years. Dick Sites says that fixing this bug paid for his salary for a decade. This is another bug where traditional sampling profilers would have had a hard time. The key insight was that the slowdowns were correlated and machine wide, which isn’t something you can see in a profile. ]]> debugging performance visualization instrumentation metrics dan-luu latency google dick-sites linux scheduler throttling kernel hangs https://pinboard.in/ https://pinboard.in/u:jm/b:670193c955f5/ 2015-03-07T21:13:54+00:00 https://github.com/gquintana/metrics-sql jmA Dropwizard Metrics extension to instrument JDBC resources and measure SQL execution times. ]]> metrics sql jdbc instrumentation dropwizard https://pinboard.in/ https://pinboard.in/u:jm/b:937f107730e0/ 2014-07-22T09:20:12+00:00 http://blog.librato.com/posts/2014/7/16/metrics-driven-development jmwe believe MDD is equal parts engineering technique and cultural process. It separates the notion of monitoring from its traditional position of exclusivity as an operations thing and places it more appropriately next to its peers as an engineering process. Provided access to real-time production metrics relevant to them individually, both software engineers and operations engineers can validate hypotheses, assess problems, implement solutions, and improve future designs. Broken down into the following principles: 'Instrumentation-as-Code', 'Single Source of Truth', 'Developers Curate Visualizations and Alerts', 'Alert on What You See', 'Show me the Graph', 'Don’t Measure Everything (YAGNI)'. We do all of these at Swrve, naturally (a technique I happily stole from Amazon).]]> metrics coding graphite mdd instrumentation yagni alerting monitoring graphs https://pinboard.in/ https://pinboard.in/u:jm/b:2bbaac35dc32/ 2013-09-14T21:47:49+00:00 http://hirt.se/blog/?p=370 jm instrumentation logging profiling java jvm ops https://pinboard.in/ https://pinboard.in/u:jm/b:3f5a76e13a69/ 2013-09-11T22:21:22+00:00 http://hirt.se/blog/?p=364 jmBuilt into the HotSpot JVM [in JDK version 7u40] is something called the Java Flight Recorder. It records a lot of information about/from the JVM runtime, and can be thought of as similar to the Data Flight Recorders you find in modern airplanes. You normally use the Flight Recorder to find out what was happening in your JVM when something went wrong, but it is also a pretty awesome tool for production time profiling. Since Mission Control (using the default templates) normally don’t cause more than a per cent overhead, you can use it on your production server. I'm intrigued by the idea of always-on profiling in production. This could be cool.]]> performance java measurement profiling jvm jdk hotspot mission-control instrumentation telemetry metrics https://pinboard.in/ https://pinboard.in/u:jm/b:cee0d58c8fe6/ 2011-11-11T08:57:22+00:00 http://www.jointhegrid.com/highperfcassandra/?p=95 jm tcp tcprstat tcp-ip networking measurement statistics performance instrumentation linux unix tools cli https://pinboard.in/ https://pinboard.in/u:jm/b:78cc56260920/ 2011-11-09T13:54:17+00:00 https://www.erlang-factory.com/upload/presentations/462/euc2011-draft2.pdf jm dtrace monitoring instrumentation debugging tracing unix erlang via:istvan https://pinboard.in/ https://pinboard.in/u:jm/b:76d5b3fb1bbf/