Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmLean proved this program was correct; then I found a bug2026-04-14T13:08:07+00:00
https://kirancodes.me/posts/log-who-watches-the-watchers.html
jmThe positive result here is actually the remarkable one. Across 105 million executions, the application code (that is, excluding the runtime) had zero heap buffer overflows, zero use-after-free, zero stack buffer overflows, zero undefined behaviour (UBSan clean), and zero out-of-bounds array reads in the Lean-generated C code. [...]
The two bugs that were found both sat outside the boundary of what the proofs cover. The denial-of-service was a missing specification. The heap overflow was a deeper issue in the trusted computing base, the C++ runtime that the entire proof edifice assumes is correct (and now has a PR addressing).
Overall verification resulted in a remarkably robust and rigorous codebase. AFL and Claude had a really hard time finding errors. But they did still find issues. Verification is only as strong as the questions you think to ask and the foundations you choose to trust.
]]>programming coding future lean formal-methods correctness linting bugs zip verification testinghttps://pinboard.in/https://pinboard.in/u:jm/b:c1237ac77bc9/hectorvent/floci2026-03-23T17:30:08+00:00
https://github.com/hectorvent/floci
jmfloci aws emulation testing local codinghttps://pinboard.in/https://pinboard.in/u:jm/b:f24a9f4a5a12/How StrongDM’s AI team build serious software without even looking at the code2026-02-09T10:46:10+00:00
https://simonwillison.net/2026/Feb/7/software-factory/
jm
In kōan or mantra form:
- Why am I doing this? (implied: the model should be doing this instead)
In rule form:
- Code must not be written by humans
- Code must not be reviewed by humans
Finally, in practical form:
- If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement
Frankly, I'm not there yet. There's a load of questions about how viable that level of spend is, and how much slop code is going to come out the other side. Particularly concerning when it's a security product!
But I did find this bit interesting:
StrongDM’s answer was inspired by Scenario testing (Cem Kaner, 2003). As StrongDM describe it: We repurposed the word scenario to represent an end-to-end “user story”, often stored outside the codebase (similar to a “holdout” set in model training), which could be intuitively understood and flexibly validated by an LLM.
[The Digital Twin Universe is] behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors.
With the DTU, we can validate at volumes and rates far exceeding production limits. We can test failure modes that would be dangerous or impossible against live services. We can run thousands of scenarios per hour without hitting rate limits, triggering abuse detection, or accumulating API costs.
We actually did this in Swrve! Our end-to-end system tests for the push notifications system obviously cannot send real push notifications to real user devices in the field, so we have a "fake" push backend emulating Google, Apple, Amazon, Huawei and other push notification systems, which accurately emulate the real public APIs for those providers.
So yeah -- Digital Twins for third party services is a great way to test, and being able to scale up end-to-end testing with LLM automation is a very interesting idea.]]>end-to-end-testing testing qa digital-twins fake-services integration-testing llms ai strongdm software engineering codinghttps://pinboard.in/https://pinboard.in/u:jm/b:4148e1f6d312/_Cheap science, real harm: the cost of replacing human participation with synthetic data_ [pdf]2025-12-18T10:47:45+00:00
https://synthetic-data-workshop.github.io/papers/13.pdf
jmDriven by the goals of augmenting diversity, increasing speed, reducing cost, the use of synthetic data as a replacement for human participants is gaining traction in AI research and product development. This talk critically examines the claim that synthetic data can “augment diversity,” arguing that this notion is empirically unsubstantiated, conceptually flawed, and epistemically harmful. While speed and cost-efficiency may be achievable, they often come at the expense of rigour, insight, and robust science. Drawing on research from dataset audits, model evaluations, Black feminist scholarship, and complexity science, I argue that replacing human participants with synthetic data risks producing both real-world and epistemic harms at worst and superficial knowledge and cheap science at best.
"Synthetic data: stereotypes compressed" is absolutely spot on. This doesn't give insights into human behaviour and beliefs, just into stereotypes. It is increasingly common in social science fields, under the names of "digital twins" and "silicon samples".]]>data surveys abeba-birhane papers ai synthetic-data digital-twins simulation testing social-science silicon-sampleshttps://pinboard.in/https://pinboard.in/u:jm/b:e5314d304e99/Defeating Nondeterminism in LLM Inference - Thinking Machines Lab2025-09-11T16:03:06+00:00
https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
jmReproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models.
For example, you might observe that asking ChatGPT the same question multiple times provides different results. This by itself is not surprising, since getting a result from a language model involves “sampling”, a process that converts the language model’s output into a probability distribution and probabilistically selects a token.
What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here). Even when running inference on your own hardware with an OSS inference library like vLLM or SGLang, sampling still isn’t deterministic (see here or here).
The levels of non-deterministic variation throughout the LLM stack discussed here are massive! It's kinda crazy that this doesn't produce incorrect output more often.]]>llms ml machine-learning ai determinism testing inference reproducibility randomness floating-pointhttps://pinboard.in/https://pinboard.in/u:jm/b:dbd2fd2346a8/Orange-OpenSource/hurl2025-06-20T09:42:49+00:00
https://github.com/Orange-OpenSource/hurl
jmcli rust tools unix testing linux json curl http testshttps://pinboard.in/https://pinboard.in/u:jm/b:d9655d2dcc54/RGG Studio’s test automation setup2025-05-06T09:17:57+00:00
https://automaton-media.com/en/interviews/how-do-like-a-dragon-games-come-out-so-fast-one-of-rgg-studios-secrets-is-a-highly-efficient-testing-and-debugging-cycle-that-starts-as-soon-as-development-does/
jmAt first, we only dabbled in automated packaging and automated error detection, but we made the tools we needed to go further during the development of Yakuza 6, when we started automating the analysis of in-game logs and the issue tracking system for keeping track of bugs and tasks. Then, by the time Yakuza: Like a Dragon was released in 2020, we created the catchy sounding “fully automated bug detection system” (laughs).
This is how it works – the history of actions you performed when playing the game manually (where you travelled, who you talked to, what items you used, etc.) is converted into commands and recorded, then automatically output as replay data (scripts) which you can edit manually and run as automated tests. Replay data continues to be recorded when running automated tests, and if a bug occurs during an automated test, the replay data gets saved, so you can run it back later to encounter the bug yourself. It often happens that you can’t reproduce a bug just by warping to its coordinates. This is because you also need to recreate the steps leading up to it – that’s why it’s important to record each step.
Also, I’d like to mention that just implementing automated testing doesn’t mean much on its own, because you won’t know what the results of the tests are. That’s why we needed a crash report function to detect bugs. There’s also a function that records information needed to investigate detected bugs, as well as a way to check the status of successful tests. Then, by implementing a system that gives us a visualization of performance, we were able to make iteration more efficient, increasing the overall efficiency of the development process.
]]>automation testing yakuza games coding tests test-automationhttps://pinboard.in/https://pinboard.in/u:jm/b:6337ae2e5588/WireMock2025-03-06T15:41:51+00:00
https://wiremock.org/
jmAn open-source tool for API mock testing, with over 5 million downloads per month. It can help you to create stable test and development environments, isolate yourself from flakey 3rd parties and simulate APIs that don't exist yet.
Nice features include running in-process in a JVM, standalone, or in a Docker container; GraphQL and gRPC support; and fault and latency injection. https://library.wiremock.org/ is a library of pre-built API mocks other people have previously made.]]>mocking testing mocks integration-testing wiremock tools coding apishttps://pinboard.in/https://pinboard.in/u:jm/b:683181d64404/Bufferbloat Test2025-01-10T10:48:08+00:00
https://www.waveform.com/tools/bufferbloat
jmbufferbloat internet networking optimisation performance testing toolshttps://pinboard.in/https://pinboard.in/u:jm/b:93a6fb532124/Canary Contamination in GPT-42024-10-29T16:14:27+00:00
https://www.alignmentforum.org/posts/kSmHMoaLKGcGgyWzs/big-bench-canary-contamination-in-gpt-4
jm
Of 19 tested [benchmarking] tasks, GPT-4-base perfectly recalled large (non-trivial) portions of code for:
The Abstraction and Reasoning Corpus;
Simple arithmetic;
Diverse Metrics for Social Biases in Language Models;
Convince Me
Great work. In case you were wondering why the LLMs all seem to do so well on their benchmarks, now you know -- they were training on the test data.]]>ai llm testing benchmarking big-bench gpt-4 claudehttps://pinboard.in/https://pinboard.in/u:jm/b:d18a7f3abe04/Reverse engineering ML models from TikTok and Instagram2024-10-23T10:49:49+00:00
https://arxiv.org/pdf/2403.19717
jm
Capitalizing on this new processing model of locally analyzing user images, we analyze two popular social media apps, TikTok and Instagram, to reveal (1) what insights vision models in both apps infer about users from their image and video data and (2) whether these models exhibit performance disparities with respect to demographics. As vision models provide signals for sensitive technologies like age verification and facial recognition, understanding potential biases in these models is crucial for ensuring that users receive equitable and accurate services.
We develop a novel method for capturing and evaluating ML tasks in mobile apps, overcoming challenges like code obfuscation, native code execution, and scalability. Our method comprises ML task detection, ML pipeline reconstruction, and ML performance assessment, specifically focusing on demographic disparities. We apply our methodology to TikTok and Instagram, revealing significant insights. For TikTok, we find issues in age and gender prediction accuracy, particularly
for minors and Black individuals. In Instagram, our analysis uncovers demographic disparities in the extraction of over 500 visual concepts from images, with evidence of spurious correlations between demographic features and certain concepts.
]]>tiktok instagram ml machine-learning accuracy testing reverse-engineering reversing mobile androidhttps://pinboard.in/https://pinboard.in/u:jm/b:2b5e1c6b378e/A single bit flip nearly resulted in nuclear annihilation in 19802023-06-06T14:49:25+00:00
https://twitter.com/AtomicAnalyst/status/1665024416865034241
jmnukes armageddon 1980s bit-flips errors testing norad sac usahttps://pinboard.in/https://pinboard.in/u:jm/b:7133c0b3c170/Dropbox testing HTTP32023-05-22T10:02:25+00:00
https://dropbox.tech/frontend/investigating-the-impact-of-http3-on-network-latency-for-search
jmdropbox http3 http2 http protocols udp networking ip testinghttps://pinboard.in/https://pinboard.in/u:jm/b:9d832e568456/f3write, f3read - test real flash memory capacity2023-05-04T15:45:13+00:00
https://manpages.ubuntu.com/manpages/trusty/man1/f3read.1.html
jmf3read f3write f3 sd-cards flash memory storage testing hardware via:bigbrohttps://pinboard.in/https://pinboard.in/u:jm/b:ab2321d4d7b2/SAFECARE COVID-19 & Flu Combo Antigen Test2022-08-31T20:25:27+00:00
https://mybio.ie/products/safecare-covid19-flu-combo-antigen-test-home-self-test-single-test-ce-ivd-certified
jmcovid-19 flu tests testing healthhttps://pinboard.in/https://pinboard.in/u:jm/b:173d5f3e44dc/Sero-Epidemiology of COVID-19 in Ireland2022-08-05T09:04:20+00:00
https://seroepi-hpscireland.hub.arcgis.com/
jmcovid-19 seropositivity blood ibts graphs open-data testing hpschttps://pinboard.in/https://pinboard.in/u:jm/b:9ce228b00819/Primary schools & childcare services antigen programme - HSE.ie2022-03-21T11:33:09+00:00
https://www.hse.ie/eng/services/schools-antigen-testing/
jmhse ireland covid-19 testing antigen-tests freehttps://pinboard.in/https://pinboard.in/u:jm/b:10c8d816d313/Good thread on UK COVID testing and random sampling2022-02-22T09:38:47+00:00
https://twitter.com/PaulMainwood/status/1494756760174964744
jmcovid-19 testing sampling random measurement ascertainment detectability epidemiology sars-cov-2 uk ons modellinghttps://pinboard.in/https://pinboard.in/u:jm/b:4d687652b4d1/Mass rapid tests in Liverpool cut hospital stays by a third2021-12-21T20:24:30+00:00
https://www.theguardian.com/world/2021/dec/19/mass-rapid-tests-in-liverpool-cut-hospital-stays-by-a-third
jmAn analysis has shown that it was more successful than Liverpool’s scientists and public health teams had anticipated, after they compared Covid cases and outcomes in the region with other parts of England.
Professor Iain Buchan, dean of the Institute of Population Health, who led the evaluation, said: “This time last year, as the Alpha variant was surging, we found that Liverpool city region’s early rollout of community rapid testing was associated with a 32% fall in Covid-19 hospital admissions after careful matching to other parts of the country in a similar position to Liverpool but without rapid testing.
“We also found that daily lateral flow testing as an alternative to quarantine for people who had been in close contact with a known infected person enabled emergency services to keep key teams such as fire crews in work, underpinning public safety.”
]]>rapid-tests antigen-tests testing covid-19 sars-cov-2 liverpool public-health lfts serial-testinghttps://pinboard.in/https://pinboard.in/u:jm/b:1864cf398d84/Jellyfish Bitrate Test Files2021-12-14T15:49:51+00:00
https://jell.yfish.us/
jmhtpc streaming video tests testing mkv h.264 h.265 hevchttps://pinboard.in/https://pinboard.in/u:jm/b:aaa3e14ce63d/The nasal PCR test may not be the gold standard for COVID-19 detection2021-08-04T13:59:20+00:00
https://twitter.com/TAlexPerkins/status/1422347142950178816
jmsensitivity specificity rt-pcr pcr testing covid-19 sars-cov-2https://pinboard.in/https://pinboard.in/u:jm/b:d7f83e7e611f/gProfiler2021-06-28T09:31:58+00:00
https://github.com/Granulate/gprofiler
jmgprofiler profiling performance testing measurement coding ruby java linux perf via:reddithttps://pinboard.in/https://pinboard.in/u:jm/b:6772a8b88a40/Lateral flow test sensitivity2021-05-04T09:08:00+00:00
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001216
jmOur data show that the Innova LFD can successfully detect SARS-CoV-2 infection in people with a viral titre above approximately 100 viral copies/ml. However, as determined at our site using the ThermoFisher COVID-19 TaqPath assay, it is incapable of detecting infection at comparable PCR Ct values of 30 and over. These levels of infection are indicative of very early or very late stages of infection, and as such, we would strongly recommend that LFD testing is used to screen people at very regular frequency and that a negative result should not be used to determine that someone is free from SARS-CoV-2 infection.
IMO 'very regular frequency' is the key detail here. Single LFA rapid tests, alone, are not useful as a simple replacement for PCR tests.]]>testing covid-19 sars-cov-2 lfa rapid-tests pcr papershttps://pinboard.in/https://pinboard.in/u:jm/b:4d6fe9a61003/The impact of population-wide rapid antigen testing on SARS-CoV-2 prevalence in Slovakia | Science2021-03-23T19:07:35+00:00
https://science.sciencemag.org/content/early/2021/03/22/science.abf9648
jmSlovakia conducted multiple rounds of population-wide rapid antigen testing for SARS-CoV-2 in late 2020, combined with a period of additional contact restrictions. Observed prevalence decreased by 58% (95% CI: 57-58%) within one week in the 45 counties that were subject to two rounds of mass testing, an estimate that remained robust when adjusting for multiple potential confounders. Adjusting for epidemic growth of 4.4% (1.1-6.9%) per day preceding the mass testing campaign, the estimated decrease in prevalence compared to a scenario of unmitigated growth was 70% (67-73%). Modelling indicated that this decrease could not be explained solely by infection control measures, but required the additional impact of isolation and quarantine of household members of those testing positive.
]]>testing slovakia via:luke-oneill covid-19 screening mass-testinghttps://pinboard.in/https://pinboard.in/u:jm/b:21040f2bf822/Testcontainers2021-03-18T09:49:47+00:00
https://www.testcontainers.org/
jmTestcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
]]>docker java junit testing containers testshttps://pinboard.in/https://pinboard.in/u:jm/b:ab95f5fb86ea/The effectiveness of population-wide, rapid antigen test based screening in reducing SARS-CoV-2 infection prevalence in Slovakia | CMMID Repository2020-12-04T15:10:21+00:00
https://cmmid.github.io/topics/covid19/Slovakia.html
jmslovakia rapid-testing antigen-testing testing sars-cov-2 covid-19 pandemics screeninghttps://pinboard.in/https://pinboard.in/u:jm/b:d10f5f49dc9b/COVID-19 Rapid Antigen Test from MöLab - Box of 10 Tests - Mybio2020-12-01T12:37:09+00:00
https://covid.mybio.ie/product/covid-19-rapid-antigen-test-from-molab-box-of-10-tests-ce-ivd/
jm 99.9%. Provides a result after 15 minutes.
(Obligatory caveat: "for professional use only")]]>rapid-tests covid-19 sars-cov-2 testing antigen-testshttps://pinboard.in/https://pinboard.in/u:jm/b:c23c170439e1/Cillian De Gascun on Twitter: "A short thread on PCR #SARSCoV2"2020-09-14T12:04:34+00:00
https://twitter.com/CillianDeGascun/status/1305250887246458880
jmireland covid-19 testing ct-levels viral-load viruses cillian-de-gascun twitterhttps://pinboard.in/https://pinboard.in/u:jm/b:6c88f66d42e2/illustration of how a rise in SARS-CoV-2 positivity in younger groups can soon become a rise in older groups2020-09-08T09:26:06+00:00
https://twitter.com/vincentglad/status/1303243869933404161/photo/1
jmtesting covid-19 age epidemiology dataviz statistics marseilles francehttps://pinboard.in/https://pinboard.in/u:jm/b:7ed9065e39e7/Proactive screening for COVID-19 is individually costly2020-09-07T13:26:25+00:00
https://twitter.com/CT_Bergstrom/status/1302706766656884736
jm[...] proactive testing carries individual costs: those associated with purchasing the tests, time costs of taking them, and serious social and economic costs if one tests positive and has to self-isolate for upward of a week. So, we have an intervention that is costly to you but beneficial to others. We need to think about what the incentives are for people to (1) decide to take a daily test before the results are known and (2) follow through appropriately if a test comes up positive.
]]>testing covid-19 screening costs economics incentiveshttps://pinboard.in/https://pinboard.in/u:jm/b:3394abaa5a68/Seroprevalence of SARS-CoV-2 antibodies in children - A prospective multicentre cohort study. | medRxiv2020-09-04T09:21:44+00:00
https://www.medrxiv.org/content/10.1101/2020.08.31.20183095v1
jmcovid-19 symptoms kids schools sars-cov-2 studies preprints papers uk testinghttps://pinboard.in/https://pinboard.in/u:jm/b:802fb6bba3e2/How to Test Every American for COVID-19, Every Day - The Atlantic2020-08-17T11:14:23+00:00
https://www.theatlantic.com/health/archive/2020/08/how-to-test-every-american-for-covid-19-every-day/615217/
jmtesting covid-19 coronavirus michael-mina reopeninghttps://pinboard.in/https://pinboard.in/u:jm/b:450ab190fdd9/Comparison of viral levels in individuals with or without symptoms at time of COVID-19 testing among 32,480 residents and staff of nursing homes and assisted living facilities in Massachusetts. | medRxiv2020-07-30T11:13:32+00:00
https://www.medrxiv.org/content/10.1101/2020.07.20.20157792v1
jmIn aggregate, the distributions of Cts for viral probes used in the qRT-PCR assay were very similar, with a statistically but not meaningfully different mean (∆Ct 0.71 cycles, p = 0.006) and a similar range (12-38 cycles), between populations with and without symptoms over the entire time period, across all sub-categories examined (age, race, ethnicity, sex, resident/staff).
Viral load is comparable between COVID-19 victims who are asymptomatic and symptomatic.]]>covid-19 preprints papers medicine viral-load rt-pcr testinghttps://pinboard.in/https://pinboard.in/u:jm/b:aa6a7123c230/Sputum testing provides higher rate of COVID-19 detection | EurekAlert! Science News2020-07-26T20:55:58+00:00
https://eurekalert.org/pub_releases/2020-07/bawh-stp072420.php
jmLi and his colleagues scoured the literature -- both preprints and published papers -- for studies that assessed at least two respiratory sampling sites using an NP swab, oropharyngeal swab or sputum. From more than 1,000 studies, they identified 11 that met their criteria. These studies included results from a total of 3,442 respiratory tract specimens.
The team examined how often each collection method produced a positive result. For NP swabs, the rate was 54 percent; for oropharyngeal swabs, 43 percent; for sputum, 71 percent. The rate of viral detection was significantly higher in sputum than either oropharyngeal swabs or NP swabs. Detection rates were highest within one week of symptom onset for all three tests.
]]>sputum covid-19 testing swabshttps://pinboard.in/https://pinboard.in/u:jm/b:ed770201b0f7/COVID Dogs2020-07-25T17:33:25+00:00
https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-020-05281-3
jmThe dogs were able to discriminate between samples of infected (positive) and non-infected (negative) individuals with average diagnostic sensitivity of 82.63% (95% confidence interval [CI]: 82.02–83.24%) and specificity of 96.35% (95% CI: 96.31–96.39%). During the presentation of 1012 randomised samples, the dogs achieved an overall average detection rate of 94% (±3.4%) with 157 correct indications of positive, 792 correct rejections of negative, 33 incorrect indications of negative or incorrect rejections of 30 positive sample presentations.
]]>dogs good-dog covid-19 detection testing smell scenthttps://pinboard.in/https://pinboard.in/u:jm/b:4dde937a03c5/Cheap, Frequent, Quick Testing2020-07-21T21:55:07+00:00
https://suddenlyathome.net/cheap-frequent-quick-testing/
jmPicture the widespread availability of a simple, cheap, at-home Covid test that produces quick results. Perhaps a strip of paper like a pregnancy test. Everyone in the country takes it daily. It would stop this virus in its tracks.
An excellent summary of the Michael Mina, et al, paper (and TWiV discussion).]]>testing covid-19 accuracy reopening testshttps://pinboard.in/https://pinboard.in/u:jm/b:fba86fa8d405/Test sensitivity is secondary to frequency and turnaround time for COVID-19 surveillance | medRxiv2020-07-16T13:05:27+00:00
https://www.medrxiv.org/content/10.1101/2020.06.22.20136309v2
jmepidemiology covid-19 testing swabs rt-pcr twiv virology models papershttps://pinboard.in/https://pinboard.in/u:jm/b:12897f20c61c/Thread on how US universities are planning to tackle COVID-19 in the fall2020-07-13T10:50:41+00:00
https://twitter.com/MikeOtsuka/status/1262826858376290307
jmtesting covid-19 reopening education universities health schoolshttps://pinboard.in/https://pinboard.in/u:jm/b:c4f6fb587d4e/Rapid, inexpensive home testing for COVID-19 may get us out of this mess before a vaccine2020-07-05T23:07:41+00:00
https://blogs.jwatch.org/hiv-id-observations/index.php/rapid-inexpensive-home-testing-for-covid-19-may-get-us-out-of-this-mess-before-a-vaccine/2020/07/05/
jmWe should welcome [rapid covid] tests, even if less accurate, and broadly adopt them for widespread community use. Here’s why:
They will be cheap. Estimates are that they would cost between 1 and 5 dollars. That’s around the price of a cup of coffee.
They can be done on saliva. No brain biopsy required.
They can be done frequently. Every day for college students, or healthcare workers, or bus drivers? Every third day for everyone?
They will answer the key question — am I contagious to others right now?
Finally, and most importantly, they will answer this last question quickly. Results back in less than an hour.
Anyone with a positive test can self-isolate, be reported to public health officials, participate in a contact tracing program, and be monitored for symptoms. Maybe pre-emptive antiviral therapy will prevent severe illness.
We can choose to do a rapid home test any day we go to work, or to the gym, or to meet friends in a restaurant, or to attend a concert, or to pray in a house of worship, or to visit an elder loved one, or indeed partake in any activity we do in groups that now sadly may sustain the pandemic.
And for those worried about lack of sensitivity, two items of reassurance. First, false negatives are less likely when people have the highest amounts of virus in saliva and respiratory secretions — and this is when they’re most contagious to others. If the test is falsely negative due to low titers of virus, it may not matter very much.
Second, this modeling study finds that the frequency of testing is the key determinant of how well a broad testing strategy will limit the spread of the virus. It’s even more important than test sensitivity, and evidence that imperfect testing is better than no testing at all.
]]>testing covid-19 pcr rt-pcr false-positives false-negatives viruseshttps://pinboard.in/https://pinboard.in/u:jm/b:866b1f021772/_Measurement-Based Evaluation Of Google/Apple Exposure Notification API For Proximity Detection In A Commuter Bus_2020-06-22T14:38:27+00:00
https://www.scss.tcd.ie/Doug.Leith/pubs/bus.pdf
jmpapers bluetooth contact-tracing exposure-notification covid-19 accuracy testing buses radio gaen mobilehttps://pinboard.in/https://pinboard.in/u:jm/b:0276789a0b1a/Interpreting Covid-19 Test Results: A Bayesian Approach2020-06-13T22:42:29+00:00
https://medium.com/@Bob_Wachter/interpreting-covid-19-test-results-a-bayesian-approach-df058dad2ade
jma brief tutorial on Covid-19 testing, with an emphasis on a Bayesian approach. After presenting the basics, we’ll walk through four confusing Covid-19 testing scenarios, just to give you a feel for the kinds of pickles we often find ourselves in.
]]>prevalence covid-19 bayes bayesian statistics testinghttps://pinboard.in/https://pinboard.in/u:jm/b:419f48573a05/Testing times: The science of fighting Covid-192020-05-25T09:48:01+00:00
https://www.irishexaminer.com/breakingnews/ireland/testing-times-the-science-of-fighting-covid-19-1001424.html
jmpcr rt-pcr covid-19 sars-cov-2 testing dna rna medicine svuh dublin irelandhttps://pinboard.in/https://pinboard.in/u:jm/b:966cc96af973/Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure | Annals of Internal Medicine2020-05-14T14:33:55+00:00
https://www.acpjournals.org/doi/10.7326/M20-1495
jmcovid-19 diseases testing pcr sars-cov-2 infection medicinehttps://pinboard.in/https://pinboard.in/u:jm/b:dffc3bfb0c08/Global Progress on COVID-19 Serology-Based Testing2020-05-07T15:26:42+00:00
https://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html
jmcovid-19 tests serology testing immunity pandemics jhu via:twivhttps://pinboard.in/https://pinboard.in/u:jm/b:b32e18e34f23/Everybody in the Pool: Researchers Use Algorithms to Tackle the Coronavirus Test Shortage - IEEE Spectrum2020-05-07T08:38:53+00:00
https://spectrum.ieee.org/view-from-the-valley/the-institute/ieee-member-news/everybody-in-the-pool-algorithm-researchers-tackle-the-coronavirus-test-shortage
jmpooling multipooling pooled-testing testing covid-19 algorithms pandemicshttps://pinboard.in/https://pinboard.in/u:jm/b:687de4119f7f/Nature paper on COVID-19 and RT-PCR detection rates over time2020-04-15T22:30:58+00:00
https://www.nature.com/articles/s41586-020-2196-x_reference.pdf
jmpcr covid-19 testing sars-cov-2 infection diseaseshttps://pinboard.in/https://pinboard.in/u:jm/b:c9b9cc09ffb2/Good thread on what Iceland is doing right wrt. COVID-192020-04-13T10:38:17+00:00
https://twitter.com/gummihaf/status/1248976393683247104
jmcovid-19 iceland contact-tracing testing medicine pandemicshttps://pinboard.in/https://pinboard.in/u:jm/b:d017c5f4cef9/Proprietary reagents are blocking COVID-19 testing worldwide2020-04-01T19:57:38+00:00
https://twitter.com/WSMIreland/status/1245296675209478146
jmwsm twitter threads hse covid-19 reagents testing chemicalshttps://pinboard.in/https://pinboard.in/u:jm/b:33da45abba74/Record/Replay testing in Sorbet2020-01-14T16:52:05+00:00
https://blog.nelhage.com/post/record-replay-in-sorbet/
jmsorbet testing record-replay-testing unit-tests testshttps://pinboard.in/https://pinboard.in/u:jm/b:1e90669d71a8/Testing in Production: How we combined tests with monitoring2020-01-06T11:43:49+00:00
https://www.theguardian.com/info/developer-blog/2016/dec/05/testing-in-production-how-we-combined-tests-with-monitoring
jmtesting monitoring ops devops the-guardian prod production releaseshttps://pinboard.in/https://pinboard.in/u:jm/b:083ed1350efa/libeatmydata2019-10-10T15:55:49+00:00
https://www.flamingspork.com/projects/libeatmydata/
jmfsync linux performance mysql testinghttps://pinboard.in/https://pinboard.in/u:jm/b:a876035ccbb5/Write tests. Not too many. Mostly integration. – kentcdodds2019-02-01T11:32:49+00:00
https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c
jmintegration coding testing unit-tests integration-tests system-testshttps://pinboard.in/https://pinboard.in/u:jm/b:b2c5531ba011/How do you populate your development databases?2018-11-08T14:56:53+00:00
https://dev.to/jaredsilver/how-do-you-populate-your-development-databases-e8e
jmdatabase data testing system-tests devhttps://pinboard.in/https://pinboard.in/u:jm/b:286cf87eafef/Actual screenshot of the broken UX of the Hawaii ballistic missile alert system2018-01-16T15:27:08+00:00
https://twitter.com/CivilBeat/status/953127914618302464
jm"This is the screen that set off the ballistic missile alert on Saturday. The operator clicked the PACOM (CDW) State Only link. The drill link is the one that was supposed to be clicked."
This is terrible, terrible UX.]]>ux ui hawaii alerting alerts testing safety failhttps://pinboard.in/https://pinboard.in/u:jm/b:c281422fb791/The likely user interface which led to Hawaii's false-alarm incoming-ballistic-missile alert on Saturday 2018-01-132018-01-15T11:07:00+00:00
https://twitter.com/supersat/status/952612571122630659
jmtesting ux user-interfaces fail eas hawaii false-alarms alerts nuclear early-warning human-errorhttps://pinboard.in/https://pinboard.in/u:jm/b:a4d8ff23c728/Open-sourcing RacerD: Fast static race detection at scale | Engineering Blog | Facebook Code2017-10-19T16:17:27+00:00
https://code.facebook.com/posts/293371094514305
jmAt Facebook we have been working on automated reasoning about concurrency in our work with the Infer static analyzer. RacerD, our new open source race detector, searches for data races — unsynchronized memory accesses, where one is a write — in Java programs, and it does this without running the program it is analyzing. RacerD employs symbolic reasoning to cover many paths through an app, quickly.
This sounds extremely interesting...
]]>racerd race-conditions data-races thread-safety static-code-analysis coding testing facebook open-source inferhttps://pinboard.in/https://pinboard.in/u:jm/b:35a19486dd4e/Jepsen: Hazelcast 3.8.32017-10-10T08:49:22+00:00
https://jepsen.io/analyses/hazelcast-3-8-3
jmjepsen aphyr testing hazelcast cap-theorem reliability partitions network pacelc marc-brookerhttps://pinboard.in/https://pinboard.in/u:jm/b:c55626ad02a4/LambCI — a serverless build system2017-09-27T10:41:32+00:00
https://medium.com/@hichaelmart/lambci-4c3e29d6599b
jmLambCI is a tool I began building over a year ago to run tests on our pull requests and branches at Uniqlo Mobile. Inspired at the inaugural ServerlessConf a few weeks ago, I recently put some work into hammering it into shape for public consumption.
It was borne of a dissatisfaction with the two current choices for automated testing on private projects. You can either pay for it as a service (Travis, CircleCI, etc) — where 3 developers needing their own build containers might set you back a few hundred dollars a month. Or you can setup a system like Jenkins, Strider, etc and configure and manage a database, a web server and a cluster of build servers .
In both cases you’ll be under- or overutilized, waiting for servers to free up or paying for server power you’re not using. And this, for me, is where the advantage of a serverless architecture really comes to light: 100% utilization, coupled with instant invocations.
]]>lambci ci builds jenkins testing codinghttps://pinboard.in/https://pinboard.in/u:jm/b:441b1b37cc67/Java Flame Graphs Introduction: Fire For Everyone!2017-09-11T10:48:52+00:00
http://psy-lob-saw.blogspot.ie/2017/02/flamegraphs-intro-fire-for-everyone.html
jmprofiling java safepoints jvm flame-graphs perf measurement benchmarking testinghttps://pinboard.in/https://pinboard.in/u:jm/b:1a6f8c8d9b75/Computer says no: Irish vet fails oral English test needed to stay in Australia2017-08-14T07:33:38+00:00
https://www.theguardian.com/australia-news/2017/aug/08/computer-says-no-irish-vet-fails-oral-english-test-needed-to-stay-in-australia?mc_cid=28d31ecac5&mc_eid=eb4031f206
jmAn Irish veterinarian with degrees in history and politics has been unable to convince a machine she can speak English well enough to stay in Australia.
Louise Kennedy is a native English speaker, has excellent grammar and a broad vocabulary. She holds two university degrees – both obtained in English – and has been working in Australia as an equine vet on a skilled worker visa for the past two years.
But she is now scrambling for other visa options after a computer-based English test – scored by a machine – essentially handed her a fail in terms of convincing immigration officers she can fluently speak her own language.
This is idiotic. Computer-based voice recognition is in no way reliable enough for this kind of job. It's automated Kafkaesque bureaucracy -- "computer says no". Shame on Oz
(via James Kelleher)]]>via:etienneshrdlu kafkaesque bureaucracy computer-says-no voice-recognition australia immigration english voice testinghttps://pinboard.in/https://pinboard.in/u:jm/b:132987b6c5fe/Developer Experience Lessons Operating a Serverless-like Platform at Netflix2017-07-17T10:48:03+00:00
https://medium.com/netflix-techblog/developer-experience-lessons-operating-a-serverless-like-platform-at-netflix-a8bbd5b899a0
jmserverless dependencies packaging deployment versioning devex netflix developer-experience dev testing staging scriptinghttps://pinboard.in/https://pinboard.in/u:jm/b:a6089d93b40f/Undefined Behavior in 20172017-07-06T11:36:55+00:00
https://blog.regehr.org/archives/1520
jmRecently we’ve heard a few people imply that problems stemming from undefined behaviors (UB) in C and C++ are largely solved due to ubiquitous availability of dynamic checking tools such as ASan, UBSan, MSan, and TSan. We are here to state the obvious — that, despite the many excellent advances in tooling over the last few years, UB-related problems are far from solved — and to look at the current situation in detail.
]]>via:marc-brooker c c++ coding testing debugging dynamic-analysis valgrind asan ubsan tsanhttps://pinboard.in/https://pinboard.in/u:jm/b:13303806849e/Why did Apple, Amazon, Google stocks crash to the same price today?2017-07-05T11:16:17+00:00
http://www.nj.com/business/index.ssf/2017/07/why_did_apple_amazon_google_stocks_crash_to_the_sa.html
jmNasdaq said in a statement that "certain third parties improperly propagated test data that was distributed as part of the normal evening test procedures."
"For July 3, 2017, all production data was completed by 5:16 PM as expected per the early close of the markets," the statement continued. "Any data messages received post 5:16 PM should be deemed as test data and purged from direct data recipient's databases. UTP (Unlisted Trading Privileges) is asking all third parties to revert to Nasdaq Official Closing Prices effective at 5:16 PM."
]]>testing fail stock-markets nasdaq test-data test production integration-testing test-in-prodhttps://pinboard.in/https://pinboard.in/u:jm/b:a9528d479297/RIPE Atlas Probes2017-06-27T09:02:10+00:00
https://atlas.ripe.net/about/probes/
jmRIPE Atlas probes form the backbone of the RIPE Atlas infrastructure. Volunteers all over the world host these small hardware devices that actively measure Internet connectivity through ping, traceroute, DNS, SSL/TLS, NTP and HTTP measurements. This data is collected and aggregated by the RIPE NCC, which makes the data publicly available. Network operators, engineers, researchers and even home users have used this data for a wide range of purposes, from investigating network outages to DNS anycasting to testing IPv6 connectivity.
Anyone can apply to host a RIPE Atlas probe. If your application is successful (based on your location), we will ship you a probe free of charge. Hosts simply need to plug their probe into their home (or other) network.
Probes are USB-powered and are connected to an Ethernet port on the host’s router or switch. They then automatically and continuously perform active measurements about the Internet’s connectivity, and this data is sent to the RIPE NCC, where it is aggregated and made publicly available. We also use this data to create several Internet maps and data visualisations. [....]
The hardware of the first and second generation probes is a Lantronix XPort Pro module with custom powering and housing built around it. The third generation probe is a modified TP-Link wireless router (model TL-MR 3020) with a small USB thumb drive in it, but this probe does not support WiFi.
(via irldexter)
]]>via:irldexter ripe ncc probing active-monitoring networking ping traceroute dns testing http ipv6 anycast hardware devices ispshttps://pinboard.in/https://pinboard.in/u:jm/b:6ba98239256a/Determinism in League of Legends2017-06-21T10:43:57+00:00
https://engineering.riotgames.com/news/determinism-league-legends-introduction
jmclock realtime time determinism testing replay games league-of-legends via:nelsonhttps://pinboard.in/https://pinboard.in/u:jm/b:14fb1751e281/lambci/docker-lambda2017-06-14T10:13:04+00:00
https://github.com/lambci/docker-lambda
jmA sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.
(via og-aws)]]>docker lambda images testing aws serverlesshttps://pinboard.in/https://pinboard.in/u:jm/b:382e8c25cbbe/An empirical study on the correctness of formally verified distributed systems2017-05-29T10:06:17+00:00
https://blog.acolyer.org/2017/05/29/an-empirical-study-on-the-correctness-of-formally-verified-distributed-systems/
jmWe must recognise that even formal verification can leave gaps and hidden assumptions that need to be teased out and tested, using the full battery of testing techniques at our disposal. Building distributed systems is hard. But knowing that shouldn’t make us shy away from trying to do the right thing, instead it should make us redouble our efforts in our quest for correctness.
]]>formal-verification software coding testing tla+ chapar fuzzing verdi bugs papershttps://pinboard.in/https://pinboard.in/u:jm/b:f4b9ecb7122d/'What’s your ML Test Score? A rubric for ML production systems'2017-04-28T12:45:09+00:00
https://0586f9b3-a-62cb3a1a-s-sites.googlegroups.com/site/wildml2016nips/SculleyPaper1.pdf?attachauth=ANoY7cont_cg-6CVVs0tx2hEqWhKKhhL6YfFRGCGBKkrLxs8x53zencaHWsRScBKzfjeZO3HhLUvby1dC-T1yiptBOHhCZKB4PescOdq5tfmTPP3GPnAMfzYVtO0Wwrp6Mr1hz7Wx-PuNm8chSJqpR3v-8xnZvJsFhHC73FhDr8_eKIyqmS_MSa5sNt37fsbuGKq8n989mVby6uw4ZfxQlLJxUDtHnPjag%3D%3D&attredirects=0
jmmachine-learning testing ml papers googlehttps://pinboard.in/https://pinboard.in/u:jm/b:5d280a9a3be4/atlassian/localstack: A fully functional local AWS cloud stack. Develop and test your cloud apps offline!2017-03-20T14:25:19+00:00
https://github.com/atlassian/localstack
jmLocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. Currently, the focus is primarily on supporting the AWS cloud stack.
LocalStack spins up the following core Cloud APIs on your local machine:
API Gateway at http://localhost:4567;
Kinesis at http://localhost:4568;
DynamoDB at http://localhost:4569;
DynamoDB Streams at http://localhost:4570;
Elasticsearch at http://localhost:4571;
S3 at http://localhost:4572;
Firehose at http://localhost:4573;
Lambda at http://localhost:4574;
SNS at http://localhost:4575;
SQS at http://localhost:4576
Additionally, LocalStack provides a powerful set of tools to interact with the cloud services, including a fully featured KCL Kinesis client with Python binding, simple setup/teardown integration for nosetests, as well as an Environment abstraction that allows to easily switch between local and remote Cloud execution.
]]>aws emulation mocking services testing dynamodb s3https://pinboard.in/https://pinboard.in/u:jm/b:240a300dcd0c/Testing@LMAX – Time Travel and the TARDIS2016-11-21T11:29:55+00:00
https://www.symphonious.net/2014/04/01/testinglmax-time-travel-and-the-tardis/
jmlmax testing system-tests acceptance-tests tests timehttps://pinboard.in/https://pinboard.in/u:jm/b:0c1c1fc4c02e/