selection bias
#test-results 2025-10-15
Wednesday49 messages11 participantstimes are UTC
Highlights from this day
- salt_bridge — it makes the dataset internally consistent and externally uncalibrated. every result is comparable to every other result and none of them are anchored to anything… 20:32
- void_volume — it is a known limitation rather than bad. if their method runs half a point generous or half a point strict across the board, every vendor in our table moves together… 20:43
- cold_chain_cmdr — agreed, and stated for the record: no evidence. structural risk only 21:12
- salt_bridge — greater than 99% is a marketing number. no lab writes greater than 99 on a report, they write a figure 22:21
slightly off topic but i had a result i am fairly sure was a handling error on my side and i logged it as such
does anyone log the negative results, the boring ones
posting my Janoshik result on lot SG-1177, what do people make of it
which lab
how do you separate batch variance from supplier variance
my vial assayed under label but the purity was fine, which matters more
weigh the vial before you reconstitute. it costs nothing and catches the common failure
weigh the vial before you reconstitute. it costs nothing and catches the common failure
*Medutest not the other one
a single sample tells you about a single vial. that is genuinely all it tells you
do the results in this channel skew because people test when suspicious
thats not great
does a bad result on one lot condemn the whole supplier
my result contradicts the supplier certificate, do i tell them
a point or two between the certificate and an independent result is inside what two labs disagree by
unrelated but a supplier disputing a result is rare here and it has never gone well for the supplier
unrelated but what interests me is when the gap goes the same direction every time. once is noise, three times is a pattern, your mileage will differ
lot bought purity assay/label
A-2418 2025-03 99.1% 9.6 / 10mg
A-2601 2025-08 98.7% 9.8 / 10mg
B-0114 2026-01 99.3% 9.4 / 10mg
B-0329 2026-04 98.9% 9.9 / 10mgunpopular observation. almost every number in our dataset comes from one lab
because they are the one that actually turns hobby samples around at a price people will pay
i know, and i am not complaining about them. i am pointing at what it does to our data
it makes the dataset internally consistent and externally uncalibrated. every result is comparable to every other result and none of them are anchored to anything outside
is that bad
it is a known limitation rather than bad. if their method runs half a point generous or half a point strict across the board, every vendor in our table moves together and the ranking survives
what does not survive is a claim like this lot is 99.2% in absolute terms
exactly. relative comparisons are robust, absolute claims are not
and the second-order problem is that everyone in this market optimises for the same report. a vendor knows which lab you will use
which does not mean they are gaming it. it means the option exists and nobody can rule it out
keeping this on the methodology and off accusations please. we have no evidence any vendor has gamed a specific lab and saying otherwise is not fair
agreed, and stated for the record: no evidence. structural risk only
👍9
the fix is boring. split samples between labs occasionally, and accept that the two numbers will not match
how far apart is acceptable
one to two points of area% between two competent labs on the same powder is unremarkable. i would want to understand three. i would want to understand five very much
*and by understand i mean look at both gradients and both integration baselines, not accuse anybody
the CPC split we did came back 1.3 apart which was the most boring and reassuring result we have ever bought
boring is what we are paying for
does the second lab need to be a specific one
it needs to publish its method and its wavelength. that is the bar. a lab that reports a number with no method line is not a comparator, it is a rumour
we have had exactly one report submitted with no method line and it got filed with an asterisk
and stays filed with an asterisk. we do not delete data, we annotate it
there is a nice example in the archive: a GGPeps result reported as greater than 99% with no chromatogram, no method, no lot
greater than 99% is a marketing number. no lab writes greater than 99 on a report, they write a figure
💯16
so if i see greater than 99 anywhere
you are reading a vendor, not a lab
the ERP sheets say not less than 98% which is a specification not a result, and that is a different trick again
a spec tells you what they promise. a result tells you what they measured. both belong on a real document and only one of them usually appears
this channel is quietly making me a much more annoying customer
good
😂18
that is the intended outcome, yes