Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmAdditional Checksum Algorithms for Amazon S32022-03-07T17:36:56+00:00
https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/
jmIt is now very easy for you to calculate and store checksums for data stored in Amazon S3 and to use the checksums to check the integrity of your upload and download requests. You can use this new feature to implement the digital preservation best practices and controls that are specific to your industry. In particular, you can specify the use of any one of four widely used checksum algorithms (SHA-1, SHA-256, CRC-32, and CRC-32C).
(via Last Week in AWS)]]>checksums integrity uploads s3 sha crc md5https://pinboard.in/https://pinboard.in/u:jm/b:3dbeb1888cbe/Historic S3 data corruption due to a fault load balancer2020-01-22T14:05:12+00:00
https://forums.aws.amazon.com/thread.jspa?threadID=22709
jmWe've isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20 [2008]. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3's total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted. Based on our investigation with both internal and external customers, the small amount of traffic received by this particular load balancer, and the intermittent nature of the above issue on this one load balancer, this appears to have impacted a very small portion of PUTs during this time frame.
One of the things we'll do is improve our logging of requests with MD5s, so that we can look for anomalies in their 400 error rates. Doing this will allow us to provide more proactive notification on potential transmission issues in the future, for customers who use MD5s and those who do not. In addition to taking the actions noted above, we encourage all of our customers to take advantage of mechanisms designed to protect their applications from incorrect data transmission. For all PUT requests, Amazon S3 computes its own MD5, stores it with the object, and then returns the computed MD5 as part of the PUT response code in the ETag. By validating the ETag returned in the response, customers can verify that Amazon S3 received the correct bytes even if the Content MD5 header wasn't specified in the PUT request. Because network transmission errors can occur at any point between the customer and Amazon S3, we recommend that all customers use the Content-MD5 header and/or validate the ETag returned on a PUT request to ensure that the object was correctly transmitted. This is a best practice that we'll emphasize more heavily in our documentation to help customers build applications that can handle this situation.
]]>aws s3 outages postmortems load-balancing data-corruption corruption failure md5 hashing hasheshttps://pinboard.in/https://pinboard.in/u:jm/b:7067b5a9a1e4/AV vendors still relying on MD5 to identify malware2015-06-10T15:07:42+00:00
http://blog.silentsignal.eu/2015/06/10/poisonous-md5-wolves-among-the-sheep/
jmmd5 hashing antivirus malware security via:fanf bugshttps://pinboard.in/https://pinboard.in/u:jm/b:11ef4e54eeb8/What's the probability of a hash collision?2014-11-18T11:50:47+00:00
http://davidjohnstone.net/pages/hash-collision-probability
jmprobability hashing hashes collision risk md5 sha sha1 calculatorshttps://pinboard.in/https://pinboard.in/u:jm/b:7941face31b6/How I created two images with the same MD5 hash2014-11-04T18:14:08+00:00
http://natmchugh.blogspot.co.uk/2014/10/how-i-created-two-images-with-same-md5.html
jmI found that I was able to run the algorithm in about 10 hours on an AWS large GPU instance bringing it in at about $0.65 plus tax.
Bottom line: MD5 is feasibly attackable by pretty much anyone now.]]>crypto images md5 security hashing collisions ec2 via:hnhttps://pinboard.in/https://pinboard.in/u:jm/b:3b301b6423b9/NYC generates hash-anonymised data dump, which gets reversed2014-06-25T15:36:55+00:00
https://medium.com/@vijayp/f6bc289679a1
jmThere are about 1000*26**3 = 21952000 or 22M possible medallion numbers. So, by calculating the md5 hashes of all these numbers (only 24M!), one can completely deanonymise the entire data. Modern computers are fast: so fast that computing the 24M hashes took less than 2 minutes.
(via Bruce Schneier)
The better fix is a HMAC (see http://benlog.com/2008/06/19/dont-hash-secrets/ ), or just to assign opaque IDs instead of hashing.]]>hashing sha1 md5 bruce-schneier anonymization deanonymization security new-york nyc taxis data big-data hmac keyed-hashing saltinghttps://pinboard.in/https://pinboard.in/u:jm/b:86f2bc539afe/Analyzing Flame's MD5 Collision Attack [slides, PDF]2012-06-11T23:36:36+00:00
http://www.trailofbits.com/resources/flame-md5.pdf
jmvia:fanf flame security malware md5 collisions hashing pki tls ssl microsofthttps://pinboard.in/https://pinboard.in/u:jm/b:1e484697f020/Stop using unsafe keyed hashes, use HMAC2009-10-30T22:23:02+00:00
http://rdist.root.org/2009/10/29/stop-using-unsafe-keyed-hashes-use-hmac/
jmhmac security crypto hashing md5 hashes sha256 sha1https://pinboard.in/u:jm/b:e18fe54cec21/details of the Markdown Javascript-escaping hole2009-09-29T10:27:14+00:00
http://blog.reddit.com/2009/09/we-had-some-bugs-and-it-hurt-us.html
jmhacks security reddit javascript md5 escaping htmlhttps://pinboard.in/u:jm/b:ec19e371bc31/