Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmHow Uber Engineering Evaluated JSON Encoding and Compression Algorithms to Put the Squeeze on Trip Data2019-12-16T11:32:03+00:00
https://eng.uber.com/trip-data-squeeze/
jmKey conclusions:
Simply compressing JSON with zlib would yield a reasonable tradeoff in size and speed. The result would be just a little bigger, but execution was much faster than using BZ2 on JSON.
Going with IDL-based protocols, Thrift and Protocol Buffers compressed with zlib or Snappy would give us the best gain in size and/or speed.
]]>compression json performance python serialization protobuf zlib snappy cbor messagepack thrift bz2https://pinboard.in/https://pinboard.in/u:jm/b:41b3c6225764/Schema evolution in Avro, Protocol Buffers and Thrift2016-01-29T13:52:24+00:00
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
jmavro thrift protobuf schemas serialization coding interop compatibilityhttps://pinboard.in/https://pinboard.in/u:jm/b:b9648b78d5fd/What Do WebLogic, WebSphere, JBoss, Jenkins, OpenNMS, and Your Application Have in Common? This Vulnerability.2015-11-09T12:31:56+00:00
http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
jmjava serialization security exploits jenkinshttps://pinboard.in/https://pinboard.in/u:jm/b:c0dc116c6113/Release Protocol Buffers v3.0.0-alpha-2 · google/protobuf2015-02-27T11:15:34+00:00
https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2
jmprotobuf binary marshalling serialization google grpc proto3 coding open-sourcehttps://pinboard.in/https://pinboard.in/u:jm/b:017754170f81/Cap'n Proto, FlatBuffers, and SBE2014-06-19T14:49:08+00:00
https://kentonv.github.io/capnproto/news/2014-06-17-capnproto-flatbuffers-sbe.html
jmserialization protobuf capnproto sbe flatbuffers google coding storagehttps://pinboard.in/https://pinboard.in/u:jm/b:ac723723a01c/FlatBuffers: Main Page2014-06-17T09:21:59+00:00
http://google.github.io/flatbuffers/
jmAccess to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility).
Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details.
Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures.
Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details.
Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you.
Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).
Looks nice, but it misses the language coverage of protobuf. Definitely more practical than capnproto.
]]>c++ google java serialization json formats protobuf capnproto storage flatbuffershttps://pinboard.in/https://pinboard.in/u:jm/b:6301528e508a/'Pickles & Spores: Improving Support for Distributed Programming in Scala2014-04-29T10:00:31+00:00
https://speakerdeck.com/heathermiller/spores-distributable-functions-in-scala
jmpickling scala presentations spores closures fp immutability coding distributed distcomp serialization formats networkhttps://pinboard.in/https://pinboard.in/u:jm/b:4d377e1b419e/"Understanding the Robustness of SSDs under Power Fault", FAST '13 [paper]2014-01-14T11:30:42+00:00
https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf
jmpdf papers ssd storage reliability safety hardware ops usenix serialization shorn-writes bit-errors corruption fsynchttps://pinboard.in/https://pinboard.in/u:jm/b:bd499b07b5d1/Simple Binary Encoding2013-12-10T10:41:27+00:00
https://github.com/real-logic/simple-binary-encoding
jmsbe encoding codecs persistence binary low-latency open-source java c++ serializationhttps://pinboard.in/https://pinboard.in/u:jm/b:312431bb5c47/Cap'n Proto2013-04-03T16:22:04+00:00
http://kentonv.github.com/capnproto/
jmCap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap’n Proto is INFINITY TIMES faster than Protocol Buffers.
Basically, marshalling like writing an aligned C struct to the wire, QNX messaging protocol-style. Wasteful on space, but responds to this by suggesting compression (which is a fair point tbh). C++-only for now. I'm not seeing the same kind of support for optional data that protobufs has though. Overall I'm worried there's some useful features being omitted here...]]>serialization formats protobufs capn-proto protocols coding c++ rpc qnx messaging compression compatibility interoperability i14yhttps://pinboard.in/https://pinboard.in/u:jm/b:63b8402f4508/Introducing BERT and BERT-RPC2009-10-22T11:04:20+00:00
http://github.com/blog/531-introducing-bert-and-bert-rpc
jmgithub bert erlang ruby rpc protocol thrift serialization networkinghttps://pinboard.in/u:jm/b:f2f0b799bc30/