Hasnain says:

I hate self promotion but since I’ve been asked a few times about what I work on - I’m glad to be able to share a bit of it.

I recently got a chance to speak on a podcast about some of my work - in particular, experience building services and rewriting things in Rust. It’s kinda off the cuff and not super polished, but the content comes across fine I’d hope. The other speakers had much better advice / experience in this domain, so it’s worth a listen either way.

Happy to take any questions here!

(excuse the incorrect title, the share scraper seems to be off, the link is right)

Hasnain says:

Great read on societal trends through a systems lens. I don’t agree with everything but I do agree with a majority of the stuff said here.

“Even the fanciest pantsed distributed databases, with all the Rafts and Paxoses and red/greens and active/passives and Byzantine generals and dining philosophers and CAP theorems, are subject to this. You can do a bunch of math to absolutely prove beyond a shadow of a doubt that your database is completely distributed and has no single points of failure. There are papers that do this. You can do it too. Go ahead. I'll wait.

Okay, great. Now skip paying your AWS bill for a few months.

Whoops, there's a hierarchy after all!

You can stay in denial, or you can get serious.”

Hasnain says:

Terrible bug aside, this was a great postmortem and actionable set of follow-ups.

inb4 rust rewrite though.

"The maximum size signature that this structure can handle is whatever the largest union member is, in this case that’s RSA at 2048 bytes. That’s 16384 bits, large enough to accommodate signatures from even the most ridiculously oversized keys.

Okay, but what happens if you just....make a signature that’s bigger than that?

Well, it turns out the answer is memory corruption. Yes, really.

The untrusted signature is simply copied into this fixed-sized buffer, overwriting adjacent members with arbitrary attacker-controlled data."

Hasnain says:

This is some really cool stuff.

“This is the best of both worlds: it is at once dynamic and general purpose with respect to what the system can run, but also entirely static in terms of the binary payload of a particular application — and broadly static in terms of its execution. Dynamic resource exhaustion is the root of many problems in embedded systems; having the system know a priori all of the tasks that it will ever see liberates it from not just a major source of dynamic allocation, but also from the concomitant failure modes. For example, in Hubris, tasks can always be safely restarted, because we know that the resources associated with a task are available if that task itself has faulted! And this eliminates failure modes in which dynamic task creation in response to load induces resource exhaustion; as Cliff has quipped, it is hard to have a fork bomb when the system lacks fork itself!”

Hasnain says:

Well this is really cool.

“Scientists discovered that the cancer does not always originate in the mature bone marrow cells where it is found and where textbooks say it originates.

…

The results are too new to have led to patient therapies. But they are leading to provocative discoveries that are expected to inspire novel methods for treating diseases.”

Hasnain says:

This was a great technical read.

"We hope hearing how we managed our Kafka-based telemetry ingest pipeline is helpful to you in scaling your own Kafka clusters. We’ve grown 10x in two years while TCO1 for Kafka has only gone up 20%—in other words, an 87%2 cumulative reduction in cost per MB of incoming data. We do not have any engineers dedicated full time to the care and feeding of Kafka, and have preserved about the same number of engineers fluent in Kafka debugging as needed through training with incidents. This means our toil has not significantly increased even as the data scope increased."

Hasnain says:

Very informative piece on wage theft and the disparities in how it’s reported versus other property crime. Obviously one has to insert the famous chart of how property crimes are very insignificant compared to wage theft - I’ll look that up again and add it in the comments.

“Shoplifting is just a small fraction of total property crime because more than half of the value of all stolen property comes from stolen vehicles and currency. Nevertheless, media coverage of shoplifting vastly exceeds media coverage of wage theft. A search of United States publications in the Nexis news database reveals 11,631 stories mentioning shoplifting so far in 2021. Over the same period, the same outlets published just 2,009 stories mentioning wage theft. “

Hasnain says:

How is this not illegal?

“Alex, a 35-year-old who recently moved to Miami, answered a posting through the online employment marketplace Snagajob offering $16 an hour to work in a technology sales position at Staples. At the end of the interview, the manager revealed that the pay was actually $10 an hour.”

Hasnain says:

Great perspective on the eternal oncall debates.

"Individuals owning things. In a healthy engineering organization, there are no gaps in coverage. Every critical component is owned by a TEAM, not a person. People practice pairing and buddying up for code reviews for just this reason, to make sure other people know about the tricky bits, the twiddly bits, the history, how to debug, how to ameliorate. The more critical the component, the more urgent this coverage becomes."

Hasnain says:

This is an interesting quote at the end for sure. Also an interesting read overall, covering some aspects of the security business I’m not super familiar with. And on what appears to be one of the (rare) legitimate defamation suits I’ve seen in the US.

“The appetite for intelligence gathering in the hypercompetitive tech world continues, though. Mr. Gicinto, the former C.I.A. officer, has a warning for any of his former colleagues considering a move to this part of the private sector, where the motivations behind a given mission are not always as clear as he found them in his past work life.

“In the government, when you’re given a mission or you’re given a task, you go and you execute on the mission,” Mr. Gicinto said. “Your experience tells you to go execute because your boss or the leadership have given you this tasking, and you worry about how to do it — not whether or not you should do it, because you’ve never had to worry about that before.””