Hasnain says:

This is pretty cool. I’ve been doing a lot of SQL lately and developer efficiency is a mess (I have a lot to learn).

“In the query on the right, the author is trying to combine timestamps in milliseconds from the table user_login_events_mobile with timestamps in nanoseconds from the table user_login_events_desktop — an understandable mistake, as the two columns have the same name. But because the tables’ schema have been annotated with user-defined types, UPM’s typechecker catches the error before the query reaches the query engine; it then notifies the author in their code editor. Without this check, the query would have completed successfully, and the author might not have noticed the mistake until much later.”

Hasnain says:

Some great wins here. When I saw people complain about JS build times and how they moved things to Rust or Go; I was (perhaps naively) assuming the tools had been optimized as much as can be in JS. Being proven wrong here.

“At this point I stopped looking, but I'd assume that you'll find more of these minor performance issues in popular libraries. Today we mainly looked at some build tools, but UI components or other libraries usually have the same low hanging performance issues.

Will this be enough to match Go's or Rust's performance? Unlikely, but the thing is that the current JavaScript tools could be faster than they are today. And the things we looked at in this post are more or less just the tip of the iceberg.”

Hasnain says:

Really cool technical story of how they built a kernel + user space driver for the M1 GPU so it could run Linux. And some great hackery in between. I promise I didn’t share this just for the Rust fanboyism.

“Normally, when you write a brand new kernel driver as complicated as this one, trying to go from simple demo apps to a full desktop with multiple apps using the GPU concurrently ends up triggering all sorts of race conditions, memory leaks, use-after-free issues, and all kinds of badness.

But all that just… didn’t happen! I only had to fix a few logic bugs and one issue in the core of the memory management code, and then everything else just worked stably! Rust is truly magical! Its safety features mean that the design of the driver is guaranteed to be thread-safe and memory-safe as long as there are no issues in the few unsafe sections. It really guides you towards not just safe but good design.”

Hasnain says:

This continues to increase my motivation to learn more about garbage collectors.

“The numbers reported represent the 50th and 99th percentile for Blink memory allocated with Oilpan across the fleet[2]. The reported data shows the delta between Chrome 105 and 106 stable versions. The absolute numbers in MB give an indication on the lower bound that users can expect to see. The real improvements are generally a bit higher due to indirect effects on Chrome’s overall memory consumption. The larger relative improvement suggests that packing of data is better in such cases which is an indicator that more memory is used in collections (e.g. vectors) that have good packing. The improved padding of structures landed in Chrome 108 and showed another 4% improvement on Blink memory on average.

Because Oilpan is ubiquitous in Blink, the performance cost can be estimated on Speedometer2. The initial prototype based on a thread-local version showed a regression of 15%. With all the aforementioned optimizations we did not observe a notable regression.”

Hasnain says:

“But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man.

Ultimately nobody really knows what they are doing, and that’s OK. But between companies not building around this observation, and individuals not self-directing their educations around this observation, it is just a bit maddening to feel stuck in stupid hell.”

Hasnain says:

Great read. Beyond the details of the static analysis itself, I enjoyed the insights into how they rolled out this program at scale and the lessons learned throughout.

"The problems outlined above are hardly specific to Meta. Unexpected null-dereferences have caused countless problems in different companies. Languages like C# evolved into having explicit nullness in their type system, while others, like Kotlin, had it from the very beginning.

When it comes to Java, there were multiple attempts to add nullness, starting with JSR-305, but none was widely successful. Currently, there are many great static analysis tools for Java that can check nullness, including CheckerFramework, SpotBugs, ErrorProne, and NullAway, to name a few. In particular, Uber walked the same path by making their Android codebase null-safe using NullAway checker. But in the end, all the checkers perform nullness analysis in different and subtly incompatible ways. The lack of standard annotations with precise semantics has constrained the use of static analysis for Java throughout the industry."

Hasnain says:

Assembly is so hard to get right. More so in a cross-platform manner.

"I decided to write this up because I thought that the journey was interesting, but also because the journey is not over. There could be other registers that are not being properly saved and restored in Chromium. There could be other projects that are making this mistake, perhaps not aware of the differences between the Linux and Windows ABI. Any software rules that are not tested and enforced are inevitably broken and I am not aware of any consistent testing to detect ABI violations. More bugs of this type seem inevitable."

Hasnain says:

Bookmarking this for later rereading. some of this is stuff that's worked well for me, and a lot of it was novel stuff. I'm a huge fan of issue driven development and documenting via commits, for what it's worth.

The quote though, uhhhhh....

"This is the most important tip: avoid side projects with user accounts.

If you build something that people can sign into, that’s not a side-project, it’s an unpaid job. It’s a very big responsibility, avoid at all costs!

Almost all of my projects right now are open source things that people can run on their own machines, because that’s about as far away from user accounts as I can get.

I still have a responsibility for shipping security updates and things like that, but at least I’m not holding onto other people’s data for them."

Hasnain says:

“If you’re wondering if you need to add a composite index, or can get away with creating to single indexes and rely on the database to use both indexes — then the rule of thumb we establish is that an index merge will be ~10x slower than the composite index. However, we’re still talking less than 100ms in most cases, as long as you’re operating on 100s of rows (which in a relational, operational database, hopefully you mostly are).”

Hasnain says:

“The idea of recognizing a sequences of bytes and finding the meaning behind them has always felt very satisfying to me.”

Hasnain says:

““Katie Meyer’s tragic death resulted from Stanford’s egregious and reckless mishandling of its disciplinary process,” attorney Kim Dougherty said in a statement. “Stanford has known for years that its disciplinary process, in its own Committee 10’s words, is ‘overly punitive’ and harmful to its students, yet the school and its administrators have done nothing to correct its procedures. Through this litigation we will not only obtain justice for Katie, but also ensure necessary change is put into place to help protect Stanford students and provide safeguards when students are in need of support.””

Hasnain says:

When twitter dies, I’ll pour one out for Dril.

“So far, Dril said, he’s enjoying the spectacle of Musk’s takeover. “Elon seems like one of the classic comedic showmen,” he said. “Everything he does is a comedic bit. He’s always trying to get a laugh, that’s why he makes all his cars suicidal. Just watching everything burn, it’s entertaining, that’s for sure.””

Hasnain says:

“Rust is the only language where I regularly write non-thread-safe data structures without caveats or fear. In cases where they apply, such data structures can be simpler and often faster than concurrent thread-safe data structures – with the drawback, of course, that you can’t use threads to speed up operations on them.

If you’re coming to Rust from another language that emphasizes concurrency, such as Java or Go or (increasingly) C, keep this in mind:

You don’t necessarily need to make every data structure thread-safe, and

If you get it wrong, you’ll get a compiler error, not a weird runtime bug.”

Hasnain says:

“This is the story of some DC reporters who got upset that they didn’t get invited to a wedding. Nothing more. And naturally, when faced with pushback (such as asking why Cook didn’t bother to read the photo caption), these journalists got even more defensive, arguing that people just don’t understand how journalism works.”

Hasnain says:

“But that displacement happens only because building dense housing is illegal in many rich neighborhoods, and because cities build so little of it overall. “If you want to build enough to really help low-income people, you’re talking about doing a lot of building,” Rick Jacobus, an expert on inclusionary housing and the principal of Street Level Urban Impact Advisors, told me.

As it turns out, two economists had, in a way, answered my question. Enrico Moretti of UC Berkeley and Chang-Tai Hsieh of the University of Chicago wanted to know how much GDP and productivity the United States gives up by throttling the housing supply in its biggest cities. In a blockbuster 2019 paper, they found that if New York, San Jose, and San Francisco—just those three cities—had the permitting standards of Atlanta or Chicago over the previous several decades, the U.S. economy would have been roughly $2 trillion bigger in 2009. American households would have earned an average of $3,685 more a year.”

Hasnain says:

This was a really interesting technical read.

“In this post, we demonstrated how we built the fastest interpreter for Lua (to date) through a novel meta-compiler framework.

However, automatically generating the fastest Lua interpreter is only the beginning of our story. LuaJIT Remake is designed to be a multi-tier method-based JIT compiler generated by the Deegen framework, and we will generate the baseline JIT, the optimizing JIT, the tiering-up/OSR-exit logic, and even a fourth-tier heavyweight optimizing JIT in the future.”

Hasnain says:

“This little trick shipped in Firefox 105 and had an enormous impact on Firefox stability on Windows. The chart below shows how many out-of-memory browser crashes were experienced by users per active usage hours

You’re looking at a >70% reduction in crashes, far more than our rosiest predictions.”

Hasnain says:

This was a great technical read that goes into a lot of distributed systems stuff.

“We mostly like Consul and would use it again in new designs. It’s easy to stand up. It’s incredibly useful to deploy infrastructure configurations. For example: we write blog posts like this and people invariably comment about how cool it is that we have a WireGuard mesh network between all of our machines. But, not to diminish Steve’s work on flywire, that system falls straight out of us using Consul. It’s great!

But we probably wouldn’t use Consul as the backing store for a global app platform again, in part because a global app platform might not even want a single globally consistent backing store. Our trajectory is away from it.”

Hasnain says:

SRO/SRE stories are always legendary

“Chasing down the Blips can teach you a lot about how a complex system functions. Sometimes it can help you find real problems before they happen. And a lot of the time, they'll teach you only one thing: That complex systems are chaotic, and sometimes things just harmlessly Blip and you'll never know why.

Our general policy was that if a Blip was big enough to set off an alert, we'd give it at least a cursory look.”

Hasnain says:

Sigh. Tucker Carlson, Fox News, Libs of TikTok and all the media commenters “just asking questions” have even more blood on their hands here. The quote below is a little misleading as the article and other reporting clarifies: the club was going to host a drag show and there has been a lot of nonsensical hatred against those recently.

“Although a motive in the shooting wasn’t yet clear, nor were the gender identities of the victims, the incident came as anti-gay rhetoric has intensified by extremists. In a statement, Club Q termed the shooting a hate attack.”

Hasnain says:

always learn something new with Amos’ posts.

“This is boring. Everything just works.

Right?? And yet I'm thrilled about it. I'm thrilled to know that, with Rust, I can do any of:

blocking I/O with threads
non-blocking I/O with tokio
io-uring with tokio-uring
eBPF with aya
just read procfs

And they all, boringly enough, work just fine. And I feel good about leaving them in production and never touching them again.

To me, that is the dream.”

Hasnain says:

Gotta love deep technical reads on fast data structures.

“Conclusions and next steps
boost::unordered_flat_map and boost::unordered_flat_set are the new open-addressing containers in Boost.Unordered providing top speed in exchange for some interface and behavioral deviations from the standards-compliant boost::unordered_map and boost::unordered_set. We have analyzed their internal data structure and provided some theoretical and practical evidence for their excellent performance. As of this writing, we claim boost::unordered_flat_map/boost::unordered_flat_set to rank among the fastest hash containers available to C++ programmers.”

Hasnain says:

Great read on how to profile, benchmark, and improve the performance of software.

“The test we used to measure performance, cating a big file, only measures performance in a very specific circumstance. There are other scenarios in which the performance of Zellij lacks or shines. It’s also important to note that because we are measuring the performance of full apps in a relatively complex and not 100% sterile environment, the measurements in this post should be regarded as loose signposts rather than exact measurements.

Zellij does not claim to be faster or more performant than any other software. Performance-wise, it only uses other software as an inspiration and something to aspire to.”

Hasnain says:

“And it wasn’t just the fault of Meta’s marketing team. Yann LeCun, a Turing Award winner and Meta’s chief scientist, defended Galactica to the end. On the day the model was released, LeCun tweeted: “Type a text and Galactica will generate a paper with relevant references, formulas, and everything.” Three days later, he tweeted: “Galactica demo is off line for now. It’s no longer possible to have some fun by casually misusing it. Happy?”

It's not quite Meta's Tay moment. Recall that in 2016, Microsoft launched a chatbot called Tay on Twitter—then shut it down 16 hours later when Twitter users turned it into a racist, homophobic sexbot. But Meta’s handling of Galactica smacks of the same naivete.

“Big tech companies keep doing this—and mark my words, they will not stop—because they can,” says Shah. “And they feel like they must—otherwise someone else might. They think that this is the future of information access, even if nobody asked for that future.””

Hasnain says:

So. Much. Corruption. Alito likes to complain when people call the court illegitimate, but he’s one of the main people that made it so!

“The ruling this year thrilled anti-abortion supporters, though it has proved deeply unpopular among the majority of Americans. After the draft was leaked, Mr. Schenck said, he felt compelled to come forward about his attempts to influence the court.

“You can position yourself in a special category with regard to the Justices,” he said. “You can gain access, have conversations, share prayer.”

Even when his group was most active at the court, he said, “I would look up at that phrase that’s chiseled into the building itself, ‘Equal Justice Under Law,’” he recalled. “I would think, ‘Not really.’””

Hasnain says:

lol.

“Multiple “critical” teams inside Twitter have now either completely or near-completely resigned, said other employees who requested anonymity to speak without Musk’s permission. That includes Twitter’s traffic and front end teams that route engineering requests to the correct backend services. The team that maintains Twitter’s core system libraries that every engineer at the company uses is also gone. “You cannot run Twitter without this team,” a departing employee said.

Several members of Twitter’s “Command Center” team, a group of engineers that is on call 24/7 and acts as the clearing house for problems internally, also tweeted about their departures. “If they go down, there is no one to call when shit breaks,” said a person familiar with how the team operates. The team that manages Twitter API for developers has also been severely gutted.

In a tweet Thursday evening, Musk said: “The best people are staying, so I’m not super worried.””

Hasnain says:

I hate sharing a twitter link but this is too funny to not share: most people called Elon’s bluff and are now quitting - and he’s begging people to stay: https://twitter.com/alexeheath/status/1593365103331577856?s=46&t=wrqExQQ5qhV_VfOsEVeM-g

And future update (will link in comments) - they’ve closed offices and locked people out, presumably fearing sabotage

Hasnain says:

"Archaeologists first came across the ancient city in 1911 after hearing reports of some brickwork in the area. However, the Archaeological Survey of India (ASI) dismissed the bricks as not having any kind of antiquity and the site remained undisturbed for several more years. It wasn't until 1922 that R D Banerji, an ASI officer, believed he saw a buried stupa, a mound-like structure where Buddhists typically meditate. This led to large-scale excavations – most notably by British archaeologist Sir John Marshall – and the eventual naming of Mohenjo-daro as a Unesco World Heritage Site in 1980. The remains they uncovered revealed a level of urbanisation not previously seen in history, with Unesco lauding Mohenjo-daro as the "best preserved" ruin of the Indus Valley.

Perhaps the city's most surprising feature was a sanitation system that was far beyond its contemporaries. While drainage and private toilets were seen in Egypt and Mesopotamia, they were luxuries of the rich. In Mohenjo-daro, concealed toilets and covered drains were everywhere. Since excavations began, more than 700 wells have been recovered, in addition to a system of private baths, including a 12m x 7m "Great Bath" for communal use. Incredibly, toilets were found in many private residences, and waste was covertly disposed of through a sophisticated, city-wide sewage system."

Hasnain says:

I am so so excited by this that words are failing me (yes, this is not a common statement).

Using git for side projects has honestly sucked after the great experiences I've had with sapling and I'm glad to see I can use it again. Can't wait to set it up now for my private repos; and ... my mind is blown that there's an accompanying review UI too that's optimized for stacks.

stacked diffs, `hg ssl`, `hg commit`, `hg absorb`, and `hg histedit`, how I missed thee.

"I hope you find Sapling as pleasant to use as we do, and that Sapling might start a conversation about the current state of source control and how we can all hold the bar higher for the source control of tomorrow.See the Getting Started page to try Sapling today."

Hasnain says:

"Additionally, more than five times as many people have been hospitalized for the flu so far this season than at the same point in any of the last 10 years. And unlike RSV, which poses the biggest threat to the youngest and oldest, the severe disease flu causes is more evenly spread across age groups. About one-third of the people who’ve been hospitalized for flu this year were 65 or older, while another quarter were ages 18 to 49."

Hasnain says:

"It's common wisdom that systems with caches can fail badly when the caches are cold. When every request misses the cache, the system is overloaded, and never recovers. What I observed was the mirror image."

Hasnain says:

My list of funny Elon related drama stories is so long it’s hard to pick what to share. This one takes the cake though - so much for free speech.

Currently tied with how he’s turning off 80% of micro services soon. 2FA already broke, locking people out.

Hasnain says:

Great read.

“To recap, we learned about the various pillars of observability and took a deep dive into the tracing pillar. We learned about setting up an Elixir application with the OpenTelemetry tooling and even deployed our application along with some supporting monitoring tools to Fly.io. We then compared trace results between two different LiveView pages and were able to see the effects that an N+1 query would have on our application's performance.”

Hasnain says:

This (along with all the news coming out of Iran the last few months) is horrifying.

“In response to the parliament vote, activist and journalist Masih Alinejad tweeted: "227 members of the 290-seat Parliament in Iran have called on the Judiciary to issue death sentences for people arrested during the ongoing uprising. They want to execute innocent protesters who chanted Woman Life Freedom. The world must stop this act of terror."”

Hasnain says:

"As that allocation was happening for every single path that was being matched against, removing it made the whole program 1.7x faster in this instance. Not bad for a one character change."

Hasnain says:

“The fact there isn’t stems, for Paul, from a deeper problem. Despite underpinning the viability of life itself, nature’s contributions are absent from most economic modelling. And unlike corporations, nature has no legal standing: a river can neither sue its polluters, nor charge for the many services it provides. This has allowed it to become a “free” externality: something neither capitalism or the state has to account for, or take seriously.

“It’s effectively stealing from nature”, Paul says. “Thames Water don’t have to pay when they do an overspill into the river. The Highways authorities don’t have to pay when they discharge stormwater into the river. They don’t pay when they extract from the river.” If the full actual cost of what companies like Thames Water take from, and put back into, the Roding were actually accounted for, Paul believes “we could easily pay for dozens of people to look after the river.”

While the idea of a river being paid for its services might seem radical, it is gaining ground. Natural capital - the idea that nature provides intrinsic economic value, either through services it provides (e.g. the pollination of crops by bees) or the preservation of life (e.g. through the carbon dioxide it absorbs) is slowly becoming more normalised in policy circles.”

Hasnain says:

Great technical analysis and write up.

“Disabling writes to the secondary superclass cache worked well in our case, and even though this might not be a desirable solution in all cases, we wanted to share our methodology, toolset and the fix in the hope that it would help others encountering similar symptoms. While working through this problem, we came across JDK-8180450 — a bug that’s been dormant for more than five years that describes exactly the problem we were facing. It seems ironic that we could not find this bug until we actually figured out the answer. We believe our findings complement the great work that has been done in diagnosing and remediating it.”

Hasnain says:

The fact that a company lawyer is writing this is pretty huge.

“All of this is extremely dangerous for our users. Also, given that the FTC can (and will!) fine Twitter BILLIONS of dollars pursuant to the FTC Consent Order, extremely detrimental to Twitter’s longevity as a platform. Our users deserve so much better than this.

If you feel uncomfortable about anything you’re being asked to do, you can call Twitter’s Ethics Hotline at (800) 275-4843 or submit a report at ethicshelpline.twitter.com. Please also note the FTC’s number is: 1-877-FTC-HELP. You may also remember that Mudge reached out to httos://whistlebloweraid.org”

Hasnain says:

To all my friends and coworkers still at Meta: sorry you’re going through this. I’m here to talk or provide help/guidance for searching for your next job (based on my current experience).

“In the blog post announcing Meta’s cuts, Zuckeberg said laid off employees in the US would receive 16 weeks of base pay plus two additional weeks for each year of service, health insurance coverage for six months, and support for finding a new career and navigating immigration issues. Zuckerberg said the company would be instituting a hiring freeze through the first quarter of 2023 “with a small number of exceptions.””

Hasnain says:

“White America has always denigrated corpulence as a sign of immorality and sin. Historians, including Stearns, have long argued that modern Western dieting and beauty standards are direct descendants of puritanical Protestantism: fasting, self-discipline, and self-denial of earthly pleasures. More recently, sociologist Sabrina Strings, PhD, delved deeper into this connection, citing the confluence of Protestantism and the trans-Atlantic slave trade. In her 2019 book, Fearing the Black Body, Strings notes that the overtly racist concept of African gluttony, sloth, and ignorance, invented by European colonists and race scientists, met its perfect match in early American Protestantism. Thus, slimness was established as a mark of righteousness and superiority, while fatness symbolized unintelligence, immorality, and “blackness.””

Hasnain says:

“Twitter Inc., after laying off roughly half the company on Friday following Elon Musk’s $44 billion acquisition, is now reaching out to dozens of employees who lost their jobs and asking them to return.

Some of those who are being asked to return were laid off by mistake, according to two people familiar with the moves. Others were let go before management realized that their work and experience may be necessary to build the new features Musk envisions, the people said, asking not to be identified discussing private information.”

Hasnain says:

Time to upgrade!

“CPython 3.11 is 25% faster than CPython 3.10 on average according to benchmarks with pyperformance. The changes that are being done are part of something that Guido called: The "Shannon Plan". For the 3.11 release, the plan was about making a lot of optimizations in two main areas: startup and runtime. The release also includes additional optimizations that are not part of the faster CPython project.”

Hasnain says:

Will definitely need to try this in the future.

“One question that might arise is: is there any point in using fuzzing alongside Kani? In other words, if we can prove a harness for all inputs with Kani, would we still want to use fuzzing? The answer is yes, because fuzzing and Kani provide complementary benefits.

Kani verifies a Rust program by symbolically analyzing its code. This allows Kani to make mathematical statements about the expected semantics of the Rust code being verified. Kani allows you to prove that, for all possible inputs, the code under verification follows its specification, assuming everything else functions correctly (e.g. the underlying hardware, the OS, etc.).

Fuzzing concretely executes the program under test. This gives you end-to-end confidence that for the set of inputs generated by the fuzzer, the code under verification follows its specification, under real-world conditions. Fuzzing and Kani fit together to give more assurance than either provides on its own.”

Hasnain says:

“Unlike his splashy announcement videos, however, when Elon Musk fails to deliver on the wild and absurd promises he regularly makes, no one in the mainstream media is there to cover it. Things are different this time, however. Now, instead of a handful of technology writers being the only witnesses to his hype collapsing, nearly 290 million Twitter users are looking on in horror while the vast majority of people who will never join the platform are laughing from afar.”