Hasnain says:

Todays fun fact of the day comes from this book I was listening to on my commute. TIL this fact (though it seems disputed)

“The Islam brought to America by enslaved Africans did not survive long, but it left traces that are still visible today. The practice of ring shout, a form of religious dance in which men and women rotate counterclockwise while singing, clapping their hands and shuffling their feet, was directly inherited from enslaved Muslims such as Bilali Mohammed and Salih Bilali in the Georgia Sea Islands. It originally mimicked the ritual circling (or shaw’t) of the Kaaba in Mecca by Muslim pilgrims.”

Hasnain says:

So many useful insights summarized concisely here. I’ll have to go over the papers again at some point to refresh my memory.

“Insight 9 : In a datacenter environment, we prefer centralized controllers over decentralized ones due to their simplicity and ability to make higher-quality decisions. In many cases, a hybrid approach—a centralized control plane combined with a decentralized data plane—provides the best of both worlds.”

Hasnain says:

Learned so much about so many things from this one. Gotta love when you see something like this a few thousand words into a post you already thought was quite interesting

“Now that the boring stuff is behind us, let’s start the fun part. Again, just as a reminder on how the intersection works: we do two phases of intersection, one for the conventional intersection and another for the bits that would cross the group boundary, and in the end, we merge these two.

In this section, we will take a look at assembly, some cool tools to analyze this assembly, AVX-512, differences in the microarchitecture of AMD and Intel chips, emulation of instructions, and a lot more. So again, sorry to bother you with all of the previous stuff, but it was important.”

Hasnain says:

“Moving data serialization format used by the Issue Service to Protobuf resulted in many improvements, including faster response time and reduced resource consumption (CPU, storage). Even though there were some challenges we had to solve during the migration, the final results were absolutely worth the effort. As we continue our work in the Issue Service and progress to handling more traffic and data, the impact of these relative improvements will continue to grow.”

Hasnain says:

“Inflation has lots of causes, it's true. But when an industry is consolidated enough to take advantage of a data brokerage or just engage in tacit collusion, any source of inflation – war, disease, weather – allows whole sectors to raise prices together, and keep them high, long after the shock has passed.”

Hasnain says:

“We need to build in the ability to reconfigure our systems in advance, without knowing exactly what sorts of changes we’ll need to make. The Canva engineers had some powerful operational knobs at their disposal through the Cloudflare firewall configuration. This allowed them to make changes. The more powerful and generic these sorts of dynamic configuration features are, the more room for maneuver we have. Of course, dynamic configuration is also dangerous, and is itself a contributor to incidents. Too often we focus solely on the dangers of such functionality in creating incidents, without seeing its ability to help us reconfigure the system to mitigate incidents.

Finally, these sorts of operator interfaces are of no use if the responders aren’t familiar with them. Ultimately, the more your responders know about the system, the better position they’ll be in to implement these adaptations. Changing an unhealthy system is dangerous: no matter how bad things are, you can always accidentally make things worse. The more knowledge about the system you can bring to bear during an incident, the better position you’ll be in to adaptive your system to extend that competence envelope.”

Hasnain says:

This was a fun read, reminding me I need to go back and keep up with the latest in succinct data structure research.

Also kinda timely quote given the recent hoopla around DeepSeek:

“Even though modern spell checkers use different techniques like edit distance and language models, the engineering insights from Unix spell remain valuable. It shows how deep understanding of theoretical concepts combined with practical constraints can lead to efficient and elegant solutions.

Most importantly, it demonstrates that some of the best innovations happen when we are resource constrained, forcing us to think deeper about our problems rather than throwing more hardware at them.”

Hasnain says:

“In 1872, Weierstrass published a function that threatened everything mathematicians thought they understood about calculus. He was met with indifference, anger and fear, particularly from the mathematical giants of the French school of thought. Henri Poincaré condemned Weierstrass’ function as “an outrage against common sense.” Charles Hermite called it a “deplorable evil.””

Hasnain says:

“The Real Culprit: setenv and getenv

setenv is not a safe function to call in a multithreaded environment. This is often a problem, and occasionally rediscovered as developers like us hit weird crashes in libc’s getenv [9], [10], [11], [12].”

Hasnain says:

This was chock full of great advice.

“Choose happiness. If you’re unhappy in life, success is much harder to achieve. When I was growing up, the American mantra was that happiness requires wealth. Wealth and happiness are two different goals; we have unhappy billionaires today! I always picked happiness over wealth when there was a choice, and I’m very glad that I did.”

Hasnain says:

TIL over half of LA’s fire trucks were out of commission during the recent fires and a nontrivial amount of the blame here goes to… private equity

“While AIP’s consolidation of economic power over fire truck manufacturing is appalling, it is not some unsolvable, intractable problem we just have to live with. State and federal antitrust laws already prohibit the kind of monopolistic roll-up that AIP perpetrated — they just need to be enforced. State AGs can bring lawsuits to force REV Group to divest the manufacturers it illegally acquired and to pay damages to fire departments for the harm that its (attempted) monopolization of the fire-truck industry has caused. Fire departments and other fire-apparatus purchasers can bring their own lawsuits to do the same. So can the FTC and the DOJ’s Antitrust Division. If state legislators or members of Congress want to pave the way for such lawsuits, they can launch their own investigations into the fire apparatus industry. And if anyone wants guidance on what a lawsuit against AIP could look like, Lina Khan left us a roadmap just before she stepped down from the FTC last week — when she sued private-equity giant Welsh Carson for rolling up Texas anesthesiology practices to drive up the price of anesthesia services to Texas patients.

We have all the tools we need to check AIP’s greed and abuse and restructure the fire-truck industry so it serves the public interest. The only question is whether our political leaders have the will.”

Hasnain says:

“The Israel Defense Forces (IDF) says the building was targeted because it was a Hezbollah "terrorist command centre" and it "eliminated" a Hezbollah commander. It added that "the overwhelming majority" of those killed in the strike were "confirmed to be terror operatives".
But a BBC Eye investigation verified the identity of 68 of the 73 people killed in the attack and uncovered evidence suggesting just six were linked to Hezbollah's military wing. None of those we identified appeared to hold a senior rank. The BBC's World Service also found that the other 62 were civilians - 23 of them children.”

Hasnain says:

I am glad this is finally out, if only because I can finally reference Mark S's famous one ampersand commit and have people believe me and not think that I'm making shit up. Great read on profilers and also TIL the code is open source.

"A seasoned performance engineer was looking through Strobelight data and discovered that by filtering on a particular std::vector function call (using the symbolized file and line number) he could identify computationally expensive array copies that happen unintentionally with the ‘auto’ keyword in C++.

The engineer turned a few knobs, adjusted his Scuba query, and happened to notice one of these copies in a particularly hot call path in one of Meta’s largest ads services. He then cracked open his code editor to investigate whether this particular vector copy was intentional… it wasn’t.

It was a simple mistake that any engineer working in C++ has made a hundred times.

So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!

Go back and re-read that sentence. One ampersand! "

Hasnain says:

This is why I always name settings that have a time component as eg “settingNameSeconds” so there is no confusion because what even is this

“Which was what the setting value was changed to in the patch that was eventually accepted. This means that setting help.autocorrect to 1 logically means "wait 100ms (1 decisecond) before continuing".

Now, why Junio thought deciseconds was a reasonable unit of time measurement for this is never discussed, so I don't really know why that is. Perhaps 1 full second felt too long so he wanted to be able to set it to half a second? We may never know. All we truly know is that this has never made sense to anyone ever since.”

Hasnain says:

Lots to ponder and think about from this rant. I do think as a society (maybe I’m just grumpy) the value of artisanal, high quality work, has really gone by the wayside. It’s so magnificent when you get to see an expert at work, someone who really cares about their craft.

“When I joined my former Big Tech job, everyone cared. Over time, incentives attracted a different set of people who didn't care as much. Eventually those people became the majority. It's painful to work with people who don't care if you care a lot, and eventually I left because of it.

Now, I'm at a small startup full of people who care. Customer bug reports go right to our chatroom. We fix them immediately. I feel guilty I wrote the bugs at all. We reach out to users to see if we can make their lives better. We care.

I want to live in a community where everyone cares.

The one place in the world you get this vibe is probably Japan. Most people just really care. Patrick McKenzie refers to this as the will to have nice things. Japan has it, and the US mostly does not.

In Japan, you get the impression that everyone takes their job and role in society seriously. The median Japanese 7-11 clerk takes their job more seriously than the median US city bureaucrat. And the result is obvious if you visit both places.”

Hasnain says:

It’s gonna be a long four years. Anxiously waiting to see what exactly gets signed in these EOs so I can prepare

“Immigration and border security will make up a major pillar of Trump's early executive actions. Trump has promised to carry out mass deportations, end birthright citizenship and "secure" the southern border.”

Hasnain says:

Even in 2024, people not baking in rollback safety into their releases..

“Before long, the buggy new app had become as pleasant as a termite infestation. There were so many complaints from disappointed customers that executives seriously considered just going back to the old app. But they couldn’t. After rigorous testing, they determined the previous version of the app was no longer compatible with the rest of its software.

All of these problems were compounded by a lack of communication. It took until July for the then-CEO, Patrick Spence, to apologize. Even when Spence detailed his plans for repairing the app, he cautioned that it would take more time. And today, it’s still not entirely fixed.”

Hasnain says:

“But in French's experience, most people eventually embrace their diagnosis and find that it helps them to put support in place and live a better life. This applies to French herself. As with many women, when she was a child her inattention was not seen as disruptive. And as she grew up and moved from France to the UK, her anxiety and depression were not linked to ADHD. It was only on moving to Australia and seeing a new GP that she was referred to an ADHD specialist. At the age of 30, when she finally received an ADHD diagnosis, there was a sense of relief: "It was a very welcome explanation to a lot of the things I was struggling with". “

Hasnain says:

Another example of why security is so hard to get right.

“The threat actors behind the Otelier breach told BleepingComputer that they initially hacked the company's Atlassian server using an employee's login. These credentials were stolen through information-stealing malware, which has become the bane of corporate networks over the past few years.

When BleepingComputer asked Otelier to confirm this information, a company representative said they could not share any further comments on the incident. However, BleepingComputer found on the Flare threat intelligence platform Otelier employee information that had been stolen by infostealer malware.

The threat actors say they used these credentials to scrape tickets and other data, which contained further credentials to the company's S3 buckets.

Using this access, the hackers claimed to have downloaded 7.8TB of data from the company's Amazon cloud storage, including millions of documents belonging to Marriott that were in S3 buckets managed by Otelier. These documents include nightly hotel reports, shift audits, and accounting data.”

Hasnain says:

"When I first started Keygen, I had this idea in my head that I could create a company where I never had to get on a sales call — or any call. Being an introvert, I absolutely hated calls. They're not only awkward, but a 30 minute call takes up hours of my headspace. I quickly learned that I didn't want to do them, and so I decided that I wouldn't.

I instituted a bonkers 'no calls' policy at work.

(Even I thought I was being crazy.)"

Hasnain says:

Great read on how to build larger projects over the long term. Some of the advice resonated with my past experiences; and the rest is stuff I’ll start applying immediately

“Instead of splitting a project up into arbitrary milestones, consider delivering incremental value by shipping concrete stepping stones. Stepping stones can serve to de-risk a project by minimizing dependencies and providing standalone checkpoints. Most importantly however they help to simplify a complex project by providing a means to structure deliverables around eliminating unknown unknowns. Formulating useful stepping stones is an art and often requires an intense focus on simplicity to avoid deviating out of the cone of strategy along the way to an end goal.”

Hasnain says:

So much great advice here. I haven’t listened to the whole thing but I did read the full summary. It’s great.

“A consequence of the interview process not changing much, but the job market becoming more competitive is how the bar to do well on these interviews went up. This is because there are more and better preparation materials, so the “average” candidate does better on these interviews than years before. Preparing for interviews at Big Tech companies and scaleups is no longer a “nice to have:” it’s a necessity for even strong engineers, who want to get a job offer.”

Hasnain says:

“The top reason those non-voters cited, above the economy at 24 percent and immigration at 11 percent, was Gaza: a full 29 percent cited the ongoing onslaught as the top reason they didn’t cast a vote in 2024.”

Hasnain says:

Cautiously optimistic here. There will be lots of gotchas and caveats. But a pause in the fighting is very welcome at this stage.

If it does turn out, as initial reports suggest, that this was because Trump pushed harder than Biden (with associated headlines / news tidbits that have been, interesting, to say the least), I will have to give credit where credit is due because at least he's stopping a genocide (even if it's for self serving aims). Hoping to learn more.

"Trump pledged last month that “all hell will break out” if Hamas doesn’t promise to release hostages by Jan. 20 and his Mideast envoy, Steve Witkoff, joined talks in the region to push for a deal."

Hasnain says:

I spent way too much time on this.

(well, the ~40 or so minutes needed to finish it)

Hasnain says:

Jeff has always been one of the great ones. This whole post is admirable and worth reading. When a few people win, we all lose. We need to uplift everyone around us.

“I think many of the Americans who did vote are telling us they no longer believe our government is effectively keeping America fair for everyone. Our status as the world's leading democracy is in question. We should make it easier for more eligible Americans to vote, such as making election day a national holiday, universal mail in voting, and adopting ranked choice voting so all votes carry more weight. We should also strengthen institutions keeping democracy fair for everyone, such as state and local election boards, as well as the Federal Election Commission.

It was only after I attained the dream that I was able to fully see how many Americans have so very little. This much wealth starts to unintentionally distance my family from other Americans. I no longer bother to look at how much items cost, because I don't have to. We don't have to think about all these things that are challenging or unreachable for so many others. The more wealth you attain, the more unmistakably clear it becomes how unequal life is for so many of us.

Even with the wealth I have, I can't imagine what it would feel like to be a billionaire. It is, for lack of a better word, unamerican.”

Hasnain says:

“Even more important, the work demonstrates that the Gowers norm can act as a powerful tool in a new domain. “Because it’s so new, at least in this part of number theory, there is potential to do a bunch of other things with it,” Friedlander said. Mathematicians now hope to broaden the scope of the Gowers norm even further — to try using it to solve other problems in number theory beyond counting primes.

“It’s a lot of fun for me to see things I thought about some time ago have unexpected new applications,” Ziegler said. “It’s like as a parent, when you set your kid free and they grow up and do mysterious, unexpected things.””

Hasnain says:

I've never seen a name put to this, though in hindsight I've seen it a lot. Worth pondering.

"In general, I think well-designed tools (and systems) should aim to minimize this effect. This can be hard to do in a fully general manner, but some things I think about when designing a new tool:

* Does it need to be configurable?
* Does it need syntax of its own?
* As a corollary: can it reuse familiar syntax or idioms from other tools/CLIs?
* Do I end up copy-pasting my use of it around? If so, are others likely to do the same?"

Hasnain says:

“The formatting function used by std.debug.print in the examples is a powerful generic function. Lots of languages parse their format strings at runtime, and possibly add some special validators to the string format to catch errors early. In Zig, the format string is parsed at comptime, creating efficient output code while also performing all validation at compile time.”

Hasnain says:

Lots to ponder in this one.

“What's taboo is to say that the source is the system we inhabit, not our personal inability to manifest god-like powers. The system works fine for the winners who twirl the dials on the narrative control machinery, and they're appalled when they suffer some mild inconvenience when the peasantry doing all the work for them break down and quit.

A tsunami of burnout and quitting, both quiet and loud, is on the horizon, but it's taboo to recognize it or mention it. That the system is broken because it breaks us is the taboo that is frantically enforced at all levels of narrative control.”

Hasnain says:

So much truth in here. I wish I could hammer home the point about consistency, it makes a world of a difference in large codebases.

Speaking here as a guy who often has to go and clean these up for various reasons,

“Why is consistency so important in large codebases? Because it protects you from nasty surprises, it slows down the codebase’s progression into a mess, and it allows you to take advantage of future improvements.”

Hasnain says:

Having verified this myself, uh, bro, what

“Microsoft is pulling yet another trick to get people to use its Bing search engine. If you use Bing right now without signing into a Microsoft account and search for Google, you’ll get a page that looks an awful lot like... Google.”

Hasnain says:

"While Yemeni coffee shops fill a need for the Muslim community, they attract a wide range of cultures, welcoming anyone looking for family-friendly outings or alcohol-free evenings.

“There’s been a huge transformation across the U.S. People are trying to transition from the nightlife bar environment,” Almatrahi says. “We provide not only the coffee experience but also the ambience.”

During the last lease negotiation for a new location, the landlord was “shocked” by Arwa’s proposed hours, according to Almatrahi. “We wanted to be open until one a.m. on the weekends. And they’re not used to that concept,” he says. “But that’s the power of cardamom.”"

Hasnain says:

This is a rough day. On one hand, I understand some of the criticisms. The fact checkers got things wrong. Facebook jail was real and appeals sucked.

But you don’t fix that by throwing the baby out with the bath water. Or making it acceptable to call women household objects. Or allowing transphobia. Or so many other things.

“The former employee I spoke with feared that whatever consequences Meta's surrender to the right on speech issues might have in the United States, its effect in the rest of the world could be even more dire.

"I really think this is a precursor for genocide," they said. "We've seen it happen. Real people's lives are actually going to be endangered. I'm just devastated”

Hasnain says:

Climate change is here. The videos I saw were scary, including the abandoned cars being bulldozed to let fire trucks through.

“A fire was burning out of control Tuesday in Pacific Palisades, destroying homes and forcing residents to abandon their vehicles and flee amid a potentially “life-threatening and destructive” windstorm.”

Hasnain says:

Lots to ponder from this piece about using LLMs for day to day productivity.

A few main takeaways for me:

* ensure you can easily verify the output
* getting over the starting hump is really valuable
* having more smaller specialized modules vs fewer larger reusable modules may be better
* use LLMs for tests!

“Let me try to motivate this for the skeptical. A lot of the value I personally get out of chat-driven programming is I reach a point in the day when I know what needs to be written, I can describe it, but I don’t have the energy to create a new file, start typing, then start looking up the libraries I need. (I’m an early-morning person, so this is usually any time after 11am for me, though it can also be any time I context-switch into a different language/framework/etc.) LLMs perform that service for me in programming. They give me a first draft, with some good ideas, with several of the dependencies I need, and often some mistakes. Often, I find fixing those mistakes is a lot easier than starting from scratch.”

Hasnain says:

Same day as unconfirmed rumors hit that 300 employees of another nationality did the same thing, I see a news story about some employees doing this. Like… why

“The former employees, over a three-year period, tricked the tech company into matching thousands of dollars in donations to children’s charities when they were not in fact donating a thing, the DA's Office said. The total take from the scheme was about $152,000”

Hasnain says:

“Mr Kingswood said he would be "surprised if the airfield hadn't met all the requirements in accordance with industry standards".
"I suspect if we went around the airfields at a lot of major international airports... we would find a lot of obstacles that could similarly be accused of presenting a hazard," he added.”

Hasnain says:

“Now one can try to reverse-engineer how the app decrypts the currency file, and developers have a wide suite of tools to practice security by obscurity. You can read the file backwards, shift everything left by 12 bits, then forward by 2, modulo every second byte with 3, subtract your birthday as a UNIX timestamp, blah blah blah.

While it certainly is fun to dig around and follow the trail to having a usable file, I enjoy actually finishing my projects much more.”

Hasnain says:

I’ve tried to do some of these on my own sites too. I miss the old web. And also working with / learning from Rachel.

“I've been thinking about things that annoy me about other web pages. Safari recently gained the ability to "hide distracting items" and I've been having great fun telling various idiot web "designers" to stuff it. Reclaiming a simple experience free of wibbly wobbly stuff has been great.

In doing this, I figured maybe I should tell people about the things I don't do here, so they realize how much they are "missing out" on.”

Hasnain says:

Great read which I’ll probably revisit over the years

“Engineering choices seem – dare I say? – more pragmatic. Alex Kotliarskyi from Secta Labs said he had to dial down his “tech purist views” from when he was at Facebook and Replit, and instead just choose “good enough” tools. At Fern Creek Software, Keith also said how he noticed how it’s VC-funded companies that aspire to technical elegance, and over engineering things.”

Hasnain says:

Honestly, there was some useful stuff to ponder after reading this, since I don't think the problems are necessarily limited to super rich people. The goals and meaning of life are issues that hit us all.

Edit to clarify: this made *me* ponder things given I was already thinking about life and everything else going on and what I want to do in my life.

So that takeaway of “money isn’t everything and an ulterior motivation in life is needed” was definitely an apt reminder, regardless of which form it was delivered in.

This specific guy’s midlife crisis is a bit too out of touch - from calling people NPCs to only being in a relationship as long as it benefited him, to, uh, whatever that was with DOGE, and I hope it goes without saying that I hope to never find myself in that situation.

"I know. This is a completely zeroth-world position to be in. The point of this post isn’t to brag or gain sympathy. To be honest, I don’t exactly know what the point of this post is. I tried to manufacture one, but I just felt like a phony. Then I recognized the irony of creating purpose out of a blog post when I don’t currently have much conviction or purpose in life."

Hasnain says:

"There’s an expansionist tendency right now. You would think that this is the time that Israel could conclude that it would expand, it could occupy and it could annex with no serious international repercussions. All of this is strange because Israel is currently facing more international repercussions or threats of international repercussions than ever before in its history, and yet those repercussions haven’t materialized."

Hasnain says:

This is from 1988. What in the world..

(Thank you Twitter for the olfactory ethics discourse that lead me to this)

“Assistant Hesperia Fire Chief Will Wentworth listened incredulously as a caller complained that the noxious black smoke pouring from a nondescript building in the desert carried the sickeningly sweet smell of burning human flesh.

“I don’t think so, it’s a ceramics shop,” Wentworth replied.

“Don’t tell me they’re not burning bodies. I was at the ovens at Auschwitz,” the man said chillingly, Wentworth recalled.”