Strobelight: A profiling service built on open source technology
We’re sharing details about Strobelight, Meta’s profiling orchestrator. Strobelight combines several technologies, many open source, into a single service that helps engineers at Meta improve effic…
Hasnain says:
I am glad this is finally out, if only because I can finally reference Mark S's famous one ampersand commit and have people believe me and not think that I'm making shit up. Great read on profilers and also TIL the code is open source.
"A seasoned performance engineer was looking through Strobelight data and discovered that by filtering on a particular std::vector function call (using the symbolized file and line number) he could identify computationally expensive array copies that happen unintentionally with the ‘auto’ keyword in C++.
The engineer turned a few knobs, adjusted his Scuba query, and happened to notice one of these copies in a particularly hot call path in one of Meta’s largest ads services. He then cracked open his code editor to investigate whether this particular vector copy was intentional… it wasn’t.
It was a simple mistake that any engineer working in C++ has made a hundred times.
So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!
Go back and re-read that sentence. One ampersand! "
Posted on 2025-01-24T05:28:29+0000
Why is Git Autocorrect too fast for Formula One drivers?
Why does Git's autocorrect wait 0.1s before executing a mistyped command? Let's dig in.
Hasnain says:
This is why I always name settings that have a time component as eg “settingNameSeconds” so there is no confusion because what even is this
“Which was what the setting value was changed to in the patch that was eventually accepted. This means that setting help.autocorrect to 1 logically means "wait 100ms (1 decisecond) before continuing".
Now, why Junio thought deciseconds was a reasonable unit of time measurement for this is never discussed, so I don't really know why that is. Perhaps 1 full second felt too long so he wanted to be able to set it to half a second? We may never know. All we truly know is that this has never made sense to anyone ever since.”
Posted on 2025-01-23T07:19:50+0000
Nobody Cares
A rant about caring
Hasnain says:
Lots to ponder and think about from this rant. I do think as a society (maybe I’m just grumpy) the value of artisanal, high quality work, has really gone by the wayside. It’s so magnificent when you get to see an expert at work, someone who really cares about their craft.
“When I joined my former Big Tech job, everyone cared. Over time, incentives attracted a different set of people who didn't care as much. Eventually those people became the majority. It's painful to work with people who don't care if you care a lot, and eventually I left because of it.
Now, I'm at a small startup full of people who care. Customer bug reports go right to our chatroom. We fix them immediately. I feel guilty I wrote the bugs at all. We reach out to users to see if we can make their lives better. We care.
I want to live in a community where everyone cares.
The one place in the world you get this vibe is probably Japan. Most people just really care. Patrick McKenzie refers to this as the will to have nice things. Japan has it, and the US mostly does not.
In Japan, you get the impression that everyone takes their job and role in society seriously. The median Japanese 7-11 clerk takes their job more seriously than the median US city bureaucrat. And the result is obvious if you visit both places.”
Posted on 2025-01-21T06:02:54+0000
Trump inauguration live updates: Day One executive orders target Alaska energy, birthright citizenship, DEI efforts
President-elect Donald J. Trump plans to sign dozens of executive orders within hours of his inauguration.
Hasnain says:
It’s gonna be a long four years. Anxiously waiting to see what exactly gets signed in these EOs so I can prepare
“Immigration and border security will make up a major pillar of Trump's early executive actions. Trump has promised to carry out mass deportations, end birthright citizenship and "secure" the southern border.”
Posted on 2025-01-20T16:22:14+0000
The $500 Million Debacle at Sonos That Just Won’t End
Companies update their apps all the time. This one annoyed customers, cratered the stock and cost the CEO his job.
Hasnain says:
Even in 2024, people not baking in rollback safety into their releases..
“Before long, the buggy new app had become as pleasant as a termite infestation. There were so many complaints from disappointed customers that executives seriously considered just going back to the old app. But they couldn’t. After rigorous testing, they determined the previous version of the app was no longer compatible with the rest of its software.
All of these problems were compounded by a lack of communication. It took until July for the then-CEO, Patrick Spence, to apologize. Even when Spence detailed his plans for repairing the app, he cautioned that it would take more time. And today, it’s still not entirely fixed.”
Posted on 2025-01-19T07:11:38+0000
Why an ADHD diagnosis can be a mixed blessing
For adults, an ADHD diagnosis can be life-saving. But it can also carry risks.
Hasnain says:
“But in French's experience, most people eventually embrace their diagnosis and find that it helps them to put support in place and live a better life. This applies to French herself. As with many women, when she was a child her inattention was not seen as disruptive. And as she grew up and moved from France to the UK, her anxiety and depression were not linked to ADHD. It was only on moving to Australia and seeing a new GP that she was referred to an ADHD specialist. At the age of 30, when she finally received an ADHD diagnosis, there was a sense of relief: "It was a very welcome explanation to a lot of the things I was struggling with". “
Posted on 2025-01-19T06:59:18+0000
Otelier data breach exposes info, hotel reservations of millions
Hotel management platform Otelier suffered a data breach after threat actors breached its Amazon S3 cloud storage to steal millions of guests' personal information and reservations for well-known hotel brands like Marriott, Hilton, and Hyatt.
Hasnain says:
Another example of why security is so hard to get right.
“The threat actors behind the Otelier breach told BleepingComputer that they initially hacked the company's Atlassian server using an employee's login. These credentials were stolen through information-stealing malware, which has become the bane of corporate networks over the past few years.
When BleepingComputer asked Otelier to confirm this information, a company representative said they could not share any further comments on the incident. However, BleepingComputer found on the Flare threat intelligence platform Otelier employee information that had been stolen by infostealer malware.
The threat actors say they used these credentials to scrape tickets and other data, which contained further credentials to the company's S3 buckets.
Using this access, the hackers claimed to have downloaded 7.8TB of data from the company's Amazon cloud storage, including millions of documents belonging to Marriott that were in S3 buckets managed by Otelier. These documents include nightly hotel reports, shift audits, and accounting data.”
Posted on 2025-01-19T01:42:40+0000
No calls
Enterprise sales don't have to be crazy.
Hasnain says:
"When I first started Keygen, I had this idea in my head that I could create a company where I never had to get on a sales call — or any call. Being an introvert, I absolutely hated calls. They're not only awkward, but a 30 minute call takes up hours of my headspace. I quickly learned that I didn't want to do them, and so I decided that I wouldn't.
I instituted a bonkers 'no calls' policy at work.
(Even I thought I was being crazy.)"
Posted on 2025-01-17T05:59:04+0000
Stepping Stones not Milestones
Structure projects around delivering incremental value and eliminating unknown unknowns.
Hasnain says:
Great read on how to build larger projects over the long term. Some of the advice resonated with my past experiences; and the rest is stuff I’ll start applying immediately
“Instead of splitting a project up into arbitrary milestones, consider delivering incremental value by shipping concrete stepping stones. Stepping stones can serve to de-risk a project by minimizing dependencies and providing standalone checkpoints. Most importantly however they help to simplify a complex project by providing a means to structure deliverables around eliminating unknown unknowns. Formulating useful stepping stones is an art and often requires an intense focus on simplicity to avoid deviating out of the cone of strategy along the way to an end goal.”
Posted on 2025-01-17T02:15:24+0000
“The Coding Machine” at Meta with Michael Novati
In today’s episode, I’m joined by Michael Novati, Co-founder and CTO of Formation. Michael spent eight years at Meta, where he was recognized as the top code committer company-wide for several years.
Click to view the original at newsletter.pragmaticengineer.com
Hasnain says:
So much great advice here. I haven’t listened to the whole thing but I did read the full summary. It’s great.
“A consequence of the interview process not changing much, but the job market becoming more competitive is how the bar to do well on these interviews went up. This is because there are more and better preparation materials, so the “average” candidate does better on these interviews than years before. Preparing for interviews at Big Tech companies and scaleups is no longer a “nice to have:” it’s a necessity for even strong engineers, who want to get a job offer.”
Posted on 2025-01-16T03:22:26+0000