placeholder

Reducing Logging Cost by Two Orders of Magnitude using CLP

Long, long ago, the amount of data our systems output to logs was small enough that we were able to retain all of the log files. This allowed our engineers to freely analyze the logs, say for troubleshooting our systems or improving applications. But as Uber's business grew rapidly, the amount of da...

Click to view the original at uber.com

Hasnain says:

Now this is some really impressive work, taking costs from $1.8M/yr to $10k/yr for log storage. I liked how it was an iterative process, massaging and moving around data till it can be compressed much better. Reminds me of some work we did back in the day to split up data a little for better compression. The wins are huge!

“We have deployed Phase 1 (i.e., the custom Log4j appender with our custom float encoding) across our entire Spark platform. We are currently working on deploying the Phase 2 compression and integrating CLP’s search capability into our analytics and observability platforms.

Result of Phase 1 compression: In a 30-day window, our entire Spark ecosystem generated 5.38PB of uncompressed INFO level unstructured logs yet our CLP appender compressed them to only 31.4TB, amounting to an unprecedented 169x compression ratio. Now with CLP, we have restored our log verbosity from WARN back to INFO, and we can afford to retain all the logs for 1 month (as requested by our engineers).

Preliminary result of Phase 2 compression: The above mentioned result is only the size of the compressed IR. We have tested a prototype of CLP’s complete compression (including both Phase 1 and 2) on a subset of our Spark logs, and CLP’s compression ratio is 2.16x higher than Zstandard’s ratio and 2.28x higher than Gzip’s ratio. This is consistent with the results reported on other log datasets. “

Posted on 2022-10-01T16:19:03+0000

placeholder

Outdated vs. Complete

On August 22nd, I got an email out of the blue from Apple that notified me that I had a new App Review message. It was for my app, WorldAnimals, a light-hearted game for guessing animal onomatopoeia sounds in different languages.Usually, you receive a message after you submit a new version to the...

Click to view the original at vivqu.com

Hasnain says:

"Day-by-day, month-by-month, the App Store will get a little less rich and vibrant as apps start being designated as outdated and get removed. Another consequence of this hostile policy is that indie and hobbyist developers may stop building mobile apps. After all, the web is fundamentally a more stable place for experimental software and “finished artworks”, since backwards-compatibility is the gold standard and apps can run indefinitely.

After 4 hours of work to re-compile my app and 44 hours waiting in the review queue, WorldAnimals is now updated to a new version. I am safe for at least another three years before getting automatically flagged for removal. Unless, that is, Apple decides there is a new threshold for “outdated” and change their policy once again."

Posted on 2022-09-27T04:31:40+0000

placeholder

Downturn career decisions.

When I joined Yahoo In 2008, I received a small number of options. I don’t remember how many–it was very few–but I do know my strike price was roughly $16. I don’t remember that because my strike price was particularly lucrative, but rather because some of my coworkers would complain about t...

Click to view the original at lethain.com

Hasnain says:

"I’ll end with some advice to remember next year as the downturn ends and the upswing begins. The personal freedom to ignore downturns comes from financial stability, and the best path to financial stability is taking money off the table whenever you can until you’ve reached financial independence. A lot of financial advice out there is written from the perspective of very wealthy folks. If you’re already wealthy, your goal is to maximize the risk-adjusted expected return of endeavors, often by taking meaningful risk. For example, if you’re wealthy, it’s almost always the right decision to early exercise your equity. If you have millions of dollars, then it’s reasonable to risk $100k now for the potential of millions in reduced tax in six years. That’s not necessarily true when you’re not already wealthy."

Posted on 2022-09-26T03:34:23+0000

placeholder

25 Sep 2022 McKinsey and Providence colluded to force poor patients into destitution – Pluralistic: Daily links from Cory Doctorow

No trackers, no ads. Black type, white background. Privacy policy: we don't collect or retain any data at all ever period.

Click to view the original at pluralistic.net

Hasnain says:

“In light of all this, it's only natural that Providence would turn to McKinsey when they needed help committing crimes and destroying thousands of people's lives. McKinsey helped Providence craft a program to coerce poor people into paying for care they were entitled to get for free. They called it "Rev-Up."

Writing in the New York Times, Jessica Silver-Greenberg and Katie Thomas reveal the full depravity of "Rev-Up." McKinsey advised Providence to train its staff to avoid truthfully answer poor patients' queries about whether they were eligible for free care.”

Posted on 2022-09-25T19:44:43+0000

placeholder

Addition vs Subtraction

“Organizations are horrible at subtraction.” Someone said that to me a couple months ago, and I haven’t been able to stop thinking about it since. It is so true. Organizations are so much better at adding things than they are at taking things away. We’re better at setting goals and talking a...

Click to view the original at mollyg.substack.com

Hasnain says:

“It is so true. Organizations are so much better at adding things than they are at taking things away. We’re better at setting goals and talking about what we’re going to do than we are at talking about what we’re NOT going to do. It's easier to add process than it is to ask why we're still doing that thing that worked great two years ago but mostly isn't relevant anymore. We’re better at adding meetings than we are at removing them. “

Posted on 2022-09-25T18:50:12+0000

placeholder

Buckle up, America: The Fed plans to sharply boost unemployment

Fed Chief Jerome Powell says curbing inflation will cause pain. Critics say putting millions out of work is unnecessary.

Click to view the original at cbsnews.com

Hasnain says:

This just seems straight up evil.

“Here's the idea behind why boosting the nation's unemployment could cool inflation. With an additional million or two people out of work, the newly unemployed and their families would sharply cut back on spending, while for most people who are still working, wage growth would flatline. When companies assume their labor costs are unlikely to rise, the theory goes, they will stop hiking prices. That, in turn, slows the growth in prices.

But some economists question whether crushing the job market is necessary to bring inflation to heel.

"The Fed clearly wants the labor market to weaken quite sharply. What's not clear to us is why," Ian Shepherdson, chief economist at Pantheon Macroeconomics, said in a report. He predicted that inflation is set to "plunge" next year as supply chains normalize.”

Posted on 2022-09-25T00:55:40+0000

placeholder

Friday Facts #370 - The journey to Nintendo Switch | Factorio

We have a long history of trying to bring Factorio to other platforms, including consoles and mobile phones (not including April Fools). We even worked with some external companies, but the projects never even got to the point where they would run technically, let alone the complicated part of makin...

Click to view the original at factorio.com

Hasnain says:

Ah, UB, the bane of every C++ programmer’s existence.

“With that out of the way, the next step was multiplayer determinism. One big goal was that I didn't want to cut multiplayer from the game. Furthermore, I wanted players on PC to play with players on Nintendo Switch. This is the first time we had to make sure the game is deterministic between ARM and x86. We should be fine, C++ is portable, right? Just don't use undefined behaviour. Turns out we use quite a lot of undefined behaviour, both in our main code and in the libraries. For example, when casting a double to an integer, if the value does not fit in the integer, it is considered undefined behaviour and the resulted value is different on ARM and x86 CPUs.”

Posted on 2022-09-24T04:56:07+0000

placeholder

The Hierarchy Is Bullshit (And Bad For Business)

My friend Molly has had an impressive career. She got a job as a software engineer after graduating from college, and after kicking ass for a year or so she was offered a promotion to management &#…

Click to view the original at charity.wtf

Hasnain says:

Great read as always on career paths.

“At some point you have to learn to tune in to your own inner compass. What draws you in to your work? What fuels your growth and success?

Being an adult means not measuring yourself entirely on other people’s definition of success. Personal growth might come in the guise of a big promotion, but it also might look like a new job, a different role, a swing to management or back, becoming well-known as a subject matter expert, mentoring others, running an affinity group, picking up new skill sets, starting a company, trying your hand at consulting, speaking at conferences, taking a sabbatical, having a family, working part time, etc. No one gets to define that but you.

You have a thirty- or forty-year adult life and career in front of you. What the hell are you going to do with all that time and space??”

Posted on 2022-09-24T01:55:59+0000

placeholder

The Most Famous Blunder Of Content Moderation: Do NOT Quote The Princess Bride

We’ve written stories about people having difficulty recognizing people joking around quoting movies. Sometimes it ends up ridiculously, like the guy who was arrested for quoting Fight Club a…

Click to view the original at techdirt.com

Hasnain says:

“But, in context, it’s quite clear that this is a joke, a quote from a funny movie.

The issue is that so much of content moderation involves context. This is something that critics of content moderation (both those who want more and those who want less) never seem to fully grasp. How does a content moderator (whether AI or human) have enough context to handle all sorts of issues like this? Do you need to train your AI on classic movies? Do you need to make sure that everyone you hire has seen every popular movie and knows them by heart and can recognize when someone is quoting them?

How do you deal with a situation where someone tries to hide behind the quote — but is actually threatening someone? (Not what Kel did here, but just noting, you can’t just say “okay, leave this line if it’s quoting a movie”).

The point is that it’s ridiculously complicated.”

Posted on 2022-09-24T00:21:24+0000

placeholder

Hasnain says:

This was a great read - that matched many of my management experiences as well.

“In my experience, the TLM role is **challenging but extremely rewarding**. You can be an Engineering Manager while still keeping your feet on the ground and solving challenging technical problems together with your team.

I’ve found that **people trusted me more to make the right decisions**. I knew how the technology stack _actually worked_ because I built a large part of it. It’s frustrating and demoralizing to report to a pure people manager who is unable to understand what people are building, and which problems they are facing, regardless of how much time you spend explaining it to them. Stepping into the TLM role is a good way to counter that, and reporting to a TLM is often deeply rewarding because they tend to care about people and technology in the right way to get amazing things done.”

Posted on 2022-09-23T00:22:14+0000