Clarence "Sparr" Risher

TL;DR: Valve could save the world about two petabytes of storage just by deduplicating the contents of their localization/translation files, and it gets worse from there.

I've recently become annoyed as I realize just how bloated many modern games have become. This was a moderate problem 5-10 years ago, but it has gotten completely out of control since then. As I was making a short list of games I've recently considered installing but skipped due to their size (4GB for a top-down low-poly low-texture platformer? 30GB for a RTS? 83GB for a MMORPG?) I found myself installing a 117MB update for Steam itself. In the software-bloat mindset, this prompted an immediate "wait, what?", leading to the writing of this post.

First, I headed over to `/Applications/Steam.app` (yes, I am on a Mac, but [almost?] everything below is platform-agnostic except for the paths). Here we have a total of 4.6MB, comprised of 3.3MB for the binary, 1.1MB for a crash reporting library, and 0.3MB for translations. I'm not going deeper into the binary or library; let's say those are reasonably sized and necessary/useful. That brings us to the translations. Translations are great! I'm definitely not saying they shouldn't be there, or that they should be less thorough. However, I'm going to be specific about this problem, because it becomes more important later. The English strings appear number_of_languages times; Do we really need 26 copies of every string to be translated, including:

%appname% has changed where it stores game content from '~/Documents/Steam Content' to '~/Library/Application Support/Steam/SteamApps'. You have game files in the original location, and Steam was not able to move the files because files already exist at the new location. You may need to move the files manually, or delete the old files and download your games again. Continue anyway?

The labels for the strings appear number_of_languages*2-1 times, making 51 copies of "SteamBootstrapper_InsufficientDiskSpace". Finally, "[english]" appears a whopping 900 times, once for every string in every non-english language. A simple .tar.gz of this folder cuts the size by 92%. Assuming there's some good reason the files aren't compressed, the content could still be deduplicated and the necessary duplication handled in whatever code loads them, which might be an 60% reduction in size. I am hopeful that all of this was compressed when downloading, but that doesn't solve the problem of it wasting space on disk, which gets more important later.

Next, I went to `~/Library/Application Support/Steam/`. I realized there's a lot of user-specific stuff in there, so I created a new user to launch Steam once and get a clean slate. 202MB of downloads leading to 757MB of disk space used. I will mention here that installing apps and games for just one user, and requiring the installation effort and disk space to be duplicated for a second user, is a terrible paradigm when it's not explicit and intentional, but that's a different rant for a different post. Let's dive into that 757MB, and I'll skip any parts that I have no objections to.

168MB for `Steam/Steam.AppBundle/Steam/Contents/MacOS/package/` which contains a bunch of archives that *seem* to match up to extracted content in the other directories. Does `package/tenfoot_fonts_all.zip.vz.SOMEHASH` contain a copy of `tenfoot/resource/fonts/*`? If the Steam client can use these files to repair itself in case of damaged or lost files, that's great. If it can't, then why are they still there?

147MB for Chromium Embedded Framework. I can't blame Valve here; the fault for not sharing libraries like this with other applications and keeping them in a central place falls on the shoulders of Apple and Microsoft and the environments they have developed. I am potentially giving Google or Apple a bit of side eye here for 14MB of translations, 200-500kB per language, depending on who is to blame for the contents of the *.lproj/locale.pak files, and how much space is wasted in there.

137MB for the tenfoot (Big Picture) interface:

63MB for `tenfoot/images/` on which I just deleted 500 words of detail here upon realizing I hadn't covered even 10% of it. TGAs that should be PNGs, PNGs that should be JPEGs, including some that appear to already have JPEG artifacts, PNGs that should be vector graphics, and so on.

31MB for `tenfoot/localization/` which has the same data duplication problems as from Steam.app, writ large. 177856 copies of "[english]" and 26 copies of:

Intended to be used with a dual-stage trigger setup, Hip Fire allows a quick pull of the trigger to engage the click without engaging the threshold. A slower pull or hover will engage the threshold action. This allows for actions such as iron-sights to be set on the threshold and fire on the click, while still allowing a quick pull to click to fire without entering iron-sights. Additionally, once the click has been hit, the threshold won't be engaged until the trigger has been released outside of the threshold range, allowing it to be primed for additional clicks. Relaxed Hip Fire mode is a larger window before engaging the threshold, allowing a slower pull of the trigger to avoid it. This means the Threshold action will be slightly less responsive when intentionally trying to engage.

11MB for `tenfoot/sounds/`, most of which is ambient background MP3, but also a bunch of WAV sound effects despite the evidenced support for MP3s.

That's all for Big Picture, back to top level Steam contents.

36MB for translations in `public/`, again with all the duplication in the previously mentioned translation/localization files.

20MB for 'graphics/' which is full of TGA files that should be PNGs and/or procedurally generated, most notably the 4.2MB music_details_mask.tga that is just a checkerboard on top of a radial gradient, and clienttexture*.tga which are just gradients on a flat background.

The rest of the smaller directories contain many more examples of poor image format selection, duplicated translated strings, etc, which I won't further enumerate here. If Valve ever decides to address the problems above, they will probably solve the smaller instances of the same problems as a side effect.

Having gotten this far, my original "wait, what?" actually still remains unanswered. Almost everything I've just described seems mostly static, images and sounds and text files and third party libraries. The binaries and first party libraries are actually pretty small. I can't point to 117MB worth of content that I expect to have changed in the update today, let alone the ~300MB that I would expect from the compression ratio evident in the original client download. http://store.steampowered.com/news/38412/ has the changelog, where I see a lot of what should be code/library/driver changes, none of which touches on all the pieces of the client that seem to take up tens to hundreds of MB of space each. Maybe I'll find some insight into the update size. Maybe I'll go do this same teardown on those three games, or at least two seeing as how I don't even have 83GB of free disk space right now.

Summary

This was my first net-negative experience at a burn or con. The things I learned will have positive value in the future, so eventually I can look back and say this weekend was worth it, but I don't know how long that will take. I've finished burning a lot of bridges and made some new friends. Firefly is still the best regional burn I've been to, despite its flaws, and I'll probably return if they let me.

The Good

I got to spend some part of a couple of days camping with some of the people whose company I enjoy most. This is a welcome reprieve from what has been mostly solitude on the west coast. There were friendly conversations as well as good hugs and cuddles. Due to some things mentioned in sections below, there's a good chance I'll get to spend more time with those people fwen I return to Firefly.

( Read more... )

LJ is the best social network. I cannot count the number of times I have seen (or made) a complaint about Facebook or G+ or Ello or Fetlife that could not be countered with "LJ does that better".

What they don't have is the best interface, or the best marketing team.

I would love to wake up tomorrow and learn about a "new" social network launching, which is really just a new domain name and front end for LJ. A modern web interface (AJAXy, dynamic, etc). A more convenient interface. A new marketing campaign, and a "new" thing to tell my friends about.

Dreamwidth isn't much better. None of the LJ clones really match what people want out of the user experience of a social network. But the back end is SO MUCH BETTER. One-way friends, per-user and per-tag subscriptions and notifications, threaded comments, markup in posts...

Some day, maybe.

Ello, if you're listening, just clone LJ.

It's one of the strongest insults in my vocabulary, but also one of the least precise in common parlance. It means "You have intentionally led me to believe, directly or indirectly, that something untrue is true". This meaning is often lost, or misinterpreted. I often reserve it as an insult for someone who uses their lie for their personal benefit and potential or actual detriment to myself or people I care about, but that selectivity doesn't change the meaning. The word itself conveys much more impact than phrasing the same accusation differently[1].

( Read more... )


If you don't want to talk to me, tell me that you don't want to talk to me, don't tell me your dog is on fire. If you don't want me to talk about a certain subject to you, or with other people, tell me that, don't lie to me in an attempt to convince me that talking about it isn't worthwhile. Lying to me is almost never going to have the desired result, and the more effectively I can convey that the more likely we are to have successful communicative interactions.

Today I encountered a community in which I cannot post under my LJ name because I am not a member of the community... but I can post with my Facebook account, or any OpenID account. WTF?

Four days ago I joined the modern era by buying a smartphone. I was an early adopter in the PDA market, with a Palm in the 90s, then again with an HP iPaq and Asus MyPal in the early 00s, but at some point in the last three years the revolution happened. I can thank Apple, however grudgingly, for that. Without the iPhone we would not have Android, and without either of them we would not have the thriving and growing ecosystem of mobile applications today that dwarfs the market for RIM/Palm/Symbian/Zaurus apps in the past. This post chronicles my joys and woes in the first half-week of use.

( Read more... )

Overall I am greatly enjoying the experience, despite the hurdles, most of which were at least somewhat expected. I think that having an internet connected phone will make my life more productive and efficient, and only slightly more distracted. Having one device to take the place of a media player, PDA, phone, and pocket camera[3] is great, but everyone who has one already knew that. I'm looking forward to starting to develop apps. I have some great ideas, and my Java skills are decent but rusty, so maybe I can even make a little money.

( footnotes )

Let's talk about tipping food service wait staff professionals. These are people you would typically call "waiter" or "waitress". I am going to systematically address every argument I have heard in favor of commonly advocated American tipping practices.

Before I start, I need to tell you that I give my waitress a larger tip than her average, on average. I tip more than most people in my party, most of the time. I am not a cheapskate. *I* am not the problem.

( Read more... )
[1] For the sake of this post, I am discussing the typical range of restaurant prices encountered by my friends (particularly of the LJ variety) on a regular basis. "cheap" is somewhere around $4 entrees and $6 meals, at places like Waffle House or Denny's. "expensive" is $20 entrees and $30+ meals, at places like Maggiano's and Ruth's Chris.
[2] Georgia has, afaik, the lowest state minimum wage in the country. It is one of a handful of states that do not improve on the federal laws in this regard. Almost every server in the country is earning more than $2.13 as their base wage.

You are shopping for a new car. The latest model from [manufacturer] has a curious "security" feature wherein the car must contact headquarters by radio before it can be started. You decide to buy it, because the other features are enticing. Six months later, you are unable to start the car due to [radio interference / headquarters being bombed / manufacturer going out of business / other] and this problem lasts a few hours, a few days, or forever. Later, you are shopping for a new car again, and you see the same feature on some of the newer options again... and you buy one of them. Why? Did you not learn your lesson? Maybe the story above happened to a friend and not yourself? Maybe the other features of the car make it worth the risk?

I put this story forward as an understandable analogy (thanks Jim!) to explain what I see as really dumb moves on the part of consumers when it comes time to buy digital media or software. This isn't purely hypothetical or a thought experiment. Over and over again we see companies turning off their DRM servers on purpose, or having them fail on accident. These aren't little companies you've never heard of, or companies you would otherwise distrust, they are companies like Google, Microsoft, Yahoo, Virgin, Major League Baseball, and now Ubisoft. Millions of honest customers have been screwed by DRM of all sorts, more every time this happens. How big of a problem does this have to become before consumers will demand better?

I don't like other people telling me when I have to stop using the things that I have bought and paid for (aside from illegal uses, that's another topic). This is why I won't be buying any new games from Ubisoft. Also why I won't buy games on Steam. It is why I won't buy a BluRay player (key revocation, anyone?), any game whose multiplayer component relies on nontrivial publisher-provided server hardware (sucks to be a Halo 2 fan right about now), or, on a note more related to the original analogy, any car that OnStar can kill remotely.

Please help me understand why you don't reach the same conclusions.

Two experiences this week with overpopular restaurants have driven me to write this post.

The Vortex (L5P) is often packed, with hour+ waiting lines out the door in the cold. They have another location about 4 miles away that is often sparsely populated at the same times. Here's a crazy idea... Offer a % off their meal, or free parking, or something of similar value, for a party of 8+ to go to the other location. You are going to lose their business when they find out there's a 2 hour wait, why not salvage it?

The Majestic is also often packed, with faster moving lines out the door also in the cold. Tonight I arrived around 3AM to find a Fulton County Sheriff manning the door. I walked in, alone, to take a seat at the counter (which had 3-4 empty seats), and he told me to get to the back of the line. The line full of parties of 2-6. WTF? Majestic, you just lost a customer, probably for a substantial number of potential future visits, until I'm desperate enough to come back.

Six guys sit around a table playing russian roulette, with the gun pointed at the guy next to them.  One of them inevitably ends up shooting another.  Has that guy committed a greater crime than the other 5?  The intent and risk was identical every time.  They all knew the chances that someone would die when they pulled the trigger, they all chose to take that risk.

Two people buy the same car.  Bob just starts driving his.  Joe does thousands of dollars in aesthetic customization first.  They get in identical wrecks, being hit by Sue and Amy (who are completely at fault).  The chance of your car being hit is predictable, just like getting struck by lightning or having a tree fall on it.  If Sue and Amy had stayed home that day, the chance of Bob and Joe's cars being hit would not have changed in any significant way.  Joe took a greater risk than Bob by putting a more expensive car on the road, but our society puts the burden of Joe's risk on Amy, who took the exact same risk as Sue.  Why is that?  Why does Joe not bear any liability for his riskier behavior?

Somewhere out there is Sam.  He has that same car again, and spends even more than Joe making it prettier.  But Sam recognizes that he is taking a greater risk by putting a more expensive car on the road, and he does the responsible thing and gets comprehensive insurance.  Why is Sam not the norm?

In almost every part of life, sensible people can come to a consensus that with greater risk comes greater liability and responsibility.  In this particular situation, and a few other similar ones, that consensus is shattered.  Why?