2014-04-23

Hyper-Optimization of PDF Scans


Mind: BLOWN.

The other night Robert threw a bunch of PDFs my way, scans from a Xerox office copier that needed to be converted to JPEG. Child's play for ImageMagick, even with dozens of files to convert. So, off we go...

The only sticking point was figuring out what DPI to rasterize them at. The nominal resolution of the PDFs appeared to be impossibly low. With default settings, ImageMagick produced a near-screen-res image that lost a ton of detail, detail I could see in the PDF. Plus, the resulting JPEG file sizes were already larger than the (tiny) PDFs. Confusing, since you'd expect a PDF of a scan to contain nothing but a bitmap; the converted JPEG file should be within a few K of the same size. But at a resolution that visually matched the PDF's detail level (300dpi), the output file was almost 10x the size of the input! Something weird was going on.

I decided to pull one of the PDFs into Inkscape (think: open-source Adobe Illustrator), to see if I could at least find the native resolution of the embedded bitmap (the one I thought should be the only thing in the file). That's when things started to get weird.

Turned out, the file didn't contain a single embedded bitmap image object. It contained NINETEEN distinct component objects. The base image was that expected embedded bitmap, which was indeed stored at the near-screen-resolution nominal dimensions. Yet, as I'd proved to myself by zooming in on the PDF, there was far too much detail for that resolution!

The trick was in those other 18 objects. I don't know if this was Xerox's code, or the latest Adobe PDF creator optimizations, but whatever the source... the image had been deconstructed, and any areas that required extra detail — mostly, the white lettering printed against the dark background art — had been stored in masked layers placed on top of the low-res image. Those layers are all much higher resolution than the background, but each one contains only a single color so they're stored as one-bit masks, requiring a few kilobytes at most to store.

Keep in mind, this optimization was all done 100% automatically by the copier's scan-to-PDF function! So, upshot is that even though the file is tiny in size, it contains all the detail of a much higher-res image. And any conversion to JPEG or other fixed-resolution format would necessarily result in a much larger file than the PDF. I ended up pegging the rasterization at 150 dpi and output 85%-quality JPEGs, which tended to be roughly 2-3x the filesize of the source.

Craziness.

The first image here is one of the output files. The second I created in inkscape, showing the structure of the input PDF — each box is a non-background layer, green for white text and blue for detail sections in the background art, and the background image is dimmed so the detail elements stand out more clearly.
The scanned image
Structure of the scanned PDF

Detail
Structure detail

2012-12-25

The Post-Faithful Human

This was written in response to a Dec. 24 New York Times op-ed piece by Rabbi Jonathan Sacks, entitled The Moral Animal. It's not entirely clear to me why the Times closed it to comments only a day after it was posted, and with only 105 responses visible, but I'll soldier on here.

The day before Christmas is probably the perfect time to publish an stirring piece on religion's value to society now and going forward, and Christmas Day itself not at all the right time to dispute that claim. Still, here we are. The rabbi walks us through the basis for religion's importance in a clear enough manner, and I even agree with his assessment — religion has helped us survive, has been a catalyst for bonding and development of altruism, has strengthened the herd as we've learned to think beyond our individual selves and to consider the good of the community. There's really no debating the role of religion, in shaping human society.

But it seems to me he overreaches, with the argument that "we will need [religion] in the future". With that statement, he makes two great leaps which, taken together, certainly support his claim... but neither is a foregone conclusion. So, it's certainly possible that religion will be important in future society — but it's also possible, in the absence of those two assumptions, that its role will diminish.

The first leap is intrinsic in his statement that, "the Neo-Darwinists have helped us understand why [religion] matters". Really, that should have read mattered. As I said, I agree with him on the history and the analysis. But it's an assumption on his part that, because religion "remains the most powerful community builder the world has known", that it will always be so. It's true that religion, in the past, has acted as a powerful catalyst for society, but it is not the only one. People need something to share, something to bring them together — that much is true. But societies and social groups are emerging centered around things other than faith.

Take as one example the gay community, a tightly-knit society fiercely protective of its members, and pretty much equally accepting of all faiths (and none). Now, it's entirely possible that, as homosexuality is increasingly accepted in mainstream society, the strength of those communal bonds may be weakened. The "gay community" as an entity may become more diluted, or less central to its members' lives. Indeed, such a shift is probably inevitable. But the same shift is occurring in religious belief. Even among many of the faithful, religion is less central to their lives today, and plays less of a role in informing how they live. Religion will increasingly have to compete with other, secular communities for influence over its members, and those other communities will increasingly play the same role as religion in building and strengthening communities. Over time, religion may become less necessary for that process to occur.

The second great leap the rabbi makes, is the assumption that religion itself remains unchanged — that its influence is as positive today as it has been in the past. He overlooks what seems clear to me, which is that religion is corrupting itself! If not religion as a whole, then some religious groups — particularly within Christianity. Now, I can understand how a Jewish leader living in London may not be very attuned to this. Judaism isn't typically known for its proselytizing or religious activism, let alone political involvement. (Western Judaism, anyway — I can't, and won't, display my ignorance by attempting to discuss Jerusalem.) So to him, religion has the same helpful, strengthening influence it's always had. But this is not necessarily the case for all religion, and it won't necessarily remain the case over time. Religion's value to society is measured in the balance of its positive and negative influences, and that balance can shift with changes in religion, or in society. I believe we are currently in the throes of one such change.

As I mentioned, like never before religion is in competition with other forces for influence over its followers. And in America, many Christian leaders leaders have reacted to this by "stepping up their game", making increasingly aggressive moves to retain faith's position of prominence in people's lives. Some members of the clergy, discovering the power of the media and politics in shaping society, even attempt to impose their views on society as a whole — Christian or otherwise — in an effort to emphasize the importance of faith as they see it. Thus, groups like the Westboro Baptist Church take horrible and painfully destructive action in their god's name. Prominent figures like Jerry Fallwell, Pat Robertson, Bob Vander Plaats, New York's Cardinals O'Connor and Dolan, and countless others take active roles in social and political discussions of all nature, speaking to the broader community and asserting what they view as the positions dictated by their faith. Meanwhile, some secular politicians increasingly hold up their faith as the source of their views, and take positions on secular matters affecting members of all faiths which are informed in whole or in part by that politician's religious beliefs.

To the extent this activism, this politicization of faith, is attuned to society as a whole, its voice would be welcome. But where religious doctrine becomes increasingly out-of-step with the views of broader society, among both members of the faith and non-believers, the church's struggle to retain influence grows ever more caustic and damaging. If the church resists the natural evolution of society, and continues struggling to impose a set of rules and beliefs so firmly rooted in the past, it may very well find itself left there, while humanity moves on to a future without it.

Posted via email from ferdnyc's posterous

2011-06-22

Community Participation, Done Right (Lawyers Too!)

The Fedora Project Contributors' Agreement (sometimes/formerly "CLA", the L is License/ing) must be signed by everyone who registers with the Fedora Account System and receives credentials allowing them to contribute to the project. (That includes software packaging, wiki editing, code checkins, and so on.) It sets the tone both for members' activities within the community, and for that community's relationship with its leaders.

For an open-source, community-driven project's organizational body to be respectful of that community, it mustn't put its own interests ahead of theirs. In fact, it shouldn't even have a concept of "its own interests" distinct from theirs; its only interests are the community's. Any goals that organization has should be narrowly defined, and any rights or authority it asserts should, when faced with conflict or overlap, be deferred. That's the model of a responsible governing body, one that wishes only to consolidate and guide the efforts of its contributors for the benefit of all involved, not appropriate those contributions for its own benefit or profit.

That's why, for Fedora contributors agreeing to & signing the CLA, almost more important than what that does mean is what the agreement doesn't mean. And why, in recognition of that fact, the text of the CLA opens with explicit statements of both points:

The Fedora Project Contributor Agreement

http://fedoraproject.org/wiki/Legal:Fedora_Project_Contributor_Agreement

Goal

We require that contributors to Fedora (as defined below) agree to this Fedora Project Contributor Agreement (FPCA) to ensure that contributions to Fedora have acceptable licensing terms.

Non-Goals

The FPCA is *not* a copyright assignment agreement.

The FPCA does *not* somehow supersede the existing licensing terms that apply to Fedora contributions. There are two important subpoints here. First, the FPCA does not apply to upstream code (or other material) that you didn't write; indeed, it would be preposterous for it to attempt to do so. Note the narrow way in which we have defined capital-c "Contribution". Second, the main provision of the FPCA specifies that a default license will apply to code that you wrote, but only to the extent that you have not bothered to put an explicit license on it. Therefore, the FPCA is *not* some sort of special permissive license granted to any party, despite the explicit choice of a more restrictive license by you or by upstream developers.

...Commendable. And why FESCo (the Fedora Engineering Steering Committee), the Fedora Project Board, and anyone else involved should be commended. In this time when free websites have Terms of Use claiming exclusive ownership of everything their users do on the site, this type of dedication to putting the community first — even in their lawyers' fine print — is all too rare.

Posted via email from ferdnyc's posterous

2011-06-11

Samsung Transform phone Review: Awesome keyboard, substandard everything-else

[Since I bothered to bang out this review of my Android phone over on Samsung's site, and I have no idea if it'll ever see the light of day there, I thought I'd mirror it here, with additional details that are apparently forbidden in Samsung-site reviews. Like prices, or mentioning any competitor products or sites.]

 ...OK, to be honest the screen's not half bad, either.

I was dead-set on replacing my previous Instinct with a phone that had a REAL physical keyboard, and the Transform's is possibly the best keyboard Samsung has ever designed. Great feel and no weird layout "stuff". (Looking at you, Moment keyboard!) But, obviously, the main selling point for the Transform is its price. An Android phone for $50? Sign me up!

Problem is, you really do get what you pay for when you buy a "cheap" Android phone. The Transform's biggest failing is simple: It. Is. SLOOOOOOW. Criminally slow. Occasionally-goes-out-for-a-walk-and-just-gets-lost slow. "Background services? Sure, I can do that... *OR* I can let you answer a call" slow.

Shortly after buying the phone, I ran some random benchmark app (all-but-useless for practical purposes, but still better than comparing clock speeds or other specs for relative comparisons) on my Transform. On whatever arbitrary scale it used, my phone's score was ONE TENTH the rating of the HTC Evo, one of Sprint's then-flagship Android phones. The Transform really did score just 10% of the Evo's benchmark result.

Now, that kind of phone is a serious mobile device with a zippy CPU; you'd expect the Transform to be slower. But TEN TIMES as slow? Which is not to say that every operation takes 10x as long on the Transform. But it does mean that it has to "ThinkRealHard!" for up to ten times as long, for each of those operations. It means that when even a few stack up, severe delays build up fast. (The ONLY thing the phone does quickly: get bogged down!)

It means that, if you're unfortunate enough to be doing anything with the phone at the moment the Alarm Clock triggers, causing the alarm popup to get lost, you will be forced to endure several seconds of loud beeping/buzzing/etc, all throughout the time it takes you to get to and then pull down the notiifcation tray, tap on the alarm notification, wait for the dialog to come up, and then select and have it process your response. (Nothing else will silence it short of pulling the battery — a better approach for particularly embarrassing situations. Of course, Google has to share some of the blame here, for an OS that violates the *one* well-established mobile convention: Whenever the phone is emitting ANY audible noise, pressing Volume Down should ALWAYS silence it immediately, without exception, no backsies.)

Because the CPU really only has two operating states — idle or near-catatonically overloaded — its power management has little practical use. Thus, battery life is... unstellar. (After learning a few new habits to save on power, though, I've found it pretty manageable.)

Lastly, factor in Samsung's unsurprising lack of enthusiasm for actively supporting a device that can't possibly represent any significant profit for them. As evidenced by the months of delays before the 2.2 update was released (though that issue affected their entire lineup), the fact that the 2.2 update that *was* finally released is actually worse in several ways (unless Samsung can explain how adding a reverse-landscape rotation orientation to a side-slider phone is somehow a feature!), and the fact that in the time since the 2.2 update finally appeared, no further update has been released or even hinted at to fix the issues with the current build.

The 2.2 update did shave several seconds off the phone's boot time — a welcome change, but still far from a gold-star feature for the phone. With the official Froyo build, time to cold-boot my Transform after an orderly shutdown, from first touch of the power button to the moment the lock screen appears: 2 minutes and 2.65 seconds. ...Yay.

Engadget and ilk have made excellent arguments against "budget" Android phones like the Transform, usually pointing out that "last year's model" of a REAL phone can be had for similar price, and will likely still be superior. I can only add that, even as a relatively-content 6-month Transform owner, I agree. The phone is "fine" for what it is, and I knew what I was getting into, but if I had it to do over again I'd look for a better solution than the Transform.

But I'd really miss that keyboard! 

Pros: Bright display, excellent keyboard 
Cons: Poor quality, slow, poor battery life

Final Verdict? No, I would not recommend it to a friend.

(...Possibly an enemy.)

Posted via email from ferdnyc's posterous

2011-01-12

For Gary, a reminder of conversations past...

Frank  WTF? NOT _AGAIN_! This happened once before on my account, and disappeared almost immediately -- the top of my feed has been replaced with a Share: box containing four links, and I have to click one of them to get an input box. It is _AWFUL_ and Assbook is *ON*CRACK* if they think this is a good idea!!
December 10, 2010 at 3:49 am
Julie  Mine has been like that since I switched tov the new profile 5 days ago. It is like that when I go to other people's walls too.
December 10, 2010 at 3:59 am
Frank  How is nobody else complaining about this, then????
...And why are you _awake_? :)
December 10, 2010 at 4:09 am
Gary  It's the new software they rolled out a few days ago. It is the new and permanent change.
December 10, 2010 at 4:14 am
Frank  Not if there's a backlash to end all backlashes.
December 10, 2010 at 4:15 am
Frank  There's no such thing as "permanent" in software, WTF does that even _MEAN_???
December 10, 2010 at 4:16 am
Gary  It won't revert back to the earlier version.
December 10, 2010 at 4:16 am
Frank  ... ... ...You work at Facebook, now? Congrats on the new job!
December 10, 2010 at 4:36 am
Gary  Thanks! I'll make sure your complaint gets routed to the appropriate individual. We are now serving number 37...your number is 354,726,586,324,591.45. Feel free to have a seat and wait.
December 10, 2010 at 4:38 am
Sean  Yea I've had that box since the new profiles were rolled out. I find the new design much more clunky but oh well they don't care.
December 10, 2010 at 8:41 am
Julie  I *was* awake giving the baby the boob!
December 10, 2010 at 10:24 am
Frank  MY BRAIN-EYES!!!!! *spaf!*
December 10, 2010 at 1:03 pm

2010-12-26

The Physical Keyboard on my Samsung Transform

Here's a look at the Samsung Transform (SPH-M920) in physical input mode. The phone, which I upgraded to last week, is an Android 2.1 (so far) device carried by Sprint since October 2010. The original image was lifted from Phandroid's Android Forums (er... thanks!), but I've adjusted it to enhance the visibility of the orange labels. (They're still somewhat indistinct, I know — my apologies.)


Overall, I lucked out with this phone in terms of keyboard. The Transform has one of the nicest keyboards Samsung's ever produced... and in a time when they've been experimenting with a lot of WEIRD designs (see below). I was dreading the thought of getting a phone with a keyboard I'd have to "learn to live with", but instead the Transform has made me even more spoiled & snobbish than I already was.

Still, there are the inevitable few annoyances, made all the more glaring by the otherwise excellent nature of the keyboard. And one or two choices about its layout are simply braindead, and make you wonder what line of reasoning could possibly have led Samsung to them.

First off, let's leap right to the single most infuriating thing about the Transform: Why, in the name of all that is good and right in the universe, is comma not bound on the period key (Fn-period)??? If you can't make the label out, comma is Fn-N. Fn-period is wasted on an insipid and questionably-useful ".com" shortcut binding. I'm guessing an association on "dot" explains the location, but it sucks. If Samsung was hell-bent on a ".com" key, they should at least have gone conceptual ("Internet") and bound it to Fn-@.

For that matter, why do the @ key and the ? have no alternate mappings at all? (Well, they do, but Fn-@ is just @, and Fn-? is still ?) What, they completely ran out of symbols that seemed useful to include? The lack of Fn-@ is particularly infuriating, as it's the obvious location for ".com" and that symbol's actual placement is so shitty. But I'm sure they could have come up with something useful for Fn-?, too. For one thing, there's no tilde to be found anywhere.

There are also visual frustrations. Why can't someone come up with alt-symbol labeling that lets you clearly differentiate, even at a quick glance, between comma (Fn-N) and apostrophe (Fn-K)? Or between hyphen (Fn-X) and underscore (Fn-V), for that matter? Both of those pairs are easily confused, when scanning for a character while typing. Even now that I'm more familiar with the layout, I still sometimes screw it up.

I have a feeling the distinctions would be far clearer, although it would mean breaking their label rules, if Samsung had placed the comma and underscore labels low on the key – to the right of the main label, instead of above-right. Since they're orange, I think it would still be clear what that means. And lower placement for those baseline symbols would avoid them appearing "up in the air", where they look confusingly like their higher-placement counterparts.

An Aside: Samsung's "Creative" Keyboard Layouts

How weird have some of Sammy's recent keyboards been? Think "space bar in the middle of the ZXCV__BNM row"-level weird! (That's the Intercept, my phone's immediate predecessor.) There are other weirdnesses, and other models with different craziness. A lot of shit you just can't imagine anyone wanting.

It's reminiscent of Nokia in the early 2000's, when they inexplicably decided to reinvent the dial pad. Suddenly they were producing phones with circular button layouts (that's right, pushbuttons arranged like a rotary phone) and similar types of insanity. That went "well" for them, as I'm sure you can imagine, so Samsung apparently chose to emulate it. :-/  The excellent, very conservative keyboard on the Transform may be a sign that Samsung's fiddling was received about as well as Nokia's.

Posted via email from ferdnyc's posterous