The Listening Machines

By now you’ve heard the beckoning of tech companies to buy a device for your home, whereby you can calmly (or loudly) speak “hey Alexa-hey Google-hey Siri” into the air and a machine, listening for the right queue, will help you do something or provide something in return. It’s the far-fetched sci-fi dream of yore — controlling your appliances throughout the home, and asking an esoteric artificial intelligence machine to do your bidding. It’s the invisible interface. But is it the final frontier for computing? And what must we sacrifice and compromise to get there?

What Exactly is Going on in the Home?

A few years ago, both Google and Apple introduced home automation frameworks in an attempt to bind several disparate Internet of Things products from third-party manufacturers. Google’s Android@Home—nowait-Brillo-holdon-Android Things and Apple’s HomeKit play important roles in centralizing control for the myriad of hardware and products that are now, for reasons of convenience (?), Internet-connected (lights, switches, locks, cameras, fans, windows, etc.). These centralized controls are found in things like the Apple Home app on your iOS device or Apple TV, since you’d probably rather use just one app that dozens of individuals apps to control your dozens of Internet-connected products.

But what makes all this even easier? An invisible interface you simply talk to, that is always on, and always at the ready. And so here is where the Amazon Echo, Google Home, and upcoming Apple HomePod enter stage left. With the innocent looks of a speaker, these are beamforming, microphone-arrayed devices that can parse out human voice through the noise of running music in the background and can respond to a variety of inputs from the user. Sure, they’re limited to what they can do, but all of them will allow for pretty consistent behavior, namely:

  • Manipulation of music, playlists, etc.
    • Mostly done natively through each company’s maintained music platform (like Amazon Music), though other music platforms can be streamed as well
  • Answering basic questions, setting timers, and so forth
  • Controlling Internet-connected devices and accessories

Conveniency and ubiquity continue to be the name of the game here. Why place an always-on listening device in your home? Because it’s more convenient to say “hey Alexa, play xx album” out loud without thinking about anything but the words than opening your phone, opening the music app, conducting a search with the software keyboard, and then hitting play. If these voice assistants can become as intricately sophisticated as we need them to be, they certainly could be the future operating systems.

Is There a Danger of Overreach?

So should we be cautious about all this new tech? Probably.

First off, having nearly everything in your home connected to the Internet could be considered dangerous in its own right. Mr. Robot has a damning episode on home automation going haywire due to malevolent hackers. It could happen. Smart home accessories have already broken down, leaving owners confused as to how to turn on a light switch.

Security is paramount. And its importance is not just integral to keeping all connected devices safe from being manipulated from the outside, but also keeping privacy intact for owners of listening devices. These devices have been raising concerns about in-home privacy more than the cameras on your laptop and phones have of late. As Alex Swoyer writes in the Washington Times:

Consumers generally are believed to have consented to a company being able to collect information based on the product’s use guidelines. But whether consumers are truly aware of what that means, and whether companies are able to share the information they collect with the government raise more questions.

These devices must listen for a key phrase in order to initiate on the user’s command, so it’s no secret that the microphone is “on” at all times. Unless you’re using a setting that requires a button press to initiate, like Siri on your iPhone. The concern of privacy and potential overreach by these devices came to the forefront of an investigation in Arkansas, late in 2015. According to NPR, we know from court documents that police confiscated an Amazon Echo at the scene of an apparent murder post-football party to potentially seek out additional information that the device may have recorded at the time of the crime. Additionally, it was stated that "investigators are also using information from a smart water meter, alleging that an increase in water use in the middle of the night suggests a possible cleanup around the crime scene”.

I’m not telling you to refrain from purchasing these kinds of products. They are, after all, extremely convenient and powerful (even in their infancy right now), and offer a pretty concrete vision of where tech companies are going in the near-future. But I am suggesting to you to think carefully about which ones you buy, and the potential unintended consequences of having one in your home.

Apple Takes the High Road

So what is the most valuable company on the planet doing? Late to the game, some may say. But at their recent Worldwide Developers Conference, Apple did announce a product launching in December called HomePod. It’s their version of the voice assistant-powered intelligent speaker, and being an Apple product, the company has a very clear idea of what its marketing message is for it.

Unlike Amazon Echo and Google Home, both of which emphasize the artificial intelligence behind the tech to drive a number of services, Apple is heavily leaning into tackling the home audio market for its HomePod (think Bose and Sonos as competitors, not Amazon or Google). They’ve called it “the new sound of home”, and it’s no mistake that they’ve put music and superior sound quality as the banner features. Ben Lovejoy has an astute write-up on the differences between Apple’s strategy here, claiming that Siri does still lag behind competitive voice assistant systems, but has a very focused direction compared to data-gathering giants like Google.

Given the comprehensive nature of the Apple ecosystem, Apple could choose to go down the same route as Google. It could use all of the data it has about me, tie Siri queries to my Apple ID and deliver the same level of intelligence and proactive suggestions as Google Home. If it did so, nobody would be saying that Siri lags significantly behind Google’s artificial intelligence.

But Apple makes a deliberate choice not to do so. When I ask Siri a question, my iPhone doesn’t attach my Apple ID to my query so that Siri’s servers can make contextual sense of it. All that is sent is a random identifier that cannot be linked to my identity in any way. The random identifier is used to help Siri learn my voice: it doesn’t know who I am, but it knows that my query came from (say) person 7582066701, and it can check back over six months to match my query against my voice file to better understand what I actually said.

Will the concessions in favor of privacy compromise Apple’s growth with Siri and its connected devices, or will the trade-off be a good middleground? I obviously am in the camp favoring data privacy, and am willing to lag behind the use-cases of competitor devices to instead wait for Apple’s cautious take on this new medium. But keep in mind that all these devices are in very early stages of their feature roadmaps, and most people don’t even know what these voice assistant-powered speakers can and cannot do. Mostly that’s because the enabled service features are still be rolled out for third-parties to use, and while Apple limits the usages to just a handful of actions, most features from Google and Amazon are used by developers but not used by the products’ users.

Recode reports”when developers for Alexa and its competitor, Google Assistant, do get someone to enable a voice app, there’s only a 3 percent chance, on average, that the person will be an active user by week 2”. It’s no surprise, then to read this:

The statistics underscore the difficulty Amazon and Google are having in getting Echo and Home owners to discover and use new voice apps on their platforms. Instead, many consumers are sticking to off-the-shelf actions like streaming music, reading audiobooks and controlling lights in their homes.

Too many choices are oftentimes too much to handle. Until these devices are ubiquitous and their broad services are well known enough to all consumers, most voice applications will probably go unused, just like applications on your phone or computer go unused either from being undiscoverable by the user, or the lack in need of its employment.

Where We Go From Here

Whether you want to call this tech transition full of overreach or not, the tale of listening “smart” speakers reinforces a few things that come with the territory of most topics I discuss on this site.

One is that we need to think through the kind of future we want. Current and future generations will probably become more accustomed to the invasiveness of these kinds of systems in our homes, and won’t think much about the privacy consequences. To them (and to many in general), it’s about convenience.

Secondly, we need to ensure that we continue to build next generation Internet-connected devices and accessories with a strong security foundation. Many security specialists, including Bruce Schneier, have advocated for a rebuilding of certain Internet protocols and security features to bake into the future of the Internet. How do we get there? Through policy and innovation. At least we hope.

Thirdly, we need to be mindful of the kinds of products we use, what the manufacturer is providing as a service, and at what cost to you. You should know by now that Google makes money off your data (they’re an advertising company); Amazon makes money off your purchases (they’re primarily a retailer); Apple makes money off your hardware purchases (they’re primarily a hardware design company). None of this may matter to you, but in the case of compromised data, hardware, and privacy risks, it’s clear that one company is probably a safer bet than the others.

Finally, someone needs to redesign the way Terms of Services are written. No consumer reads this shit thoroughly, and most of us don’t even know what we’re signing up for or handing over to various companies and third-parties. It’s an epidemic, and we need some sound policies enacted to clear up the mess for everyday people. You know, for us. We aren’t all lawyers, and we don’t have hours to read through and verify we’re good with these conditions. Leave that to Norwegian slow TV.

Faster Web & Less Bullshit, Please

It wasn’t long ago we were witnessing a cosmic shift in web development to accommodate the influx of computational powerhouse smartphones chugging through at-the-time bloatful websites. Those sites back in the mid-2000s were getting chunky with all the 2.0 insanity, and while the iPhone (in its release year of 2007) could render these sites on its 3.5” screen, it still wasn’t a great way to experience web pages. While most websites did have mobile versions of their core, desktop-friendly sites, they were woefully under-designed and lacked modern features to harbor modern conveniences (like ecommerce and rich media).

In the transitional years from the early smartphone era to now, sites tried finding a middle ground in design between too mobile-friendly (stripped down and hardly functioning) and too desktop-reliant (don’t just design sites for a large screen and tons of Internet bandwidth). This middle ground ended up becoming “responsive design”, an approach to web development that attempted to streamline page weight (for mobile) but have the flexibility of displaying the same amount of content, and typically loading the same number of scripts, across device screen sizes. For most circumstances, this was the right path to take. It wasn’t a mobile vs desktop world we were heading towards; it was a mobility world we had already entered, where the only thing that really differentiated access to websites and apps was the size of the screen and the interface accessibly (finger touch vs mouse click).

Unfortunately for everybody, this was (perhaps unintentionally) interpreted by developers that they no longer had to worry about page loading, script-rendering, and other complexities in web design contributing to page speed because an iPhone was just as powerful as your everyday, off-the-shelf laptop. Oh, and don’t mind the increasing complexity of ad networks and the growing inundation of ad placements and tracking scripts to load — any smartphone can handle those, too.

Except that this shift has left the web wounded. Everything seems to take longer to load, websites break easily, taps on mobile don’t register sometimes, and register other times, and so on and so on. I’ve written about site speed and performance before. It’s a growing problem. So much of a problem that the tech titans have taken note. Facebook attempted to remedy this and save the publishing industry by pushing hard on its Instant Articles initiative, a closed-garden approach to offering publishers a speedy alternative to their own laggard websites’ article templates and Facebook-sized reach. Apple built-in an iOS app called ‘News’, offering its take on the age-old RSS feed readers, but layering on pretty templates that were fast. And Google, the all-mighty search behemoth and purveyor of results that include the news, has aggressively pushed publishers, retailers, and websites of all kinds towards its Accelerated Mobile Pages (AMP) initiative, which is essentially an open source project encouraging the creation of streamlined HTML pages to reduce clutter and external JavaScript but while also running Google-only JavaScript and reassuring full reader analytics.

So How are Things Going?

Two years later, Instant Articles don’t seem to be working out as planned, as The Verge contemptuously bemoans:

But it's unclear if any huge advantage ever materialized. Facebook decided from the start that publishing a story using the Instant Articles format would not automatically improve its ranking in the News Feed. In practice, Instant Articles typically do reach more people, because people are more likely to read and share them. But as the format spread, competition increased, and any advantage to using Instant Articles was blunted within months. Given that Instant Articles were designed to carry less advertising than mobile web articles, broad reach was essential to ensure publishers would profit from the format. The reach just never arrived.

Apple’s ‘News’ app was initially off to a rocky start) in usage, but not much has been reported since. While arguments have risen about Apple’s role of gatekeeper in the news ecosystem, it seems that most publishers have welcomed it as an easy secondary publishing platform that permits a “bring your own advertising” model and subscription service options that are hard to ignore.

But what about Google. Google’s AMP project is more controversial than both Facebook and Apple’s forays, as it threatens web development integrity on the open web. A rant from The Register describes the plight as thus:

Announced in 2015, duly open sourced and integrated into Google’s mobile search, Google has pitched AMP as a way to speed the mobile web. It employs something the ads slinger calls AMP HTML that the firm describes as a “new open framework built entirely out of existing web technologies.”

What it is, is a way for Google to obfuscate your website, usurp your content and remove any lingering notions of personal credibility from the web.

If that appeals to you, here's what you need to do. First, get rid of all your HTML and render your content in a subset of HTML that Google has approved along with a few tags it invented. Because what do those pesky standards boards know? Trust Google, it knows what it's doing. And if you don't, consider yourself not part of the future of search results.

Sure, you might say: making the web faster is a noble vision. And yes, we unanimously agree, a faster web is better. But as the Register points out, “as with anything that eschews standards for its own modified version thereof, it's about lock-in. Tons of pages in Google AMP markup mean tons of pages that are optimized specifically for Google and indexed primarily by Google and shown primarily to Google users.” AMP is primarily a way for Google to combat lock-in systems from Facebook and Apple. The tech giants want everybody’s attention. But if you have an app feeding off standards (like Apple News), there isn’t a threat to disrupting the entire Internet’s web standards and rallying them around a controlled framework. We all want the Internet to be decentralized, right? Then you have to look at adopting AMP as an opposite way to do that. AMP is a choice for [Google search] inclusion, and there are monetary and attention-capturing benefits to doing so for brands and publishers. But forking your web development to accommodate a tech company’s recommended framework, a framework that is favored by that tech company’s mysterious organic algorithm for surfacing news results, is something else entirely. We’ve already seen what reckless strains of SEO has done to the web. Let’s not repeat those mistakes with reckless adoption of Google’s AMP HTML framework.

AMP also is a branding nightmare. Tapping a link from Google search results (again, the only way to access these versions of canonical pages) loads the page from Google's cached AMP index nearly instantaneously. Sharing that page simply shares the Google cached URL of the article, and trying to read more from that author/publisher is a frustration in interaction design -- the permalink button to go to the brand's actual domain is an unintuitive icon, and branding itself is obfuscated by the AMP framework's content-first philosophy. So what's in it for brands aside from handing over the keys to Google, and continuing to strain their own websites' development with the same shitty inundation of scripts, ad networks, unfriendly mobile paradigms, and page speed performance?

This debate has only just begun. But several of the Internet’s finest warriors are working on alternative solutions. The first of this anti-AMP movement is brought to you by a thoughtful fuck you project by Pinboard’s founder, Maciej Ceglowski. He basically re-created Google’s original AMP demonstration page without any of the forced Google scripts, and it represents the same performance. Maybe if we encouraged web developers to focus on leaner, cleaner designs (melding the pre-iPhone days with a more careful post-iPhone responsive design mantra) we could get to a better place for everyone. I’ll leave you with Ceglowski’s snarky comment at the bottom of his faux-AMP demo site:

Dozens of publishers and technology companies have come together to create this unfortunate initiative. However, it is 2015, and websites should be small and fast enough to render on mobile devices rapidly using minimal resources. The only reason they are not is because we are addicted to tracking, surveillance, gratuitous animation, and bloated, inefficient frameworks. Requiring a readable version of these sites is a great idea. Let's take it one step further and make it the only version.

Update: May 25, 2017

A mildly-related update here from TechCrunch on Facebook's plans for support for Google AMP and Apple News. Essentially they're trying to make it easier (and their own solution interoperable between competing formats) for publishers to more easily manage these specially-formatted content distribution channels. This comes in the form of an Instant Articles SDK (software development kit), enabling developers to "take the markup that’s used to build Facebook’s Instant Articles and use it to create the code that’s needed to build for AMP and Apple News." Note that Facebook would prefer you start with content distribution and formatting within its ecosystem, and use the Instant Articles SDK to output to competitor ones.

TechCrunch points out:

[T]he extension’s launch also comes at a time when a number of high-profile publishers have begun to abandon Facebook’s format, due to its lack of monetization options.

In April, for example, it was reported that Forbes, Hearst, The New York Times and others have backed out of Instant Articles. Other major media organizations including Bloomberg, The WSJ, ESPN, CBS News, NPR, Financial Times, and VICE News have also been holdouts, running little to no content in Facebook’s format. Others who have used the format have been winding down their support; and last month, The Guardian pulled out of both Facebook’s Instant Articles and Apple News.