Menu

Colin Devroe

Reverse Engineer. Blogger.

What Photos for OS X and iOS will be able to automatically detect in iOS 10

Alternate title: My hopes are low for object detection in the new Photos but I still have hope

Reddit user vista980622 dig some digital sleuthing and may have come up with the list of over 4,000 objects, memories, and facial expressions that Photos for iOS and OS X will be able to mine all on its own with Apple’s Advanced Computer Vision technology announced at WWDC. The user then wrote this about the landmark detection on Ev’s blog:

Additionally, you can search for various landmarks. For example, Photos can respond for search query of “Maho” (beach in Saint Martin), despite Photos is not programmed or trained to understand specific landmarks. Behind the scenes, Photos app first generates a generic categorization for the scene, “beach”, then searches through a built-in dictionary for all landmarks that has the name “beach” in its definition.

This is smart approach.

It reminds me of something Craig Federighi (Hair Force One to me) mentioned during John Gruber’s live Talk Show event during WWDC. There are a lot of ways to teach Apple’s Advanced Computer Vision system that do not need to involve sending your photos to them. They know what a beach or mountain or forest looks like. They have access to the location of the photo. And they have access to the world’s knowledge via the web. Combining those things they can find a huge amount of information in your photos that can be used to discover them without ever needing to look at the photos themselves.

In my wish list for Photos for OS X, iOS, and iCloud Photo Library I mentioned that I wanted to be able to search for objects. I wrote:

Sort of related to the auto-generated albums above, I’d love to be able to search for “red” or “lake” or “tree” and get results. Google is killing Apple at this. And it just makes so much sense. The more the application does for you the less classification you have to do manually. I tag my photos with things like “cat” or “ants” or “beetle” or “snake” because I want to be able to search for these things later. And adding my own layer of taxonomy on top of my library should always be an option … but for objects that are easily identifiable these days (like lakes or cats) it just makes sense.

It appears I’ll be getting that. I noticed a lot of object and animal specific terms in the list that vista980622 shared. One stood out; “arachnid”. I hope, and am pretty sure I will be able to, still search by “spider” though it isn’t listed. Which brings us to the discoverability of these types of searches. I hope Apple doesn’t only provide a search box but that they also suggest searches or create pseudo-albums for you.

For example, Google Photos creates albums (sort of) by simply giving you a way to find those objects in your library without searching for them. They aren’t albums so much as links to search results that look like albums. I hope Apple builds in a discovery mechanism too. And it’d be great if it were based on what I took photos of the most.

Looking through my Library it’d be easy to see that I take a lot of photos of lakes and rivers (kayaking), bees, barns, and buildings. I also visit a lot of wineries and breweries. It’d be nice if Apple simply had “pseudo albums” or saved searches at the ready for me for all of these things. And then they could throw in a few for fun like cats, pink, beach, panoselfie.

One tidbit about the assumed facial expression detection in Photos… They seem to be using this to create memories. Who wants memories of a bunch of angry people? So I’m guessing that if they want to make a bunch of happy memories for people they needed to go beyond just detecting the people in the photos but also what mood they were in.

OK, one more tidbit about face detection. I’m skeptical that this will be any good. But I hope I’m wrong.

Currently there are two kinds of face detection. The first involves determining that there is a face in the photo. You’ll see iOS’s camera app doing this live while you’re shooting. A yellow box will surround people or objects in a photo to get a good focal length to make sure your subject(s) are in focus. That is face detection that simply says “we think this is a face”. Then there is face detection that involves determining the actual person in the photo. Photos for OS X has this currently… though it needs to be improved a lot. Like, a real lot. Check out this example from this weekend:

Photos OS X Face Detection Error Fail

You can click the image to zoom in a bit.

On the left, my friend Matt (who has a face). On the right, a vending machine (which does not have a face).

Photos for OS X believes that the vending machine is a face. This is technology that Apple has been mucking around with for at least 8 years as it was debuted in iPhoto in 2009. It didn’t suggest any names for the face (it rarely does, which I’ve covered here), but it doesn’t even see Matt’s face.

I’m sure that Apple’s new Photos for iOS and OS X will be better than what we currently have but I’ll wait and see before I get excited. Because so far they’ve yet to be great at this and Google and Facebook kill them at it.

I’m anxious to play with iOS 10 and macOS Sierra. But not anxious enough to install the betas on my hardware. So I’ll be writing a lot more about this in the fall.

Random subtle updates to Apple software

Apple could not possibly cover every update to iOS, macOS, tvOS and watchOS in their Keynote. So as the nerds have been picking through the trash in and around San Francisco they’ve been able to dig up several subtle changes that are worth noting.

Here are a few of them that I’ve found via Twitter and scouring the blogosphere. Some of them were mentioned in passing in the Keynote as well but I thought I’d highlight those too.

  • For developers, Apple made a variant of the San Francisco font called SF Mono. Yes!
  • Apple Maps can now find gas stations and restaurants along your route of travel. This will be enormously helpful.
  • macOS gives a tabbed interface to nearly every application out of the box without the developer needing to make an update. This will end up being a bigger deal than it may seem.
  • Creating sticker packs for iMessage requires no code. So expect a lot of sticker packs.
  • iMessage allows for read receipts to be sent, or not, on a per conversation basis.
  • iOS 10 allows you to remove default Apple apps (like Tips, Stocks, etc) from your Home Screen. It doesn’t delete the app, however.
  • The News app now allows you to subscribe to specific publications rather than only selecting them as possible sources. RSS reader?
  • iOS 10’s Phone application can make Skype (and, presumably in the future, Slack, Google Hangouts, etc.) a first-class citizen and add their call lists, voice mails, etc. to the Phone app. Think of how Facetime works now. It is built into Phone.
  • Apple Pay on the web will make paying for things incredibly easy. It would be possible for the worst shopping experiences ever to become one or two taps.
  • iOS 10 lets you edit Live Photos and even stabilize them (like Google Motion Stills).
  • Universal Clipboard (macOS, iOS) will allow you to copy text, images, or video from one device and paste them on another. The engineering to make this happen must be amazing yet the feature is completely invisible.
  • Picture-in-Picture on macOS. Right now I use a bookmarklet in Safari to force a YouTube video to pop-up and be in its own window. This way I can continue working or using Safari. Picture-in-Picture will allow me to do this, even in full screen apps.
  • tvOS’s Single Sign-on feature makes a 5 step process (or more) and makes it zero steps. No more going to CBS News’ website and typing in a 4 digit code to get access.
  • AppleTV will now support 4 game controllers, not just 2.

I’m sure more and more smaller, subtle things will come to light as Apple puts the finishing touches on these releases for the Fall.

Addendum: Mac Rumors has a few good ones. Notably, the Wake Alarm and Flashlight intensity settings.

Second addendum:

A few new tidbits emerged overnight. Partially due to John Gruber’s live The Talk Show with Hair Force One and Phil. Namely:

  • iOS 10 will support shooting and editing in RAW
  • And though some Apple apps can be removed (see above) they can’t be independently updated and will not be available in the App Store. Which is sort of a pity because that could have meant faster update cycles for most used apps.
  • Apple still loves the Mac App Store. Yeah, we’ll see.
  • Safari 10, in macOS Sierra, will turn all plugins (Flash, Quicktime, etc.) off by default.

Third addendum:

  • If you use an external keyboard on iOS one of the keyboard shortcuts you may use is “CMD+Tab” which lets you cycle through the apps you currently have open. In iOS 10 the Home Screen is now an option on that.

Fourth addendum (honestly, I could keep going and going):

Fifth addendum:

  • In iOS 10 a “magnifier” can be turned on. Let the Sherlock Holmes jokes run amok.

If you notice anything else, send me a tweet or an email or something.

WWDC murdered my wish list

In a good way.

Yesterday I scrawled a few comments during the WWDC Keynote, and did 1 second reviews of the announcements on Snapchat, but I thought I’d jot down the tally of things I had hoped for against what was actually announced.

First, however, let me just say that the amount of work Apple showcased yesterday is just staggering. No doubt thousands and thousands of designers, engineers, operations, and more people made all of that possible. And watching the Keynote back this morning I’m left wondering if any of them have slept in the last year.

OK. Let’s start with the wish list:

  • “I hope Siri can do a lot more” – Yes.
  • Big changes to macOS – I think what I meant, and what we got, are different. However, what we got is pretty great. Some of the demos on the Apple web site are better than those that were shown on stage. And the “tabs everywhere” feature is bigger than it seems.
  • Displays? Nope. But I don’t really care, honestly.
  • Updates to tvOS? Yep. Big, big updates here. Looks like now is the time to buy an Apple TV.

So, pretty much everything I wished for.

Now, onto the things I wanted to see in Photos and iCloud Photo Library specifically. When I had written that post I separated the improvements that I hoped to see into two categories. I had a list for things I thought we’d definitely see and a list of things that I was skeptical that we’d see (due to the amount of work they are to do). Well, it turns out that I’ll be getting a lot of the things I thought were “pie in the sky” for Apple to release this year.

  • Facial recognition more liberal – Yes. I think we got this and much more with all of the Google Photos-like “advanced computer vision” features. I can’t wait to see how well this works. If it works as good or better than Google Photos it will be mind blowing to a lot of iOS users.
  • Sync Smart Albums – Unknown.
  • Auto-generated Albums (for things like water, etc.) – Yes.
  • Face tagging on iOS – Yes.
  • Map view – Yes.
  • Tagging on iOS – Unknown. (But I still hope)
  • Spotlight – Perhaps. But it appears I can search for photos with Siri. “Show me photos I took yesterday” So I’ll take it.

Many of the other things that I had hoped they’d improve are under-the-hood improvements so I’ll need to wait and see if I get them.

As I was watching yesterday’s WWDC Keynote I felt like Apple was reading my blog and simply checking off the boxes.

That was a strong WWDC Keynote. Huge leaps forward in software and services. A near impossible amount of work has been accomplished.

Well all of the Photos updates I wanted for iOS 10 look like they are happening. Wow.

macOS looks like a great update. Hair Force One continues to be Apple’s best presenter.

This tvOS update is also pretty great. Looks like it might be time to update.

This watchOS update is an incredible improvement. Apple must have a huge amount of resources on Watch.

MSFT’s E3 Xbox Keynote has had gallons and gallons of more blood in it than Apple’s WWDC Keynote will. Turning off E3 Keynote.

WWDC 2016 wish list

Wish lists have been swirling around these last few days and many of them are quite good. But none of them are mine. So here is my wish list, not my predictions, for what will be announced today at WWDC.

  • I hope Siri can do a lot more – I think we’re overdue on being able to say things like “Hey Siri, send the photo I just took to my wife.” Or “Hey Siri, open Spotify and play Jack White’s Blunderbuss.” Or “Hey Siri, find a note in Simplenote that I created on May 15th”. Or, even, “Hey Siri, show me all of the photos I’ve taken in Hawaii.”
  • I already wrote my list for Photos and iCloud Photo Library. Any updates to these apps and services whatsoever would be nice since I use them so much. One thing I left out of that post; slightly better video editing. I don’t need iMovie. But a few more video editing tools built into the Photos app would be nice.
  • I’m hoping for big, big changes in the new macOS (formerly Mac OS X). macOS is long overdue for a big update. I wouldn’t mind, even, if it sort of felt like a do-over.
  • My only wish for the much rumored Apple Displays is that they are affordable. But I won’t hold my breath.
  • Lastly, if there are some nice updates to tvOS I’ll pick up the new Apple TV.

Anything other than the above will be icing on the cake. I love Keynote days.