Alternate title: My hopes are low for object detection in the new Photos but I still have hope
Reddit user vista980622 dig some digital sleuthing and may have come up with the list of over 4,000 objects, memories, and facial expressions that Photos for iOS and OS X will be able to mine all on its own with Apple’s Advanced Computer Vision technology announced at WWDC. The user then wrote this about the landmark detection on Ev’s blog:
Additionally, you can search for various landmarks. For example, Photos can respond for search query of “Maho” (beach in Saint Martin), despite Photos is not programmed or trained to understand specific landmarks. Behind the scenes, Photos app first generates a generic categorization for the scene, “beach”, then searches through a built-in dictionary for all landmarks that has the name “beach” in its definition.
This is smart approach.
It reminds me of something Craig Federighi (Hair Force One to me) mentioned during John Gruber’s live Talk Show event during WWDC. There are a lot of ways to teach Apple’s Advanced Computer Vision system that do not need to involve sending your photos to them. They know what a beach or mountain or forest looks like. They have access to the location of the photo. And they have access to the world’s knowledge via the web. Combining those things they can find a huge amount of information in your photos that can be used to discover them without ever needing to look at the photos themselves.
In my wish list for Photos for OS X, iOS, and iCloud Photo Library I mentioned that I wanted to be able to search for objects. I wrote:
Sort of related to the auto-generated albums above, I’d love to be able to search for “red” or “lake” or “tree” and get results. Google is killing Apple at this. And it just makes so much sense. The more the application does for you the less classification you have to do manually. I tag my photos with things like “cat” or “ants” or “beetle” or “snake” because I want to be able to search for these things later. And adding my own layer of taxonomy on top of my library should always be an option … but for objects that are easily identifiable these days (like lakes or cats) it just makes sense.
It appears I’ll be getting that. I noticed a lot of object and animal specific terms in the list that vista980622 shared. One stood out; "arachnid". I hope, and am pretty sure I will be able to, still search by "spider" though it isn’t listed. Which brings us to the discoverability of these types of searches. I hope Apple doesn’t only provide a search box but that they also suggest searches or create pseudo-albums for you.
For example, Google Photos creates albums (sort of) by simply giving you a way to find those objects in your library without searching for them. They aren’t albums so much as links to search results that look like albums. I hope Apple builds in a discovery mechanism too. And it’d be great if it were based on what I took photos of the most.
Looking through my Library it’d be easy to see that I take a lot of photos of lakes and rivers (kayaking), bees, barns, and buildings. I also visit a lot of wineries and breweries. It’d be nice if Apple simply had "pseudo albums" or saved searches at the ready for me for all of these things. And then they could throw in a few for fun like cats, pink, beach, panoselfie.
One tidbit about the assumed facial expression detection in Photos… They seem to be using this to create memories. Who wants memories of a bunch of angry people? So I’m guessing that if they want to make a bunch of happy memories for people they needed to go beyond just detecting the people in the photos but also what mood they were in.
OK, one more tidbit about face detection. I’m skeptical that this will be any good. But I hope I’m wrong.
Currently there are two kinds of face detection. The first involves determining that there is a face in the photo. You’ll see iOS’s camera app doing this live while you’re shooting. A yellow box will surround people or objects in a photo to get a good focal length to make sure your subject(s) are in focus. That is face detection that simply says "we think this is a face". Then there is face detection that involves determining the actual person in the photo. Photos for OS X has this currently… though it needs to be improved a lot. Like, a real lot. Check out this example from this weekend:
You can click the image to zoom in a bit.
On the left, my friend Matt (who has a face). On the right, a vending machine (which does not have a face).
Photos for OS X believes that the vending machine is a face. This is technology that Apple has been mucking around with for at least 8 years as it was debuted in iPhoto in 2009. It didn’t suggest any names for the face (it rarely does, which I’ve covered here), but it doesn’t even see Matt’s face.
I’m sure that Apple’s new Photos for iOS and OS X will be better than what we currently have but I’ll wait and see before I get excited. Because so far they’ve yet to be great at this and Google and Facebook kill them at it.
I’m anxious to play with iOS 10 and macOS Sierra. But not anxious enough to install the betas on my hardware. So I’ll be writing a lot more about this in the fall.