Snapshot Serengeti Talk

Why don't we have an "Unidentifiable" option when classifying photos?

  • dms246 by dms246 moderator

    Those of us who enjoy projects like this tend to like to be accurate and precise in our classifications, and the notion that we should submit outright guesses when there isn't enough information in an image really goes against the grain. Our gut instinct is that that's just plain wrong! What earthly use is a guess to anyone?! So when we first come across the night photo with a vague, shadowy shape and two shining eyes, or a close-up photo of an eye peering back at us, or a generally tan coloured rump and nothing else visible, our first instinct is to want to click on a button that says "This image is wayyyy too vague/blurry/weird for me to stand any chance of identifying what's here!". Only problem is, there isn't one. So our next thought is "That's kinda dumb - why haven't they allowed for the possibility that some photos are just plain unidentifiable?!". So maybe we ask about this in one of the forums, and we're told - just make your best guess. And then we're back to thinking "What earthly use is a guess to anyone?!".

    Well, counter-intuitive though it seems at first, in fact a guess can convey quite a lot of information, when analysed in conjunction with the guesses made by the others who are presented with the same photo to classify. And that's why the scientists want us to make our best guess on those vague/blurry/weird photos.

    For example:

    • If ten people all classify a photo as "1 elephant, eating", the photograph can confidently be filed as "1 elephant, eating".

    • Alternatively, if ten people classify a photo as containing a single creature, moving, but all of them pick a different species, the photograph can be filed as "1 indistinct creature, moving". Further, if all of the species that people picked are of small animals, then it can be filed as "1 small indistinct creature, moving". And now, without anyone else having to look at the photo, it can be excluded from data sets relating to large mammals. So if a researcher wants to examine data relating to elephants, they can confidently exclude that photo's data, again without having to check it. Sometimes the power of collective "best guesses" is that they can say what isn't in a photo, even if they can't say accurately what is there.

    • And finally, if ten people classify a photo as containing varying numbers of creatures, of widely varying species, the photo can simply be filed as "unclear". Then it's up to the scientists whether simply to discard that photo from the overall data set, or to have someone sit down and review all such "unclear" images to see if there is in fact useful data that can be extracted from them.

    If we had an "I don't have a clue what this is!" button, there would be a tendency for us to use it whenever we're unsure, and that would simply produce lots of useless classifications that convey absolutely no information at all.

    One of the scientists, Margaret Kosmala (@kosmala on here) has written a really interesting blog post that goes into this in more detail - I highly recommend reading it (and the other posts in the project blog!) :

    http://blog.snapshotserengeti.org/2012/12/14/we-need-an-i-dont-know-button/

    Posted

  • chaoticidealism by chaoticidealism

    Good to know this. I feel better about having made guesses now.

    Do you guys use some kind of statistical analysis to try to determine the odds that it's one animal or the other?

    Posted

  • Jihang by Jihang

    So if there's a large, shadowed, camera-blocking creature with no distinct features whatsoever, what's the most useful way to respond? Go with the odds and call it a wildebeest?

    Posted

  • chaoticidealism by chaoticidealism

    I can often narrow it down by fur color, but yeah--I usually go with wildebeest, if it's a big animal of the right general color and fuzziness. But I'm surprised sometimes how little information I need to identify an animal. I got one elephant by just the tip of its tusk in the edge of the picture.

    I don't know what to do with images that are absolutely black, though, with no way to tell whether an animal is the thing blocking the camera. I mark them as "nothing here". But there are only a few of those. Most of the time you can tell if it's an animal that's way too close.

    It can help to look at all the frames. My first day here I had this vague blob I thought was a branch and almost put down to "nothing here"... but I took a good look at the third frame and figured out it was an elephant, probably exploring the camera with its trunk. Wish I'd favorited it now. Blurry as it was, it was unmistakeable once I knew what I was looking at.

    Yeah, can you tell I like elephants? 😃

    Posted

  • MarySkater by MarySkater

    For a daylight shot with a few black dots on the horizon, or a night shot with one eye-shine, I click "nothing here." (Nothing identifiable, is my logic.)

    For a lump of black blocking the frame, with no clear shape or colour - I opt out of classifying, and leave that for someone else!

    Posted

  • DruidDriver by DruidDriver

    So what's is the consensus or the preference of the organizers? If there isn't enough information,to even get to the correct body type let alone a guess, are we to make it a nothing picture?

    Posted

  • davidbygott by davidbygott moderator

    If there's definitely an animal in front of camera, it can't be nothing. Just give it your best guess. Other people may guess something else. The 'system' will flag anything that all of you volunteers are very ambiguous about, and experts will re-examine it. Does that help?

    Posted

  • SafariLiz by SafariLiz

    I can see this 'aggregate of ten' point, but what if "I" (or any other person) gets the same photo several times - say I make an extreme guess that it's one thing and tag it as that three, four or more times - doesn't that imbalance the results? I've definitely had instances of the same very distinctive photo coming to me at least four times, and sometimes there's a very unidentifiable 'something' in the image (e.g. deeply in shadow/totally blown out that PS can't rescue).

    Posted

  • crackwilding by crackwilding

    Er, if everyone is being told to guess "wildebeast" when they absolutely can't make it out, that's going to give you a lot of unambiguous results for ambiguous pictures.

    Posted

  • mistyfriday by mistyfriday

    Did you read the post by moderator dms246? As well as the link @ the bottom of it? So much of this is very well explained. Read the comments on that link tool. Get a feel for how others have resolved this issue.

    The person saying they click widlebeest stated their own response. Others may identify something else. But if you pick a big animal such as the wildebeest, you for some reason have eliminated the smaller and medium animals.

    Thirdly, this was set up by scientists and others who have done some study in scientific analysis and set it up the way it is for a reason.

    I tend to be a perfectionist and had difficulty resolving it for me if I could not properly ID an image. But now after having read through these arguments have accepted the rational. I have relaxed, and decided to enjoy the images. Obviously you get better the more you do. I surprise myself with what I can identify now that I could not before.

    Enjoy and have fun. If it bothers you too much perhaps this project isn't for you. Disclaimer: I am in no way associated with this project and these are my observations and suggestions. I hope it is helpful.

    Also images you are not sure about take the best guess and then bring it over here to discussions and see what others say.

    Posted

  • avsjen by avsjen

    This is really helpful to know -- best guesses to come!

    Posted

  • Mary_C by Mary_C

    I really need an unindentifable button. When all there is is a single leg or a dark mass moving across the shot at night my option seems to be to label none to be able to move onto the next photo. Am I missing a way to skip over the photo and move on without an identification?

    Posted

  • areinders by areinders moderator

    No, you can't skip over an image. Just do your best, as explained in the first post of this thread. If you really can't tell give it your best shot. Chances are you are not the only one to have this problem and the identifications will vary wildly for this image. Then an expert can have a look and try to identify this one. But many seemingly impossible images can actually be identified with some practice. So don't agonize over it, there is no penalty if you make a mistake - just enjoy your armchair safari!

    Posted

  • heidigard by heidigard

    Why isn't there at least a butten marked "other" insteady of "unidentifiable"? I just saw wild mules in a picture but there is no category for those. For lack of a better option I eventually filled them as zebras because that's about the closest thing, even though I know it's not correct.

    Posted

  • areinders by areinders moderator

    You can always also tag an image in the discussions, that gives others the chance to have a look and maybe find interesting species that are not listed.

    Posted

  • Envirogirl667 by Envirogirl667

    Or just choose the species that looks the most similar to it.

    Posted

  • areinders by areinders moderator

    Yes, absolutely choose the closest ID you can find, instead of "nothing there" because something is there. I just meant that you can then in addition add a comment explaining what you think it is (or that you were stumped).

    Posted

  • Chinabob by Chinabob

    I agree that there should be an extra button. Why? out of the 30+ images I have just looks at 11+ where toooooo far away even to make a guess much less an educated one. One other you need a button in the horns selection that is "no horns".

    Posted

  • samhe by samhe

    i get the idea that if it is unidentifiable then people will choose differently, but there are some cases where the animal is too unrecognizable to even make a guess, if it is dark, a mass of fur, especialy in bad light, or too far away. I understand that it is better than i dont know, but in the cases where the animal is completely unguessable there should be some choice.
    I also think there should be a no horns button, and what about ears, which can be very distinctive, and sometimes the only bit in the photo

    Posted

  • areinders by areinders moderator in response to samhe's comment.

    The problem with having an unidentifiable option is that people have different ideas about when they call an image unidentifiable. Some are very worried about giving a "wrong" answer and might therefore chose that option too often, others are happy to make a guess. So by forcing everyone to guess because there is no other option the scientists actually end up with more useable answers. And those that are truly unidentifiable will get sorted out this way as well. But really, we are all so much better at identifying animals even by an ear or a tail than we think!

    Posted

  • sboylan by sboylan

    Wouldn't an option of "herd in distance" be useful?

    Posted

  • libervurto by libervurto

    What about when there are animals in the distance? I really can't tell what they might be on even their size, all I see is that they are alive and moving or eating.

    Posted

  • aliburchard by aliburchard scientist, translator

    Animals in the distance are a tricky one. The short answer is that we aren't too worried about animals that are really really far away in the distance. And if you wouldn't be able to distinguish the animal from a rock except for the fact that it's in a slightly different position from one frame to the next, then you can go ahead and ignore it. Our cameras are triggered by heat and motion of something up close -- so those animals far far away are captured incidentally.

    On the upside, because we don't have an "I don't know" button (see http://blog.snapshotserengeti.org/2012/12/14/we-need-an-i-dont-know-button/), we get a lot of information about what the photo might contain from the variation in the answers. If half the folks say "there's nothing here" and quarter say "15 wildebeest" and a quarter say "15 zebra", chances are it's a herd in the distance that is too far away to really tell. And we can then decide what we want to do with that information in our analyses.

    Posted

  • DZM by DZM admin

    The blog and discussion here are excellent answers for why a "skip" or "unclear" button would do more damage than help. It might be nice to give these more visibility... right now, your average classifier won't see them, and may get a bit frustrated, as I did.

    I especially like how the blog shows that even a collection of random guesses gives some information!

    Posted

  • SarahJenny by SarahJenny

    MY SYSTEM FOR ID NIGHT SHOTS W EYES: Ive found one can deduce a LOT by looking at the eyes and combining the size, color reflected, shape of the eyes/how high off the ground the eyes are (also if they are 2 in front of face or how far to the sides of the head they are) WITH the Classifying Filters such as "Size" and "Looks Like" and when theres only a few choices left, I look at the example pictures to see which eyes look most like the ones in the picture. Its surprising how much those slight differences in eye shape - that dont stand out in day shots - really stand out when the eyes are all you see!

    Sounds complicated, but it goes quick with the Filters and soon you start remembering animals by there eyes only! Hope that helps 😃

    Posted

  • DZM by DZM admin

    That's a really useful idea, @SarahJenny ... thank you! 😃

    Posted

  • KaiYeves by KaiYeves

    What if we can make out that there is an animal of, say, the deer/antelope type, but not enough detail to narrow it down further? Could there be some kind of option to classify as "unidentified deer/antelope"?

    Posted