Label

Unlabeled

Person

Edge Case

How to Handle It

Example

“Good data?” attribute when a person is

being covered by another object.

In these cases if a person is being covered by an object (Ex. A desk, a table, a chair, a blanket, etc) and you cannot see at least a small part of their body touch the edge of the photo then it is bad data (Good data?: No)

You should consider the person’s torso and not just their arms for it to be considered as “covered”

If the person is being covered but their arms are reaching the lower edge of the photo it should still be considered as bad data (Good data?: No)

Unless their arms are being used to cover most of the item that is covering them and creating the illusion that all of their body is touching the lower edge of the photo

The photos on the first column are examples of someone being covered and it should be bad data (Good data?: No)

In this example the subject’s arms are touching the lower edge of the photo but we can see that her torso is not touching the edge, so in this case it is also bad data (Good data?: No)

Annotation should not go through objects

like this, this is INCORRECT

Broken images / artifacts in the pictures

Remember the 12 Out of Scope Criteria that the project has.

If the picture is presenting any artifacts within the picture regardless of where it is, then it is bad data (Good data?: No)

The artifacts can be in the subject or in the environment around the subject, regardless it is considered as bad data

YOU SHOULD NOT CONFUSE ARTIFACTS WITH LOW QUALITY IMAGES

This is a low quality image that should be annotated

Artifacts can appear at varying degrees distorting the image a lot, or not so much, regardless of the amount of artifacts, it should still be considered as bad data

Objects in front of the person and accessories

• Label anything in front of a person as Person when that Person is the background of that thing.

• Label any head accessories on the person as Person

• DO NOT label anything the person is holding as Person unless it is directly in front of the person

Example: headphones/earphones (excluding wires), glasses, and hat

Cables from headphones or earphones

DO NOT label the wires if no human background in the back

Artifacts in pictures

Please keep estimates of the remainder of the person's body.

In this case, this tasks is not considered as corrupted since the artifact is only present in a minimal portion of the task

Thin hair / Messy hair

Objects thinner than 2 pixels SHOULD NOT be annotated

Gaps between thick strands in hair

Gaps LESS than 5 pixels can be IGNORED.

Gaps WIDER than 5 pixels SHOULD BE LABELED as background.

Overlapping textures/colors

If there are any overlapping textures/colors between “Person” and “unlabeled,” please label as accurately as possible.

In the example to the right, the subject’s headset should be labeled as “Person” while the monitor behind him should be left unlabeled.

- Notice how the color of subject’s headset blends in with the monitors behind him. Exclude the monitors from the “Person” label.

Image Blur and Quality

If there are any images that have poor quality or image blur please annotate as accurately as possible. Annotate as close to the refined pixels as possible.

- Notice how the subject’s hand is blurred and might give us difficulty when trying to annotate. Try your best to annotate as accurately as possible in these situations, and make sure you don't leave them without annotation

Subject in the dark

Do NOT annotate these type of tasks

This type of annotation is wrong.

What to do if the annotation is good

quality but the task is “good data? No”
(For Reviewers)

Sometimes taskers annotate the task even though the task is bad data (good data attribute is marked as “no”) and should not be annotated 

If the annotation is incorrect or low quality then the annotation should be deleted

But if the annotation is correct or good quality then it is not required to delete the annotation, you can leave it annotated!

You should only be sure that the global attribute of “good data?” is marked as “No”

In this case, the task is bad data due to the subject’s knee being in frame and according to Criteria #5 of the Out of Scope Criteria it should be bad data

Image

Criteria

Criteria 5 &6:

No eye in image.
Belly button in frame.

Criteria 8:

90 degrees rotated.

Criteria 4:

Too large hands.

Criteria 11:

Broken image.

Criteria 11:

Broken image.

Criteria 10:

Too dark.

Criteria 7:

Person occluded by object (keyboard)

Criteria 5:

Knee in frame.

Criteria 12:

Padded image where the foreground is cropped by the black padding.

Criteria 3:

Too large foreground, covering more than 95% of the image area.

Criteria 6:

No eye in image.

Criteria 5 & 7:

Knee in frame.
Occluded by objects

Criteria 10:

Too dark.

Even if no eyes are visible, the position of at least one eye is within the image.

Even if the exposure is poor, it is possible to label this image correctly.

There can be people in the background with faces smaller than 20% of the image height. Only label the gentleman front and center of the image

Rule

Description

Example

“Person” vs. “Unlabeled”

- Please annotate any visible part of the subject as “Person”

- Additionally, if the subject is wearing any headsets/headphones, glasses, microphones, accessories or jewelry - please include them in the “Person” annotation

- Gaps or spaces between the subject and the objects they are wearing (such as a headset seen in the image to the right) should be left unlabeled

- If the subject is attached to or holding an object, only label the parts of the object that occlude the person. Do not label other parts of the object. See example here.

Here is an example of what should be labeled as “Person” and what should be left unlabeled

**notice the microphone included in the headset should also be labeled as person

“Person” vs. “Unlabeled” cont.

(For objects occluding the subject)

- Accessories or objects that occlude (overlap) with the subject should be labeled as “Person”

- If an object is partially occluding the subject, only the part that is in front of the subject should be labeled as “Person”

Additional Subjects

If there are other people in a task, please annotate as ‘Person’ only if they are approximately <4 ft. from the camera (ego) AND looking at the camera.

If the subject is too far away or too unclear to accurately annotate - they can be left unlabeled

Details

Particular things to annotate as “Person”:

• hands/fingers
• glasses/eyewear
• headphones/microphones
• accessories/jewelry
• Parts of an object that directly occlude the person)

You DO NOT need to label thin strands of hair in the “Person”. See example to the right for reference.

Here is an example of how to annotate accessories (headset, eyewear, etc.) as “Person”

Correct annotation example:

Notice how we include the microphone of the head et, but NOT the cable

Incorrect annotation example:

Here are examples of how NOT to annotate thin strands of hair:

CORRECT annotation example:

INCORRECT annotation example:

Pixel Threshold (+/- 2px)

- Please make sure annotations are within a 2 pixel threshold in relation to the subject!

- Each annotation should be as accurate as possible to avoid segmentation bleeding or misclassification errors

Example: Below is an example of a task that stays true to the +/- 2 pixel threshold

CORRECT:

INCORRECT:

Labels/Attributes

Description

Example

Person

- Any visible part of the subject’s body, including head and torso area(sometimes legs)

- Additionally, if the subject is wearing any headsets/headphones, glasses, microphones, accessories or jewelry - please include them in the “Person” annotation

- Gaps or spaces between the subject and the objects they are wearing (such as a headset seen in the image to the right) should be left unlabeled

• If an object is attached or held by the subject, only label the parts of the object that occlude(overlap)the person. Do not label other parts of the object.

Unlabeled

- Miscellaneous objects that are not attached or held by the subject

- Wires to headphones/headsets

- Gaps between headset/headphones and subject

- People far away from camera (> 4 ft away)

- People whose face is outside of image

- Furniture or other objects the subject might be sitting in/on (includes the chair a subject might be sitting in)

Critical Errors

Non-Critical Errors

- Make sure all parts of a subject is labeled as “Person”

- Make sure you do not label gaps in between headsets, and other head-wear as ‘Person’

- Make sure you do not label chairs, couches, or other pieces of furniture as “Person”

-Make sure you label the microphone included in the headset even if they don't occlude the person

- Exclude wires from headsets/headphones as “Person” (remember they should be unlabeled if they are not occluding the subject!)

- Stray hairs are non-critical