Designing for Trust, Safety, and Privacy #
Guest lecture: Karen Maxim, Policy & Law Enforcement Response Team Lead at Zoom Trust & Safety
- Initially: Zoom as a B2B company – for companies to meet other companies + clients
- “Closed meeting space” – not designed for wider things like classrooms and city council meetings
- Did not need much in the way of T&S function
- First set of reports: “Illicit Content and Substances” (ICS), sex/drugs parties, needed to shut down, but predictable work
- Zoom T&S: established in response to crisis of zoom-bombing @ beginning of pandemic
Basic definitions #
- Surface: a technical “entry point” to interact with a system
- e.g. Facebook, public profiles, private groups, live video streaming, private messaging
- For users and abusers: think about entry points for abuse
- What design elements do people create content in vs consume content from?
- What are potentials for abuse while creating content?
- Features -> Affordances -> Outcomes
- Features: design elements that offer specific types of capabilities offered by the system
- Affordances: possibilities for action available in a given environment
- Outcomes: actions or other behaviors connected with the goals of the user
ABC framework #
- Actors: abusive actors
- Who are users? What is intent? What networks are they situated in?
- Behaviors: deceptive behaviors
- What do actors do? What are actions that repeat offenders take?
- Content: harmful content
- What kinds of content is created or viewed that can be harmful?
Measurement for internal management #
- T&S perceived by execs as a “cost center” and drag on product
- Goal of measurement: support investment in T&S work
- Goals: user trust, civil communication, healthy interactions, sense of safety
- How can we know T&S is doing a good job? That users trust platform and feel safe?
- User surveys?
- But data is subjective and noisy
- Abuse leads to poor experience, has impact on uesr trust, and can lead to user harm
- Measurement via internal and external methods
Precision and Recall #
- Precision: proportion of positive identifications that are correct – TP/(TP + FP)
- Recall: proportion of actual positives that were identified correctly – TP/(TP + FN)
- Accuracy: fraction of predictions the model got right – (TP + TN)/(TP + FP + TN + FN)
- Prevalence: proportion of bad content that got past the system – FN/(TP + FP + TN + FN)
Using measurements to make decisions #
- Responsibility of product
- Harm: from bad product experience, to death or serious injury