Design for Trust and Safety

Designing for Trust, Safety, and Privacy #

Guest lecture: Karen Maxim, Policy & Law Enforcement Response Team Lead at Zoom Trust & Safety

  • Initially: Zoom as a B2B company – for companies to meet other companies + clients
    • “Closed meeting space” – not designed for wider things like classrooms and city council meetings
    • Did not need much in the way of T&S function
  • First set of reports: “Illicit Content and Substances” (ICS), sex/drugs parties, needed to shut down, but predictable work
  • Zoom T&S: established in response to crisis of zoom-bombing @ beginning of pandemic

Basic definitions #

  • Surface: a technical “entry point” to interact with a system
    • e.g. Facebook, public profiles, private groups, live video streaming, private messaging
    • For users and abusers: think about entry points for abuse
      • What design elements do people create content in vs consume content from?
      • What are potentials for abuse while creating content?
  • Features -> Affordances -> Outcomes
    • Features: design elements that offer specific types of capabilities offered by the system
    • Affordances: possibilities for action available in a given environment
    • Outcomes: actions or other behaviors connected with the goals of the user

ABC framework #

  • Actors: abusive actors
    • Who are users? What is intent? What networks are they situated in?
  • Behaviors: deceptive behaviors
    • What do actors do? What are actions that repeat offenders take?
  • Content: harmful content
    • What kinds of content is created or viewed that can be harmful?

Measurement for internal management #

  • T&S perceived by execs as a “cost center” and drag on product
  • Goal of measurement: support investment in T&S work
    • Goals: user trust, civil communication, healthy interactions, sense of safety
  • How can we know T&S is doing a good job? That users trust platform and feel safe?
    • User surveys?
    • But data is subjective and noisy
  • Abuse leads to poor experience, has impact on uesr trust, and can lead to user harm
  • Measurement via internal and external methods

Precision and Recall #

  • Precision: proportion of positive identifications that are correct – TP/(TP + FP)
  • Recall: proportion of actual positives that were identified correctly – TP/(TP + FN)
  • Accuracy: fraction of predictions the model got right – (TP + TN)/(TP + FP + TN + FN)
  • Prevalence: proportion of bad content that got past the system – FN/(TP + FP + TN + FN)

Using measurements to make decisions #

Axes:

  • Responsibility of product
  • Harm: from bad product experience, to death or serious injury