Intro

Trust and Safety Engineering #

What is Trust and Safety? #

  • The study of how people abuse the results to cause harm
    • Often using products the way they are designed to work
  • Crosses between specialties – requires understanding of society and humanity
  • Dynamic and unpredictable

Course goals #

  • Have empathy for a broad cross section of the people who use your products and the risks they face
  • Explain to software engineers and product managers the common ways that internet technologies are used to cause harm
  • Recognize the pattern of how long-existing societal challenges (hate speech, disinformation, child abuse) can be amplified by internet technologies
  • Understand how to anticipate safety risks for a proposed product
  • Design and implement a functional abuse reporting flow powered by a machine learning classifier

T&S Challenges #

  1. Scale
    • How to reason about human-created issues at billions-scale?
    • Especially with lopsided ratio of T&S (or really any) employees to users
  2. Non-diverse studies and solutions
    • Solutions are broken – because procedures only reflect assumptions about people similar to designers
  3. Measurement and definition challenges
    • Can’t fix anything without measurement
    • But measuring abuse is extremely difficult!
  4. Privacy vs Safety
    • How to keep people safe while trying to respect privacy of their information?
    • End-to-end encryption adds challenges
  5. Information sharing and division of responsibility
  6. Government vs private action
    • In both security and T&S, private actors do 90% of the work and make most of the decision
    • Law enforcement has an important, but limited role
    • How to act in a quasi-governmental way while respecting privacy of users (many of whom do not live in democracies)?
  7. Fairness in ML solutions?
    • ML is critical to any reasonable T&S program
    • But: predictability and fairness is unclear
  8. Freedom of expression
    • T&S issues create broad agreement on certain positions (e.g., banning CSAM)
    • But: some issues are much more split and controversial (e.g. vaccine misinfo)
    • When acting in a quasi-governmental context, how to reason around people’s freedom of expression?

Areas of T&S work #

Policy and research #

  • Defining abuse types
  • Building measurements and metrics
  • Performing field studies
  • Interviewing users and victims
  • Working with governmental affairs teams
  • Provides data and ideas to product

Product and engineering #

  • Red teaming product designs
  • Designing UX to encourage good behavior
  • Building detection and moderation mechanisms
  • Building and training ML

Operations #

  • Defining appropriate behavior
  • Building operational pipelines
  • Sorting and handling billions of events
  • Implementing constant improvement through self-testing and QA

Investigations #

  • Investigates worst cases or most effective adversarial actors
  • Handles incoming LE requests and external referrals