Trust and Safety Engineering
What is Trust and Safety?
- The study of how people abuse the results to cause harm
- Often using products the way they are designed to work
- Crosses between specialties – requires understanding of society and humanity
- Dynamic and unpredictable
- Have empathy for a broad cross section of the people who use your products and the risks they face
- Explain to software engineers and product managers the common ways that internet technologies are used to cause harm
- Recognize the pattern of how long-existing societal challenges (hate speech, disinformation, child abuse) can be amplified by internet technologies
- Understand how to anticipate safety risks for a proposed product
- Design and implement a functional abuse reporting flow powered by a machine learning classifier
- How to reason about human-created issues at billions-scale?
- Especially with lopsided ratio of T&S (or really any) employees to users
- Non-diverse studies and solutions
- Solutions are broken – because procedures only reflect assumptions about people similar to designers
- Measurement and definition challenges
- Can’t fix anything without measurement
- But measuring abuse is extremely difficult!
- Privacy vs Safety
- How to keep people safe while trying to respect privacy of their information?
- End-to-end encryption adds challenges
- Information sharing and division of responsibility
- Government vs private action
- In both security and T&S, private actors do 90% of the work and make most of the decision
- Law enforcement has an important, but limited role
- How to act in a quasi-governmental way while respecting privacy of users (many of whom do not live in democracies)?
- Fairness in ML solutions?
- ML is critical to any reasonable T&S program
- But: predictability and fairness is unclear
- Freedom of expression
- T&S issues create broad agreement on certain positions (e.g., banning CSAM)
- But: some issues are much more split and controversial (e.g. vaccine misinfo)
- When acting in a quasi-governmental context, how to reason around people’s freedom of expression?
Areas of T&S work
Policy and research
- Defining abuse types
- Building measurements and metrics
- Performing field studies
- Interviewing users and victims
- Working with governmental affairs teams
- Provides data and ideas to product
Product and engineering
- Red teaming product designs
- Designing UX to encourage good behavior
- Building detection and moderation mechanisms
- Building and training ML
- Defining appropriate behavior
- Building operational pipelines
- Sorting and handling billions of events
- Implementing constant improvement through self-testing and QA
- Investigates worst cases or most effective adversarial actors
- Handles incoming LE requests and external referrals