Data Safety with AI

Recap

recap · parts 01 + 02

You can conduct agents and encapsulate skills.

Agentic Basics

Agents build. You steer.

Skill Usage and Design

Design once, summon by name.

Part 03: the agent meets your real data.

Today

Three habits keep AI-assisted data science safe.

Scripts, Not Results

Ask how the answer is made

Keep Real Data Out

Structure goes to the agent, content stays home

You Are the Pilot

AI expedites the work; you are responsible for it

Use agents wisely

Mentors

instructors · managers

"Please do not use AI"

Impossible to enforce

Makers

students · engineers

Already using AI in learning and work

Engaging is the smart move

How to avoid AI?

→

How to use AI wisely?

Use GitHub

Manual cloud storage

Nothing uploads until you commit and push

Traces every change

Review or discard file edits

A place to share

Share your works to the community by public repos

Download GitHub Desktop to get started.

GitHub Extensive Features

Branches

Merges

Forks

Pull Requests

Issues

Pipeline: Data Viz based on raw data

The structure

project/

input/

data.csv

script.R

output/

figure.png

The script

script.R

library(tidyverse)

data <- "input/data.csv" %>%
  read_csv() %>%
  count(group, choice)

data %>%
  ggplot(aes(group, n)) +
  geom_col(aes(fill = choice))
ggsave("output/figure.png")

Pipeline: Data Viz based on raw data

data.csv

figure.png

group	choice
A	yes
A	maybe
B	no
⋮

script.R

Same workflow, with or without AI

data.csv

script.R

figure.png

Then

handcrafting

Set up the work tree by hand

Build the scripts by hand

Run it by hand

Verify the output by hand

Now

agentic crafting

Set up the work tree with AI

Build the scripts with AI

Run it with AI

Verify the output still you

Enhanced by agents: writing code

Agents rarely make a syntax mistake.

no typos

no missing brackets

instant setup scripts

but...

You are the gatekeeper for the coding results.

Enhanced by agents: organizing files

input/

acs-2019.csv

acs-2021.csv

nhts-2022.csv

evse-2023.csv

45 more

scripts/

01-clean.R

02-merge.R

03-visualize.R

04-model.R

README.md

output/

summary.csv

figure.png

model.rds

report.html

table.tex

Enhanced by agents: sanity checks

sanity-checks.R

rows in = rows out 1,200

join key unique no dupes

share column in 0 to 1 runs 0 to 100

Written and run in seconds. You decide what is plausible.

You will use agents

Doing all the data work by hand is no longer realistic

The strength

Hours where hand work took weeks

Iterates tirelessly until the pipeline runs

The risk

Sensitive data can leak into the chat

Numbers nobody can trace or defend

The script, not the result

Asking for results

Clean this data and show me the table. What is the effect of X on Y?

✗ A number you cannot trace

Asking for scripts

Plan the cleaning steps and write an R script. I will run it and read it.

✓ A pipeline you can audit

Not what the answer is. How the answer is made.

What the agent hands back

The scripts

Python or R: readable, rerunnable

The structure

Source, generated, support: what feeds what

The judgment

Is the processing reasonable? Yours to answer

Can't point to all three? You got a result, not an analysis.

A structure you can audit

project/

data/raw/ source · read-only

scripts/01-wrangle.R the product

scripts/02-visualize.R

scripts/03-model.R

data/derived/ generated

output/figures/ · output/tables/

The test: delete everything generated. Can the scripts rebuild it all?

Every stage, same rule

One question at every stage: can you trace how this was made?

Wrangling

Scripts, not pasted-back tables

Visualization

Scripts, not one-off images

Modeling

Scripts, not quoted coefficients

Some data can never leave

Government records

Classified or restricted microdata

Health and personal data

HIPAA, IRB protocols, PII

Proprietary business data

Contracts and NDAs bind it

Human subjects research

Consent never covered a model provider

Safe assumption: anything that enters the chat has left the building.

Leaks happen quietly

Paste a few sample rows, just so it sees the format

Let the agent read freely in the folder holding the real files

Upload the CSV just this once, to debug a parsing error

Screenshot the spreadsheet to ask about one weird value

Each one feels harmless. Each one is the whole leak.

The fake-data workflow

1

Fake the content, keep the structure: same columns, fabricated rows

2

The agent plans, writes, and tests the whole pipeline on fake data

You run the real data yourself: same scripts, offline, no agent

The agent got the structure. It never saw a single real row.

Same structure, fake content

Stays with you

benefits.csv

idageincomeenrolled
103824782,4001
103836139,7500
............

Real rows: read only by scripts you run

Goes to the agent

fake_benefits.csv

idageincomeenrolled
900013551,0001
900025267,5000
............

Same columns, same types, fabricated values

Pipelines run on structure, not on content: swap the path and it just works.

More shields

De-identify first

Strip names, IDs, exact dates before sharing

Schema only

Column names and types, zero rows

Deny by config

Block the data folder from the agent outright

Keep it local

On-prem models: nothing leaves the machine

Shields stack: layer them to match the sensitivity.

The principle

"

Give the agent your structure. Never your content.
The pipeline transfers. The rows do not.

Co-pilot, not autopilot

The co-pilot: AI

Drafts the scripts. Remembers the syntax. Flies fast.

in command

The pilot: you

Sets the course. Makes the calls. Owns the landing.

Your name on the results

Data

→

Wrangle

→

Visualize

→

Model

→

Conclude

Every arrow must survive your questions.
When someone asks why, the answer comes from you, not a chat log.

The hand-craft test

Could you do this step yourself, without AI?

If yes

Delegate freely: it types, you think

You know what correct looks like

If no

Stop: learn the method first

You cannot verify what you could not craft

AI changes the speed. It does not change your responsibility.

How it goes wrong

friday · 4:55 pm · report due monday

# a restricted benefits database, a deadline, an agent
> here are 200 rows from benefits.csv so you get the data [pastes real rows]
  ✗ restricted data is now in the chat
> what's the effect of the program on income? just give me the number
  ✗ asks for the what, not the how
The effect is +$1,240 (p = 0.03).
> perfect, thanks! [pastes the number into the report]
  ✗ ships a number nobody can defend

Nothing errored. Nothing warned. The work looks done.

Three violations, one chat

✗

Real rows left the building

Breaks Keep Real Data Out: one paste was the whole leak.

✗

Asked for the number, not the pipeline

Breaks Scripts, Not Results: no script exists, nothing can be reproduced.

✗

Shipped a result they cannot defend

Breaks You Are the Pilot: the co-pilot flew, nobody was piloting.

The fix costs minutes: fake data, real scripts, run it yourself.

Three takeaways

✓

Scripts, not results

Ask how the answer is made; judge the reasonableness yourself.

✓

Keep real data out

Structure goes to the agent; content stays home.

✓

You are the pilot

AI expedites the work; everything ships under your name.

Thank you

"

AI is a brilliant co-pilot: it makes the whole flight faster.
But the destination, the judgment calls, and the landing
belong to the pilot. That is you.

thank you

Pingfan Hu · Dr. John Helveston

   Agentic Workflows with Claude Code  ·  The George Washington University
  