1
AutoBash: Improving Configuration Management with Operating System Causality Analysis
Ya-Yunn Su, Mona Attariyan, and Jason Flinn University of Michigan
Motivation
• Configuration management is frustrating!
• Users may have to
– Change environment variables – Edit configuration files
– Manage inter-application dependencies
• Current approach:
– Ask friends, search on-line, read manual, … – Try potential solutions
– Carefully undo wrong solutions
3
AutoBash solves these problems Problems with current approach
• Applying solutions is time-consuming
• Undoing a wrong solution can be hard
• Hard to know how a problem was solved
• A “solution” may cause new problems Automatically tries many solutions
Provides undo capability Explains solution to user
Automatically runs regression tests
- ( •
• - ) • •
•
AutoBash overview
Health Monitoring Mode Periodically tests
system health Problem
Detected
Replay Mode
Automatically searches for a solution
Least user effort
Observation Mode Helps user fix the
problem Substantial user
involvement
5
Outline
• Motivation
• AutoBash design and implementation
– Observation mode – Replay mode
– Health monitoring mode
• Evaluation
• Conclusion
Observation mode
• A modified bash shell
– User types in commands to solve the problem
% command 1
% test if app works
% undo testing
% undo command 1
% command 2
7
Verifying a solution is tedious
• AutoBash automatically tests using predicates
• Predicate:
– Tests if an application functions correctly – Returns true/false if the test passes/fails
wget http://localhost/~$USER Predicate example for a web server
% command 1
% test if app works
% undo testing
% rollback command 1
% command 2
Undoing testing is tedious
• Predicate testing has no side effects
– Executed speculatively and rolled back
• Speculator [SOSP ‘05]
– Process-level speculative execution
• Speculative execution makes predicate testing safe
% command 1
% test if app works
% undo testing
% rollback command 1
% command 2
9
Undo can be hard
• AutoBash speculatively executes each action
– Light-weight checkpoint and rollback
• Speculative execution makes undo easy
% command 1
% test if app works
% undo testing
% undo command 1
% command 2
% rollback command 1
Regression testing is hard
• AutoBash automatically runs regression tests
– Executes predicates in the predicate database – Ensures all predicates pass
Predicate Database
P0 P1 P2
P3 P4 P5
CVS predicates
GCC cross-compiler predicates
Web server predicates
P6 P7 P8
11
Regression tests can be slow
• Problem: running all predicates can be slow
• Only need to run predicates affected by an action
– Uses causality tracking to find affected predicates
Tracking causality
• Output set
– kernel objects an action causally affects
• Input set
– kernel objects a predicate causally depends on Predicate: grep “test” bar
Output set = {file foo}
Action: touch foo
Input set = {file bar}
13
Analyzing causality
• AutoBash calculates the intersection
– Determines which predicates to run
Action:
touch foo
file: foo file: bar
Predicate:
grep “test” bar
Do not run predicate
sh sh
exit
touch
fork & exec
touch
exit
Tracking output sets
• An output set is tracked for each action
Process sh
Output set
file foo
create
Process touch File metadata foo
File content foo Directory entry foo
Action: sh create_file.sh create_file.sh: touch foo
15
sh Predicate: sh testfile.sh sh
testfile.sh: grep “test” bar
Process grep Process sh
File metadata bar File content bar Directory entry bar
Process grep’s input set
Process grep Process sh
grep
fork & exec
grep
exit
Tracking input sets
• An input set is tracked for each predicate
Predicate’s input set
file bar
read
Process grep File metadata bar
File content bar Directory entry bar
Process sh
File metadata bar File content bar Directory entry bar
Understanding solutions can be hard
• AutoBash generates causal explanation
– Analyzes input and output sets
17
Causal explanation
ls –l
/home/$USER
metadata:
/home/$USER/
public_html
wget index.html
metadata:
/home/$USER chmod +x /home/$USER
chmod +x /home/$USER/
public_html
wget
~$USER/cgi-bin/test.pl wget
~$USER/index.html
Actions
Kernel objects
Predicates
Outline
• Motivation
• AutoBash design and implementation
– Observation mode – Replay mode
– Health monitoring mode
• Evaluation
• Conclusion
19
Replay mode
• Problem: finding a solution is time-consuming
• Automatically searches for a solution
– No user input needed
• Speculative execution provides isolation
– User continues foreground task
– AutoBash runs replay mode in background
(1) Initial predicate testing:
• Tracks input set for each predicate
• Determines passed/failed predicates S2
S1 S0
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
P0 P1 P2 P3 P4
21
(2) Solution execution:
• Speculatively executes a solution
• Tracks solution output set S2
S1 S0
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
S0 (3) Verifying solution:
• Calculates intersection
• Runs predicates with intersection
P0 P2 S2
S1
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
Predicate fails
23
P0 P2 S2
S1 S0
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
Discards solution with no intersection
P1 P3 P4 P0 P2 P0 P2 (4) Regression tests:
• Calculates intersection
• Runs predicates affected by solution S2
S1 S0
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
P0 P1 P2 P3 P4
Predicates pass Predicate passes
25
• Speculative execution provides safety
• Causality analysis provides speed S2
S1 S0
P0 P2
How replay mode works
P0 P1 P2 P3 P4
Predicate Database Solution Database
S0 S1 S2 S3 S4 S5
P0 P1 P2 P3 P4
P1 P3 P4
P0 P1 P2 P3 P4
Predicate passes S2
Health monitoring mode
• Periodically executes all predicates
• If any predicate fails, AutoBash
– Runs replay mode to search for a solution
– Reports to the user to run observation mode
27
Outline
• Motivation
• AutoBash Design and Implementation
– Observation mode – Replay mode
– Health monitoring mode
• Evaluation
• Conclusion
Evaluation
• Questions:
– What is the overhead of speculative execution?
– How effective is causality analysis?
• Methodology:
– Evaluated CVS, gcc cross compiler, web server – Manually created 10 bugs and 10 solutions
– Manually created 5-8 predicates
29
0 20 40 60
Bug
Time (seconds)
Predicate re-testing Solution execution
Initial predicate testing
Total replay time (GCC)
Without speculative execution With speculative execution
Speculative execution overhead is negligible
1 2 3 4 5 6 7 8 9 10
0 20 40 60
Bug
Time (seconds)
Predicate re-testing Solution execution
Initial predicate testing
Total replay time (GCC)
Causal analysis improves predicate re-testing time by 67-99%
Without causality analysis With causality analysis
1 2 3 4 5 6 7 8 9 10
31
