%Master document for Workflow project. Begun on 2006-03-03.
%\input jslfloat
%\input jsl1in


\documentclass[11pt]{book}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{setspace}
\usepackage{makeidx}
\usepackage{amssymb}
\usepackage{eurosym}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{alltt}

\setcounter{MaxMatrixCols}{10}
%TCIDATA{OutputFilter=LATEX.DLL}
%TCIDATA{Version=5.00.0.2606}
%TCIDATA{<META NAME="SaveForMode" CONTENT="1">}
%TCIDATA{BibliographyScheme=Manual}
%TCIDATA{Created=Friday, March 03, 2006 09:15:48}
%TCIDATA{LastRevised=Thursday, June 26, 2008 18:08:35}
%TCIDATA{<META NAME="GraphicsSave" CONTENT="32">}
%TCIDATA{<META NAME="DocumentShell" CONTENT="Standard LaTeX\Standard LaTeX Book">}
%TCIDATA{Language=American English}
%TCIDATA{CSTFile=jslLaTeX Book.cst}
%TCIDATA{<META NAME="PrintViewPercent" CONTENT="100">}

\setcounter{topnumber}{1}
\renewcommand{\topfraction}{.99}
\setcounter{bottomnumber}{1}
\renewcommand{\bottomfraction}{.99}
\setcounter{totalnumber}{1}
\renewcommand{\textfraction}{.0}
\setlength{\topmargin}{-0.5in}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{9.0in}
\setlength{\oddsidemargin}{0.01in}
\setlength{\evensidemargin}{0.01in}
\raggedbottom
\newenvironment{proof}[1][Proof]{\noindent\textbf{#1.} }{\ \rule{0.5em}{0.5em}}
\input{tcilatex}
\pagenumbering{arabic}

\begin{document}

\frontmatter
\title{The Workflow of Data Analysis: Principles and Practice}
\author{J. Scott Long \\
%EndAName
jslong@indiana.edu}
\date{May 11, 2008}
\maketitle
\tableofcontents

\chapter{COPY\ FROM\ Edited tex for help fles}

\chapter{To clean}

\section{Automation}

\subsection{Information returned by Stata}
\begin{verbatim}
local mean_age = r(mean)
local sd_age = r(sd)
display "The mean of age `mean_age' (sd=`sd_age')"
\end{verbatim}

\subsection{Loops: foreach and forvalues}

foreach

\ \ \ \ foreach\ i\ in\ 1\ 2\ 3\ 4\ 5\ \{ ... \}

\ \ \ \ foreach\ i\ in\ var1\ var2\ var3\ var4\ var5\ \{ ... \}

\ \ \ \ foreach\ var\ of\ varlist\ lfp-inc\ \{ ... \}

\texttt{foreach} \textit{local-name} \texttt{of numlist} \textit{numlist} 
\texttt{\{} \} see help numlist

forvalues$^{{}}$

\texttt{forvalues} i = 1(1)10 \{...\}

\texttt{forvalues} i = 1/10 \{...\}

\texttt{forvalues} i = 1 3 to 11 \{...\}

Set wffrag for: counters in loops; loops to save a matrix

\section{Using multiple languages\label{wf5-language-sec}}
\begin{verbatim}
    . use wf-languages-spoken, clear
    (Workflow data with spoken languages \ 2008-04-03)
    . label language english
    . tabulate male, missing
\end{verbatim}

\texttt{\ \ .\ label\ language\ french}

\texttt{\ \ .\ tabulate\ male, missing}

\texttt{\ }

\texttt{\ \ \ \ \ Genre\ de\ \TEXTsymbol{\vert}}

\texttt{\ \ \ \ r\'{e}pondant\ \TEXTsymbol{\vert}\ \ \ \ \ \ Freq.\ \ \ \ \
Percent\ \ \ \ \ \ \ \ Cum.}

\texttt{\ \ ------------+-----------------------------------}

\texttt{\ \ \ \ \ 0-Femmes\ \TEXTsymbol{\vert}\ \ \ \ \ \ 1,227\ \ \ \ \ \ \
53.51\ \ \ \ \ \ \ 53.51}

\texttt{\ \ \ \ \ 1-Hommes\ \TEXTsymbol{\vert}\ \ \ \ \ \ 1,066\ \ \ \ \ \ \
46.49\ \ \ \ \ \ 100.00}

\texttt{\ \ ------------+-----------------------------------}

\texttt{\ \ \ \ \ \ \ \ Total\ \TEXTsymbol{\vert}\ \ \ \ \ \ 2,293\ \ \ \ \
\ 100.00}

\paragraph{Note on variable and value labels}

There is an important difference in how variable and value labels are
treated with languages. After changing to the \texttt{analysis} language, I
created new variable labels. For value labels, I\ must define labels with
different names than they had before. For example, in \texttt{%
wf-languages-analysis.dta}, the label assigned to \texttt{warm} was named 
\texttt{Swarm} (where \texttt{S} indicates that this is the source label).
In the \texttt{analysis} language, the label was named \texttt{Awarm}. With
multiple languages you must create new value label definitions for each
language.

\chapter[Cleaning your data \TEXTsymbol{\backslash} 9 May 2008]{Cleaning
your data \TEXTsymbol{\backslash} 9 May 2008\label{wf6-chapter}}

\section{Importing data\label{wf6-import-sec}}

\subsection{Verifying data conversion}

\paragraph{Step 1: Compare descriptive statistics from the source and
destination}

\paragraph{Step 2: Examine the distribution of missing values}

\paragraph{Step 3: Convert the data two ways}

\section{Verifying variables\label{wf6-verify-sec}}

\paragraph{Values review}

A values review examines whether the values held by each variable are
appropriate and whether all values that should be represented are found in
your data. For example, if you expect that some individuals will have a
Ph.D., but the variable for education has no cases in that category,
something is probably wrong.

\paragraph{Substantive review}

While all of the values that you find might be appropriate, is the
distribution of values substantively reasonable? Are there too many or too
few cases for some values? Does the range correspond to what you expect? Is
the variable appropriately related to other variables? While this step is
similar to a values review, it requires a substantive understanding of the
data.

\paragraph{Missing data review}

Do the data have the types of missing data that you expect and are the
number of cases for each type reasonable? Are the skip patterns from the
survey reflected in the missing value codes? For example, if a respondent
indicates that she did not attend college, the question on the type of
college attended should be skipped. This can lead to complex patterns of
missing data that you should verify.

\paragraph{Internal consistency review}

Information on one variable can have logical implications for the values of
other variables. For example, someone who did not attended college should
have missing values for a question about how she paid for college. In other
situations, there might be strong evidence from past research on how two
variables will be associated. Checking that responses are consistent across
variables can uncover subtle problems with the data.\medskip

\noindent Each type of review is discussed along with examples using Stata
to uncover problems.

\subsection{Values review}
\begin{verbatim}
    . codebook, compact
    . tab1 art, missing
    . stem cit
        dotplot `var'
 
\end{verbatim}

\subsection{Substantive review}

\subsubsection{Examining high frequency values}
\begin{verbatim}
  . tabulate phd if phd>4 & phd<4.5
\end{verbatim}

\subsubsection{Links among variables}
\begin{verbatim}
  . label var phd "phd: PhD prestige"
    . label var job "job: Prestige of first job"
    . dotplot phd job, xlabel(,labsize(medium))
    . scatter job phd
    . scatter job phd, msymbol(circle_hollow) jitter(8) ///
    >     ylabel(, grid) xlabel(, grid) aspectratio(1)
\end{verbatim}

\subsection{Missing data review\label{wf6-missingreview-subsec}}

Since problems with missing data occur in many subtle ways, the distribution
of missing values should be checked carefully. Let's start by discussing how
Stata handles missing values. The \textit{system missing value}, sometimes
called the \textit{sysmiss}, is indicated by \textquotedblleft \texttt{.}%
\textquotedblright . For example,%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    generate var01 = .
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
assigns all observations for \texttt{var01} to be missing. Stata also has 26 
\textit{extended missing values} indicated as \texttt{.a}, \texttt{.b},
through \texttt{.z}. Missing values cannot be used in numerical
calculations. For example, if you add \texttt{.a} to another number the
result is a missing value. Since missing values are not valid numbers,
commands such as \texttt{summarize} or \texttt{regress} cannot include a
missing value in numerical computations. This is in contrast to software,
especially older programs, where any number (e.g., 9 or 99) could be used as
a missing value and the user had to indicate which numbers were missing. If
you forgot to specify which values were missing, missing values were
included in the computations with sometimes disastrous results. If you
import data from a program in which valid numbers can be used to indicate
missing values, be careful that these values are recoded as missing values
in Stata.\noindent 

\subsubsection{Using extended missing values}

Data can be missing for many reasons and extended missing values can be used
to differentiate among the reasons for being missing. For example, a
response could be missing because:

\begin{description}
\item \textit{Refused}: The respondent refused to answer the question.

\item \textit{Don't know}: The respondent did not know the answer.

\item \textit{Not applicable}: The question was not appropriate for the
respondent (e.g., the question \textquotedblleft Where did you go to
college?\textquotedblright\ is not applicable if a person did not go to
college).

\item \textit{Not asked}: In a split ballot design, the respondent was not
asked the question.
\end{description}

\noindent Three commands are useful for examining the distribution of cases
among types of missing values. The \texttt{tabulate} and \texttt{tab1}
commands with the \texttt{missing} option tabulate both numeric and missing
values. If I want the distribution of \textit{only} missing values (e.g.,
for a variable that has thousands of valid values), I can add an \texttt{if}
condition (\textit{File}: \texttt{wf6-review-missing.do}):%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    tab1 phd if missing(phd), miss
\end{verbatim}

\subsubsection{Verifying and expanding missing data codes}

\subsection{Internal consistency review}

When there are logical links among variables, you should verify that your
data are consistent. For example, if I assume that formal education does not
begin until age five, years of education should be at least five years less
than a person's age. If not, there is probably an error in the data. Or, if
someone is not in the labor force, they should not report a wage. In other
situations, I expect a particular relationship between variables although
the relationship is not logically required. For example, I expect responses
to questions about attitudes on working mothers to be consistent in the
sense that people who are positive on one question will be positive on
related questions. Or, I do not expect a scientist to be hired in a
department that is substantially more prestigious than her doctoral origin.
In this section I illustrate some ways to look for such problems.

\subsubsection{Consistency in data on the scientific career}
\begin{verbatim}
    . assert cit==0 if art==0
    . assert art==0 if cit==0
    . tab cit if art==0, missing
   . compare job phd
\end{verbatim}

\subsection{Verifying that variables are correct\label%
{wf6-verify-vars-subsec}}

\subsubsection{Checking the code}

\subsubsection{Listing variables}
\begin{verbatim}
  . sort fincome
    . list fincome finc_mid if xselect<20, clean
 
              fincome   finc_mid
      92.      2_3-5K          4
     211.      3_5-7K          6
     242.      4_7-9K          8
     333.     5_9-10K        9.5
     479.    8_12-13K       12.5
     727.   12_17-20K       18.5
     819.   13_20-22K         21
     876.   14_22-25K       23.5
     930.   14_22-25K       23.5
    1105.   15_25-30K       23.5
    1118.   15_25-30K       23.5
    1174.   16_30-35K       32.5
    1236.   16_30-35K       32.5
    1338.   17_35-40K       37.5
\end{verbatim}

\ \ (\textit{output omitted})\medskip

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
I use this list to verify the values of the new variable.

\subsubsection{Plotting continuous variables}

For continuous variables, a plot is a useful way to verify a transformation.
For example,%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    generate inc_sqrt = sqrt(inc) if !missing(inc)
    label var inc_sqrt "Square root of inc"
    scatter inc_sqrt inc, msymbol(circle_hollow)
\end{verbatim}

\chapter{Analyzing data and presenting results \TEXTsymbol{\backslash} 10
May }

\section{Organizing do-files\label{ch7-organize-sec}Documentation for
statistical analysis}

\subsection{Documenting the provenance of results}

\subsubsection{Captions on graphs}

\section{Analyzing data using automation\label{ch7-analyze-sec}}

\subsection{Locals to define sets of variables}
\begin{verbatim}
    local varset "female male tenure year yearsq select articles prestige"
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
Then, my program becomes:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    //  #5
    //  desc statistics for men & women combined
    codebook `varset', compact
 
    //  #6
    //  desc statistics for women
    codebook `varset' if female, compact
 
    //  #7
    //  desc statistics for men
    codebook `varset' if male, compact
\end{verbatim}

\noindent ds are not changed, I know that the models are correctly specified.

\subsection{Loops for repeated analyses}

In statistical analyses, I often use the same command or series of commands
with multiple variables. An efficient way to do this is with loops which I
illustrate by computing a series of $t$-tests and estimating regression
models.

\chapter[Protecting your files \TEXTsymbol{\backslash} 11 May 2008]{%
Protecting your files \TEXTsymbol{\backslash} 11 May 2008\label{wf8-chapter}}

\section{Murphy's law and rules for copying files}

Any workflow to protect files must take Murphy's law very seriously.
Murphy's law states: \textquotedblleft If anything can go wrong, it
will.\textquotedblright\ Much can be learned from the context in which this
famous law emerged.\footnote{%
My history of the law is based on
www.murphys-laws.com/murphy/murphy-true.html which reproduces the article
\textquotedblleft Murphy's Law\textquotedblright\ from the March 3, 1978
issue of \textit{Desert Wings}.} In 1949 at Edwards Air Force Base, Murphy's
Law was named after Captain Edward A. Murphy who was an engineer working on
a project to see how much sudden deceleration a person can stand in a crash.
Dr. John Paul Stapp, who rode a sled that created 40Gs of deceleration and
lived, noted that the good safety record on the project was due to a
fundamental belief in Murphy's Law and \textit{extraordinary efforts to
circumvent it}. He coined Stapp's Ironical Paradox: \textquotedblleft The
universal aptitude for ineptitude makes any human accomplishment an
incredible miracle.\textquotedblright\ When it comes to saving your work,
expect things to go wrong, expect that you will delete the wrong file at the
worst possible time, and expect a hose to be left on in the room above your
computer. If you expect the worst, you might be able to prevent it. Based on
this notion and the sometimes painful experience of others, the following
rules should be followed no matter what approach you take to preserving your
files.

\paragraph{Rule 1: Make at least two copies}

Have at least two copies plus the original of all files.

\paragraph{Rule 2: Store the copies at different locations}

Many disasters affect everything at a location, so store copies in different
buildings. External drives can be stolen, so keeping all backup drives in
the same room is bad idea. Similarly, fire and water damage often affect
everything in a room.\footnote{%
Disasters can occur in places that seem very well protected. ICPSR recently
suffered damage when maintenance left a hose running on the floor above
their server room (Guttman, 2005, personal communication).}

\paragraph{Rule 3: Verify that copies are exact duplicates}

\section{A workflow for file protection}

\subsubsection{Part 1: Mirroring active storage}

\paragraph{Configurations for mirroring active storage}

The specific way that you mirror active storage depends on how many
computers you use, what storage devices are convenient, and the software you
use. Here are some alternative configurations, all of which provide
excellent protection.

\textit{Single computer mirror}: If you always work on the same computer, it
is easiest to keep your active files on an internal hard drive and to mirror
files to an external drive, a LAN, or other storage connected to the
computer.

\textit{Multiple computer mirror with LAN}: If you use two or more computers
that are connected to the same LAN, you can keep active files on the LAN and
use the internal hard drives on each computer to hold mirrored copies of the
active files from the LAN. Since you are using multiple computers, you have
extra security since files are mirrored multiple places. If the computers
you use do not allow you to save files to the internal drive (e.g., in a
computing cluster), you can create a mirror on an external or portable drive
that you carry with you. Alternatively, you can use the next configuration.

\textit{Multiple computer mirror with portable drive}: The best solution for
the way I work is to use a portable drive for active storage and carry the
portable with me as I\ move among computers. For years I transferred changed
or new files between computers using a ZIP disk or memory stick, but
regularly forgot to transfer the file I\ needed or was confused about which
computer had the latest version of a file. By using a portable drive for
active files and the internal drives on each computer to hold my mirror, I
always have the latest version of my files (unless I forget the portable).
On each computer's internal drive, I have a directory \texttt{\TEXTsymbol{%
\backslash}- Portable mirror} that is used for a continuous mirror when the
portable is attached to that computer. When I leave work, I eject the
portable drive, put it in my briefcase, and take normal precautions not to
drop the briefcase. At home, I plug in the portable drive and follow the
same procedures. When traveling, I take the portable drive with me and make
a mirror to the internal laptop drive. If I travel without a laptop, I take
the portable drive with me and use a memory stick to make duplicates of new
files I create.

\textit{Multiple computer mirror using the Internet}: If you use multiple
computers that are on the internet, you can use Microsoft's Foldershare or a
similar program to mirror files over the internet. Suppose you use one
compute at work and another at home. When using the computer at work,
Foldershare continuously synchronizes you active files to your home
computer. When you are home, you use the files on that computer as your
active storage and Foldershare mirrors changes to your work computer. I have
experimented with Foldershare and it is easy to use and efficient with a few
exceptions. First, the beta software limits you to 10,000 files per library
and 10 libraries. Second, if the internet connection goes down, the other
computer is turned off, or the software has a problem (which I found to
occur regularly with the beta version), you need to work without a mirror
until the problem is fixed. Third, if you use a laptop that is not always
connected to the internet, you must remember to update files to the laptop
before you use it. And fourth, if you work on computers that do not have
Foldershare installed, you will not have access to your files.

Other configurations can work equally well depending on how you work and
what technology you have available.

\subsubsection{Part 2: Offline backups}

The second part of the workflow makes backup copies of posted files and
snapshots (defined below) of files from active projects.

\paragraph{Posted files}

Posted files include the do-files, log files, datasets, text files and
others that you have used in your writing or shared with others so that they
should no longer be changed. The rule for posted files is simple: Once a
file is posted, it should \underline{\textit{never}} be changed. See Chapter
2 (page \pageref{wf2-posted-dir-page}) and Chapter 5 (page \pageref%
{wf5-posting-subsec}) for further details on posting. While I continue to
work on the project, I keep posted files from that project in my active
storage, but since these files are part of results that I have distributed
or shared with coauthors, I want to give them an extra level of protection
against hardware failures, viruses, or accidental deletion. About once a
month or when I finish a major task, I copy newly posted files from active
storage to a backup drive. If my archival files are not very large, I often
back them up along with the posted files.

The key to backing up posted files is to automate the process so that the
backup software can determine which files need to be copied. If you must
manually determine which files need to be copied, your workflow won't work
unless you are very patient and exceptionally thorough. The easiest method I
have found is to put posted files in subdirectories named \texttt{%
\TEXTsymbol{\backslash}Posted}. Suppose that I have the following
directories on my computer (where for simplicity I use a simple directory
structure):%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    \Projects
        \COGA
            \Posted
            \Work
        \EPSL
            \Posted
            \Work
    \Workflow
        \Posted
        \Work
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
My backup program only copies files from \texttt{\TEXTsymbol{\backslash}%
Posted} directories and automatically reproduces the directory structure
from active storage:%
%TCIMACRO{\TeXButton{single space}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    \Posted files
        \Projects
            \COGA
                \Posted
            \EPSL
                \Posted
        \Workflow
            \Posted
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5 spacing}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
If I add another \texttt{\TEXTsymbol{\backslash}Posted} folder to my active
storage, the next time I run the backup utility it will create the same
folder within \texttt{\TEXTsymbol{\backslash}Posted files} and will copy the
newly posted files.\footnote{%
The software I use can copy only files in directories that contain a
particular phrase, such as \textquotedblleft posted\textquotedblright .
Accordingly, it will copy files from \texttt{\TEXTsymbol{\backslash}Posted}, 
\texttt{\TEXTsymbol{\backslash}Text-posted}, etc, but not from \texttt{%
\TEXTsymbol{\backslash}Work}. I also use the rule that if a directory name
ends with \texttt{+} (e.g., \texttt{\TEXTsymbol{\backslash}Data+}), that
directory is treated as posted. See the Workflow website for details on
software.}

This plan for backing up posted files only works if your backup program can
automatically select all files located in directories named \texttt{%
\TEXTsymbol{\backslash}Posted}. On my computer, it takes my backup software
about a minute to search 4,000 folders to find which files need to be backed
up. If your software cannot select files to backup in this way (and not all
backup programs can), you need to organize your files some other way that
makes it easy to determine which files to back up. For example, you could
create a single folder \texttt{\TEXTsymbol{\backslash}Posted} and create
subdirectories for each project. I don't find this approach convenient, but
you might.

Before proceeding, I want to emphasize that this plan only works if you
stick to the posting principle: once a file is posted it cannot be changed.
For years I made the mistake of changing, renaming, or relocating
\textquotedblleft posted files\textquotedblright . Then, I needed to backup
the files again.\ What about the files I had backed up before with different
names. If I deleted them, old do-files would not work and my research logs
might point to a file that no longer exists. If I didn't delete them, I had
multiple files with the same information but different names. This process
wasted time and wasn't reliable. The solution was to follow the simple
principle for posting a file: \textit{After a file is posted, it can no
longer be changed. }If I find a mistake in a posted file, I can create a new
file with a new name, but I cannot change the original file. If I decide
that I no longer need a posted file, I can delete it but cannot change it.

\paragraph{Snapshots}

I also periodically take \textquotedblleft snapshots\textquotedblright\ of
all files, posted or not, associated with a project. For example, when I
completed the first draft of the workflow book on January 14, 2007, I backed
up all of the files in \texttt{\TEXTsymbol{\backslash}Workflow} to \texttt{%
\TEXTsymbol{\backslash}Snapshots\TEXTsymbol{\backslash}Workflow\TEXTsymbol{%
\backslash}2007-01-14}. If I later need a file that was used for this draft,
but that was not a posted file and was later deleted (e.g., a figure I did
not use), I check here. I also take snapshots before I start cleaning files
at the end of a project. That way if I make a mistake, I can easily recover
the file. My snapshot folder might look like this:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    \Snapshots
        \Workflow
            \2006-12-12
\end{verbatim}

\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (\textit{copy of all workflow
files on this day})
\begin{verbatim}
            \2007-01-14
\end{verbatim}

\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (\textit{copy of all workflow
files on this day})
\begin{verbatim}
        \EPSL
            \2004-06-02
\end{verbatim}

\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (\textit{copy of all epsl files on
this day})

\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ . . . . . .

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
I might keep the snapshots for a long time or delete the files after a few
weeks.

While I keep at least two copies of files located on different devices
stored in different locations, I do not worry about keeping the different
drives continuously synchronized. While keeping them synchronized would be
better, I find that this is impractical. Instead, I synchronize the drives
every few months. When a backup drives is full, I buy a new drive. With the
rapidly increasing capacity of hard drives, the new drive is usually at
least twice as large so it can hold all of my current files and have
capacity for at least that many more files. Using bit verification, I copy
all files from the current device to the new one. The old drive is stored as
\textquotedblleft deep backup\textquotedblright .

\paragraph{Configurations for backup storage}

You should have at least two copies of each file stored on media located in
different physical locations to protect against local disasters such as
water damage or theft. The drives are only plugged in when making copies or
recovering files, so they are not vulnerable to power surges or viruses.
Here are some configurations you can consider for your backup storage.

\textit{Backups using external drives}. My preferred approach is to use
multiple external drives to hold the two backup copies of files. These
drives are inexpensive, have fast read/write speeds, and are easy to move.
When I want to synchronize the drives,\ I carry the drive I\ store at home
to work and bring it back home after the two drives are synchronized.

\textit{Backups using a LAN}. If you have sufficient space on your LAN, you
can store one copy on the LAN\ and the other on an external drive or an
internal drive (assuming that drive isn't used for active storage).

\textit{Backups using enterprise mass storage}. If your organization
provides enterprise storage, such as tape backups, one copy could be stored
there and the other copy on another device.

\textit{Backups on the internet}. One backup copy could be saved over the
internet with a company that sells storage. As of mid-2008,
\textquotedblleft unlimited\textquotedblright\ storage costs less than \$5 a
month. The catch is that transfer speeds limit you to coping a maximum of
about 7GB of files a day (i.e., two weeks to copy 100GB, one year to copy
2.5TB). After initial copies are made, only changed files are transferred.
If you have a hard disk failure, it could take weeks to restore files over
the internet. Or, you can pay to have DVDs made and mailed. The software for
these sites only allows simple criteria for selecting files, so selecting
only files in \texttt{\TEXTsymbol{\backslash}Posted} folders would not be
possible. To find companies that sell internet storage, do a web search of
\textquotedblleft internet storage\textquotedblright , \textquotedblleft
online backup\textquotedblright ,\ or \textquotedblleft online
storage\textquotedblright . Keep in mind that if the company selling
storages goes out of business, you could lose your backups.

\textit{Backups on DVDs}. DVDs have the advantage that once you write to
them, the files cannot be deleted (assuming you avoid the multiple write
DVDs which provide more expensive and less stable storage). But, DVDs are
slow, they only hold about 5GB so you end up with a lot of disks to
organize, and quality DVDs are no longer cheaper per GB than hard drives.

\subsubsection{Stata directories}

Most of the time, you do not need to know where the Stata executable and
ado-files are located. But, sometimes this information is critical,
especially when something goes wrong or you are writing your own ado-files.
The \texttt{sysdir} command tells you where Stata stores files. On my
computer,%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    . sysdir
       STATA:  D:\Stata10\
     UPDATES:  D:\Stata10\ado\updates\
        BASE:  D:\Stata10\ado\base\
        SITE:  D:\Stata10\ado\site\
        PLUS:  D:\Stata10\plus\
    PERSONAL:  D:\Stata10\personal\
    OLDPLACE:  D:\Stata10\personal\
\end{verbatim}

\noindent

\section{Customizing Stata\label{wfa-customize-sec}}

\subsubsection{Options that can be set permanently}

For some \texttt{set} commands you can use the \texttt{permanently} option
so that the change will be remembered the next time you run Stata.

\paragraph{Format used for log files}

Log files can be text files or smcl files. To set the option:

\medskip\ \ \ \ \texttt{set logtype} \{ \underline{\texttt{t}}\texttt{ext} 
\TEXTsymbol{\vert} \underline{\texttt{s}}\texttt{mcl} \} [\texttt{, 
\underline{\texttt{perm}}anently} ]\medskip

\noindent Since I prefer text files, I run the command:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set logtype text, perm
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\paragraph{The number of commands in the Review window}

The number of commands that are saved in the Review window is controlled by:

\medskip\ \ \ \ \texttt{set reventries} \# [\texttt{, \underline{\texttt{perm%
}}anently} ] where 50$\leq $\#$\leq $500.\medskip

\noindent Unless you have very little computer memory, I suggest:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set reventries 500, perm
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\paragraph{Scrollback buffer in the Results window}

The number of lines of output that can be retrieved by scrolling the Results
Window is controlled by:

\medskip\ \ \ \ \texttt{set scrollbufsize} \# where 10,000$\leq $\#$\leq $%
500,000.\medskip

\noindent Unless you have very little computer memory, I suggest:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set scrollbufsize 500000
\end{verbatim}

\noindent%
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
This command is permanent even though it does not have a \texttt{permanently}
option.

\paragraph{Maximum size of matrices}

The largest matrix that Stata will use is controlled by:

\medskip\ \ \ \ \texttt{set matsize} \# [\texttt{, \underline{\texttt{perm}}%
anently} ] where 10$\leq $\#$\leq $1100.\medskip

\noindent If you are estimating complex models, the default \texttt{matsize}
might be too small. If you have enough memory, I suggest:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set matsize 1100, perm
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\paragraph{Memory}

Stata keeps all data in memory, rather than reading data from disk every
time it needs the information. If your dataset is large, you might run out
of memory. The amount of memory that Stata uses is controlled by:

\medskip\ \ \ \ \texttt{set memory} \#[ \texttt{b} \TEXTsymbol{\vert} 
\texttt{k} \TEXTsymbol{\vert} \texttt{m} \TEXTsymbol{\vert} \texttt{g} ] [%
\texttt{, \underline{\texttt{perm}}anently} ]\medskip

\noindent where \# is specified in terms of bytes, kilobytes, megabytes, or
gigabytes. If you routinely get an out of memory error, run \texttt{query}
to find out how much memory has been set. Then increase it until the out of
memory error is resolved. Once you find out how much memory you need, you
can make this permanent. For example,%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set memory 10m, perm
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\subsubsection{Options that need to be set each session}

These \texttt{set} commands need to be run each time Stata is started. They
can be put in your \texttt{profile.do} or \texttt{myprofile.do} (discussed
in Section \ref{wfa-profile-sec}).

\paragraph{Line size}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

The length of a line before wrapping is controlled by:

\medskip\ \ \ \ \texttt{set linesize} \#\medskip

\noindent I prefer:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set linesize 80
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\paragraph{Position of variable labels in Variables window}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

The distance from the variable name to the variable label in the Variables
window is controlled by:

\medskip\ \ \ \ \texttt{set varlabelpos} \textit{\#} for 8 $\leq $ \#$\leq $
32.\medskip

\noindent Where you set this depends on how long your variable names are. I
generally use:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    set varlabelpos 15
\end{verbatim}

%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\subsection{profile.do\label{wfa-profile-sec}}

When Stata is launched, it looks for a do-file named \texttt{profile.do}. If 
\texttt{profile.do} is found in one of the system directories, Stata runs
it. This allows you customize Stata by including commands in the \texttt{%
profile.do} that you want to run each time you start Stata. While you should
consult [GS] \textit{Getting Started with Stata} for full details or enter 
\texttt{help profile}, the following example shows you some things that I
find useful:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    //  these settings must be run each time Stata is loaded
        set linesize 80
        set varlabelpos 15
    //  function key definitions (discussed below)
        global F3 "codebook, compact"
    //  change to the working directory I want to use
        cd d:\stata_start
        exit
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\paragraph{myprofile.do for a network}

If you are working with a networked version of Stata, you probably are not
allowed to change the \texttt{profile.do}. As an alternative, you can create
a personal profile file, perhaps called \texttt{myprofile.do}, that you
store in your default working directory. When you load Stata and are in your
working directory, enter \texttt{do myprofile} .

\subsubsection{Function keys}

Stata allows you to associate strings of text with the function keys \textsf{%
F2} through \textsf{F9}. When you press one of these keys, that string is
inserted into the Command Window. To set this up, you create global macros
named \texttt{F2} through \texttt{F9}. When you press a function key, Stata
checks if the global has been defined. If it has, the content of the global
is pasted into the Command window. For example, to assign \texttt{codebook,
compact} to the \textsf{F3} key, I create the global:%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    global F3 "codebook, compact"
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion
After this global is defined, if I press \textsf{F3}, then \texttt{codebook,
compact} is written in the Command window. If I want \textsf{F3} to work
this way every time I load Stata, I can add this command to my \texttt{%
profile.do} (or \texttt{myprofile.do} if I am working on a networked version
of Stata).

If you use a different working directory for each of your projects,
functions keys are a convenient way to change directories. For example,%
%TCIMACRO{\TeXButton{1.0}{\singlespacing}}%
%BeginExpansion
\singlespacing%
%EndExpansion
\begin{verbatim}
    global F3 "cd e:\workflow\work"
    global F4 "cd e:\analysis\coga\work"
    global F5 "cd e:\teaching\s650\work"
\end{verbatim}

\noindent 
%TCIMACRO{\TeXButton{1.5}{\onehalfspacing}}%
%BeginExpansion
\onehalfspacing%
%EndExpansion

\end{document}
