Using Files

from Essential Mathematica for Students of Science
© James J. Kelly, 1998
Last revised: January, 2006

Initialization

ClearAll["Global`*"];
Off[General::spell, General::spell1]

Needs["Graphics`"]

The following strings identify some of the directories used in this lesson.  You will need to modify them for your own system!

EssentialMathematica = "C:\Documents and Settings\Jim\My \
Documents\My eBooks\EssentialMathematica";
testdir = "C:\Documents and Settings\Jim\My \
Documents\ScratchTest";

File management

All modern computer systems group files in a hierarchical organization of folders or directories, but the details of accessing these files depends upon the operating system.  Mathematica provides functions which handle most of the system-dependent aspects of file management in a uniform fashion, except that pathnames remain system dependent.

Directories

First, we need functions which access the directories.  Most of these can be found using the information escape with wildcards.

?*Directory*

First, we ask for the current directory — your answer will almost certainly be different and provides a template for proper specification of other directory locations on your operating system.

Directory[]

Next, we choose a directory using SetDirectory.  To do this you must use notation for a path that is appropriate to the operating system on your computer.  On a Windows system you might enter
	SetDirectory["C:\My Notebooks"]  
if that folder exists.  Try setting the directory to the one containing the notebooks for this course.  As an example, here is the directory on my laptop.

SetDirectory[EssentialMathematica]

You can use ResetDirectory[] instead to restore the previous definition, which works if the directory has not been changed more than once.  Sorry for the inconvenience of resetting the working directory, but Mathematica does not support symbolic names within hyperlinks.

Files

To obtain a list of most of the available file-manipulation functions, we again request information using wildcards.

?*File*

Use FileNames with an empty argument to list all the files in the current directory:

FileNames[]

If a name or pattern is given, only matching files are listed.  The wildcard character (*) can be used to identify files by extension or by partial names.

FileNames["*.nb"]

FileNames["a*"]

FileNames[forms,directories,n] searches a list of directories, including n levels of subdirectories, for files matching any member of list of forms.  The curly braces may be omitted when a list contains only one element.

FileNames[{"*.nb", "*.txt"}, {"Applications", "Additional \
topics"}] // ColumnForm

FileNames["*.m", ".", 2]

Create a new directory and manipulate some files

CreateDirectory[testdir];

SetDirectory[testdir]

FileNames[]

\(CopyFile[
      EssentialMathematica <> "\Applications\Julia.nb", "temp.np"]\)

FileNames[]

RenameFile["temp.np",  "temp.nb"]

Paths

Files can often be found without specifying their locations completely if the directory containing them is included in $Path.  Thus, normally we need not specify the directory for one of the standard packages or for one of the extra packages installed with Mathematica because the relevant directories are usually included in the path.  At the time this was written, the path was as follows.

$Path

Suppose that you have created a directory that contains packages you have either written or acquired from external sources.  You may then find it convenient to append that directory to the path string.  For the purposes of illustration, we copy a file into testdir and add that directory to the path.

CopyFile[EssentialMathematica <> "\Additional topics\Fractals.m", 
      testdir <> "\Fractals.m"]

AppendTo[$Path, testdir]

$ContextPath

Any package in that directory can now be loaded without including the directory name.  If you know the context, you can use Needs; otherwise use Get (<<) to load the appropriate file and then inspect $ContextPath to determine the context.

Needs["Fractals`"]

$ContextPath

We now find several new functions, some with usage messages.  Options are also listed but usage messages were not provided for them.

?Fractals`*

DisplayKoch[{{0. , 0. }, {0.5, ((Sqrt[3])/2.) }, {1. , 0. }}, 4, 
        ShowIterations -> False, KochType -> "sieve", 
        CloseFigure -> True];

Create a new directory and add it to the path.

Files using Mathematica formatting

Storing data

The command Put[expr, "filename"] puts the result of evaluating expr into a file with the name filename.  If the file does not exist, Put creates it in the directory returned by Directory[].  If the file does exist, Put deletes its contents and replaces them with the evaluation of expr.  If you want to append the evaluation of expr to the contents of an existing file without losing the original contents of that file, you must use PutAppend[expr, "filename"].

A convenient shorthand for Put is expr>>filename and for PutAppend is expr>>>filename.  Note that quotes are not needed for these shortcuts.  To display but not read (input) the contents of a file, use the command !!"filename".

Exercise:  write an unformatted file.

data = 
        Table[Sin[2  π 220 t]  +  Sin[2  π 220.5 t], {t, 0, 5, 
            (10^(-3))}];

Reading data

The command Get["filename"] reads a file and provides input for defining a symbol that stores the contents of the file.  Evaluation of
	data = Get["filename"]
will store the contents of filename in the symbol data.  The shorthand for this expression is
	data = <<"filename"

Exercise:  read from an unformatted file.

ListPlay[data];

Storing definitions

The assignment statement of a symbol is not saved with Put (>>); only its evaluation is saved.  The assignments of symbols (variables and functions) can be saved to a file by this command.
    	Save["filename", {s1, s2, s3,⋯ }]
Symbols can represent simple assignments, functions, procedures, graphical data, or any other Mathematica object. Note that the name of the file and the symbol to be saved are passed to Save in the opposite order from Put and PutAppend.  Also, note that Save appends definitions to a file; it does not replace the file contents with the definitions.  When the file is read into Mathematica using
	<<"filename" 
all these assignments will be activated. If you Save a symbol whose definition uses other symbol definitions, then Save also saves these definitions.  (Section 2.11.1)

Exercise: storing and retrieving definitions

Z1rot[y_] := 
      NSum[(2 j + 1) Exp[-j (j + 1) y], {j, 
          0, ∞}]

F[T_, Z_] := -N (Subscript[k, B]) T Log[ Z[T] ];
S[T_, Z_] := -(∂_T) F[T, Z];
U[T_, Z_] := F[T, Z] + T S[T, Z];
CV[T_, Z_] := (∂_T) U[T, Z]

TableZ1RotLowT = 
        Table[{T, Z1rot[y] /. y -> (1/T)}, {T, 0.05, 0.5, 
            0.05}];

TableZ1RotMidT = 
        Table[{T, Z1rot[y] /. y -> (1/T)}, {T, 0.6, 2.9, 
            0.1}];

TableZ1RotHighT = 
        Table[{T, Z1rot[y] /. y -> (1/T)}, {T, 3, 10, 0.5}];

TableZ1Rot = 
        Join[TableZ1RotLowT, TableZ1RotMidT, TableZ1RotHighT];

Z1RotInterpolatingFunction = 
        Interpolation[TableZ1Rot, InterpolationOrder -> 5];

SetDirectory[testdir];

Save["zrot", {Z1rot, CV, TableZ1Rot, 
        Z1RotInterpolatingFunction}]

!! zrot

Remove["*Z1*", T, S, U, F, CV]

?*Z1*

<< "zrot";

?Z1rot

?CV

?F

\(\(Plot[
        Evaluate[CV[T, Z1RotInterpolatingFunction]\/\(N\ k\_B\)], {T, 0.05, 
          5}, 
	PlotLabel -> "Rotational Heat Capacity", 
	
        PlotRange -> {{0, 5}, {0, 1.2}}, 
	GridLines -> None, 
        Frame -> True, 
	
        FrameLabel -> {\*""T/(Subscript[T, rot])"", \*""(Subscript[C, V])/(N (Subscript[k, B]))""}];\)\)

Hence, this function is now available for further use here or in other notebooks.

ASCII SDF Files

The simplest type of data files are written in plain text (ASCII format) and consist of a sequence of lines, called records, each containing the same number of entries in a common format.  Formats of this type are described as sequential data format (SDF) and can be read easily by a large variety of programs or software packages.  Variations of sequential data format might also include titles and column headings, often called headers, and perhaps endnotes, often called trailers.  Sometimes the sequence of header, data, trailer is repeated several times in a blocked SDF file.  However, these variations entail complications of the reading procedures and require more specialized software.  Therefore, in this section we consider only the simplest type of sequential data files in which numerical data is arranged in columns without headers or trailers.  More general file formats are discussed in the next section.

Writing an SDF file

The simplest way to store a data table in an SDF file is to use TableForm to format the data and use OutputForm to deliver the data to the file as it would appear on the screen.
	OutputForm[TableForm[data]] >> "filename"

For example, here we create an ASCII SDF file that contains a multiplication table

data = Table[i j, {i, 12}, {j, 12}];
OutputForm[TableForm[data]] >> "multiplication.dat"

and examine the contents of that file.

!! "multiplication.dat"

To gain more control over the output format, use OpenWrite["filename",options] to prepare a file to receive data.  Note that if the file already exists, OpenWrite erases the file and prepares a fresh version; if you need to append data to the end of an existing file, use OpenAppend instead.  Often one chooses FormatType->OutputForm and then does not need to use OutputForm on each expression.  Sometimes it is desirable to increase PageWidth, perhaps to ∞, to ensure that each row of table occupies a single line or record without undesirable line feeds.

Options[OpenWrite]

Then use one or more Write["filename",expr1,⋯] statements to save your data, and finally release the file using Close["filename"].

Create a 100×100 matrix of random integers in the range {1,100}.  Open a file using FormatType->OutputForm and PageWidth->∞, write the matrix to the file using TableForm, close the file, and then check its contents and format.  How does it differ from a similar file written with default options?

Reading an SDF file

To read numerical data from a file with each record stored as a sublist, you can use the function ReadList["filename", Number,RecordLists->True].  Here, Number identifies the data entries to be numerical (exact or approximate) and RecordLists->True indicates that each record is to be returned as a sublist.  Variations of ReadList which handle more general data types are discussed in the next section.

ReadList["multiplication.dat", Number,  
      RecordLists -> True]

Gently down the stream

Streams and data types

Perhaps the most flexible file concept is that of a stream which contains a sequence of data of various types.  A file can be attached to an input stream using OpenRead with basic syntax
	streamname = OpenRead["filename"]
A single datum is then obtained using Read[streamname].  Alternatively, Read[streamname,type] reads one expression of specified type while  Read[streamname,{type1,type2,⋯}] reads a sequence of expressions of specified type.  A list of the available data types is given in the following table.  The default data type for Read is Expression.

({{Data Types}, {({{Type, description}, {Number, approximate or exact number}, {Real, approximate real number}, {Byte, single byte returned as integer}, {Character, single character returned as string}, {Word, \(character\ sequence\ delimited\ by\ word\ separators\)}, {Record, \(character\ sequence\ delimited\ by\ record\ separators\)}, {String, string terminated by newline}, {Expression, complete Mathematica expression}, {Hold[Expression], unevaluated Mathematica expression}})}})

To illustrate these data types, examine the file "sample data types.txt" — you will need to specify the appropriate directory on your system.

SetDirectory[EssentialMathematica];

\(\(!! Additional\ topics\sample\ data\ types.txt\)\)

Attach this file to an input stream.

samples = 
      OpenRead["Additional topics\sample data types.txt"]

We read the first line using a sequence of data types.  Notice that the first datum is interpreted as an integer because it is declared as type Number, which includes both exact and approximate numbers, while the second datum is interpreted as an approximate real number because it is declared as type Real even though the input did not include a decimal point.  Also notice that both Fortran and C formats for real numbers are recognized and interpreted properly.

Read[samples, {Number, Real, Real, Number, Number}]

The next line contains Mary Poppins' favorite word, which is returned as a string.

Read[samples, Word]  // FullForm

Additional words on the same line can be read, but care must be taken with the delimiters.  The next word does not include the surrounding spaces because it is read with Word type, but the final word includes a leading space because it is read as a String which is terminated only by the end of line.  However, if we had read it as a Word, the end-of-line character would have been the next item in the stream.  Also note that reading Byte and Character data can be tricky because spaces, tabs, carriage returns, and line-feeds are represented as byte data.

Read[samples, {Word, String}]  // FullForm

Read[samples, String] // FullForm

Read[samples, Record] // FullForm

Mathematica expressions are evaluated

Read[samples, Expression]

unless explicitly held unevaluated.

Read[samples, Hold[Expression]]

Functions like Read, Skip, and Find operate in a sequential fashion, advancing the stream pointer as data is read.  You can position the stream pointer manually using SetStreamPosition[streamname,n] where n is the position in bytes relative to the beginning of the stream.  Stream position 0 indicates the beginning and ∞ the end.

SetStreamPosition[samples, 0];
Read[samples, Record] // FullForm

SetStreamPosition[samples, 75];
Read[samples, Record] // FullForm

Manipulating formatted data

Often one needs to extract data from a file with an inconvenient format, perhaps prepared using a text editor.  Although there is no general solution to this problem, we illustrate some possible strategies by extracting population data from a file downloaded from the web site of the U.S. Census Bureau.

First, open an input stream.

pop = OpenRead["Additional topics\popclock.txt"]

Next, read the entire file into a list of records.

poprecords = ReadList[pop, Record, RecordLists -> True]

Examining this file we find that there are 5 records containing titles and column headings followed by about 98 records containing data in a fixed columnar format.  The file concludes with a set of notes and bibliographic information we do not need.  Hence, the next step is to drop the records containing uninteresting text.

popdata = Drop[Drop[poprecords, 5], -11]

Note that each data record is represented as a list containing a single string.

FullForm[popdata[[1]]]

FullForm[
      popdata[[2, 1]]]

The commas used to format dates and numbers are rather inconvenient, but can easily be removed by mapping StringReplace onto a list of strings.  Note that one level of listing is removed first using Flatten.

\(\(\(\ \)\(newpopdata = \(\(\((StringReplace[#, "," -> ""] &)\) /@ \ Flatten[popdata]\)\(\ \)\)\)\)\)

Next, each string can be treated as an input stream using StringToStream and then read using Read.  Minor problems occur at the end because of the change of format, but they can be ignored.

newerpopdata = (Read[#, {Word, 
                Table[Number, {5}]}] &) /@  (StringToStream  /@  
            newpopdata)

Finally, we can extract the items we want; namely, the year and the total population.

data = Part[#, 2, {2, 3}] &  /@  newerpopdata

ListPlot[data];

This entire procedure can now be packaged into a single module in case we might need to apply it to a file of similar format.  Additional arguments can be added to specify the desired columns.  With relatively minor modifications this module can be adapted to handle files of similar structure.

\(GetPopData[file_String, nSkipFront_Integer, nSkipBack_Integer] := 
      Module[{pop}, 
        pop = OpenRead[
            file]; \(Part[#, 
              2, {2, 3}] &\)\  /@ \((\((Read[#, {Word, 
                      Table[Number, {5}]}] &)\) /@ \ \((StringToStream\  /@ \ \(\((StringReplace[#, "," -> ""] &)\) /@ \ 
                    Flatten[
                      Drop[Drop[
                          ReadList[pop, Record, RecordLists -> True], 
                          nSkipFront], \(-nSkipBack\)]]\))\))\)]\)

popdata = 
      GetPopData["Additional topics\popclock.txt", 5, 11]

Plot the annual change in population, either absolute or relative, using these data.

Import and Export

Versatile built-in functions for importing and exporting data using standardized file formats became available in Version 4 and more recent versions handle a greater variety of file formats.  In fact, there are so many formats and conversion options that we must refer you to the reference guide for the details of Import and Export.  The import formats available on your machine are listed by

$ImportFormats

and a similar list of export formats can also be obtained in the obvious manner.  To illustrate these functions, let us first export the population data from the previous section to a data files in our test directory.

SetDirectory[testdir];
Export["popdata.dat", popdata];

The default format is based upon the file extension, which here simply contains a column formatted table, as shown below.

!! popdata.dat

Alternatively, we can export this data to an Excel file as follows.  Open the output file with Excel to verify that this operation succeeded.  Obviously, this method is easier that stream input when the file formats are simple or standard; on the other hand, stream input/output is more flexible.

Export["popdata.xls", popdata, "XLS"];

Similarly, we can import pictures or other specialized files either with standard filename extensions or by specifying the appropriate format as follows.

me = 
        Import[EssentialMathematica <> "\Additional \
topics\jjk.jpg"];
Show[me];

Finally,

Export["jjk.eps", me];

converts a JPEG file to encapsulated postscript; again, verify using a suitable viewer.