6.3 Master Label Files
6.3.1 General Principles of MLFs
Logically, the organisation of data and label files is very simple. Every data file has a label file of the same name (but different extension) which is either stored in the same directory as the data file or in some other specified directory.
ice cream
(a) 1-alternativ
e, 1-level
ice cream
(b) 1-alterna
tive, 2-level ay
s
k
r
iy
m
ey
es cream
(c) 3-alterna
tive, 1-level
ice cream
I
scre
am
Fig. 6.1 Example Transcriptions
This scheme is sufficient for most needs and commendably simple. However, there are many cases where either it makes unnecessarily inefficient use of the operating system or it seriously inconveniences the user. For example, to use a training tool with isolated word data may require the generation of hundreds or thousands of label files each having just one label entry. Even where individual label files are appropriate (as in the phonetically transcribed TIMIT database), each label file must be stored in the same directory as the data file it transcribes, or all label files must be stored in the same directory. One cannot, for example, have a different directory of label files for each TIMIT dialect region and then run the HTK training tool HERest on the whole database.
All of these problems can be solved by the use of Master Label Files (MLFs). Every HTK tool which uses label files has a -I option which can be used to specify the name of an MLF file. When an MLF has been loaded, the normal rules for locating a label file apply except that the MLF is searched first. If the required label file f is found via the MLF then that is loaded, otherwise the file f is opened as normal. If f does not exist, then an error is reported. The -I option may be repeated on the command line to open several MLF files simultaneously. In this case, each is searched in turn before trying to open the required file.
MLFs can do two things. Firstly, they can contain embedded label definitions so that many or all of the needed label definitions can be stored in the same file. Secondly, they can contain the names of sub-directories to search for label files. In effect, they allow multiple search paths to be defined. Both of these two types of definition can be mixed in a single MLF.
6.3 Master Label Files 83 MLFs are quite complex to understand and use. However, they add considerable power and flexibility to HTK which combined with the -S and -L options mean that virtually any organisation of data and label files can be accommodated.
6.3.2 Syntax and Semantics
An MLF consists of one or more individual definitions. Blank lines in an MLF are ignored but otherwise the line structure is significant. The first line must contain just #!MLF!# to identify it as an MLF file. This is not necessary for use with the -I option but some HTK tools need to be able to distinguish an MLF from a normal label file. The following syntax of MLF files is described using an extended BNF notation in which alternatives are separated by a vertical bar |, parentheses ( ) denote factoring, brackets [ ] denote options, and braces { } denote zero or more repetitions.
MLF = “#!MLF!#”
MLFDef { MLFDef }
Each definition is either a transcription for immediate loading or a subdirectory to search.
MLFDef = ImmediateTranscription | SubDirDef
An immediate transcription consists of a pattern on a line by itself immediately followed by a transcription which as far as the MLF is concerned is arbitrary text. It is read using whatever label file “driver” routines are installed in HLabel. It is terminated by a period written on a line of its own.
ImmediateTranscription = Pattern Transcription
“.”
A subdirectory definition simply gives the name of a subdirectory to search. If the required label file is found in that subdirectory then the label file is loaded, otherwise the next matching subdirectory definition is checked.
SubDirDef = Pattern SearchMode String SearchMode = “->” | “=>”
The two types of search mode are described below. A pattern is just a string Pattern = String
except that the characters ‘?’ and ‘*’ embedded in the string act as wildcards such that ‘?’ matches any single character and ‘*’ matches 0 or more characters. A string is any sequence of characters enclosed in double quotes.
6.3.3 MLF Search
The names of label files in HTK are invariably reconstructed from an existing data file name and this means that the file names used to access label files can be partial or full path names in which the path has been constructed either from the path of the corresponding data file or by direct specification via the -L option. These path names are retained in the MLF search which proceeds as follows. The given label file specification ../d3/d2/d1/name is matched against each pattern in the MLF. If a pattern matches, then either the named subdirectory is searched or an immediate definition is loaded. Pattern matching continues in this way until a definition is found. If no pattern matches then an attempt is made to open ../d3/d2/d1/name directly. If this fails an error is reported.
The search of a sub-directory proceeds as follows. In simple search mode indicated by ->, the file name must occur directly in the sub-directory. In full search mode indicated by =>, the files name, d1/name, d2/d1/name, etc. are searched for in that order. This full search allows a hierarchy of label files to be constructed which mirrors a hierarchy of data files (see Example 4 below).
Hashing is performed when the label file specification is either a full path name or in the form
*/file so in these cases the search is very fast. Any other use of metacharacters invokes a linear
6.3 Master Label Files 84 search with a full and relatively slow pattern match at each step. Note that all tools which generate label files have a -l option which is used to define the output directory in which to store individual label files. When outputting master label files, the -l option can be used to define the path in the output label file specifications. In particular, setting the option -l ’*’ causes the form */file to be generated.
6.3.4 MLF Examples
1. Suppose a data set consisted of two training data files with corresponding label files:
a.lab contains
000000 590000 sil 600000 2090000 a 2100000 4500000 sil b.lab contains
000000 990000 sil 1000000 3090000 b 3100000 4200000 sil
Then the above two individual label files could be replaced by a single MLF
#!MLF!#
"*/a.lab"
000000 590000 sil 600000 2090000 a 2100000 4500000 sil .
"*/b.lab"
000000 990000 sil 1000000 3090000 b 3100000 4200000 sil .
2. A digit data base contains training tokens one.1.wav, one.2.wav, one.3.wav, ..., two.1.wav, two.2.wav, two.3.wav, ..., etc. Label files are required containing just the name of the model so that HTK tools such as HERest can be used. If MLFs are not used, individual label files are needed. For example, the individual label files one.1.lab, one.2.lab, one.3.lab, .... would be needed to identifiy instances of “one” even though each file contains the same entry, just
one
Using an MLF containing
#!MLF!#
"*/one.*.lab"
one .
"*/two.*.lab"
two .
"*/three.*.lab"
three .
<etc.>
avoids the need for many duplicate label files.
6.3 Master Label Files 85 3. A training database /db contains directories dr1, dr2, ..., dr8. Each directory contains a subdirectory called labs holding the label files for the data files in that directory. The following MLF would allow them to be found
#!MLF!#
"*" -> "/db/dr1/labs"
"*" -> "/db/dr2/labs"
...
"*" -> "/db/dr7/labs"
"*" -> "/db/dr8/labs"
Each attempt to open a label file will result in a linear search through dr1 to dr8 to find that file. If the sub-directory name is embedded into the label file name, then this searching can be avoided. For example, if the label files in directory drx had the form drx xxxx.lab, then the MLF would be written as
#!MLF!#
"*/dr1_*" -> "/db/dr1/labs"
"*/dr2_*" -> "/db/dr2/labs"
...
"*/dr7_*" -> "/db/dr7/labs"
"*/dr8_*" -> "/db/dr8/labs"
4. A training database is organised as a hierarchy where /disk1/db/dr1/sp2/u3.wav is the data file for the third repetition from speaker 2 in dialect region dr1 (see Figure6.2).
u1.lab u2.lab u3.lab u4.lab . . . sp1 sp2 sp3 sp4
dr1 dr2 dr3 dr4 . . .
. . . db
u1.w av u2.w av u3.w av u4.w av . . . sp1 sp2 sp3 sp4
dr1 dr2 dr3 dr4 . . . disk1
/
disk2 disk3
. . .
Fig. 6.2 Database Hierarchy: Data [Left];
Labels [Right].
Suppose that a similar hierarchy of label files was constructed on disk3. These label files could be found by any HTK tool by using an MLF containing just
#!MLF!#
"*" => "/disk3"
If for some reason all of the drN directories were renamed ldrN in the label hierarchy, then this could be handled by an MLF file containing
#!MLF!#
"*/dr1/*" => "/disk3/ldr1"
"*/dr2/*" => "/disk3/ldr2"
"*/dr3/*" => "/disk3/ldr3"
etc.
6.4 Editing Label Files 86