Python has as one of its basic data types a dictionary, which is a key to value mapping. Internally it is implemented as a hash table, so the key can be of any data type that can be hashed, including numbers and strings. If you enumerate the keys that are present in a dictionary, you will get them in an unpredictable order. Historically it was unpredictable but deterministic, based on the hash values.

For my purposes, the dictionary enumeration order being unpredictable is fine, but I want it to be deterministic, so that multiple runs of the program with identical input will produce identical output.

It turns out that starting with Python 3.3, by default the hash function used is randomized, so the same program with the same dictionary contents will result in different enumeration orders on different runs:

http://stackoverflow.com/questions/14956313/why-is-dictionary-ordering-non-deterministic

The hash randomization is claimed to be a security feature.  I’m unclear on what security it provides.  It is possible to disable hash randomization using an environment variable passed to the Python interpreter.

I’m using Python 3.5.  Starting with Python 3.6, the implementation of dictionaries is different, so even though hash randomization is still the default, the dictionary enumeration order is no longer based on the hash, and is once again deterministic.

At some point, Python added support for a new dictionary type OrderedDict, which enumerates in the order of element insertion, which happens to be the same thing that Python 3.6 does with normal dictionaries. After changing one line in my program to initialize a particular variable as an OrderedDict rather than a “plain” dictionary, my program is once again deterministic, even with hash randomization enabled.

Suppose your USB media is drive /dev/sdb, and your ISO image is at /path/to/some-image.iso. If you have an actual file system mounted (e.g., /dev/sdb1), unmount it from the command line. (Do NOT unmount it from a GUI, e.g, Gnome, as that may not leave it in a state where it is usable over USB.)

Now run the command: livecd-iso-to-disk --format --reset-mbr /path/to/some-image.iso /dev/sdb1

Previously I spent a bunch of time trying to run the command without the --format option, and got all sorts of complaints about problems with my USB medai, partition table, filesystem, etc. I’m not convinced that the error messages were accurate. The --format option simplifies everything.

The command above will default to formatting the USB media with an ext4 filesystem, which works fine with Linux, but may be inconvenient for use with Windows. You can apparently use the poorly-named --msdos option (which should have been --fat or perhaps --vfat) to use a FAT filesystem on the USB media rather than ext4. However, I haven’t verified that this works.

This application is titled “Quick filename lookup using name hash”.  Based on the title, it sounded like they are doing what TRS-DOS 2.0 did back in 1978, which is putting on the disk a hash table of filenames which then refer to directory entries.  TRS-DOS did that so that it usually only needed to read two sectors to look up a file, the HIT (Hash Index Table) sector, and the actual directory sector containing the file’s directory entry.  Otherwise they might have had to read multiple directory sectors to find the file if it existed, and all of the directory sectors if it did not.

The claims of the patent are a little difficult to interpret.  They refer to “a first one or more computer readable storage media having computer executable instructions…”.  This is basically referring to the disk/flash/etc. the operating system is booted from.  They refer to “a second one or more…” which is the disk/flash/etc. which holds the file system in question.

Here’s are two of the independent claims:

1. A first one or more computer readable storage media having computer executable instructions that, when executed on at least one processor, configure the at least one processor to perform a method of detecting if a target file name exists on a second one or more computer readable storage media, the method comprising:

(A) determining a name hash from the target name;
(B) determining if the name hash corresponds to a directory entry set name hash value, the directory entry set name hash value corresponding to one of a plurality of directory entry sets, each of the plurality of directory entry sets stored on the second one or more computer readable storage media;
(C) determining if the target name matches a directory entry set name corresponding to the one of the plurality of directory entry sets after step (B) determines the name hash corresponds to the directory entry set name hash value; and
(D) indicating that the target name exists after step (C) determines the target name matches the directory entry set name.

19. A method of detecting if a target file name exists, the method executing on one or more processors, the method comprising:

(A) determining a file name hash from the target file name;
(B) determining if the file name hash corresponds to a directory entry hash value, the directory entry hash value corresponding to one of a plurality of directory entries;
(C) determining if the target file name matches a file name, the file name corresponding to the one of the plurality of directory entries after step (B) determines the file name hash corresponds to the directory entry hash value; and
(D) indicating that the target file name exists after step (C) determines the target file name matches the file name corresponding to the one of the plurality of directory entries.

These seem to be to be to be *exactly* what TRS-DOS 2.0 did as early as 1978, so it seems possible that TRS-DOS could be used as prior art to invalidate at least these independent claims, and quite possibly some of the dependent claims as well.

So my question is, are there any published works documenting the TRS-DOS file system on-disk format, especially the use of the HIT table, other than “TRS-80 Disk and Other Mysteries” by H. C. Pennington?

The obvious thing to do was to change the spec from:

BuildRequires: qt-devel

to

BuildRequires: qt-devel >= 4.6

However, Koji still tries to use qt-devel-3.x!

A Google search revealed an explanation in the rpmfusion-developers list from Kevin Fenzi:

That doesn’t work because it’s missing the Epoch. “>= 1:4″ would work, but it’s better to use the virtual qt4-devel Provides, which is backwards-compatible with EPEL 5 (which had an actual qt4-devel package), which will keep working in the future when Qt 5 will be the default and which avoids the pesky Epoch.

Now on to the next problem!

sudo groupadd foo
sudo usermod -a -G foo user
newgrp foo

Note that this makes the new group effectively the user’s primary group ID in the current shell and any new descendents. If you don’t want that, do another newgrp back to the original group. The new group will still be in the process’ group list.

Note that the “-a” (append) option to usermod may not exist in all Linux distributions.

Oddly enough, this was exactly the opposite of what Apple did in Apple DOS.  Apple published the APIs to read and write sectors (RWTS), but never published the “File Manager” APIs that allowed access to the file system through means other than passing commands through the character output vector (e.g., the BASIC statement PRINT CHR$(4);”OPEN FOO”).

I’ll mostly describe how things work in Model II TRSDOS 1.2, the earliest version I’ve been able to obtain.  I haven’t studied 2.0 nearly as much yet.  The TRSDOS 1.2 “kernel” consists of three parts, while later versions are more monolithic.

The Model II boot ROM loads all of drive 0 track 0 (single density, 26 sectors of 128 bytes) into memory starting at 0e00.  First it looks for the four characters “DIAG” at 1400 and “BOOT” at 1000.  If either are missing, it refuses to proceed.  It calls the code at 1404, which in TRSDOS is a simple hardware diagnostic.  When that returns, it jumps to the first stage boot loader code at 1004.  Some other operating systems don’t bother with a diagnostic, and just start their boot code at 1404, never returning to the ROM.

The first stage boot loader actually understands the TRSDOS filesystem enough to find the directory entries of files in TRSDOS load module format, and load them into memory.  In 1.2, it loads “IODVRS/SYS” and then “TRSDOS/SYS”, and jumps into the latter.  The Model II TRSDOS filesystem is similar in many regards to that of Model I TRSDOS, but not enough to actually be compatible. Unsurprisingly, it looks like an intermediate step in the evolution from Model I TRSDOS to Model III TRSDOS.  As in Model III TRSDOS, files can only have a single directory entry, with a limited number of extents.

IODVRS/SYS contains, as the name implies, the low level I/O drivers for the system, including the keyboard, display, printer, and floppy drives, the dispatching for system (SVC) calls, and a few utility SVCs.  However, it only contains the SVC handlers for services 0-28, the I/O functions and basic utility SVCs.  Note in particular that it contains no file system code.  IODVRS/SYS is conceptually similar to the CP/M BIOS, though lacking CP/Ms charming simplicity.  IODVRS/SYS provides several undocumented SVCs for internal use by TRSDOS, including floppy subsystem initialization (13), floppy sector read (14), and floppy sector write (16). Note that at the time IODVRS/SYS is loaded, no call is made into it to initialize it.

TRSDOS/SYS, however, is called after being loaded.  It basically performs the TRSDOS initialization that only has to happen at boot time.  It has another implementation of filesystem reading and load module format handling, very similar to what is present in the stage 1 boot, but now instead of talking to the FDC and DMAC directly, it uses the undocumented floppy SVCs described previously.  After various initialization, it loads SYSRES/SYS and jumps into it.

SYSRES/SYS contains the filesystem code and other relatively high-level TRSDOS infrastructure code.  It generally relies on SVC calls into IODVRS/SYS to perform all I/O, and has very little other dependence on IODVRS/SYS internals.  This is conceptually similar to the CP/M BDOS.  It loads system overlays to handle some SVCs and user commands.  Overlays SYS0/SYS through SYS9/SYS are small overlays, occupying one disk granule (five sectors) and loading into 2200-26ff.  Other overlays may be larger, and load at 2800 or higher.  Many of the overlays do depend on knowledge of the internals of SYSRES/SYS, directly accessing subroutines and data structures without the use of vector tables or the like.  This means that SYSRES/SYS and the overlays must have been built at the same time, and would generally not be interchangeable with earlier or later releases.

Anyhow, getting back to the paranoia part.  Someone apparently decided that simply not documenting the SVCs that provide sector-level access to the floppy was not sufficient to thwart those that might want to bypass the filesystem.  After TRSDOS/SYS uses those SVCs for its part in the boot process, it actually removes them from the SVC vector table, and sets up jumps to them at undocumented internal TRSDOS locations 1130 (read sector) and 1133 (write sector).

In TRSDOS 1.2, access to all of the system files, including overlays, is done through the file system.  The system files have normal file system entries. Unlike Model I TRSDOS, neither the system file directory entries nor the file contents need to be in any special location on the disk.

In TRSDOS 2.0, things are much more monolithic.  The stage 1 boot code only loads and jumps into a single file, SYSRES/SYS.  The boot code does not care where this file is located, but other parts of the system do.  All of the overlays, small and large, are stored in a single file, SYSTEM/SYS, which is required to start on the track after the primary directory.  The first sector of SYSTEM/SYS contains a kind of overlay directory that gives the track and sector numbers at which each overlay starts.

There is perhaps some advantage to putting all of the overlays in a single file, since the number of directory entries on the diskette is limited to 96.  However, the need for a second, special directory mechanism for overlays is ugly, even if it is only a simple one.  Requiring the system files to be at fixed locations on the disk (at least relative to the primary directory) might be a reasonable requirement if it yielded some performance gain, but it generally doesn’t.  (With 1.2, the system files are set up when the disk is formatted, so even though they could be anywhere, in practice they are grouped together.)

TRSDOS 2.0 introduced changes to the disk organization, such that TRSDOS 1.2 and 2.0 diskettes are not interchangeable, except that the 2.0 XFERSYS utility can convert a 1.2 diskette to 2.0 format.  The disk organization changes are basically gratuitous, and don’t provide any benefit to the user, while obviously being a great inconvenience to users with TRSDOS 1.2.  They mashed the GAT (granule allocation table) and HIT (hash index table), which were sectors 1 and 2 of the directory track in 1.2, into just sector 1 in 2.0.  In 1.2, the directory occupied sectors 3-26, while in 2.0 it occupies sectors 2-25.  The only apparent rationale for doing this is to free up sector 26 on the directory track.  In TRSDOS 1.2, sector 26 was not used on any track but the directory track, for any purpose.  In TRSDOS 2.0, sector 26 of every track is used to store five bytes of unique disk ID, to better detect disk changes.  (it has been suggested that those bytes might also have been used for software copy protection.)  However, rather than mashing the GAT and HIT together, which made it impossible to support larger disks such as double-sided disks, they easily could have special cased the directory track(s) and stored the disk ID in either the GAT or HIT sector.

TRSDOS 4.0 introduced much larger changes to the disk organization, in order to support double-sided disks and hard disks.  I haven’t yet begun to dig into the 4.0 code.

In fact, FSK modulation is almost trivial, so I had the encoder written in less than an hour.

For decode, I decided to use the Goertzel algorithm, which is a simplified DFT that only looks for a single frequency.  I’ve used the Goertzel algorithm before on fixed-point DSP processors. There are app notes on using it for DTMF detection.  For FSK demodulation, I just use two Goertzels, one for the space frequency and one for the mark frequency.  The audio samples are dealt with in windows that are a fraction of a bit time, and the levels indicated by the two Goertzels are compared to decide which frequency predominates in that window.

Implementing the decoder program took a few hours last night and a few hours today.  It’s now basically working, at least for the perfectly sinusoidal waveforms generated by my encoder.  I still need to test it on some real tapes.

I’m currently using an 8 kHz sample rate for my audio files, although the encoder can be told to use a higher sample rate if desired, and the decoder will work with whatever sample rate the audio file uses.  Since the FSK frequencies are 1200 Hz and 2400 Hz, there should be no advantage to using a sample rate higher than 8 kHz.

I’m currently using libsndfile to read and write audio files.  I’d like to also use PortAudio to support direct access to the audio hardware, but I haven’t yet tried that.  While I’m doing the development on Linux, in principle the programs should work if compiled on Windows or MacOS.

I’ll release the programs under the GPLv3 license when I’ve verified that they work reasonably well.

I spent over an hour trying to figure out why libsndfile was claiming that it couldn’t recognize the format of a perfectly good .wav file.

I’m using the open source FatFs code to implement the FAT filesystem.  It’s configured to work with two “drives”, the internal SPI FRAM chip in the DIY4, and the SD card.  I used the SD interface code from the “generic” example, with only minor changes necessary.

I still have to write a -41 microcode ROM to provide the equivalent of the HP-IL module’s mass storage commands, but talking to the SD card rather than HP-IL.

My first attempt at making the PWM work failed because I had overlooked a relatively obscure note in the EFM32G reference manual which observed that the REP0 (repeat 0) register needs to contain a non-zero value, even though it is not counting down in the PWM mode. Once I added code to initialize REP0 to 1 (or any non-zero value), I could read the timer value in software, and verify that it counts down, and when it underflows, reloads the appropriate value from the COMP0 register (which sets the PWM period). However, the actual PWM pin was never changing.

I spent a lot of time studying the LETIMER chapter of the reference manual, and corrected several bugs in my code, but still didn’t get the pin to change. Finally I looked at the LETIMER application note. The text of the application note didn’t help much, but I noticed that their overly complicated example code was initializing REP1 to a non-zero value, while the reference manual had only indicated that REP1 was used in some other auto-reload mode and did not indicate that it was used or necessary for PWM.

I added a few lines to initialize REP1 to 1, and the PWM started working.  Now I can adjust the display contrast, from too light to too dark, in 16 steps. Fortunately there are midrange values that are somewhat reasonable.

Both the reference manual and the application note could certainly use some more thorough explanation of the requirements to make PWM mode work.