What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

And now it’s all this

I just said what I said and it was wrong. Or was taken wrong.

JSON


Quicker linking to apps in Drafts

Permalink - Posted on 2020-06-03 18:29

Today on the Automators forum, there was a question about using Shortcuts to get the current version of an app in the App Store. When I read the answer, from Stephen Millard (@sylumer), I immediately knew I could modify his shortcut to speed up linking to apps, something I do quite often here.

Up until now, this is how I’ve been linking to an app:

  1. Open the App Store app.
  2. Search for the app I want to link to.
  3. Get the app’s URL from the Share Sheet.
  4. Return to Drafts, which is where I’m writing the post, select the text I want to turn into a link (typically the name of the app), and run the Markdown Reference Link action from @pdavisonreiber.

Because the App Store doesn’t open in search mode, this can take half a minute to a minute to do. Even worse is the context shift as I switch from Drafts to the App Store and back. By stealing Stephen’s shortcut and making a Drafts action that glues it to @pdavisonreiber’s action, my workflow has been reduced to

  1. Type the name of the app and select it.
  2. Run the App Store Link action.

which runs in just a few seconds, requires no intervention on my part, and has no context shift other than me watching the screen change from Drafts to Shortcuts and back.

Here’s my adaptation of Stephen’s shortcut. It grabs the App Store URL of the given app and puts it on the clipboard

StepActionComment
0 App Store URL Step 00 I won’t be running this from the Share Sheet, but I have to set it up as if I will so it accepts text.
1 App Store URL Step 01 Search the App Store for the given text. We have to have some faith that the search will work.
2 App Store URL Step 02 We also have to have some faith that the app we’re looking for is at the top of the found list.
3 App Store URL Step 03 Get the URL of the app.
4 App Store URL Step 04 Put it on the clipboard so it’s ready to be used in the next step of the Drafts action.

The App Store Link action consists of two steps. The first runs the App Store URL shortcut, passing it the selected text.

App Store Link Shortcut step

The second step runs the Markdown Reference Link action, which uses the URL on the clipboard to turn the selected text into a link.

App Store Link Include Action step

So by doing virtually no work of my own, I have a new Drafts action that will save me upwards of a minute every time I need to link to an app and doesn’t take me out of the flow of writing.


[If the formatting of equations looks odd in your feed reader, visit the original article]


Drafts and iCloud Drive

Permalink - Posted on 2020-05-30 16:24

I didn’t renew my Dropbox Plus account when it expired a month or two ago. For several months I’d had both it and a 2 TB iCloud account active at the same time and had been moving my files and my way of working from one to the other. This was mainly due to my shift over the past four years from being Mac-only to Mac-mainly to iPad-mainly. While the Finder is equally adept at managing files in either cloud, Files is much better with files on iCloud Drive.

My concern was with Drafts. There are programmatic ways to save a draft to a specified file in Dropbox—I use a “filelines” system similar to Vim’s modelines in which the file path is embedded in the text of the draft—but sandboxing prevents that working with file paths in iCloud Drive. Here’s a Drafts forum thread in which Greg Pierce (agiletortoise) explains the problem.

As it turned out, my concern was overblown. Shortly after writing this post in which I talk about writing almost everything in Drafts, I started write almost everything that needs to be saved as a file in Textastic. Textastic is file-based and can read and write anywhere in iCloud Drive (or anyplace Files can access) without continual user approval.

Still, I did occasionally miss being able to use filelines to save drafts directly to iCloud Drive. It wasn’t until this past week that I realized there was a way around it by using Scriptable.

Scriptable has a feature called “bookmarks,” which are ways to access files (and folders) outside of the Scriptable sandbox. You create them by choosing File Bookmarks in Settings.

Scriptable settings

Then tap the + button in the upper right corner and choose Pick Folder from the popup menu.

Add Scriptable bookmark

You’re now presented with the usual Files-style list of file providers. Choose iCloud Drive.

Choose folder from Files picker

Finally, you’re asked to choose a name for this bookmark. I chose “iCloud.”

Bookmark naming

You can now use this bookmark and Scriptable’s FileManager library to access any folder or file in iCloud Drive. The great thing about this is that you’re not limited to just the top level of iCloud Drive—any file or folder at any level in the hierarchy below iCloud Drive is accessible.

With this bookmark defined, I can create a system that looks for an iCloud fileline, a line in a draft that contains

icloud:/path/to/subfolder/filename.txt

and saves the contents of the draft to a file in the given location. Typically, this line will be in a comment, so it will be

# icloud:/path/to/subfolder/filename.py

for a Python file or

% icloud:/path/to/subfolder/filename.tex

for a LaTeX file.

The system consists of two parts:

  1. A Drafts action that gets the fileline, extracts the path, and sends the path and the full draft contents to a shorcut.
  2. A shortcut that takes the information from the previous step and writes the contents of the draft to the given file.

The Drafts action (which you can download), starts with this JavaScript step,

javascript:
 1:  // Get the folder and filename from the dbox line.
 2:  var d = draft.content;
 3:  
 4:  // Regex for fileline.
 5:  var fileRE = /icloud:\s*(.+)\s*$/m;
 6:  if (fileRE.test(d)) {
 7:    // Get the path. Add a slash to the front if I forgot.
 8:    var path = d.match(fileRE)[1];
 9:    if (path[0] != '/') {
10:      path = '/' + path;
11:    }
12:  
13:    // Create a JSON template to pass to Scriptable via Shortcuts
14:     var fileinfo = {"path":path, "text":d};
15:    
16:    // Create a template tag from the dictionary.
17:    draft.setTemplateTag("fileinfo", JSON.stringify(fileinfo))
18:   
19:  } else {
20:    alert("No icloud: line");
21:    context.fail("No icloud: line");
22:  }

which extracts the path from the iCloud fileline. If there is no fileline, it stops the action with an alert. If there is a fileline, it creates a JSON dictionary with two entries: the path from the fileline and the full text of the draft. The dictionary is saved to a template tag in Line 17 so it can be passed to the next step of the action.

The second and final step in the Drafts action is this Run Shortcut step,

Run Shortcut step

which passes the dictionary created in the first step to a shortcut entitled “Save to iCloud.”

Here’s the “Save to iCloud” shortcut:

StepActionComment
0 Save to iCloud Step 00 Even though we won’t be using it from the Share Sheet, we define it this way so it can accept the text passed to it from Drafts.
1 Save to iCloud Step 01 The inline JavaScript, shown in full below, extracts the path and the file contents from the JSON dictionary and writes the draft contents to the given file. The “Run in App” setting has to be on for the bookmark to work.

Here’s the inline JavaScript:

javascript:
1:  // The argument passed in from Shortcuts should be a
2:  // JSON dictionary with "path" and "text" entries.
3:  var d = args.shortcutParameter;
4:  
5:  // Set up connection to the root level of iCloud Drive
6:  // through a Scriptable bookmark and save the text.
7:  var myFM = FileManager.iCloud();
8:  var iCloud = myFM.bookmarkedPath('iCloud');
9:  myFM.writeString(iCloud + d['path'], d['text']);

Line 3 gets the argument passed in and saves it to the JavaScript dictionary, d. The rest of the script gets the iCloud bookmark, appends the filepath to it, and writes the text of the draft to that file.

If you don’t feel like writing it yourself, you can download the shortcut.

My understanding (based on a post from its developer) is that Toolbox Pro is beta testing a similiar bookmarking system. If that gets released, it may simplify the shortcut.

There is a hitch in this system that’s unlikely to affect me but might affect you. Saving a draft from my iPad to iCloud drive works perfectly, and every time I edit the draft and resave it, the file is overwritten with the latest version of the draft, just as you would expect. But if I edit that draft on my iPhone and save it, a new file gets created in iCloud Drive. If the original file saved from the iPad was

/path/to/subfolder/filename.txt

the file saved from the iPhone will be

/path/to/subfolder/filename 2.txt

even though the fileline is still

icloud:/path/to/subfolder/filename.txt

I assume this has something to do with Scriptable bookmark on the iPhone being somehow different from the bookmark on the iPad, despite them both pointing to the same iCloud Drive location.

As I say, I don’t think this will affect me, as the writing I do on my phone tends to be ephemeral stuff. But it is one of those things that shows how iCloud Drive isn’t quite on the level as Dropbox (or Box or Google Drive or…) when it comes to working with Drafts.


[If the formatting of equations looks odd in your feed reader, visit the original article]


COVID antibody testing and conditional probability

Permalink - Posted on 2020-05-28 03:52

In a Slack channel that I frequent and he infrequents, podcasting magnate Lex Friedman linked to this CNN article on the effectiveness of COVID-19 antibody tests:

CNN article that says antibody tests may be wrong half the time

The concern is that people who’ve had a positive result from an antibody (serologic) test may be fooling themselves into thinking they’re immune. Unfortunately, there’s a numerical problem with this:

The CDC explains why testing can be wrong so often. A lot has to do with how common the virus is in the population being tested. “For example, in a population where the prevalence is 5%, a test with 90% sensitivity and 95% specificity will yield a positive predictive value of 49%. In other words, less than half of those testing positive will truly have antibodies,” the CDC said.

Although CNN doesn’t link to it, the quote comes from this CDC page.

Let’s go through the numbers and see how 90% turns into 49%.

First, there are a couple of terms of art we need to understand: sensitivity and specificity. Sensitivity is the accuracy of positive test results; it is the percentage of people who actually have the condition that will test positive for it. Specificity is the accuracy of negative test results; it is the percentage of people who actually don’t have the condition that will test negative for it.

With those definitions in mind, we’ll create a 2×2 contigency table for a population of 100,000 people in which the prevalence, sensitivity, and specificity are as given in the quote.

                  Have      Don't have
               antibodies   antibodies
-------------------------------------------------
Test positive    4,500        4,750    |   9,250
Test negative      500       90,250    |  90,750
-------------------------------------------------
                 5,000       95,000    | 100,000

5,000 people actually have the antibodies (5% of 100,000) and 95,000 do not. Of the 5,000 with antibodies, 4,500 (90% of 5,000) will test positive for them and the remaining 500 will test negative for them. Of the 95,000 without antibodies, 90,250 (95% of 95,000) will test negative for them and the remaining 4,750 will test positive for them.

So of the 9,250 people who test positive for the antibodies, only 4,500 actually have them. That’s where the 49% figure comes from in the quote.

This is a very common sort of problem to spring on students in a probability class who are learning about conditional probability and Bayes’s Theorem. The key thing is to realize that the probability that you have antibodies given that you had a positive test is definitely not the same as the probability that you had a positive test given that you have antibodies. When you hear about a test’s “accuracy,” it’s usually the latter that’s presented even though you as a patient are only interested in the former. After all, if you already knew you had the condition, you wouldn’t need to have the test.

Although the CNN article makes it sound as if antibody testing is useless, there are ways of using multiple tests (of different types) to achieve a high predictive value:

Algorithms can be designed to maximize overall specificity while retaining maximum sensitivity. For example, in the example above with a population prevalence of 5%, a positive predictive value of 95% can be achieved if samples initially positive are tested with a second different orthogonal assay that also has 90% sensitivity and 95% specificity.

We can see how this works by putting the 9,250 population of positive tests through a second round of independent testing. Here’s the contingency table for the second test:

                  Have      Don't have
               antibodies   antibodies
-------------------------------------------------
Test positive    4,050          238    |   4,288
Test negative      450        4,512    |   4,962
-------------------------------------------------
                 4,500        4,750    |   9,250

Here we see that of the 4,288 people who test positive in the second test, 4,050 (94%) actually have the antibodies. The second test works so well because it is performed on a population that has a much higher prevalence of people with antibodies. It benefits from the screening done by the first test


[If the formatting of equations looks odd in your feed reader, visit the original article]


Turning bookmarks into Markdown reference links

Permalink - Posted on 2020-05-25 20:29

For some time now, I’ve been writing almost all of my blog posts in Drafts on an iPad. I usually have Drafts and Safari set up in Split View, and I copy the URL of the current Safari page whenever I need to add a link to a post. I’ve always preferred Markdown’s reference format for links, and this action inserts reference links in Drafts the same way an AppleScript I wrote a long time ago does it in BBEdit.1

With a URL on the clipboard, I select the text that’s going to be the link and run the action. It surrounds the selected text with brackets, puts a numbered reference after it,

after reading [this John Voorhees review][3] in MacStories.

and then adds the reference and URL at the bottom of the draft:

[3]: https://www.macstories.net/reviews/highlights-for-iphone-and-ipad-an-excellent-companion-for-researchers/

The reference number is incremented each time.

Sometimes, though, I have a bunch of websites open in tabs in Safari, and because I know I’ll be linking to each of them, I’d like to add all the references to the bottom of the draft in a single step. I can do this via AppleScript on the Mac, but Shortcuts currently isn’t smart enough to do this sort of thing. So I decided to use the Mac to get around iPadOS’s limitations.

First, on the iPad, I bookmark all the tabs to a “Blogging” sublist I’ve set up in my Favorites (which is the Bookmarks bar on the Mac). This can be done in a single step by long-pressing the bookmark button and choosing the Add Bookmarks for # Tabs command from the popup menu.

Long press on Bookmarks button

Because iCloud syncs my bookmarks, all of these links are now saved in the ~/Library/Safari/Bookmarks.plist file on my Mac. If I have a script on my Mac that can extract them from that file, I can run it from my iPad via the Shortcuts Run Script Over SSH command.

Such a script isn’t that hard to write in Python, as the standard library has a plistlib module that knows how to read and write plist files.

Here’s the mdbookmarks script that pulls out all the URLs in the “Blogging” sublist and prints them in Markdown reference format:

python:
 1:  #!/usr/bin/env python
 2:  
 3:  import plistlib
 4:  import os
 5:  from docopt import docopt
 6:  
 7:  # Handle the command line options
 8:  usage = '''Usage:
 9:    mdbookmarks [options]
10:  
11:  Options:
12:    -n NNN  Starting number for reference links [default: 1]
13:    -h      Show this help message
14:  
15:  Return a set of lines with Markdown reference links to all the
16:  bookmarks in the "Blogging" section of Safari's Bookmarks bar.'''
17:  
18:  args = docopt(usage)
19:  n = int(args['-n'])
20:  
21:  # Load the Safari Bookmarks plist
22:  fn = os.environ['HOME'] + '/Library/Safari/Bookmarks.plist'
23:  with open(fn, 'rb') as f:
24:    bm = plistlib.load(f)
25:  
26:  # Work through the plist and assemble all the URLs in the
27:  # "Blogging" section of the Bookmarks bar into a list
28:  blist = []
29:  for a in bm['Children']:
30:    if a['Title'] == 'BookmarksBar':
31:      for b in a['Children']:
32:        try:
33:          if b['Title'] == 'Blogging':
34:            for c in b['Children']:
35:              blist.append(f"[{n}]: {c['URLString']}")
36:              n += 1
37:        except KeyError:
38:          pass
39:  
40:  # Print the list, one per line
41:  print('\n'.join(blist), end='')

As usual, I use the nonstandard docopt module to handle the command-line switches.

The trickiest part of this script was figuring out where the “Blogging” bookmarks are kept within the labyrinth of Bookmarks.plist. Lines 28–38 were built up through trial and error. I dug through the layers of the plist, testing each layer using type and keys to eventually find my way to the URLs.

Although this script is written for Python 3.7, it shouldn’t be too hard to get it to run on the Python 2.7 that comes preinstalled on the Mac. I think you can just change the f-string in Line 35 to a format call.

I ran mdbookmarks on my Mac to test it, but it’s meant to be run via a shortcut on my iPad. The shortcut that does it is called “Blogging Bookmarks,” and it has only two steps:

StepActionComment
1 Blogging Bookmarks Step 01 Get the starting number from the user.
2 Blogging Bookmarks Step 02 Log into my Mac and run the mdbookmarks script. Use the starting number from the previous step.

You’ll note that this shortcut doesn’t do anything with the output of mdbookmarks. That’s because it’s meant to be run from a simple Drafts action that can be downloaded from here. There are only two steps in the action:2

  1. Run the “Blogging Bookmarks” shortcut. This makes the output of mdbookmarks script available in the [[shortcut_result]] template.
  2. Insert [[shortcut_result]] at the current cursor point. The intention is to run this action with the cursor at the bottom of the draft.

With all three pieces in place, adding a bunch of references to blog post is fairly easy. I just add bookmarks to the “Blogging” sublist as I do my research, then run the “Blogging Bookmarks” action in Drafts to insert all the references when I start writing the post.

I can make this system sound ridiculously complex: It’s a Drafts action that runs a shortcut that connects to my Mac over SSH and runs a Python script that reads a plist file and extracts information that then works its way back along the chain to insert text with that information into Drafts. But each link in the chain is straightforward. Only mdbookmarks took more than a couple of minutes to write, and that was because the Bookmarks.plist file is a nested mess.

The biggest limitation of this system is that I have to clean out the “Blogging” sublist whenever I’m done with a post. Otherwise, links from older posts will be inserted into newer posts.

I’m pleased I was able to build this system, but I wish I didn’t have to. I’d be much happier if Apple made all of this obsolete by adding features to Shortcuts that, among other things, allow us to get all of Safari’s tab links directly.


  1. Going even farther back, I had a similar system set up in TextMate. 

  2. Unfortunately, Drafts actions aren’t especially screenshotable. You have to go at least three levels deep to see all of an action’s properties, and I’m just not interested in trying to figure out a good way to present all that scattered information. It’s better to just download the action and see it within Drafts itself. 


[If the formatting of equations looks odd in your feed reader, visit the original article]


Highlighting with Highlights and LiquidText

Permalink - Posted on 2020-05-23 15:07

I decided to get a copy of Highlights after reading this John Voorhees review in MacStories. After trying it out on a few PDF documents that needed to be summarized for my job, I learned that it wasn’t going to work for me. Ironically, it was very bad at highlighting text in the kinds of documents I deal with. But my experiment with Highlights led me to giving LiquidText another try, and with a new perspective on how to use it, LiquidText fits my highlighting needs pretty well.

Highlights is a focused app with a straightforward user interface for highlighting and commenting on text in a PDF, and it can export the highlighted text as plain text.1 It seems like a perfect fit for how I want to work. I subscribed to the Pro version (necessary to get plain text exporting) so I could give it a try.

As luck would have it, I had just finished a couple of projects in which I had done a fair amount of document summarizing. So I put copies of those documents in a new folder and started going through them in Highlights to see how much more effective it would be than my current system.2

On the very first document I tried, Highlights wouldn’t select the text I wanted it to, overselecting in certain areas and underselecting in others. (I can’t show screenshots of it because that would expose the client’s work product.) The trouble seemed to be most commonly associated with footnotes. The selection jumped around in unpredictable ways whenever a footnote or endnote was within the desired selection or nearby.

My history with Microsoft products leads me to believe that this PDF, which I know started its life as a Word document, has a convoluted internal structure, and that may be part of the reason Highlights had so much trouble with it. But I don’t have any control over the history of PDFs I need to summarize, and documents with footnotes that were written in Word make up a large enough percentage of the material I get from clients to make Highlights effectively useless to me. I cancelled my Pro subscription.

Shortly after my Highlights experiment, this thread about PDF note-taking apps appeared on the Mac Power Users forum. It reminded me of that copy of LiquidText I got a long time ago and decided not to use. Maybe it was worth another shot.

The signature feature of LiquidText is the ability to grab excerpts from PDFs and combine them into new documents that show the linkages between different PDFs and different sections of the same PDF. It’s very impressive but struck me as more a presentation tool than a research tool. What I learned from the forum (and some new testing of my own) was that I don’t have to use LiquidText’s cool linking feature; I can just highlight and comment on text as I read along and generate a summary of the highlights and comments when I’m done. And, most important, LiquidText is much better at selecting the text I want than Highlights is. Not perfect—I have found a couple of glitches—but definitely good enough that I expect the average PDF to give me no trouble at all.

After the text is highlighted, in needs to come out, and if you’ve looked through LiquidText’s sharing options, you might think it’s not possible to get a plain text summary out of it.

LiquidText sharing options

But if you choose the “Notes Outline” option, which is intended to create a Word file, you’ll see another window appear that lets you put the notes onto the clipboard instead of into a DOCX file.

LiquidText export window

Copying those notes into Drafts makes for a pretty decent summary.

LiquidText noted copied into Drafts

As with the Markdown export from Highlights, I’m not thrilled with the way the notes are formatted, but I’ve written a Drafts action that cleans it up, distinguishing between highlights and comments and getting rid of the extra spaces that often appear in selections from fully justified text.

Summary notes after cleanup

(I should point out that this particular PDF had several equations that I had highlighted. They’re hard to express in plain text and will need to be cleaned up.)

The Drafts action that does the cleanup consists of just one JavaScript step:

javascript:
 1:  var summary = editor.getText();
 2:  
 3:  // Inexplicably, LiquidText uses CRs as line endings.
 4:  var reformatted = summary.replace(/\r/g, '\n');
 5:  
 6:  // Get rid of the second header.
 7:  reformatted = reformatted.replace(/\nNotes in Document \n[^\n]+\n/, '');
 8:  
 9:  // Reformat the comments and highlights.
10:  function noteReplace(full, m1, m2, m3, offset, string) {
11:    if (m1 == "Highlight") {
12:      return 'Page ' + m3 + ' quote:\n' + m2;
13:    }
14:    else {
15:      return 'Page ' + m3 + ' summary:\n' + m2;
16:    }
17:  }
18:  
19:  reformatted = reformatted.replace(/(Highlight|Comment):\s?:?\s?(.+)\u2028\(.+p.(\d+)\)/g, noteReplace);
20:  
21:  // Extra spaces are probably from full justification.
22:  reformatted = reformatted.replace(/  +/g, ' ');
23:  
24:  editor.setText(reformatted);

A couple of comments on the script:

  • As you can see in Lines 3–4, the notes from LiquidText use carriage return (CR) characters as line endings. Not linefeed (LF) characters, as would be expected on iOS. Not the CRLF combination common to Microsoft products. No, it’s the bare CR that Macs used to use back in the pre-OS X days. Bizarre.
  • Nearly as odd is the use of the Unicode LINE SEPARATOR character (U+2028) within each note. That gets cleaned up in the large regex in Line 19.

I don’t expect the summary that comes out of this process to be in final form, but my early experiments have shown that there’s less editing needed in these summaries than in those I dictate.

Overall, I like the experience of summarizing a document with LiquidText. I can still sit with my iPad on my lap, and I prefer swiping with the Pencil to reading text into my phone. In those places where I need to make a comment instead of a highlight, I can still dictate—I just dictate into the iPad instead of my phone. I no longer have to wake up a phone that’s gone to sleep between comments. Unless I run into a showstopper, LiquidText is how I’ll be summarizing from now on.


  1. The exported plain text is Markdown with a header structure that I wouldn’t want to use, but I don’t consider that a significant problem. It’s easy to write a filter that reformats well-structured text to get the output I want. 

  2. Which is to dictate quotes and comments into Drafts on my phone as I read a document on my iPad. 


[If the formatting of equations looks odd in your feed reader, visit the original article]


Siri and context, four years on

Permalink - Posted on 2020-05-22 13:38

Reading the latest Daring Fireball post this morning, I immediately thought of Effingham and my frustration, four years ago, with Siri’s inability to make reasonable guesses as to what we want from it no matter how many contextual clues it has.

I was driving up through Central Illinois… My iPhone was charging and sitting upside-down in a cupholder in the center console. I pushed the home button, waited for the Siri beep to come through my car’s speakers, and asked “How far is it to Effingham?”

Siri’s response: “Which Effingham? Tap the one you want.”

On the positive side, Siri recognized the word “Effingham” and recognized it as a place name. But those successes made its two context failures even more annoying.

First, I’m driving north on I-57 in Illinois between Mount Vernon and Effingham. Which effing Effingham do you think I want?!

And Siri knows damned well I’m driving. It’s connected to my car via Bluetooth. It can use its GPS to figure out I’m moving 80 mph. It has no business asking me to tap on a choice.

The interesting difference between my 2016 experience and John Gruber’s and Nilay Patel’s 2020 experiences is that I did want the nearest city with the name I gave. It’s fun to see the wide variety of ways in which Siri manages to choose the worthless answer, but we really should have a better assistant by now.

Baby Groot choosing a button


[If the formatting of equations looks odd in your feed reader, visit the original article]


Derangement extra postage

Permalink - Posted on 2020-05-22 04:16

I left something out of last night’s post on derangements: the Python program that generated the table of values I used to search the OEIS for the sequences associated with one, two, three, etc. fixed points. It was a deliberate omission, as I thought the post was long enough as it was. Consider this post the equivalent of Numberphile’s “extra footage.”

Recall that the table looked like this (you may have to scroll left to see it all)

                                   m
           0      1      2      3      4      5      6      7      8
     ----------------------------------------------------------------
  2 |      1      0      1
  3 |      2      3      0      1
  4 |      9      8      6      0      1
n 5 |     44     45     20     10      0      1
  6 |    265    264    135     40     15      0      1
  7 |   1854   1855    924    315     70     21      0      1
  8 |  14833  14832   7420   2464    630    112     28      0      1

where \(n\) is the number of cards we’re shuffling and \(m\) is the number of fixed points, that is, the number of cards that are in their original position after the shuffling.

Shuffling cards is a permuting operation, and the itertools module in Python’s standard library has a permutations function that returns all the permutations of the list (or other iterable) passed to it. The program I wrote does nothing more than call that function to generate the permutations and then search through them, counting up the number of fixed points in each. It does that for \(n = 2 \ldots 8\).

Here it is:

python:
 1:  #!/usr/bin env python
 2:  
 3:  from itertools import permutations
 4:  
 5:  def countMatches(o, p):
 6:    count = 0
 7:    for i in range(len(o)):
 8:      if p[i] == o[i]:
 9:        count += 1
10:    return count
11:  
12:  # Table header
13:  print(' '*3, end='')
14:  for m in range(9):
15:    print(f'{m:7d}', end='')
16:  print()
17:  print(' '*3 + '-'*64)
18:  
19:  # Table body
20:  for n in range(2, 9):
21:    cards = [chr(i+65) for i in range(n)]
22:    counts = [0]*(n+1)
23:  
24:    for a in permutations(cards):
25:      matches = countMatches(a, cards)
26:      counts[matches] += 1
27:    
28:    print(f'{n} |', end='')
29:    for i in range(n+1):
30:      print(f'{counts[i]:7d}', end='')
31:    print()

The key to the script is the countMatches function in Lines 5–10. When given the original list and a permutation, it marches through each and returns the number of fixed points. I’m sure there’s a more clever way of doing this, but I wasn’t interested in spending the time to be clever. Since the program as a whole is a brute force approach to the problem, I didn’t care if there was a brute force subroutine lurking within it.

Lines 12–17 just set up the table’s header, with the values of \(m\) and a row of hyphens. The rest of the program loops through all the values of \(n\) to create the row labels and body of the table.

Those of you who know your ASCII will recognize that the list of cards created in Line 21 consists of n capital letters in alphabetical order, starting with A. Line 22 initializes the counts list to a bunch of zeros.

The loop in Lines 24–26 goes through all the permutations and uses the countMatches function to increment the appropriate member of counts. For each item in counts, the index is the number of fixed points, \(m\), and the value is the number of permutations with that many fixed points.

Finally, Lines 28–31 print out the row of the table associated with the current value of n.

The “m” label above the top header row and the “n” label to the left of the row labels were added by hand. It was easier than programming them in.

You’ll note that some of the printing is done with f-strings, so this will only run in Python 3.6 and above. But you could rewrite those lines to use the format method if you’re really stuck with an older version of Python.

If you ever run into a problem that requires rearranging or combining lists, keep itertools in mind.


[If the formatting of equations looks odd in your feed reader, visit the original article]


Arrangements and derangements

Permalink - Posted on 2020-05-21 03:19

I’ve never thought of myself as particularly good at combinatorics or integer math in general. Most of my formal math study and the math I use in my professional life deals with the full continuity of real numbers, so that’s what I’m best at. But I do enjoy playing around outside my comfort zone.

I recently watched this Numberphile video from a few years ago. It’s about rearranging a set of items, with the goal being to determine the likelihood, after a random shuffle, that none of the items are in their original position.

When James Grime said the probability was 37%, I knew he was really talking about an asymptotic approach to \(1/e\). Not because I’m a math genius, but because I’ve seen that come up many times in dealing with the binomial distribution and its asymptotic movement toward the Poisson distribution as the number of “events” gets large.

The description of why the probability of derangement (a lovely word choice there) tends toward \(1/e\) as the number of shuffled items gets larger is explained, although not fully explained, in the video’s extra footage.

While it’s nice to see the series expansion of \(e^{-1}\), what got me curious was the form of the individual terms of the series. It was the formula

\[\frac{n!}{k!}\]

for the alternating terms of the series. While writing out these terms (starting at about 3:30 in the second video), Dr. Grime says “You might have to think about this” and “Have a check about that.” So I did, and now I’m going to inflict my thinking on you.

I generally approach combinatoric problems by enumerating all the possibilities of a small number of items and then looking for the patterns of interest. Here, I started with a set of four cards, labeled A, B, C, and D. If we start with them in that order and shuffle, we could end up with any of the following 24 permutations:

A B C D        B A C D
A B D C        B A D C
A C B D        B C A D
A C D B        B C D A
A D B C        B D A C
A D C B        B D C A

C A B D        D A B C
C A D B        D A C B
C B A D        D B A C
C B D A        D B C A
C D A B        D C A B
C D B A        D C B A

Jumping immediately to the final answer of the first video, we can see there are 9 permutations in which none of the letters are in their original place, so the likelihood of a well-shuffled set of four cards coming out with none of the cards in their original place (i.e., zero fixed points) is

\[\frac{9}{24} = \frac{3}{8} = 0.375\]

which is, as Dr. Grime says, pretty close to \(1/e \approx 0.3679\) or about 37%.

Now let’s try to come up with the individual terms of the series he wrote out in the second video. First, we’ll count up all the examples of where some of the cards are in their right places.

Card(s) in right place Count
A 6
B 6
C 6
D 6
A and B 2
A and C 2
A and D 2
B and C 2
B and D 2
C and D 2
A and B and C 1
A and B and D 1
A and C and D 1
B and C and D 1
A and B and C and D 1

Grouping these, we see that we have four 6s, six 2s, four 1s, and then another 1. These counts, of course, are not mutually exclusive. For example, the two cases in which A and B are in the right place overlap with cases of A being in the right place and B being in the right place. So if we wanted to count the permutations for which A or B are in their right place, we’d have to subtract off the overlap:

\[6 + 6 - 2 = 10\]

You can confirm this by going back to the list of all permutations. You’ll see that there are 10 with A or B in the right place: all six from the subset with A in first position, two from the subset with C in the first position, and two from the subset with D in the first position.

Returning to our table of counts, we can figure out the number of permutations with at least one fixed point:

\[4 \cdot 6 - 6 \cdot 2 + 4 \cdot 1 - 1 = 24 - 12 + 4 - 1 = 15\]

Therefore, the number of permutations with zero fixed points is

\[24 - (24 - 12 + 4 - 1) = 24 - 15 = 9\]

which is what we got by looking through the list directly.

How can we generalize this to any number of cards, \(n\)? The leading term, the one outside the parentheses, is the total number of permutations, so we already know it’s general form: \(n!\).

Now let’s think about the first term within the parentheses, which represents all the permutaions in which at least one card is in the right spot. We’ll start by consider the cases where the A card is in the right spot. The number of arrangements in which A is in the right spot is the number of permutations of the other cards, which is \((n - 1)!\). We can say the same thing for each of the \(n\) cards, so the number of permutations for which at least one card is in the right spot is

\[n (n - 1)!\]

In our example with \(n = 4\), that’s the four 6s (\(4\cdot3!\)) at the top of the counts table. (Yes, we could simplify this to \(n!\), but we’ll see soon that it makes more sense to leave it in this form).

Now we move on to the permutations for which at least two cards are in their original spots. If there are two cards in their original places, there are \(n - 2\) other cards and \((n - 2)!\) ways to arrange them. And how many ways can 2 cards out of \(n\) be in their right places? It’s the number of combinations of \(n\) items taken 2 at a time. Therefore, the second term inside the parentheses is

\[\binom{n}{2} (n - 2)!\]

which matches the six 2s we got for \(n = 4\).

At this point, we should step back and realize that our expression for the first term in the parentheses,

\[n (n - 1)!\]

could have been written as

\[\binom{n}{1} (n - 1)!\]

The pattern should be clear. Each term for \(k\) fixed points is

\[\binom{n}{k} (n - k)!\]

and the terms have alternating signs for the reasons given by Dr. Grime in the second video.

Now let’s look again at the leading term outside the parentheses: \(n!\). It may seem silly to do so, but this can be written as

\[\binom{n}{0} (n - 0)!\]

because

\[\binom{n}{0} = 1\]

for all values of \(n\).

The method to this madness is that we now have all the terms—the leading term outside the parentheses and all the terms inside the parentheses—in the same form and we can bundle them together into nice compact summation:

\[\sum_{k=0}^{n} (-1)^k \binom{n}{k} (n - k)!\]

where the \((-1)^k\) term handles the alternating signs.

We still haven’t reached the form Dr. Grime showed. For that, we need to expand out the binomial terms and do a little algebra.

Recall that

\[\binom{n}{k} = \frac{n!}{k! (n - k)!}\]

So

\[\sum_{k = 0}^n (-1)^k \binom{n}{k} (n - k)! = \sum_{k = 0}^n (-1)^k \frac{n!}{k! (n - k)!} (n - k)! = n! \sum_{k = 0}^n (-1)^k \frac{1}{k!}\]

The summation part is

\[\sum_{k=0}^n \frac{(-1)^k}{k!} = \frac{1}{0!} - \frac{1}{1!} + \frac{1}{2!} - \frac{1}{3!} + \dots \pm \frac{1}{n!}\]

which is exactly what the second video shows (at about 4:35).

Numberphile equation screenshot

This is a series that tends toward \(1/e\) as \(n\) gets “large.” Since the denominators are growing by factorials, “large” isn’t very large at all. As we can see for \(n = 4\), the series is already within 2% of \(1/e\).

Now that we know the formula for the number of permutations with zero fixed points, we should be able to figure out the formulas for the numbers of permutations with exactly one, two, three, etc. fixed points. And we can.

For exactly one fixed point, the number of permutations is

\[n! \sum_{k = 0}^{n-1} (-1)^k \frac{1}{k!}\]

For exactly two fixed points, it’s

\[\frac{n!}{2} \sum_{k = 0}^{n-2} (-1)^k \frac{1}{k!}\]

For exactly three fixed points, it’s

\[\frac{n!}{6} \sum_{k = 0}^{n-3} (-1)^k \frac{1}{k!}\]

You have the pattern now. For exactly \(m\) fixed points,1 it’s

\[\frac{n!}{m!} \sum_{k = 0}^{n-m} (-1)^k \frac{1}{k!}\]

I’d like to say I worked these formulas out on my own, but I didn’t. I cheated. I do think, though, that I cheated in an interesting way.

I wrote a short, brute force Python program that calculated the number of fixed permutations for \(n\) from 2 to 8 and \(m\) from 2 to \(n\). It used the itertools library to generate all the permutations and looped through them, counting all the occurrences of zero, one, two, three, etc. fixed points. Here they are:2

                                   m
           0      1      2      3      4      5      6      7      8
     ----------------------------------------------------------------
  2 |      1      0      1
  3 |      2      3      0      1
  4 |      9      8      6      0      1
n 5 |     44     45     20     10      0      1
  6 |    265    264    135     40     15      0      1
  7 |   1854   1855    924    315     70     21      0      1
  8 |  14833  14832   7420   2464    630    112     28      0      1

I then took these sequences (the columns) and searched for them in the Online Encyclopedia of Integer Sequences. The sequence for one fixed point is A000240, the one for two fixed points is A000387, the one for three fixed points is A000449, and the general one for any number of fixed points is A008290.

The nice thing about the OEIS is that it doesn’t just identify the sequences, it also gives you formulas and recurrence relations in several forms, tables, graphs, references, connections to other sequences, and even musical interpretations.

Apart from helping me find the formulas in the OEIS, the table showed a few other things:

  • The numbers on the main diagonal (\(m = n\)) are all ones. It should be obvious that there’s only one way to arrange all the cards in their original places, but it’s nice to see that the math works out that way.
  • The numbers immediately to the left of the main diagonal (\(m = n-1\)) are all zeros. This is only slightly less obvious: there’s no way to have just one of the cards out of its original place.
  • The numbers in the 0 and 1 columns always differ by one. It’s clear from the formulas why this is so, but I can’t think of a simple narrative explanation.

I think I’ve mentioned before that my doctor likes his patients to do puzzles and other mental stimulation as they move toward senior citizen status. Whether the “use it or lose it” theory is valid, it can’t hurt. Maybe I’ll show him this post at my next checkup as proof that I’m following doctor’s orders.


  1. I’ve carefully included the word “exactly” in these descriptions because I want to emphasize that they’re not the “at least” calculations we did earlier. Now that you’ve seen it a few times, we’ll take the “exactly” as given for the remainder of the post. 

  2. Sorry, I was too lazy to turn this into an HTML table. 


[If the formatting of equations looks odd in your feed reader, visit the original article]


Sort of handy

Permalink - Posted on 2020-05-19 22:38

This afternoon, I wanted to see how much disk space I had left on the server that runs this blog. As is often the case with Unix/Linux commands, after I getting the needed information, I started thinking about other ways to do things. And, as is also often the case, I learned something new. New to me, anyway.

First, I logged into the server and ran the df command:

df -h .

The output, which summarizes the disk usage of the file system containing the given file (in this case, the current directory, ., was my home directory) was

Filesystem                 Size  Used Avail Use% Mounted on
/dev/disk/by-label/DOROOT   25G  9.5G   14G  41% /

This shows I’m using 41% of the 25 GB I’m paying for. The -h option told df to use a “human” format for the output, Instead of showing the usage in “1-K blocks,” it shows it in kilobytes, megabytes, and gigabytes. Lots of GNU utilities have an -h option that works this way.

This disk usage includes everything on my virtual server—all the executables, libraries, and support files in addition to the files specific to the blog. I wanted to refine this to see just what the blog was using. That called for the du command:

du -hd 1 .

Once again, the -h option meant “human formatted values.” The -d 1 option told du to go only one directory level deep. The output was

8.0K    ./.gnupg
68K     ./pagelogs
114M    ./all-this
9.0M    ./.local
116K    ./php-markdown
1.5M    ./.cache
68K     ./.ipython
20K     ./.pip
8.0K    ./.ssh
522M    ./tmp
16K     ./bin
8.0K    ./.conda
1.1G    ./public_html
4.0K    ./.nano
3.4G    ./anaconda3
5.1G    .

The line for public_html was what I was looking for: 1.1 GB. So relatively little of the space on the server is being used for the blog.

As I looked at the du output, I thought it would be more useful to have it in numerical order. I thought about adding an -s option to du, but that doesn’t work. The du man page shows no option for sorting the output.

The standard Unix way of doing things would suggest piping the output to sort, but I was sure that wouldn’t work here. Because although sort has an -n option for sorting numerically, the numbers in du’s human output weren’t what needed to be sorted. It’s the quantities I wanted sorted, and that means the suffixes had to be accounted for. A test with

du -hd 1 . | sort -n

gave me

1.1G    ./public_html
1.5M    ./.cache
3.4G    ./anaconda3
4.0K    ./.nano
5.1G    .
8.0K    ./.conda
8.0K    ./.gnupg
8.0K    ./.ssh
9.0M    ./.local
16K     ./bin
20K     ./.pip
68K     ./.ipython
68K     ./pagelogs
114M    ./all-this
116K    ./php-markdown
522M    ./tmp

which confirmed my suspicions. Perfect for sorting the numbers but useless for sorting the quantities.

I could use a different output switch for du:

du -kd 1 . | sort -n

The -k tells du to output the sizes in kilobytes. The sorted output is

4       ./.nano
8       ./.conda
8       ./.gnupg
8       ./.ssh
16      ./bin
20      ./.pip
68      ./.ipython
68      ./pagelogs
116     ./php-markdown
1448    ./.cache
9184    ./.local
115896  ./all-this
534104  ./tmp
1098316 ./public_html
3470796 ./anaconda3
5333936 .

which is great until the numbers get up past five or six digits and you lose track of the order of magnitude.

But here comes the part where I learn something. It turns out the GNU folks recognized the need to read human-formatted values as well as write them, and they added an -h option to sort. So

du -hd 1 . | sort -hr

gives

5.1G    .
3.4G    ./anaconda3
1.1G    ./public_html
522M    ./tmp
114M    ./all-this
9.0M    ./.local
1.5M    ./.cache
116K    ./php-markdown
68K     ./pagelogs
68K     ./.ipython
20K     ./.pip
16K     ./bin
8.0K    ./.ssh
8.0K    ./.gnupg
8.0K    ./.conda
4.0K    ./.nano

which is exactly what I wanted: easy to read and properly sorted. (The -r switch tells sort to reverse so the biggest directories come first.)

The -h option was added to GNU sort in 2009, four or five years after I moved back to the Mac and stopped being as intense a command line user as I had been. I don’t feel too bad about not knowing of it.


[If the formatting of equations looks odd in your feed reader, visit the original article]


Timing in SSH

Permalink - Posted on 2020-05-19 15:42

A lot of the work I do “on” my iPad consists of me essentially using the iPad as a terminal to edit files and run commands on one of my iMacs. Editing files is done in Textastic and synced to the Mac through iCloud (used to be through Dropbox). Running commands is done through either Textastic’s internal terminal, which works well enough for simple stuff, or Prompt, which has more robust terminal emulation and is better for more complex sessions. Since a lot of the terminal work I do isn’t complex, and consists of repeated invocations of

pdflatex report

and

python script.py

the responsiveness of the SSH connection to the Mac isn’t important to my productivity. The most common key I use is ↑ to rerun the last command.

But when I do need to construct and run longer commands, I’ve noticed a lag between the keyboard and the screen. I am by no means a fast typist, but I regularly get ahead of the display. This leads to typos that take a while to correct because repetition on the ⌫ key also outruns the display. I often just don’t know where the cursor is on the Mac because it’s not necessarily where I see it on the iPad.

This is not a bug in Prompt or Textastic. When I log into the Linux box that hosts this blog, both terminal programs are smoothly responsive. That the web server is halfway across the country makes it all the more frustrating when I have a choppy terminal session to the Mac that’s no more than 20 feet away from me.

When I decided to try to fix this annoyance, my working assumption was that there was something in the SSH server configuration causing the hiccups. So that was where my Googling started. I found an older post with a suggested change to the DNS settings in /etc/ssh/sshd_config, but that didn’t help because the change had already been made in more recent versions of macOS.

Somehow, though, I ran into this answer from Pistos on StackExchange. Apparently, in its neverending quest to save battery, Apple is powering down the wifi system between packets, which means a delay when new packets arrive or need to be sent. This doesn’t materially affect file transfers or streaming because the packets keep coming, but it plays havoc with intermittent communication like a terminal session.

Pistos’s solution was to set up two connections: one that keeps up a constant, albeit low volume, flow of bytes between the Mac and whatever was connected to it; and another for what he really wanted to do. I took his solution and turned it into this short shell script, which I called nolag:

#!/usr/bin/env bash
while true; do echo -n .; sleep 0.5; done

Pistos used a sleep argument of 0.1 seconds, but I have found 0.5 works just as well.

Now what I do is start a Prompt connection to my Mac and run nolag. Then I duplicate that session in Prompt and do my real work there.

Prompt session menu

This works perfectly and saves me much frustration.

You might be wondering why I don’t just plug my iMac into a wired network. It’s because my home wired network was strung back in the 90s and has bottlenecks that make it slower than my current wifi setup. A better question might be why Apple is trying to save battery life on a Mac that doesn’t run on battery.


[If the formatting of equations looks odd in your feed reader, visit the original article]


Timing is everything

Permalink - Posted on 2020-05-16 23:16

Most people describe Shortcuts as being like Automator, but that misses the mark. Yes, there is a resemblance to Automator because of the block-like visual way you program them and I would never deny the connection, but Shortcuts’ deeper similarity is with AppleScript. Both Shortcuts and AppleScript have significant capabilities on their own, but their real power comes from exploiting hooks into apps and the underlying operating system. And, significantly, those hooks have to be programmed into the apps by the developers. For ages, Mac users have bemoaned the lack of AppleScript support in many of our apps; in the past few years, iOS users have had the same frustrations with apps and Shortcuts.

Recently, I’ve run across another connection between AppleScript and Shortcuts: they both have filtering operations that are really easy to use but which can be unbelievably slow. In AppleScript, there’s the whose construct, which provides a very compact and English-like way of filtering lists.

applescript:
tell application "ThisApp"
  set aVariable to every appClass whose property is value
end tell

The property is value part can be changed to any legal AppleScript condition:

applescript:
property is less than value
property contains value
property starts with value

And so on.

I use variations on this snippet quite often, both in one-off scripts to solve a single problem and in automations that run frequently. A couple of months ago, I wrote about how slow this kind of filtering can be with Calendar events, but it isn’t confined to that app. Another slowpoke is part of my system for following up on unpaid invoices at work.

I don’t want to get into the details of my invoicing system here. Suffice it to say that I have an Invoices list with an entry for every outstanding invoice. These entries have both a due date and a recurrence relation that keeps reminding me to send followup emails to the client until the invoice is paid. Whenever I send a followup email, I click or tap the completion button in Reminders, which

  1. Marks that entry as completed.
  2. Uses the recurrence relation to create an identical entry with a due date of (typically) two weeks later.

Note that marking an entry as complete does not delete it from the Invoices list. Over time, completed entries can build up unless they’re deleted.

The automation that goes along with this system runs at 5:00 am. It figures out which active invoice reminders will be due that day and generates the followup emails that will be sent to the appropriate clients. The emails are saved as drafts and are available on all my devices through iCloud. That way, whenever I get a notification to send a followup email, the email is already written and waiting for me.

The snippet of AppleScript that figures out which invoices need a followup is this:

applescript:
set tonight to (current date)
set time of tonight to 18 * 60 * 60
tell application "Reminders"
  tell list "Invoices"
    set duns to name of every reminder whose (due date < tonight) and (completed is not true)
  end tell
end tell    

The filtering, which is based on both the due date and the completed status, can take a surprisingly long time to run. With just a couple hundred entries in Invoices (all but a dozen or so of which are completed), it can take over 10 seconds to run this little snippet of AppleScript (on my Late 2012 27″ iMac).1

Now, as a practical matter, I don’t care how long this automation takes, because it runs on my office computer while I’m still at home in bed. But it is weird that a list of only a couple hundred entries takes so long to filter. And I’ve used this same filtering construct in other AppleScripts that I run while I’m sitting at the computer, and it’s always frustrating to wait for something you know shouldn’t take that long.

I learned that Shortcuts can have this same problem after listening to Episode 49 of the Automators podcast. David and Rosemary’s guest was Scotty Jackson, and one of the shortcuts he discussed needs certain contact information for people he works with. He’s using Data Jar for storing this information, which struck me as odd, because surely all that information is already in Contacts. Couldn’t he just organize his coworkers in groups and use the group names to filter them according to context?

So after listening to the episode, I decided to see if a shortcut that used Contacts would work. I have a “Work” group in Contacts, so I wrote a shortcut with just this one step:

Contacts Filter

You can’t beat the simplicity, but on my 2018 iPad Pro, with about 950 entries in Contacts and 5 people in the Work group, this takes 13–15 seconds to run. That’s way too long and explains why Scotty J violated the DRY principle and duplicated some of his Contacts information in Data Jar. The time he spent duplicating that data will be made up for in faster run times and less frustration.

As I learned a couple of months ago, the slowdown in filtering Calendar events via AppleScript has to do with the interaction between the Open Scripting Architecture and the Calendar app. I suspect that basically the same interaction bottleneck happens with Reminders, as Reminders is at least partially built on Calendar. And maybe there’s a similar structure to Shortcuts’s interaction with Contacts that makes it so slow.

Whatever it is, it’s especially annoying when you know how little time it should take. To get a sense of this, I exported my contacts as a CSV file and ran this little Python script:

python:
#!/usr/bin/env python

import pandas as pd
df = pd.read_csv('contacts.csv')
print(df[['First name', 'Last name']][df.Group=='Work'])

It took about 0.7 seconds to run on my 2012 iMac and I bet most of that was importing the Pandas library. There’s no excuse for Apple giving us tools that run slower than something an amateur like me can whip up in a couple of minutes.


  1. You might well ask, “Why don’t you clean up your completed reminders using that script you wrote a few years ago?” All I can say is we are all sinners. 


[If the formatting of equations looks odd in your feed reader, visit the original article]


Acceptance

Permalink - Posted on 2020-05-08 03:06

There’s a certain type of work I do for which I write a short proposal in single-page letter format and send it to the prospective client as a PDF attached to an email. When I started doing this type of work a few years ago, my thinking was that the client would simply email me back saying they accept the proposal and that I could start the work. But many clients wanted something more formal, or at least more analog.

They wanted something they could sign and send back to me. So I made a small PDF image I could use as a sort of stamp.

Acceptance stamp

After writing the proposal in LaTeX and generating a PDF, I’d open it in PDFpen and add the stamp. At first I’d just open the stamp file, copy it out of that document, and paste it into the proposal. Then I got more clever and added the stamp to my PDFpen Library, so there was no need to open a second file.

The problem with adding the stamp this way was twofold: I had to remember to do it, and then I had to do some manual work in PDFpen. Given that the stamp would always be in the same spot—lower right corner, with a ¾″ margin from the page edges—it seemed like I could get rid of the manual part.

I probably could have written an AppleScript to control PDFpen, but when I finally go around to automating it, I used PDFtk and the stamp option I wrote about last year. What I built was a short shell script that was basically the same as the overlay script I described in that post, but with one of the PDF files being set to the acceptance stamp file, which I had to “grow out” to be a full page.

Acceptance overlay page

This PDFtk automation didn’t last long. It solved the manual placement problem, but I still had to remember to run the script, and it turned out that that was the more important problem to solve. I began searching for a simple way to add the stamp directly in LaTeX.

There are undoubtedly ways to add the stamp using TikZ or some similar drawing package, but I didn’t want to learn a graphics macro language just to remake a drawing I already had. What I ended up using is the pdfoverlay package, which let me put my acceptance stamp in the background and layer the proposal letter over it.1

Pdfoverlay is included in recent TeX Live and MacTeX distributions. I don’t remember how old my installation was, but I had to update it to get pdfoverlay installed and working. Once I did, though, these two lines were all I needed to add to the preamble to my proposal letters:

\usepackage{pdfoverlay}
\pdfoverlaySetPDF{/path/to/acceptance.pdf}

I edited the TextExpander snippet I use for my proposal template to include these two lines, and now I don’t have to remember anything. Which is one of the main goals of automation, right?


  1. So it’s more like the inverse of a stamp. 


[If the formatting of equations looks odd in your feed reader, visit the original article]