# JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

And now it’s all this

I just said what I said and it was wrong. Or was taken wrong.

JSON

# Brace yourself, I’m in an expansive mood

Permalink - Posted on 2019-09-07 18:08

A longstanding truth of this blog is that whenever I write a post about shell t̸r̸i̸c̸k̸s̸ features, I get a note from Aristotle Pagaltzis letting me know of a shorter, faster, or better way to do it. Normally, I add a short update to the post with Aristotle’s improvements, or I explain whay his faster way wouldn’t be faster for me (because some things just won’t stick in my head). But his response to my jot and seq post got me exploring an area of the shell that I’ve seen but never used before, and I thought it deserved a post of its own. I even learned something useful without his help.

Here’s Aristotle’s tweet:

@drdrang Bash/zsh brace expansion can replace 99% of jot/seq uses (though bash < 4.x doesn’t support padding or step size 🙁). Your last example becomes much simpler:

printf '%s\n' 'Apt. '{2..5}{A..D}

You even get your preferred argument order:

printf '%s\n' {10..40..5}

Brace expansion in bash and zsh doesn’t seem like a very important feature because it takes up so little space in either manual. The brief exposure I’ve had to it has been in articles that talked about using it to run an operation on several files at once. For example, if I have a script called file.py that generates text, CSV, PDF, and PNG output files, all named file but with different extensions, I might want to delete all the output files while leaving the script intact. I can’t do

rm file.*


because that would delete the script file. What works is

rm file.{txt,csv,pdf,png}


The shell expands this into

rm file.txt file.csv file.pdf file.png


and then runs the command.

This is cute, but I never thought it worth committing to memory because tab completion and command-line editing through the Readline library makes it very easy to generate the file names interactively.

What I didn’t realize until Aristotle’s tweet sent me to the manuals was that the expansion could also be specified as a numeric or alphabetic sequence using the two-dot syntax. Thus,

mkdir folder{A..T}


creates 20 folders in one short step, which is the sort of thing that can be really useful.

And you can use two sets of braces to do what is effectively a nested loop. With apologies to Aristotle, here’s how I would do the apartment number generation from my earlier post:

printf "Apt. %s\n" {2..5}{A..D}


This gives output of

Apt. 2A
Apt. 2B
Apt. 2C
Apt. 2D
Apt. 3A
Apt. 3B
Apt. 3C
Apt. 3D
Apt. 4A
Apt. 4B
Apt. 4C
Apt. 4D
Apt. 5A
Apt. 5B
Apt. 5C
Apt. 5D


just like my more complicated jot/seq command.

The main limitation to brace expansion when compared to jot and seq is that you can’t generate sequences with fractional steps. If you want numbers from 0 through 100 with a step size of 0.5,

seq 0 .5 100


is the way to go.

And if you’re using the stock version of bash that comes with macOS (bash version 3.2.57), you’ll run into other limitations.

First, you won’t be able to left-pad the generated numbers with zeros. In zsh and more recent versions of bash, you can say

echo {005..12}


and get1

005 006 007 008 009 010 011 012


where the prefixed zeros (which can be put in front of either number or both) tell the expansion to zero-pad the results to the same length. If you run that same command in the stock bash, you just get

5 6 7 8 9 10 11 12


Similarly, the old bash that comes with macOS doesn’t understand the three-parameter type of brace sequence expansion (mentioned by Aristotle), in which the third parameter is the (integer) step size:

echo {5..100..5}


which gives

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100


in zsh and newer versions of bash. Old bash doesn’t understand the three-parameter at all and just outputs the input string:

{5..100..5}


We’ve been told that Catalina will ship with zsh as the default shell, which means we shouldn’t have to worry about these deficiencies for long. Because I don’t want to learn a new system of configuration files, I’m sticking with bash, but I have switched to version 5.0.11 that’s installed by Homebrew. My default shell is now /usr/local/bin/bash.2

One more thing. I said last time that seq needs a weirdly long formatting code to get zero-padded numbers embedded in another string. The example was

seq -f "file%02.0f.txt" 5


to get

file01.txt
file02.txt
file03.txt
file04.txt
file05.txt


What I didn’t understand was how the %g specifier works. Based on my skimming of the printf man page, I thought it just chose the shorter output of the equivalent %f and %e specifiers. But it turns out to do further shortening, eliminating all trailing zeros and the decimal point if there’s no fractional part to the number. Therefore, we can use the simpler

seq -f "file%02g.txt" 5


to get the output above. Because printf-style formatting is used in lots of places, this is good to know outside the context of seq.

Of course, now that I understand brace expansion, I wouldn’t use seq at all. I’d go with something like

echo file{01..5}.txt


1. I’m using echo here to save vertical space in the output.

2. Fair warning: I will ignore or block all attempts to get me to change to zsh. I’m glad you like it, but I’m not interested.

[If the formatting looks odd in your feed reader, visit the original article]

# Making lists with jot and seq

Permalink - Posted on 2019-09-02 15:12

The Mac comes with two command-line tools for making sequential lists: jot and seq. In the past, I’ve tended to use jot, mainly because that’s the one I learned first. But seq can do almost everything jot can, and its argument ordering makes more sense in many situations.

The point of both commands is to generate a list of strings that include sequential numbers.1 At their most basic, jot and seq each produce a set of lines with numbers that count up from one. Both

jot 5


and

seq 5


produce the same output:

1
2
3
4
5


More interesting results come from adding arguments and options.

## Arguments

jot can take up to four arguments; seq up to three. Here’s a table of how the arguments are ordered.

Arguments jot seq
1 count last
2 count first first last
3 count first last first step last
4 count first last step

As you can see, additional arguments get added to the end for jot but are inserted for seq. As we’ve seen above, for the one-argument case, jot’s count and seq’s last are effectively the same, because both commands have default starting points and step sizes of 1.2

For the two-argument case, both commands keep the default step size as 1 but allow you to change the starting point. To generate numbers from 5 through 10, you’d use either

jot 6 5


or

seq 5 10


To me, the seq arguments are easier to remember and a more natural expression of what’s intended. It’s easy to make an off-by-one error and try

jot 5 5


which will get you only up to 9.

As far as I’m concerned, jot with three arguments is worthless. If I know the starting and stopping points, and the step size is still at its default of one, the count parameter is redundant. You can avoid doing the subtraction in your head (and then remembering to add 1 to avoid the off-by-one error) by filling the count slot with a hyphen:

jot - 5 10


This is fine, but seq 5 10 is simpler.

The three-argument seq and the four-argument jot allow you to specify the step size. Again, jot’s count is redundant and it’s best to put a hyphen in its place to allow the computer to do the arithmetic. To get from 10 to 40 counting by 5s, enter

jot - 10 40 5


to get

10
15
20
25
30
35
40


The seq command to get the same result is

seq 10 5 40


This is the one place where I find seq’s arguments hard to remember. So many programming languages use the order first, last, step that my fingers do that before I can stop them. There’s nothing illogical about the first, step, last ordering, but it’s not what I’m used to.

## Options

Just printing out lines with numbers is dull. The real value of jot and seq comes from applying options (often called switches) that allow us format the numbers and include them in other text. Here are the options for jot and seq:

Option jot seq
output format -w string -f string
repeated string -b string
equal width -w
separator -s string -s string
omit final newline -n
precision -p decimals
characters -c
random -r

The option I use most often is -w for jot and the nearly equivalent -f for seq. In both cases, the string argument is a printf-style formatting string. So if I wanted to generate a bunch of sequential file names, I could use

jot -w "file%02d.txt" 5


or

seq -f "file%02.0f.txt" 5


to get3

file01.txt
file02.txt
file03.txt
file04.txt
file05.txt


Here’s a situation where jot is a little more sensible. Even though both commands treat the numbers they generate as floating point, jot recognizes when the numbers have no decimal part and allows you to use the %d format specifier. seq forces you to use a floating point specifier, like %f or %e, which then forces you to explicitly state that the number of digits after the decimal point is 0. If you don’t, you’ll get a mess. For example,

seq -f "file%02f.txt" 5


returns

file1.000000.txt
file2.000000.txt
file3.000000.txt
file4.000000.txt
file5.000000.txt


The -b option for jot will repeat the given string as many times as specified. The string isn’t a formatting string and won’t include the numbers generated by jot.

jot -b Hello 4


returns

Hello
Hello
Hello
Hello


which I guess could be useful. You might think you could use the -w option with a string that doesn’t include a formatting specifier. But if you try that, jot will add the number to the end anyway.

jot -w Hello 4


returns

Hello1
Hello2
Hello3
Hello4


which is kind of presumptuous of it.

seq is better behaved.

seq -f Hello 4


returns

Hello
Hello
Hello
Hello


just as you’d expect.

By the way, if you want just a list of numbers, but you want them zero-padded to be of the same character length, seq has you covered:

seq -w 8 12


returns

08
09
10
11
12


You could achieve the same thing with a formatting string, but the -w option is a little shorter.

We’ve been looking at output where each item in the sequence is on its own line—i.e., separated by newlines—but both commands have an -s option for specifying another separator. The option works a little differently for the two commands.

jot -s, 5 8


gives

8,9,10,11,12


whereas

seq -s, 8 12


gives

8,9,10,11,12,


You see that seq’s “separator” (that’s what it’s called in the man page) is really a suffix. To get the same output as jot, you can pipe the output through sed to delete the last character:

seq -s, 8 12 | sed 's/.$//'  Kind of annoying. One advantage seq has over jot with regard to separators is that it understands tabs. seq -s "\t" 8 12 | sed 's/.$//'


will give you the numbers separated by tab characters. But jot treats the backslashed t literally.

jot -s "\t" 5 8


will give you

8\t9\t10\t11\t12


One more thing related to -s. Because jot treats the separator as an actual separator, the newline that’s added by default to the end of the output isn’t handled by -s. If you don’t want jot to add it, including the -n option will omit it. This is just like the -n option to echo.

As mentioned above, both commands handle floating point numbers, but they have different defaults.

seq -w 7 .5 9


returns

7.0
7.5
8.0
8.5
9.0


as you would expect. But the similar

jot - 7 9 .5


returns

7
8
8
8
9


which seems bizarre. The trick is that jot has a default precision of zero decimal places, so it’s rounding to the nearest integer. The default precision can be changed with the -p option.

jot -p1 - 7 9 .5


returns

7.0
7.5
8.0
8.5
9.0


which is almost certainly what you wanted.

Finally, I want to talk about one clear advantage that jot has over seq: the ability to generate sequences of characters using the -c option.

jot -c 5 65


returns

A
B
C
D
E


What it’s doing is interpreting each number as an ASCII code and returning its corresponding character. Even better, you can use character arguments:

jot -c - a e


returns

a
b
c
d
e


You can get the same effect by using the %c formatting specifier in the -w option. I’ve used this feature of jot to generate apartment addresses when working with a building that uses letters for its units. For example, say I am dealing with a five-story building in which the apartments start on the second floor and are given the letters A through D. I want to quickly generate a list of all the apartment addresses. The command is

for n in seq 2 5; do jot -w "Apt. $n%c" - A D; done  and the output is Apt. 2A Apt. 2B Apt. 2C Apt. 2D Apt. 3A Apt. 3B Apt. 3C Apt. 3D Apt. 4A Apt. 4B Apt. 4C Apt. 4D Apt. 5A Apt. 5B Apt. 5C Apt. 5D  As a practical matter, with only 16 apartments, I probably wouldn’t bother with seq or jot. It’s faster to just type out the apartment addresses when the number is small. But with larger buildings, a command like this saves a lot of time and insures that I don’t skip or duplicate addresses. As with any automation, accuracy is as important as the time saved. 1. jot also has an option for generating random numbers, but I’ve never had any use for that. 2. That the counts start at 1 instead of 0 is an indication that these commands were written for normal humans, not programmers. 3. The quotation marks aren’t necessary in either of these cases because the shell won’t process the formatting string before sending it to jot or seq. But I have a hard time remembering which characters are special to the shell, so I tend to use quotation marks any time I have a string that includes potentially special characters like % [If the formatting looks odd in your feed reader, visit the original article] # How to mock your Apple Card Permalink - Posted on 2019-08-22 14:01 I feel the need to expand on this tweet from last night: I use thousands of dollars of equipment from the company that wrote this. The quote comes from Apple’s “How to clean your Apple Card” support document, which went up earlier this week. The one-paragraph jump from “leather and denim may stain your card” to “keep your card in your wallet or your pocket” generated lots of complaints on Twitter, mostly of the form “That’s Apple, putting form over function.”12 My complaint is not that the Apple Card may lose its luster in a wallet. I’m not sure anything will maintain its looks when put between sheets of leather and compressed by my butt. My complaint is that Apple wrote a support document that looks absurd and invites snarky comments. Everything Apple does generates derision from Apple haters; this generated derision from Apple’s best customers. The support document is, in fact, putting function over form. Apple wants to tell its customers that the card won’t look brand new forever and advise them on the best way to store it. That’s the function of the document. But through bad writing—how many people read this before it was published?—it looks like Apple made a fragile card and is advising you to store it in a way that will destroy it. Instead of invoking Louis Sullivan, we should be be turning to Casey Stengel: Can’t anybody here play this game? 1. If Louis Sullivan knew how often his words would be abused by people with no sense of form or function, he might have bit his tongue. As reader Scott Wright said, whatever staining might occur doesn’t affect the function of the card. 2. Apple critics would argue that the real function of the Apple Card is not to pay for things but to look cool. If that’s the case, though, form and function are the same, and Apple can’t put one over the other. [If the formatting looks odd in your feed reader, visit the original article] # Unpaid shortcut redux Permalink - Posted on 2019-08-10 12:54 There’s an old programming adage that says you can write Fortran in any language. Although I’ve made progress, I’m still writing parts of my Shortcuts in Fortran. After reading my last post, Majd Koshakji tweeted me a couple of suggestions: By following this advice, I was able to make the Unpaid shortcut both shorter and easier to understand. Here’s the before and after: The first thing to notice is that the action near the bottom of the shortcut—the one with the regular expression that uses a lookahead—has been replaced with the far simpler action. There are two things to say about this: 1. I didn’t know there was a action. Discovering what’s available in Shortcuts is still a struggle for me. 2. Even if I’d known about it, I’m not sure I would have guessed that adds thousands separators. There’s no checkbox for it in the action itself (as you see in Numbers, for example), nor is there any mention of it in the highly detailed popup documentation. But using simply replaces one action with another. The real shortening comes from using Majd’s other suggestion. The action performs any one of several calculations on a list of numbers. I knew about when I wrote the original version of Unpaid, but I didn’t think it would simplify the shortcut. I thought I’d have to build up the list of numbers in a variable that was updated with each pass through the loop. That meant initializing a variable before the loop and adding a term to the end of it within the loop. Because this is basically what I was doing with the total variable, I didn’t see any advantage. As is so often the case with Shortcuts, I underestimated the power of magic variables. The loop creates a list on its own; with each pass through the loop, it adds the output of the last action within the body of the loop to that list. That magic variable, called Repeat Results, is the output of the loop. So there’s no need to build a list of numbers explicitly, Shortcuts is doing it for me. All I need to do use is pass it into the action. It’s shown up above, but let’s show the new version of Unpaid its own:1 There are lots of ways to get a program to work. But every language has certain characteristic features that give it syntactical advantages. You need to learn those features if you want to program in a style that’s efficient for that language. Perl has a magic variable, $_, that is—or can be—the implicit argument to lots of functions; idiomatic Perl uses that magic variable to streamline its code. Idiomatic Python uses list comprehensions to eliminate certain types of loops.

I’ve been getting better at using magic variables in shortcuts. I don’t put the output of scalar operations (like the action) into explicit variables anymore. But I didn’t understand how loops output lists until now.

Thanks to Majd for the suggestions on this specific shortcut and for helping me become a better Shortcuts programmer in general.

1. Let me forestall questions on Twitter: the long screenshots, including the iPhone frames, are assembled in PicSew

[If the formatting looks odd in your feed reader, visit the original article]

# A Shortcut for Reminders

Permalink - Posted on 2019-08-06 13:16

I’ve written several times about how I keep track of and follow up on unpaid invoices, most recently here. It’s a system that’s evolved over time to become both more automated and less reliant on me being in my office in front of my Mac. This past weekend, I wrote a Shortcut that allows me to quickly calculate the total of my outstanding invoices from an iOS device., something I’ve been able to do on my Mac for quite a while.

To recap, every time I generate and send out an invoice at work, an item is added to my Invoices list in Reminders. The Reminder is set to follow up with the client 45 days1 after the invoice is issued (assuming it hasn’t been paid) and includes the project name, project number, invoice number, and invoiced amount.

What I wanted was a quick way to get the total of all those dollar amounts in parentheses. The solution was a Shortcut, named Unpaid, that was saved to my home screen and then stored in a folder with other work-related items.

Tapping the Unpaid icon runs the shortcut, which finishes by displaying a dialog box with the number of unpaid invoices and their total.

It’s obviously not good when this number gets too high, but it’s also bad when it gets too low. That usually means I’ve been delinquent in sending out invoices.

The Unpaid shortcut, which is on all my iDevices, looks like this:

The first two steps initialize the variable total to zero. This is where we’ll accumulate the sum of all the invoiced amounts.

The next step gets all the items in the Invoices list that are unchecked. I should mention that all these reminders are set to repeat. Checking an item on the list doesn’t mean the bill has been paid, it means that I’ve sent a followup email on the bill, and it creates an new item to follow up again two weeks later. When an invoice is paid, I delete its reminder from the list.

We then get a count of the reminders, which we’ll use later.

The bulk of the work is done by a loop that goes through each item in the list, pulling out the title (), extracting the parenthetical with a regular expression (), deleting the extraneous characters—parentheses, dollar sign, comma—from that string ( and ), and adding that amount to the running total ( and ).

Finally, total is formatted with commas () and included with the item count in message to the user ().

Overall, it’s a pretty simple shortcut. The trickiest parts are the regular expressions. If you don’t care about regexes, you can consider the post finished right here. The rest is a discussion of how each of them work.

The first regex, in , is this:

$$\[^)]+$$


It’s messy because it has to escape the dollar sign and the parentheses at the beginning and end. But the idea is simple: find any string that starts with an opening parenthesis and a dollar sign and collect every character through the first closing parenthesis.

The second regular expression, in the first step, is this:

[()$,]  It defines a character class consisting of opening and closing parentheses, dollar signs, and commas. Any instance of this class is replaced by an empty string, i.e, deleted. I do this because the amount of the invoice has to be parsed as a number, and these extraneous characters prevent that.2 An unstated feature of Shortcuts’s step is that it performs a global replace. All instances of the find string are replaced, not just the first one. In Perl terms, this is like using the g flag. Finally, we have the hardest regex, the one in the final : (\d)(?=(\d{3})+\.)  I adapted this from a Stack Overflow answer that handled integers. It starts with a digit and then the (?=… ) construct, which is called a positive lookahead. The lookahead finds groups of three digits (one or more of such groups) followed by a period. The key is that the regex engine considers the lookahead to have zero width, which has two effects: 1. It allows us to use the simple replace string, $1,


which puts a comma after the digit captured in the first set of parentheses. We don’t have to worry about the part of the string matched in the lookahead because the engine considers the match to have ended just after the captured digit.3

2. It means that after the first replacement, the regex continues its search after the comma. This allows it to put commas between all the groupings of three digits. The pattern will not only convert 51600.61 to 51,600.61, it will also convert 51600600.61 to 51,600,600.61, and so on, into the billions, trillions, etc. If the search pattern had been

(\d)((\d{3})+\.)


and the replacement pattern

$1,$2


we’d turn 51600.61 into 51,600.61 as before, but 51600600.61 would be turned into 51,600600.61 because the search would pick up after the period and skip the second comma. The zero-width makes all the difference.

You could, of course, argue that this application has no need to format numbers in the millions or higher. I’ve never even had as much as a six-figure accumulation of unpaid invoices, let alone a seven-figure one. So a simpler pattern, one that anticipates no more than one comma separator, would work. But it’s more fun to explore the dark corners and learn new things.

1. Strictly speaking, it’s the first Tuesday that comes after 45 days, but that detail isn’t important here.

2. Actually, Shortcuts seems to be able to parse commas in numeric strings, but I deleted them anyway to be on the safe side.

3. Oddly, you can capture things by using parentheses inside the lookahead construct, but the regex engine still considers the match to have ended just before the lookahead.

[If the formatting looks odd in your feed reader, visit the original article]

# Gonna Roll the Bones

Permalink - Posted on 2019-08-03 17:02

Early last month, James Thomson released a new iOS dice-rolling app called, with an eye to the App Store’s search function, Dice by PCalc. Being James, he didn’t just hook up a random number generator to an animation of dice, he used a physics engine to simulate the mechanics of rolling dice.

My first thought was “how random will this simulation be?” What we think of as randomness in dice-rolling and coin-tossing is really based on the chaos inherent in the dynamics of these actions. A coin flip, for example, is a completely deterministic process and seems random to us (and can pass statistical tests) only because the results change significantly due to small changes in the initial conditions.1 There’s a great Numberphile video in which Persi Diaconis discusses this. In fact, you should just watch all the Diaconis videos, including the two on fair dice.

So my question was really about how good the physics engine was at simulating real chaotic dynamic processes. Would the rolls that come out of Dice pass the kind of statisical tests that rolls of fair physical dice would?

The best way to check this would be to generate a bunch of rolls in Dice and then run a statistical test on the results. Here is where my laziness kicked in. Sure, I was interested in this, but was I interested enough to do all the tedious work necessary to collect the data?

Yes and no. I certainly wasn’t going to roll with Dice and then type in the number that came up. Even with two iOS devices running in parallel, one for the rolling and one for the typing, that was too painful to contemplate.

I then thought about dictating the numbers. I’ve had success dictating measurements while I’m working in the lab. But then I realized I didn’t have to do the dictation myself.

Dice has a setting for speaking the results. By turning that on, I could put my iPad and iPhone near each other and have Dice running on the iPad dictating its results to Drafts running on the iPhone.

I figured Drafts was the best app to dictate into because it’s more forgiving of pauses than other apps and there were definitely gaps between rolls. Even so, Drafts would typically time out after 20–25 rolls, so I got in the habit of stopping dictation when the line of numerals got to that length.

By continually tapping Dice’s Reroll button, I soon had a list of about 1000 rolls (1005, to be precise) of a single six-sided die collected in Drafts.

(The short lines came when I mistapped on one of the two devices and I had to restart the dictation on a new line.)

Now it was time to analyze the data. First, I cleaned up the data by searching for all the newline characters and deleting them. That gave me one long string of numerals that I could paste into my Python analysis script.

The purpose of the script is to count all the occurrences of each number. We can then use the chi-square test to see if the counts are close enough to equal to be considered uniformly distributed.

Here’s the script (where I’ve broken up the dice string to make it easy to read):

python:
1:  from collections import Counter
2:
3:  dice = '62331245646253365416252416456666441662363644345422542256142\
4:  1466414261214312335455454662646535364552643553665562651445113223516\
5:  4345133236466256615163133461424555341161364531342162154345456123551\
6:  5423652314323336623453164254465211353346441264444255242555423541323\
7:  6533463525333334261214625566242633555332152324134625433336551162653\
8:  6315124456213426444412453433411545664123666142441221443216112523321\
9:  6152221326121156452653165253554144341516263223352541216535363436646\
10:  4541465526654644463253423326446544441415433335134414135322626155446\
11:  4312665234231443443266324544222633214232324134645313425461251615143\
12:  6632166254234354361564226654553242645146115336541241611551536125452\
13:  5232345614355646146336344234364241521341565322613665651434435414414\
14:  1232452266522616432354611625545222424146665126511162164412245423651\
15:  2432445513445453562253441623145615244351443355253425216214125633642\
16:  4532111621412634643555163546232311251341431622614114561262153142162\
17:  5161465461436565564566513311562131144611523336626164155421421515335\
18:  53622163'
19:
20:  n = len(dice)
21:  m = n/6
22:  x = Counter(dice)
23:
24:  chi2 = 0
25:  for i in range(1, 7):
26:    k = str(i)
27:    x2i = (x[k] - m)**2/m
28:    print(f'{k}:  {x2i:.4f} ({x[k]}/{m:.2f})')
29:    chi2 += x2i
30:  print()
31:  print(chi2)


Lines 20–21 calculate the expected number of each count if the die is fair. Line 22 then uses the Counter class from the collections module to create a dictionary, x, for the counts of each number.

We then loop through the possibilities, 1–6, and sum up the chi-square statistic,

where the $O_i$ are the observed counts collected in x and the $E_i$ are the expected counts, which in this case are all the same value, m. You can see the Python expression of this formula in Lines 27 and 29.

As we go through the loop, Line 28 prints the observed and expected values. When the loop is finished, Line 31 prints the $\chi^2$ value.

The results are:

1:  0.9328 (155/167.50)
2:  0.5388 (158/167.50)
3:  0.1209 (163/167.50)
4:  4.8493 (196/167.50)
5:  0.0015 (167/167.50)
6:  0.0134 (166/167.50)

6.456716417910448


The count for 4 (196) looks a little suspicious, and we see that it’s the main contributor to the $\chi^2$ value of 6.457. As we can see from the formula, higher values of $\chi^2$ mean more observed counts further from the expected values. But how high is too high?

Back in the pre-computer days, we used to look up values of the chi-square distribution from tables in the backs of our textbooks. Now we can do the calculations directly. Here are the results from an online chi-square calculator:

Before discussing the results, let’s talk about “degrees of freedom.” Recall that we were able to calculate the expected number of rolls for each value (167.5) because we knew the total number of rolls (1005). And since we know the total number of rolls, the six individual occurrence counts are not independent: they must add up to the known total. If we know five of the individual counts, the sixth is automatically determined by subtraction. Therefore, this set of counts is said to have only five degrees of freedom. The number of degrees of freedom is the parameter that governs the chi-square distribution.

OK, so now let’s look at the results. In the bottom two lines, we see that

This means that if we had a run of 1005 rolls from a fair die, there is a 74% chance that they would be more uniform than what we got in our set of 1005 rolls and, conversely, a 26% chance that they would be less uniform.

Is this evidence of an unfair die? No. This is like flipping two coins and getting two heads—not unusual at all. People typically start considering unusual behavior to be statistically significant when the probability of it happening by chance is less that 5%. In our problem, that would correspond to a $\chi^2$ value of 11.1 or higher.2

So the upshot is that James’s dice rolls look to be as random as any real dice rolls. You can use Dice with impunity.

1. Science fiction and fantasy readers may recognize that I stole the title of this post from a well-known story by Fritz Leiber in which the main character is a craps player who, when he’s “on,” is able to control the initial conditions so well that he can roll whatever he wants.

2. You might be wondering why I didn’t use the SciPy library’s chi2 distribution functions or, better yet, its chisquare test function, which would have done all the calculations for me in a single step. It’s because I was doing this on the iPad in Pythonista, and Pythonista doesn’t include SciPy. And doing it this way made the process more explicit. Black box solutions are best to use only after you understand what’s going on inside the box.

[If the formatting looks odd in your feed reader, visit the original article]

# The intersection of technology and liberal arts

Permalink - Posted on 2019-08-03 02:01

That Federico Viticci has quite an eye.

This week’s episode of Connected started with the boys talking about Stephen Hackett’s reversion to his Genius days at the Macstock conference. On Saturday, he did a RAM upgrade and replaced a hard drive with an SSD in Father John’s iMac. On Sunday, he did a RAM upgrade on Rosemary Orchard’s new Mac mini. While teasing Stephen for his good deeds, Federico said the photo of the latter upgrade (linked in the show notes) reminded him of a Renaissance painting. I was stunned. I knew he had to be talking about my photo, because that was exactly the effect I was going for.

Here’s the photo.

I took the picture specifically because the scene reminded me of The Last Supper. The foreshortening of the table draws your eye down to where the action is and is similar to the Last Supper’s ceiling. The people watching Stephen are like the apostles.

I disagree with Federico’s comment about the golden spiral, though.1 I was going for triangular construction, just like da Vinci. Note the three heads of the gentlemen standing at the other end of the table. The near edge and long sides of the table also suggest a triangle. The best triangle would have been the head and folded arms of David Sparks, but he turned bit too far sideways before I could snap the photo.

You might think I’d end with a comment about the relationship between the golden spiral and the Fibonacci sequence, which the boys were fumbling around with, but the internet is loaded with articles about that. I assume you know how to Google.

1. And Ed Cormany will be hearing from my lawyers regarding his defacing of my masterpiece.

[If the formatting looks odd in your feed reader, visit the original article]

# File extensions and colors

Permalink - Posted on 2019-07-29 16:54

For the past couple of months, I’ve been testing out iCloud Drive as a replacement for Dropbox. This was inspired by my growing tendency to work on my iPad Pro and the improvements to the Files app. While Dropbox has long been the standard way for iOS text editors to sync files, that environment is clearly shifting and strong integration with Files (where Dropbox will always have second-class status compared to iCloud Drive) is the way of the future. But Files still presents obstacles to the way I work; luckily I have a way around one of those obstacles, thanks to David Sparks.

I write my reports in LaTeX and generate PDFs to send to my clients. In the process of creating the PDF, LaTeX and its helper programs also create several other files. For example, if my LaTeX source code is test.tex, after compilation the folder will include test.aux, test.fls, test.out, test.fdb_latexmk, and test.log in addition to test.tex and test.pdf. All these files with the same base name but different extensions is no problem when working in the Finder on a Mac because I can see the extensions. But Files doesn’t show extensions.

The little icons help distiguish the PDF from the rest, but the files that Files sees as text tend to look the same, as do the file with extensions it doesn’t understand at all. So if I need to look through the log file to figure out an error, I’m likely to open the wrong one.1

I was complaining about this to David Sparks during a break at this weekend’s MacStock conference. He asked the very reasonable question “Isn’t there a setting that will let you see the extensions?” As far as I’ve been able to tell, the only way to see a file’s extension is to long-press on it

and select the Info from the popup.

This three-step process—long-press for popup, tap Info, dismiss Info window—is not something I want to keep doing.

“What about assigning colors to the different extensions?” said David.

“Oooh,” said I.

As a general rule, I don’t think much of tags as an organizational strategy. I’m much more comfortable with folders and subfolders. But in this case, they’re a great workaround because I’d only need a few, colors are already provided by default, and I’d be using them only in my work project folders.

So when I got home that night, I went to Hazel and made up some rules. I have individual folders for each work project, but they’re all inside a folder called projects. So all my new rules are assigned to that folder.

The first rule makes sure the color-tagging rules are applied to all the subfolders.

The other rules assign colors based on the file extension.

You’ll note that I have a rule for coloring Python source files. Obviously, LaTeX doesn’t generate a test.py file, but when I write a program to do analysis for a report, I often give the program and report the same base names. I suppose I could force myself out of this habit, but why should I have to?

The Hazel rules run on my Mac, of course, but the changes they make propagate across iCloud. Now I can distinguish at a glance the LaTeX source, the Python source, and the LaTeX log files.

I can’t remember the last time I looked at an .aux, .out, or any of the other files, so I didn’t bother making rules for them. I can do that later if needed.

This workaround doesn’t let Apple off the hook. The extension is part of a file’s name, and we should be able to see it directly from within Files, just as we can in the Finder. It’s not like there isn’t room for the extension, especially in list view. My guess is that this is not something we’ll be seeing in iOS/iPadOS 13, so the workaround is likely to be useful for quite while. Thanks again, David!

1. I should mention here that while I’m editing and reviewing my report on my iPad, I’m compiling it on one of my Macs via an SSH connection. Sometimes I make this connection through Prompt and sometimes through the Terminal feature of Textastic

[If the formatting looks odd in your feed reader, visit the original article]

# Zoom out

Permalink - Posted on 2019-07-09 17:55

This vulnerability in the Macintosh client for the Zoom audio/video conferencing system has gotten a lot of attention since it was published yesterday. In a nutshell:

1. Safari forces users to click a button to confirm they want an app launched.1 This would mean an extra click for Zoom users.
2. To get around this terrible imposition on their users, Zoom installed a webserver on their computers without telling them. A webserver that remains present and running, even when the Zoom app is not running. Even if you uninstall the Zoom app.
3. There is a set of commands you can run to kill the webserver and prevent its relaunch, even if you still want/need to use Zoom. The commands are near the end of the Medium post by Jonathan Leitschuh, who discovered all this. Ironically, the commands sometimes don’t appear in Safari on the Mac, apparently because of some incompatibility between the Safari and the GitHub embedding code used in the post.

I don’t pretend to follow all of Leitschuh’s explanation of the vulnerability, but I do understand the commands for the fix. I thought I’d talk about what they do. Also, there’s a cut-and-paste solution getting some traction on Twitter that I want to talk about.

First, here’s what Zoom says about the installation of the webserver:

Second, when Zoom is installed on a Mac device by the user, a limited-functionality web server that can only respond to requests from the local machine is also installed on the device. This is a workaround to a change introduced in Safari 12 that requires a user to confirm that they want to start the Zoom client prior to joining every meeting. The local web server enables users to avoid this extra click before joining every meeting. We feel that this is a legitimate solution to a poor user experience problem, enabling our users to have faster, one-click-to-join meetings. We are not alone among video conferencing providers in implementing this solution.

There’s no question that Mac users have a tendency to complain about changes Apple makes to the Mac’s OS and first-party software (I may have even done some complaining myself). But generally speaking, we think Apple’s focus on security is a good thing, and we certainly don’t want apps installing servers on our computers without telling us about it. Even if it means the “poor user experience” of (horrors!) an extra click when we want to use video. Note that Zoom doesn’t discuss the poor user experience of finding your computer’s camera turned on when you visit a malicious website.

Assuming you want to keep Zoom on your computer, the first step in Leitschuh’s fix is to make sure Zoom’s “Turn off my video when joining a meeting” setting is clicked on:

Let us bask for a moment in the warm glow of a user interface that asks you to turn something on in order to turn something off. And remember, Zoom is very concerned about poor user experiences.

With that bit of GUI work done, we have to move to the command line to rid ourselves of the unwanted webserver. These steps will be needed even if you deleted the Zoom app.

The unwanted webserver uses port 19421. We can find which process is using that port through the lsof command:

lsof -i :19421


lsof means “list open files,” and it’s using the expansive Unix definition of a file, which is basically anything that can read or write a stream of bytes. An internet address that’s listening for input, which our webserver uses, fits within that definition. The -i option to lsof allows us to specify as much of an internet address as necessary. We could, for example, give the protocol (TCP) and host (localhost) along with the port, but in this case only the port number (which has to come after a colon) is necessary.

The output is a table of processes that fit the specification

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ZoomOpene 59742 drang   7u  IPv4 0x31175d99fb28ad5b      0t0  TCP localhost:19421 (LISTEN)


The header line tells us what’s in the columns below. In this case, there’s only one entry. You can see from the COMMAND that it’s related to Zoom (the full name of the command is trucated to fit) and from the USER that it’s running as me. The key item is the PID, or process identification number. That’s the ID of the webserver we want to stop.

Stopping a process in Unix means using the kill command. Despite its name, kill normally just sends a signal to the given process to “please finish up what you’re doing and then stop, if you don’t mind.” But you can give it an option to be more forceful. The -9 option is aggressive; the process can’t ignore it and shuts down immediately. So our next command is

kill -9 <PID>


where <PID> is number we got from the lsof output.

This tweet from Mateusz Stawecki uses a Unix pipeline to get the PID from the lsof command and feed it to the kill command.

Given that the point of this exercise is to enhance the security of our computers, it’s hard to justify copying and pasting a command from Twitter. But Stawecki’s command does work. Let’s see why.

Ignoring for a moment everything that comes after the first semicolon, the pipeline is

lsof -i TCP:19421 | awk 'NR > 1 {print $2}' | xargs kill -9  As we’ve seen, the TCP part of the lsof command is an unnecessary but harmless addition. The output of lsof is fed to the awk command, which skips over the header line (NR > 1) and outputs the second field (print$2) of the next line. As we’ve seen, this second field is the PID of the process we want to kill. The xargs command takes the PID and turns it into an argument for the kill command.

There’s an easier way to do this, eliminating the awk command entirely. lsof has a “terse” option, -t, which reduces the output to just the PID(s) of the matching process(es). So we can cut Stawecki’s pipeline down to just two steps:

lsof -ti :19421 | xargs kill -9


And since there’s only one matching process, we could reduce the command even further:

kill -9 $(lsof -ti :19421)  The $(command) construct says “run this command and put its output here.” Because I like building up my commands step by step, I prefer the xargs solution, but either will work.

Now we’ve killed the web server, but we need to do more to prevent it from coming back. Zoom put a folder named .zoomus in your home directory, and inside that folder is an app called ZoomOpener. This is the app that launches the web server.

We get rid of the app and its enclosing folder through

rm -r ~/.zoomus


Leitschuh’s article and Stawecki’s pipeline both use rm -rf, but the force (-f) option is unnecessary. The Unix rm command is powerful and a little scary, especially when combined with the recursive (-r) option. You may prefer to use the Finder to throw the .zoomus folder into the Trash and empty it. If so, you’ll have make the folder visible (files and folders that start with a period are invisible by default). This is done by pressing ⇧⌘. (that’s Command-Shift-Period). Press it again to go back to the usual Finder view.

With the ZoomOpener app deleted, we must be finished, right? Wrong. The Zoom app will recreate the folder and the ZoomOpener app the next time you launch it unless you do this:

touch ~/.zoomus


The touch command is meant to update an existing file’s access and modification dates without changing its contents, but if the file you give it as an argument doesn’t exist, it will create it. For reasons I don’t quite understand, putting a file named .zoomus in your home directory prevents the Zoom app from creating a .zoomus folder and installing ZoomOpener in it.

If you go back to Stawecki’s tweet, you’ll see that he has the rm and touch commands in there after the pipeline. Again, I don’t blame you for being leery of using his commands, but they do work and won’t ruin your system.

Of course, the real question is why should you trust me? Or anyone? You may want to read Ken Thompson’sReflections on Trusting Trust.”

1. Thanks to Jeff Johnson for clearing up my initial confusion about the confirmation. Jeff makes StopTheNews, an app that handles Apple News URLs and is the sort of app that gets launched from a URL and requires a confirmation. Pretty sure Jeff hasn’t added a web server to get around that.

[If the formatting looks odd in your feed reader, visit the original article]

Permalink - Posted on 2019-07-04 13:57

When in the course of human events it becomes necessary for a person to dissolve the computational bands which have connected him with an app and to assume among the powers of the earth, the separate and equal station to which the Laws of Mathematics and of Polynomials entitle him, a decent respect to the opinions of other users of the app requires that he should declare the causes which impel him to the separation.

A few months ago, I explained why I wrote my own combinations function for PCalc. Today I declare my independence from its quadratic formula function.

We all know the quadratic formula. It works on equations that look like this,

and provides the two solutions:

It’s the ± that gives us two solutions. One of the solutions uses the minus and the other uses the plus.

PCalc comes with a quadratic solver function. You can find it in the Special section of functions, and it has a very long and explicit name.

What the name is telling you is to put the a term in the X register, the b term in the Y register, and the c term in the memory (or the M0 memory if you have multiple memories turned on). Then the two solutions, $x_1$ and $x_2$, will come out in the X and Y registers.

This solution has the advantage of being usable to every PCalc user. It works whether you have multiple memories on or not. More important, it works whether you’re in algebraic or RPN mode. It’s also presented in a form that should be familiar to anyone who knows what the quadratic formula is. PCalc’s developer, James Thomson, was careful to set it up with universal applicability.

But I don’t care about universal applicability. I care about what seems natural to me, and the placement of the a, b, and c terms has never seemed right to me. So I wrote my own quadratic solver function that fits with the way I think.

First, there’s no need for three inputs. Any quadratic can be put in the form

without any loss of generality by dividing through by the leading coefficient. This means the inputs to the function can fit in the X and Y registers—no need to use one of PCalc’s memory registers.1

Second, I prefer to enter the b term in the Y register and the c term in the X register. In RPN mode, I enter them onto the stack in what I consider their natural order: b first and then c. Even if I were in algebraic mode—which I never am—it would seem more natural to me to enter b first, press the key to put it in the Y register, and then enter the c term.

So with that design of the inputs, the PCalc function for the quadratic formula looks like this (the long screenshot was stitched together with PicSew):

If you don’t want to enter the steps yourself, you can download it.

Note that if the equation has complex roots, my function will throw an error. This is consistent with PCalc’s built-in quadratic function and with how it handles the square roots of negative numbers. PCalc is a real calculator.

To solve a quadratic, I just enter b and c in that order and call the function. Here’s the input and output for the equation

I hold that for the kinds of work I do with quadratic equations, the superiority of my quadratic solver function is self-evident.

1. You may think that dividing through by the leading coefficient is extra work. Not really. Most of the quadratics I deal with already have a leading coefficient of one, so I seldom need to divide. And even when I do, the two divisions can be done quickly with RPN stack manipulation. It’s harder for me to remember which term goes into memory than it is to do two divisions.

[If the formatting looks odd in your feed reader, visit the original article]

# PDF overlays

Permalink - Posted on 2019-07-03 02:34

Today I needed to combine two PDFs into one. Not by concatenating the pages; that would be easy using any number of apps, of which Preview is probably the most obvious. No, I had two single-page PDFs, each of which was meant to be printed on Avery 5161 label stock.

One of the PDFs had printing for the first three row of labels and the other had printing for the fourth and fifth rows. I could, of course, have printed the first page and then taken the sheet and fed it back through the printer for the second page, but I didn’t want to do it that way because

1. Every time you pass label stock through a laser printer, it picks up a bit of toner on all the labels, making the remaining (unprinted upon) labels dingier.
2. Where’s the fun in taking the easy way out?

What I wanted was to create a third PDF with all five rows on a single page, an overlay of the second PDF onto the first. There are ways to do this in PDFpenPro and PDF Expert, but a command-line solution seemed more efficient in the long run.1

For me, the Swiss Army knife of PDF manipulation is the so-called server version of PDFtk, so I checked its man page to see if it had an overlay command. It has two:

• background, which puts one PDF behind the other.
• stamp, which puts one PDF in front of the other.

You might think these are redundant operations, but because of scaling, rotation, and transparency differences, you might find that one of these commands can do what you want and the other can’t. For the PDFs I was working with, either would do.

The command that did what I wanted was

pdftk first.pdf stamp second.pdf output labels.pdf


I then printed labels.pdf and got all the labels in a single pass through the printer. To cut down on the amount of typing I’d need to do in the future, I made a simple script,2 called overlay:

#bash
#!/bin/bash

pdftk "$1" stamp "$2" output -


Now I can use this command with a syntax similar to cat:

overlay first.pdf second.pdf > labels.pdf


To make it really like cat, I’d have to generalize overlay to handle any number of inputs, not just two. I’ll leave that, as the math textbooks like to say, as an exercise for the reader.

1. And there is a long run. I’ve had this situation come up before where I either did the two-passes-through-the-printer thing or used a GUI app to make the combined PDF. It was time to create an automated solution.

2. Thanks to Vitor Galvão for reminding me to wrap the argument placeholders in double quotes. I always forget that.

[If the formatting looks odd in your feed reader, visit the original article]

# Cheesegraters and crystals

Permalink - Posted on 2019-06-22 19:50

Earlier this week, Stephen Hackett wrote a nice article on the design of the new Mac Pro’s grill and how it’s related to older and simpler Apple grill designs. It led me to Apple’s page on the design of the Pro and this language:

The lattice pattern on Mac Pro is based on a naturally occurring phenomenon in molecular crystal structures. A network of three-dimensional interlocking hemispheres, it increases the surface area, optimizing airflow and structural rigidity.

… The result is a lightweight lattice pattern that maximizes airflow while creating an extremely rigid structure.

This isn’t just marketing blather. The hole are a negative-space representation of a common atomic arrangement in metallic crystals, something that most engineers learn about in their introductory materials science class.

Atoms are hard to see, but crystal structures bear a geometric relationship to the packing of spheres, which are easy to see. We start with a layer of spheres and then set another layer on top of it, with the balls of the second layer fitting into valleys formed by the first layer.

• First, when discussing sphere packing, the balls in each layer are normally shown touching each other because that gives the densest arrangement. I’ve shown them slightly separated for reasons that will become apparent later when we return to the Mac Pro grill.
• Second, each layer can be thought of as extending out infinitely in each direction. The sphere arrangement isn’t limited to the balls shown in the drawings.

We can now set a third layer on top of the second, but there are two ways to do it. In one (on the left), the balls of the third layer are directly above those of the first. In the other (right), the balls of the third layer are set in the openings that run through the first two layers.

The left method is called the “hexagonal close packed” (hcp) structure, and the right is called “face-centered cubic” (fcc).

The hcp structure is fairly easy to visualize. Image a layer of seven balls arranged in a hexagon, then a layer of three, then a layer of seven again.

In materials science books, the hcp crystal structure, in which atoms arrange themselves like the spheres we’ve been looking at, is usually shown in an oblique view, like in the drawing at the lower right. That shows you the three layers and all the atom locations in a single image.

The fcc structure isn’t as obvious because it’s harder to visualize cubic arrangements in our layers of balls. I find it easiest to imagine four layers: three layers like those shown above and another that’s just like the green layer but placed below the blue layer.

The trick is to recognize that the faces of the cube aren’t aligned with the layers of balls. We’re looking down the diagonal of a cube, with the green balls at the farthest and nearest corners.

The drawing on the right is the fcc crystal structure as it would commonly be shown in a materials science book.

Now we can see how the cheesegrater grill of the Mac Pro is related to crystal structures. Here’s Apple’s animation of the machining process, which gives a good 3D view of the holes.

A frame capture from the video shows how the hemispherical holes machined into the aluminum plate are arranged like the blue and red layers in our sphere-packing drawings.. The holes on the near side are like the layer of red spheres, and the holes on the far side are like the layer of blue spheres.

Now you can see why I drew the spheres of each layer slightly separated from one another. It was to match the separation of the holes on the two sides of the plate. The holes need to be separated to keep the grill from falling apart.

The other difference between the holes in the Mac Pro grill and the packing of spheres is that the holes from the two sides don’t simply touch, they intersect. Otherwise, there’d be no airflow.

So, the lattice pattern on Mac Pro really is “based on a naturally occurring phenomenon in molecular crystal structures.” I have no idea how this “maximizes airflow”—I suspect maximum airflow would come if the holes from one side were drilled straight through. But if the holes were drilled straight through it wouldn’t look as cool. Also, Stephen and I would have had nothing to write about.

One last thing, which I’m sure delights the metallurgists at Apple. Aluminum has an fcc structure, so the geometry of the grill mimics—in part, at least—the atomic arrangement of the material from which it’s made.

[If the formatting looks odd in your feed reader, visit the original article]