Mac OS X Web Programmers Toolkit

Free (gratis)

• Fugu - Secure FTP. SCP etc. Hasn't really been updated in a while but it works pretty well. Allows you to edit remote files with your favorite local text editor.
• TextWrangler - Decent text editor from the makers of BBEdit. Major deficiency is that it doesn't support nested languages (e.g. Javascript within HTML, HTML within PHP, etc.).
• RegExhibit - Interactively test regular expressions
• Locomotive - Ruby on Rails in a box. A single standalone application that includes Ruby, MySQL, a web server, and an application to manage them all.
• MySQL 5.0
• PostgreSQL
• CocoaMySQL
• Aquamacs Emacs - Holy union of emacs and Aqua. Normal emacs key bindings work (e.g. C-e end of line). Normal Aqua keybindings work (e.g. Cmd-a select all). Includes various elisp functions to bring emacs closer to the 21st century. All in a nice OS X native application with lots of polish.

Not Free

• TextMate - Best text editor ever. (Sorry emacs).
• Transmit - Best FTP/SFTP/SCP client ever. Allows me to edit remote filew seemlessly with TextMate. These days I'm often connected to the internet intermittently or with high latency. Editing files in this idiom (which seems to be almost unique to the Mac) is the best way around that.

emacs deficiencies

I want to like emacs more than I do. But it's hard. It's missing all kinds of features that I've gotten used to from more modern text editors. There are many basic operations that, although common in other editing programs and easy enough to implement in emacs-lisp, have never found their way into standard emacs. For instance, why aren't there built-in functions for:

• count-words-[buffer|region]
• unfill-paragraph
• kill-entire-line (this function is standard in xemacs)
• kill-to-beginning-of-line (equivalent to M-0 C-k)

I mean honestly, it's the feakin' 21st century here. It's not like counting the number of words in your buffer is a fringe use-case.

Also, how hard would it be to support soft-wrapped text? And why can't I bind commands to the capital letter key combos (e.g. C-N in addition to C-n)? longlines-mode will get you a close-enough approximation of soft-wrapped text and it should be possible to bind commands with the shift modifier. I just can't get it working on my Mac but I think it's a Terminal.app configuration issue.

The biggest issue I have with emacs however is that it doesn't support multiple major modes at once. This is a big problem for anyone programming with embedded languages – particularly web programmers – as it means that when you're editing HTML with Javascript or CSS, emacs can only be in one (major) mode at a time. So even if it has support for all the different languages in a given buffer, emacs can't switch modes as you move around in the buffer from one language section to another.

There is a project called MMM (Multiple Major Modes) that's trying (or tried, it hasn't been updated since 2004) to remedy that problem but it's an add-on, has limited language support, and requires customization for every language combination that you want to use.

Okay, enough with the complaining. Let's try something a little more constructive. Here are some useful bits of emacs-lisp to rectify some of my complaints. I found all of them with Google so I can't take credit for anything below. Apologies for the line truncation that's happening in the <pre> tags. Something lame with blogspot templates going on there.

Kill Away

(defun unfill-paragraph ()
  (interactive)
  (let ((fill-column (point-max)))
  (fill-paragraph nil)))

;; No longer neccessary. See update below
(defun kill-entire-line ()
  "Kill this entire line (including newline), 
regardless of where point is within the line."
  (setq kill-whole-line t)
  (interactive)
  (beginning-of-line)
  (kill-line)
  (setq kill-whole-line nil))

;; If you care to bind it
(global-set-key [(control ,)] 'kill-entire-line)

Counting words

;; This one is nice and simple but has a 
;; slightly awkward output. E.g.
;; Word count: 325 occurances.
(defun wc ()
  (interactive)
  (message "Word count: %s" 
           (how-many "\\w+" 
                     (point-min) (point-max))))

;; Nice version from Robert J. Chassell's 
;; /Programming in Emacs Lisp/
;; Available at http://www.gnu.org/software/emacs/emacs-lisp-intro/
(defun count-words-region (beginning end)
  "Print number of words in the region."
  (interactive "r")
  (message "Counting words in region ... ")

;;; 1. Set up appropriate conditions.
  (save-excursion
    (let ((count 0))
      (goto-char beginning)

;;; 2. Run the while loop.
      (while (and (< (point) end)
                  (re-search-forward "\\w+\\W*" end t))
        (setq count (1+ count)))

;;; 3. Send a message to the user.
      (cond ((zerop count)
             (message
              "The region does NOT have any words."))
            ((= 1 count)
             (message
              "The region has 1 word."))
            (t
             (message
              "The region has %d words." count))))))

;; Count the words in the entire document
(defun count-words-buffer ()
  "Count all the words in the buffer"
  (interactive)
  (count-words-region (point-min) (point-max) )
)

Longlines.el (aka soft-wrap)

Most excellent elisp code to simulate soft-wrapping in emacs. Basically, hard-returns are inserted when text is loaded or typed into a buffer but all such returns are removed when the buffer is saved to disk.

Shell Mode Improvements

I have my shell set to use a colorized prompt. With default setting, emacs shell mode renders a colorized prompt pretty poorly. Amit Patel's blogI has some useful settings

(add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)
(setq comint-prompt-read-only t)

Update: (2008-09-09)

Well eat me and get big: emacs22 addresses some of my complaints. First, kill-entire-line is a standard command. It's supposed to be bound to C-S-<backspace> but I can't get it to work on my Mac in Terminal.app. I suspect Terminal.app is doing something non-kosher to the shift-ctrl combo but I'm not sure. In any case, it should be possible to bind commands to capital letters.

Second, longlines support is standard. M-x longlines-mode and you're set.

Haven't checked it out yet but Flymake seems like it should be awfully cool. It's supposed to do on the fly compile/syntax checking.

You can cycle through open buffers with C-x left and C-x right.

Where art thou iTerm?

UPDATE(2009-05-16): I found a fix for the broken meta-key problem on the iTerm bug tracker. Check out: http://sourceforge.net/tracker/index.php?func=detail&aid=1294192&group_id=67789&atid=518973

UPDATE(2008-09-26): iTerm released an update on Sept 24th. It seems to fix the mouse selection problem mentioned below but I still can't get the alt key to act as a Meta key in either local or remote connections. Key combos using alt just print out non-ASCII characters.

Leopard has a pretty decent terminal emulator in Terminal.app and I use it all the time. But before Leopard, Terminal.app left much to be desired and so I used a third party freebie called iTerm. The biggest advantage over Terminal.app was that you could open a bunch of terms in tabs instead of needing a separate window for each. Now Terminal.app has tabs and that seems to have killed interest in iTerm. As of this writing, the "News" section on the iTerm webpage hasn't had any postings in a year and a half.

I find this a little sad. iTerm still has some things going for it that haven't found their way into Terminal.app (and may never). I miss Full screen mode. I miss iTerm's ability to override the color values for ANSI colors. The default blue color (used for directories) is essentially unreadable on a black background so with iTerm I lightened up the blue. I can't do this in Terminal.app so I'm back to squinting at directory names. I also think iTerms handling of bookmarks and tabs was more intelligent than Terminal but that's almost a quibbling issue.

So why don't I still use iTerm? Three reasons. First, iTerm has weird mouse selection behavior. When you select a line of text with the mouse moving from left to right, you have to be sure to position the cursor all the way to the left of the first character you want to select. I used to end up missing the first character all the time until I managed to override my habits. Right to left selection works fine. Second, I could never get the meta/alt/option key to work correctly in remote terms. I use Meta-d and Meta-backspace in bash quite often and I could never get it to work on remote connections to my Linux machines. It's easy to get working on Terminal.app. Lastly, iTerm has never supported .term files. These are a little known secret of Terminal.app that have been around since at least 10.3 (though they may go all the way back to 10.0). They allowed me to create files that would initiate ssh connections to particular machines. I used to keep a directory in my dock with a bunch of .term files in it so that I could right-click and jump to any machine that I needed to work on. Then I started using Quicksilver and dispensed with the Dock folder altogether. I could open a new Terminal.app window with an ssh connection to my workstation just as quickly as I could type Cmd-Space ws [enter].

So iTerms's source is freely available on sourceforge. Maybe when I get several months of free time I can hack in those features. Go open source. When you find yourself frustrated with a product, all it takes is a degree in Computer Science, years of professional experience, and several months to get up to speed on a codebase.

ANSI Color Codes in C++

I finally got around to figuring out how to get C++ to output text in different colors (assuming your output system (terminal) supports it). It's not at all difficult , it just requires knowing the string representations of the ANSI color codes. I haven't tried it with C/printf but I can't see why it wouldn't work the same way.

HP 35s

I've been wanting another RPN calculator to use at work so after months of ignoring the HP35s that was sitting in my Amazon wish list, I finally ponied up and bought one.

It arrived today and boy is it disappointing. I don't know if it's supposed to harken back to HP's glory days but its RPN mode is pretty hobbled. The stack is only 4 levels deep(!) and the top and bottom positions clone when the stack is shifted up and down. Very annoying and quite different from the HP that I bought about 15 years ago (and still use at home).

Old issues of Wired

Fuck. A bad day.

One positive thing: I drew up a list of old issues of Wired that I’d like to have because they featured articles by Bruce Sterling or Neal Stephenson. Then, I ransacked the library of back issues that someone has discarded here at the office. I found most of the isses I was looking for. The articles themselves I’ve mostly already read on-line but this way I get the photos and layout.

David Foster Wallace article on talk radio

Brilliant.

Desk

The desk I’m stitting at is a featureless white rectangle with rounded corners. It supports a pair of giant flat-panel monitors, taller than they are wide and with edges flush, so the pair looks like a giant, luminous book, opened before me. The rest of the surface looks like a battle ground between the Rectangles and the Cylinders. The Rectangles have piles of paper, a notebook, the monitors, a clipboard and no fewer than three sizes of sticky paper pads. The Cylinders count among their members: a cup, a tin of tea leaves, a medicine bottle (with one tiny cyindrical pill inside), a can of compressed air and a handful of pens. I seem to be in league with the Rectangles. Every day I produce more of them and drain the contents of the Cylinders.

On the prospects of creating a headless Firefox

I've often wondered how hard it would be to make a headless version of Firefox. HTTPUnit is nice and all but if you want to ensure that everything is behaving as it does in a real browser there's nothing better than using a real browser.

So setting aside all questions as to difficulty does the idea of a headless Firefox even make conceptual sense? You'd at least need some way to script or control the behavior externally. But let's imagine you designed a feature where by hovering over an event you triggered an Ajax request or caused some DOM fragment to be constructed on the fly. In order to get the headless browser to do that you'd need to either have to tell it to simulate the mouse movement or tell it to artificially fire the hover event for the element in question.

(Lesser known) things I like about Leopard

1. Calculator.app sports a fully functional RPN mode. It's had RPN mode since Panther but it always seemed like a shoehorned afterthought. Switching to RPN never produced any change in the UI and RPN really demands a multi-line display. Now it has one. There's one quirk I've noticed: when you type a number and press enter, the number gets pushed up the stack (as it should) and then gets duplicated on the line below (not as it should). As soon as you begin to type a new number, the doubled entry disappears but it should never have appeared in the first place.
2. Calculator.app can now also display digit seperators.
3. The Airport menu (from the drop-down in the menu bar) now shows lock icons next to those networks which are protected. You can see at a glance if there are any available networks that are open.
4. Hitting command-backspace from any NSTextEdit field will delete from the cursor to the beginning of the line. This means TextEdit gets the "feature" I most miss when I'm away from TextMate.
5. iCal.app no longer changes the size of your window when you switch from (say) month view to week view.

ECMAScript 4 Overview (final draft)

The final draft of the Overview that describes what's going to be in Javascript 4 (E4) is online as a pdf.

Interestingly, this represents a point of departure from previous Javascript updates in that the group will provide a "reference implementation" in ML! And I though all that time I spent working in ML back in college would surely go to waste.

You can grab the standard SML release for Mac (as a DMG even) from here.

Bike to Work

I biked to work last Friday. No different from any other Firday at previous jobs except that I now work 42 miles from home. So my commute took 3 hours.

View Larger Map

Download the KML file.

New Job (@ Google!)

I've started a new job at Google!

BBEdit Comes Around

We programmers spend a lot of time editing text and tend to feel pretty strongly about the relative merits of various text editors. For the better part of two jobs - spanning nearly four years - I lived and breathed in BBEdit, “the programmer’s text editor”. It’s always seemed like a good product but even then it was already quite old and it seemed like its designers were resting on their laurels somewhat. The interface looked about the same as it had when my friend was using it in college on OS 9 back in 1998. There didn’t seem to be much work going on to create innovative features or even to copy ones that had started to crop up in other editing programs. But it still had an impressive feature list and at the time was by far the best tool for creating HTML/XML/CSS on Mac OS X.

But somewhere along the way I noticed TextMate. I saw a screencast for some Ruby on Rails demo and they were using snippets - type a keyword, hit some expansion key, get a whole bunch of templated text for free. I didn’t know the TextMate name for it but I knew the feature: it was the thing I most missed from a Java editing program that was a one time favorite on Windows. I decided to give TextMate a try even though it wasn’t free and even though I’d already shelled out big bucks for BBEdit (well, truth be told, the Reagents of the University of California shelled out the big bucks for BBEdit).

Over the years I’ve probably married about a dozen different text editors so it’s not a tremendously big deal for me to switch to something new. The muscle memory takes a little overcoming but it’s nothing horrific, especially if you’re not also switching platforms to something where the modifier keys are totally different.

It didn’t take long before I was hitched to TextMate. With the exception of occasional forays into emacs I’ve been using it exclusively for about 2 years. I switched and never looked back. Until today.

I was reading a blog which suggested that BBEdit had finally updated their interface so that it looked less out of place on OSX than does a disco ball at a funeral. So I downloaded the trial version (8.6) and gave it a spin.

There’s plenty to like

1. The icon looks less like ass. I was using BBEdit when they switched from their chestnut of an icon to something that looked like a beta placeholder for the image it was replacing.

   

   Either the whole thing was a play from the people that brought us New Coke and Coke Classic or they’ve seen the light: the newest version looks like the old except rotated a few degrees.

2. Rumors of interface polishing are true. The changes are rather minor but still welcome.

3. They’ve revamped their Glossary tool. Now it’s called Clippings and it works much like snippets in TextMate. The expansion key (F5) is a little awkward, especially on laptops. And where TextMate allows you to cycle through various pre-defined locations in the expansion text by repeatedly pressing the the same key (tab) that expanded the text in the first place, BBEdit uses Ctrl-` to cycle through positions. F5 still seems like a bad choice but having a different key command to cycle through the positions actually seems like a good idea. One annoying thing about TextMate’s snippet system is that you can’t expand a snippet if you’re still tabbing through the positions of a snippet you just expanded. It will interpret “tab” as a request to move to the next position instead of searching for an expansion. The BBEdit team seems to have made a better choice in this matter.

4. They’ve added support for a few more languages including Markdown and, get this, SQL. Yeah, it’s 2007 and they’re just getting around to adding SQL. Still they did it well and have support for multiple SQL variants like PL/SQL and MySQL.

5. They’ve added a preview window just like TextMate and SubEthaEdit. If you keep it open while you type it will refresh the preview automatically whenever you stop typing for a second or two. You don’t need to save the document to use it. Yeah, it’s just like TextMate and yes, it feels like they’re copying but it’s still a worthwhile addition.

6. Code folding. I make very little use of this feature but I’m sure it has its fans.

As long as they were out copying features I wish they’d copied a little more. There is still no equivalent to TextMate’s themes. The syntax highlighting engine still doesn’t support things like making keywords boldface. It still has pretty poor support for editing files with mixed languages (like PHP inside HTML). It still needs an organizational concept as functional and powerful as TextMate’s bundle system.

I’m not going to move back to BBEdit any time soon but these recent changes are enough to entice me into checking back on it in a few months. If nothing else, BareBones seems to have recognized that they’ve been squandering their name recognition and an enormous lead and that if they want to remain competitive they’ll need to start making radical changes. I think they made a couple of large mistakes when they moved to OS X. They ported the code but didn’t seem to do much to avail themselves of all that Objective C and Cocoa would provide. Even if they wanted to exploit the undeniable advantage of their extensively optimized and highly debugged text engine, they should have recognized that the arrival of OS X meant that others less encumbered than they would be able to make full use of everything from XCode to Cocoa to Objective C to the UNIX shell, and that this might allow a hypothetical upstart to move so quickly that any initial lead could evaporate before you could say “reality distortion field.”

Pardon the interruption

I am typing this entry using MarsEdit 1.1.5(207), of which I just downloaded a trial version. It seems you can edit your posts in whatever text editor you choose but I need to see if I can specify that markdown formatting is to be applied as part of the post procedure or if that’s something that’s only available while you’re previewing your post in MarsEdit.

This sentence should begin a new paragraph and this word should be italicized.

Update: it turns out markdown syntax is recognized by MarsEdit’s built-in preview window but there doesn’t seem to be any way to configure it to run your post through markdown prior to submitting it to your blog. Even more mysterious, there doesn’t seem to be any way to manually convert your markdown-formatted post into HTML suitable for posting. It’s lucky that I have a licensed copy of TextMate and that MarsEdit will allow me to edit my post there.

iTerm goes full screen

I started using iTerm pretty regularly at work a few months ago because it supported multiple tabs. I flirted with the program long before that but it seemed too unstable back then so I gave it up and went back to Terminal.app. The stability has improved a lot since then but there has always been one feature I found missing: full screen support.

Until now.

I didn't even notice when it happened but the most recent versions of iTerm now sport a great full screen mode.

Like a lot of programmers (and sysadmins) I spend a good deal of time working in a terminal. At work, a lot of us do. The crowd seems to be split between:

1. those who think full-screen mode is the coolest thing since screensavers
2. those who make fun of the first group

I'm such a fan of full-screen mode that I've long kept GLTerminal on my machine just for those instances where I want to work in a terminal without any distractions.

XSL[T|-FO]

My friend (and co-worker, and constant source of distraction) Kirt was poking around the company intranet today and came across a document detailing interview questions for “XSLT Developers”. When I was doing Java for a living I used XSLT quite a lot and eventually became quite good at using it effectively. Naturally, I was curious to see what we had to say about the qualifications of a good XSLT developer, especially since as far as I know, we don’t now, nor have we ever, used the slightest bit of XSLT for any part of our entire system.

Using the document to conduct an actual interview would have made for a pretty miserable experience which - since we’ve never used it and are unlikely to ever use it - I’m okay with. But it reminded me that there is a lot of confusion about what XSLT is and isn’t. XSLT and XSL are not synonymous. XSLT is a language for transforming XML documents. XSLT is but one of two components of the larger XSL, the other being XSL-FO: which is a language for specifying visual display. XSL is just a unifying term, a shorthard for saying “XSLT plus XSL-FO.” Wikipedia’s article on XSL disagrees with at least two points here but I won’t go into it other than to say that Wikipedia (though I love it so) is wrong.

I think that part of the reason why XSL and XSLT are used interchangeably is that in the beginning, XSLT pretty much was all of XSL. Sure, XSL-FO was envisioned early on but it took a long time to get a standard and it took a really long time to get an implementation. XSLT had a standard and multiple implementations while XSL-FO was nothing more than a dim idea. If you started using XSLT back then, you could be forgiven for thinking that was all there was to XSL.

The other point of confusion about XSLT is that it’s nothing more than a glorified variable substitution engine with horrid syntax. It’s syntax is retched and you certainly can use it to simply plug variables into an output template but XSLT is a lot more capable than that. It took me a long time to see and even longer to really understand that XSLT is basically a functional programming language.

lisp on OS X

I’ve decided (finally) to learn lisp. I don’t know if it will make me a better programmer but if even half of what Paul Graham says about it is true, it has to be the best programming language ever written. =)

Mac OS X doesn’t ship with a lisp interpreter but there are several free implementations out there. After ruling out the ones that cost money and the ones that have icky IDEs that run in X11 and the ones that depend on things like Fink, you’re really only left with two options.

1. Clozure CL (formerly known as OpenMCL)
2. SBCL (Steele Bank Common Lisp)

I went with SBCL because it works on both PowerPC and intel. Installing it is pretty easy. You do need to have the Developer Tools installed on your computer as the install process makes use of gcc.

1. Download the “binary” for the platform of your choice
2. Uncompress/untar the bz2 file
3. Fire up Terminal.app
4. cd to the directory that you extracted from the tar file
5. Run the following commands:

   INSTALL_ROOT=/usr/local
   sudo sh install.sh

That’s all there is to it. At this point you can run the lisp interpreter (sbcl) or bring up the man page for it.

UPDATE (27 August 2008): Clozure CL now supports both PPC and x86-64. CCL has always had better integration support for Mac OS X so it may now be a better choice for Mac users than SBCL. I haven't used it much but I plan to put it through it's paces.

Sunnis and Shiites

If you pay attention to the news, it’s nearly impossible to go a day without hearing about Sunnis and Shiites. Assuming you start out knowing nothing about them, it only takes a few news stories to figure out the most pertinent facts: they’re Islamic groups that seem to hate each other. But the news outlets never tell you anything about why Sunnis and Shiites don’t get along. If you want to understand that, you have to do a little research of your own.

The best and most succinct description I’ve read comes from Thomas L. Friedman’s From Beirut to Jerusalem

The Muslims of the world have long been divided between Sunnis, who are the majority, and Shiites. In the seventh century, shortly after the death of Islam’s founder, the prophet Muhammad a dispute arose over who should be his successor as spiritual and political leader, known as caliph. One group, the majority, argued that Muhammad’s successor should be appointed through the process of election and consensus by the elders of the community, as was the tradition of the desert. Sunna in Arabic means tradition, and those who held this view became known as the Sunnis.

A minority faction, however, argued that Muhammad’s successors should come exclusively from his own family and their descendants. They insisted, therefore, that his first cousin and son-in-law—Ali—be appointed as leader of the community. Those who held this view became known in Arabic as the Shia or “partisans,” of Ali. The Shiites were clearly influenced by the notion of divine-right monarchy of pre-Islamic Persia (Iran). The Sunnis eventually defeated the supporters of Ali and installed their own chosen caliphs. Nevertheless, the Sunni-Shiite split has continued down through the ages of Islam, and a whole body of theological and even cultural differences developed, distinguishing Shiites from Sunnis. Summarizing these differences, Islam expert Edward Mortimer observed in his book Faith & Power: “Sunni Islam is the doctrine of power and achievement. Shi’ism is the doctrine of opposition. The starting point of Shi’ism is defeat: the defeat of Ali and his house…. Its primary appeal is therefore to the defeated and oppressed. That is why it has so often been the rallying cry for the underdogs in the Muslim world … especially for the poor and dispossessed.”

Iraq happens to be one of only two Islamic countries (Iran is the other) where the Shia constitute a majority of the population (about 60 percent). This does nothing to help explain the Iran-Iraq war but does help illuminate why many other Islamic states seem supportive of the Sunni insurgency.

Mac Pro Optical Drive Door Problem

I recently became the proud new father of a Mac Pro 2xDual Core Xeon. It basically rocks the pants off the machine that it replaced - a poky 533Mgz G4.

But it has developed a slight problem with the little sliding door that protects the optical drive innards. It normally slides down when the tray slides out and then back up again after the tray goes back in. But mine now gets stuck when it's down. Like a depressive alcoholic, it just sits down there and sulks long after the tray has gone back home.

I can trick it back up again by raking my finger over the left side of the door a few times. I can't be the only one having this problem so I thought I'd pass it along.

Update 2007-09-19

I have not had a problem with the door for several months. Not sure what caused the problem to go away.