word_freq.lisp in Lisp

Counts word frequencies in an inline passage and prints a ranked table.

;; word_freq.lisp: word frequency report over an inline passage.

(defparameter *passage*
  "The quick brown fox jumps over the lazy dog. The dog barks
   and the fox runs. Quick thinking saved the fox that day.")

(defun word-char-p (ch)
  (or (alpha-char-p ch) (char= ch #\')))

(defun split-words (text)
  "Return TEXT as a list of lowercase words with punctuation dropped."
  (let ((words '())
        (start nil))
    (dotimes (i (length text))
      (let ((ch (char text i)))
        (cond ((and (word-char-p ch) (null start))
               (setf start i))
              ((and (not (word-char-p ch)) start)
               (push (string-downcase (subseq text start i)) words)
               (setf start nil)))))
    (when start
      (push (string-downcase (subseq text start)) words))
    (nreverse words)))

(defun tally (words)
  "Return (word . count) pairs, most frequent first, ties alphabetical."
  (let ((counts (make-hash-table :test #'equal))
        (pairs '()))
    (dolist (word words)
      (incf (gethash word counts 0)))
    (maphash (lambda (word count) (push (cons word count) pairs)) counts)
    (sort pairs (lambda (a b)
                  (if (= (cdr a) (cdr b))
                      (string< (car a) (car b))
                      (> (cdr a) (cdr b)))))))

(let* ((words (split-words *passage*))
       (ranked (tally words))
       (singles (remove-if-not (lambda (pair) (= (cdr pair) 1)) ranked)))
  (format t "~d words, ~d unique~%~%" (length words) (length ranked))
  (format t "~12a ~5a~%" "word" "count")
  (dolist (pair (subseq ranked 0 (min 6 (length ranked))))
    (format t "~12a ~5d~%" (car pair) (cdr pair)))
  (format t "~%~d word~:p appeared exactly once~%" (length singles)))

How it works

1. split-words scans the text one character at a time, cutting words on anything word-char-p rejects.
2. tally counts into a hash table built with :test #'equal, then maphash collects pairs to sort.
3. Ties break alphabetically with string<, and ~12a ~5d aligns the printed rows.

Keywords and builtins used here

• and
• car
• cdr
• char
• cond
• cons
• count
• defparameter
• defun
• dolist
• dotimes
• equal
• format
• gethash
• if
• incf
• lambda
• length
• let
• maphash
• min
• not
• nreverse
• null
• or
• push
• setf
• sort
• subseq
• when

The run, in numbers

Lines

44

Characters to type

1490

Tokens

356

Three-star pace

75 tpm

At the three-star pace of 75 tokens a minute, this run takes about 285 seconds.

Type this snippet

Step 1 of 3 in Encore, step 25 of 27 in Language basics.

← Previous Next →