Category Archives: macOS (Mac OS X)

Installing a MySQL server on Mac OS X 10.10 Yosemite or 10.11 El Capitan with or without Server

Apple recently phased out MySQL in favor of PostgreSQL.  There are still valid reasons why we might prefer MySQL, such as compatibility with legacy code.

Here are my instructions on how to install MySQL as a permanent service on OS X. These instructions work with or without OS X Server installed. For the most part, installation is straightforward, but read this carefully because there are gotchas where it comes to file permissions.

I’ve tested these instructions several times on OS X 10.11 El Capitan and 10.10 Yosemite. I think these instructions will probably work on 10.8 Mountain Lion and 10.9 Mavericks.

Backup your MySQL databases.

You want to back up your MySQL databases before doing a OS X upgrade. The easiest way is to create a mysqldump file. Databases can also be restored from data files, but this is harder and I don’t recommend it.

$ mysqldump -uroot --all-databases > ~/Desktop/dump_all.sql

Install or upgrade OS X.

According to your plan.

Download Xcode from the App store and install the command line tools.

Xcode is a dependency for Homebrew, which uses Xcode’s gcc compiler to compile everything from source. Once you have Xcode installed, you need to install the command line tools for the gcc compiler to work. From Terminal:

$ xcode-select --install

Check the systemwide PATH variable.

Open a Terminal window:

$ cat /etc/paths

Make sure that /usr/local/bin occurs before /usr/binIf they don’t, then you need to change this order. Edit /etc/paths using vi or your favorite text editor. I love and use TextWrangler. Close your Terminal window and open a new Terminal window for this change to take effect.

Obtain Homebrew. 

Homebrew is a great package manager for OS X that installs everything in /usr/local/bin and does not require sudo.It then symlinks to the expected locations so that the packages can find one another. Because the packages are centralized in the Cellar, they are easily updated and removed.

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Fix any problems that Homebrew detects.

$ brew update
$ brew doctor

Follow brew doctor‘s instructions. brew doctor usually complains about Xcode. If I’m guessing the error correctly, here is the solution that brew doctor will suggest (assuming OS X 10.10 – note the version since this affects what you will type into Terminal):

$ cd /Applications/Xcode.app/Contents/Developer/Toolchains/
$ sudo ln -s XcodeDefault.xctoolchain OSX10.10.xctoolchain
# note the version (10.10) and modify accordingly.

Run brew doctor again. Continue to follow brew doctor‘s instructions until it tells you that “Your system is ready to brew.”

Install MySQL.

$ brew install mysql
# ignore homebrew's instructions to symlink the .plist file!

Homebrew has just installed MySQL to run under the current user, which is not what we want for our server, but we will get to that in a moment. First, we will test the mysqld service to see if it launches. In newer versions of MySQL, the command to launch the service is $ mysql.server start.

$ mysql.server start
$ mysql.server stop 

Modify the .plist file to improve logging.

Using vi or your favorite text editor, modify the .plist file.

$ vi /usr/local/opt/mysql/homebrew.mxcl.mysql.plist
# use vi or your editor of choice 

Add these lines within the <dict> block:

<key>StandardErrorPath</key>
<string>/var/log/homebrew.mxcl.mysql.err</string>
<key>StandardOutPath</key>
<string>/var/log/homebrew.mxcl.mysql.log</string>

Set up the MySQL service to launch at boot time as part of a server.

Homebrew’s instructions, which I asked you to ignore, would have installed a LaunchAgent for your current user account, so that the mysqld service would start whenever you logged in. This would work great for a personal development machine, but it’s not not ideal for a server.

For a server, what we want is for mysqld to start up at boot time by the root account. So, we need to make two changes:

  1. the .plist must link into /Library/LaunchDaemons and have the appropriate permissions to be launched by root.
  2. the mysql database files in /usr/local/var/mysql must all be owned by  _mysql. You might wonder why the owner must be _mysql, since the server starts up as root. This is because whenever mysqld detects that it is being run as root, the process steps down to user _mysql as a security measure. This is typical behavior for services. Apache, for example, steps down to user _www.

Open a Terminal window, and enter:

$ sudo ln -sfv /usr/local/opt/mysql/*.plist /Library/LaunchDaemons
$ sudo chown root:wheel /usr/local/opt/mysql/*.plist
$ sudo chmod 600 /usr/local/opt/mysql/*.plist
$ sudo chown -R _mysql:wheel /usr/local/var/mysql

Important: Now that mysql belongs to root, do not run the $ mysql.server command again! Don’t do it. This will result in an aborted launch with permissions errors. You will need to delete the resulting .pid files in /usr/local/var/mysql before mysqld will be able to run again, even as root. Again, don’t do it.

Verify that the MySQL service starts up at boot time.

We will reboot the server machine. At boot time, launchctl should load our modified .plist and run mysqld. We will then run the mysql command from Terminal to interface with the mysqld service and verify that it’s running.

 $ sudo shutdown -r now

After the machine restarts, re-enter Terminal and type:

</pre>
<pre>$ mysql -uroot 

If the mysql command fails, then we know that either the LaunchDaemon didn’t kick in, or that mysqld failed to start successfully. Again, if this happens it is usually because of a permissions issue. To troubleshoot, check the log files at:

/var/log/homebrew.mxcl.mysql.out
/var/log/homebrew.mxcl.mysql.err
/usr/local/var/mysql/*.err

Restore your MySQL databases.

$ mysql -uroot < mydumpfile.sql
$ mysql_upgrade

A full dump from a recent version of MySQL should successfully restore all databases plus the privilege table, which controls users. If there are problems with the privilege table after restoring, then edit out the privilege table from your mysql dump file, and re-create your users manually using a tool like phpmyadmin.

I hope this post helped someone. Happy monkeying!

Advertisements

Fancy html mail signatures in macOS (OS X) Mail

This post has been updated for macOS 10.12 Sierra.

I wanted to set up a nice html signature for macOS’s Mail app, but the editing options in the Preference > Signatures pane of Mail are painfully limited. Also, even after discovering that I could use cmd-T to change fonts, I found that the fonts that I chose didn’t always display correctly. There was also no way to add <img> tags directly in the Mail app. Obviously, I needed to be able to edit the source HTML directly. Here’s how to do it.

Create a draft signature.

Open Mail. Go to Preferences > Signatures, and create a new signature. Type in some draft text. This creates a file that you can find again through Terminal.

Find your draft signature file through Terminal.

Open Terminal, and type:

$ cd ~/Library/Mail/V4/MailData/Signatures
$ ls *.mailsignature

Edit your signature file in your favorite editor.

I love and use TextWrangler, but you could use vi or any other editor.

If you have multiple signatures, you might need to open each file until you figure out which one to edit.

Consider locking your signature.

I don’t do this, but you might want to. macOS will sometimes change some aspects of your signature to match the font of the rest of your email. This might help you or hurt you, depending on how complicated your signature is. You can lock your signature either through the UI, or in Terminal with this command:

$ chflags uchg *.mailsignature

Restart mail.app.

Your signature should now work.

For including images, it’s better to load the image from a permanent location on the web using <img> tags instead of attaching an image file to each email. This cuts down on the size of each of your emails, sparing your outbox and your recipients’ inboxes. Also, it prevents your email from being flagged in recipients’ inboxes as containing an attachment when in spirit, you don’t have one.

Good luck!

I didn’t come up with all of this on my own. I acknowledge:

Syntax highlighting for Mac OS X Mail and applications

Code is a lot harder to read without syntax highlighting, and I needed highlighting for emails and presentations. So, I cobbled this solution from what I could find on the net. This OS X service allows me to select and highlight text in many OS X applications. It works for Pages, Keynote, and Apple’s Mail. It does not work, as written, in Microsoft Office, although you can still copy highlighted text from Pages to Word or PowerPoint, or perhaps modify this script to make it work in Office.

You will need two things to get this service working on your computer. First, download and install a command-line based syntax highlighter. Second, you need to set up an Automator workflow, to enable the service itself.

Install a command-line highlighting utility.

I use highlight. A strength of Pygments is that it can auto-detect languages, while highlight does not. But, Pygments often doesn’t guess the correct language if the highlighted code is short, because today’s popular languages use highly similar syntax.

I installed highlight using Homebrew:

$ brew install highlight

To install Pygments:

$ pip install pygments

Create your Automator workflow.

Open Applications > Automator. Choose “Service” as your new workflow type. Choose “Run Applescript” as your one and only action. Above this action, you will see some options. Set this workflow to receive text in any application. Do not check the box that says ‘Output replaces selected text’ because then this script won’t work.

Copy and paste this code.

In your run Applescript box, copy and paste:

-- inspired from http://apple.stackexchange.com/questions/94222/how-to-get-automator-to-treat-text-as-rtf/94246#94246

-- limitation: does not auto-detect syntax (which is iffy with short sections of code, anyway); therefore this action is limited to Python syntax

on run {input, parameters}
    try
        set old to the clipboard as record
    end try

-- the tr function is necessary because some applications copy rtf as OS 9 CRs instead of Unix LFs.

    do shell script "echo '" & input & "' | tr '\\r' '\\n' | /usr/local/bin/highlight --syntax=Python --out-format=rtf  --style=edit-matlab --font=courier | pbcopy"

--do shell script "echo '" & input & "' | tr '\\r' '\\n' | /usr/local/bin/pygmentize -l python -f rtf | pbcopy"

-- copying and pasting via the clipboard necessary because text replacement bypasses the word processor's automatic conversion of RTF

    tell application "System Events" to keystroke "v" using command down
    delay 0.5

    try
        set the clipboard to old
    end try
end run

Save this service.

I use the name “Python Syntax Highlighting” but you can call your service whatever you want. Whenever you select text in any application, your new Service will appear on the menu bar as [Application] > Services > Python Syntax Highlighting [or whatever name you saved it as]

This service works well for programs that support RTF. I use it all the time in Keynote, Pages, and Apple’s Mail. It does not work in Microsoft Office, though, although it may be possible to modify the script above to make it work in Office. Good luck!

Installing scientific Python with NumPy, SciPy, matplotlib, PyMC, and Basemap/geos on Mac OS X 10.9 Mavericks, 10.10 Yosemite or 10.11 El Capitan

I love OS X for day-to-day work, especially compared to its main alternative, Windows. But Apple neglects data monkeys who install tools under the hood. For my research, I use Python, NumPy, SciPy, matplotlib, PyMC, and Basemap/geos, among other modules. These tools used to break with each new version of OS X, although as OS X has matured, the disruptions with each new release have become less.

I’ve tested these instruction on OS X 10.8 Mountain Lion, 10.9 Mavericks, 10.10 Yosemite, and 10.11 El Capitan when they were reasonably current (I usually stay about one OS X release behind). Over the years these instructions have evolved, and Homebrew formulae have been added and improved. I haven’t been going back and re-testing these instructions on older OS X releases, so your mileage might vary if you do this.

Consider starting with a clean install of OS X.

The dirty upgrade works great if you’re an everyday user who stays on the UI level, but in the distant past it made a terrific mess of my tools installed under the hood. I’ve ended up with multiple MySQL servers and multiple apache servers. But, as OS X has become more mature, I’m noticing less of a need to do this.

If you decide to take my advice and do a clean installation, first make sure that you have a backup. Time Machine works spectacularly well for this purpose, or you can install onto a new hard drive or carbon copy your old drive. Erase your working hard drive, install OS X, then have Migration Assistant import user accounts and applications from the backup (but leave the Other Files & Folders box unchecked). Migration Assistant takes care of restoring ordinary documents, which Apple does well, yet it provides a relatively blank slate under the hood to freshly install tools.

If you decide that you don’t want to start with a clean install of OS X, then go through these instructions anyway. For each package, use brew to uninstall, and then reinstall each package in sequence. Use my directions to keep you on track as you update each package individually. Don’t forget to also test each package after each reinstallation, as I have demonstrated in my instructions, so that if there are problems, you know exactly what the problem is.

Download Xcode from the App store and install the command line tools.

Xcode is a dependency for Homebrew, which uses Xcode’s gcc compiler to compile everything from source. Once you have Xcode installed, you need to install the command line tools for the gcc compiler to work. The quickest way to do this is from Terminal:

$ xcode-select --install

Check the systemwide PATH variable.

Open a Terminal window, and type:

$ cat /etc/paths

Make sure that /usr/local/bin occurs before /usr/bin. If they don’t, then you need to change this order. Edit /etc/paths using vi or your favorite text editor. I love and use TextWrangler. Close your Terminal window and open a new Terminal window for this change to take effect.

Obtain Homebrew. 

Homebrew is a great package manager for OS X that installs everything in /usr/local/bin and does not require sudo.It then creates symlinks to the expected locations so that the various tools can find one another. Because the packages are centralized in the Cellar, they are easily updated and/or removed.

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Fix any problems that Homebrew detects.

$ brew update
$ brew doctor

Follow brew doctor‘s instructions. brew doctor usually complains about Xcode. If I’m guessing the error correctly, here is the solution that brew doctor will suggest (assuming OS X 10.10 – note the version since this affects what you will type into Terminal):

$ cd /Applications/Xcode.app/Contents/Developer/Toolchains/
$ sudo ln -s XcodeDefault.xctoolchain OSX10.10.xctoolchain 
# note the version (10.10) and modify accordingly.

Run brew doctor again. Continue to follow brew doctor‘s instructions until it tells you that “Your system is ready to brew.”

Install Homebrew’s gcc.

This step, overlooked in the other blogs I’ve seen, has been required in the past for successfully installing SciPy and PyMC (newer Homebrew formulae may have fixed this problem). Without this, installation has failed with a vague gfortran error.

$ brew install gcc

Install Homebrew’s Python.

$ brew install python

Update OS X’s Python symlink to point to Homebrew’s Python.

This symlink controls the version of Python that runs when a user types python in Terminal. These directions should have appeared at the end of your brew installation, but in case you missed them, here they are (These instructions assume Python 2.7.6 – note the version since this affects what you will type into Terminal):

$ cd /System/Library/Frameworks/Python.framework/Versions
$ sudo rm Current
$ ln -s /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/Current
# note the version (2.7.6) and modify accordingly.

Now, we will verify homebrew’s Python installation. If there’s a problem, we need to fix it now before proceeding with installations.

$ which python
# should see /usr/local/bin/python

Don’t touch or relink OS X’s system Python. Change your hashbang habits.

This is another place where this blog differs from other blogs.

OS X exasperatingly contains yet another copy of Python, called system Python, located at /usr/bin/python. The standard hashbangs !#/usr/bin/python or !#/usr/bin/env python will call Homebrew’s Python when the script is run by the local user, but will call system Python if the script is called by a launchctl daemon/agent. Other blogs have recommended deleting system Python and symlinking the deleted file to homebrew’s Python installation, but this solution creates serious problems with OS X Server, which depends on the original system Python installation being left intact.

Therefore, just change your habits to always use this hashbang: #!/usr/local/bin/python and all of these problems go away, no matter where or who your script is executed by. Easy-peasy.

Install NumPy, SciPy, and matplotlib.

First try each installation using homebrew. In the past, Homebrew didn’t yet have formulae for some of these, so I used pip whenever a formula was not available.

$ brew tap homebrew/python
$ brew install numpy
$ brew install scipy # this module used to require homebrew's gcc to succeed.
$ brew install matplotlib
# let's test our installations
$ python
>>> import numpy
>>> import scipy
>>> import matplotlib
>>> exit()

Install Basemap.

Basemap depends on geos, which needs to be installed first.

$ brew install geos

Download the latest non-Windows Basemap tarball. Unpackage it by double clicking on the compressed file on your desktop (which is easier) or use tar -xzf. Then, cd into the directory.

$ python setup.py install
$ cd examples
$ python simpletest.py 
# this tests your installation

Install PyMC.

This module is for Markov chain Monte Carlo (MCMC) simulations. If you don’t use MCMC then you should skip this. PyMC won’t install without a successful SciPy installation, which, in turn, has depended upon Homebrew’s gcc compiler.

$ git clone https://github.com/pymc-devs/pymc.git
$ cd pymc
$ python setup.py build
$ python setup.py install
$ python # let's test our installation
>>> import pymc
# if you get a warning about statsmodels and/or patsy you can probably ignore it. 
# I did, and my simulations have worked fine.
# (I hope!)
>>> exit()
$ rm -R pymc

You’re done! Happy programming!

I didn’t come up with all of this on my own. I acknowledge: