Cocoa Emacs 23 as a Python IDE on MacOS 10.6 Snow Leopard: Part Two

Background

I was introduced to Emacs early on in my career, and had familiarized myself with enough of its commands that I could perform basic editing tasks, but never learned to truly appreciate its power until a few years ago, when a pair programming session with Malthe Borch opened my eyes to how learning a single text editor well can significantly enhance productivity and make programming more enjoyable. Malthe’s workflow incorporated Emacs in a way that made programming seem effortless and fun, at least compared to my programming workflow when using Eclipse PyDev or TextMate. I was a TextMate user at the time, and I found myself feeling clumsy and slow when using the mouse as part of my TextMate workflow. During our pair programming session Malthe challenged me to use my editor without using the mouse. I was surprised at how much I actually depended on the mouse and how much that dependency was slowing me down. I then resolved to learn the keybindings for TextMate, and that definitely helped speed up my workflow when using that particular editor. I still found myself using the mouse for some things in TextMate such as navigating the project tree and open file buffers. I also missed several features of Emacs that Malthe had turned me on to such as multiple ‘windows’ per ‘frame’ and running shells inside buffers. I decided to give Emacs another try and really learn how to use a larger portion of its features. I am still learning, and it seems the learning won’t end, which I don’t mind. In fact, I enjoy learning, and with Emacs, there is always a new feature or mode to explore. There are limitless possibilities for configuration as well. Needless to say, many yaks (or gnus) have been shaved in preparation for this tutorial.

General Configuration

Like many Emacs enthusiasts, I have spent a good deal of time hacking a configuration to my liking. I have crafted my configuration files over several years, starting from the sections of .emacs files of colleagues such as Malthe that I have cargo-culted. Some of the core bits of functionality that I rely on every day come from that pair programming session at a sprint in the Austrian Alps, back in 2008.

I have checked my Emacs configuration into bitbucket, feel free to check it out. It should run out of the box with Emacs 23 on a Mac, with the caveat that you need all of the third party libraries installed that I’ve got installed. We’ll get to that part. For now, you should be safe just loading the dbrenneman-init.el file with the lines loading other dbrenneman-*.el files commented out with semicolons.

There are many options for configuring Emacs, and for this section of the tutorial, I’m going to stick to what I consider my base configuration.

I won’t be going over my emacs configuration line by line, but most of my config is fairly well commented. Cargo-cult to your liking.

Your Emacs configuration lives in ~/.emacs by default. You can edit your ~/.emacs configuration file in place. When you are ready to test your changes, simply save the file:

C-x C-s

Then enter the command:

M-x eval-buffer

This will reload your configuration without restarting Emacs.

I have opted for the ‘write your own config’ method of configuring Emacs, for the most part, rather than using the customize function from within Emacs.

One of the first options I configure in my .emacs file is the load path for my custom configuration files:

(add-to-list 'load-path "~/.emacs.d")

This allows me to put any Emacs elisp libraries in ~/.emacs.d/ and easily load them into Emacs.

I also start Emacs in server mode with the following directive:

(server-start)

With an Emacs running in server mode, you can use emacsclient to open files in your running emacs, you can even set emacsclient as your default EDITOR and use it for commit messages, etc. I have created a shell script that I use to call emacsclient with the proper flags for the given editing situation.

Here’s my ~/bin/emacsclient.sh:

#!/bin/bash
/Applications/Emacs.app/Contents/MacOS/bin/emacsclient -c

I have added the directory ~/bin/ to my PATH, and have made the ~/bin/emacsclient.sh file executable.

I have aliased the executable to ec in my shell initialization file, so I can then run ‘ec filename.py’ and have the specified file open in a new buffer in my running Emacs, and have the focus switch to the new buffer in Emacs as well.

I run midnight mode in the default configuration, which will run a command at midnight to clean up old and unused buffers.

I also load a library called lorem-ipsum, which allows one to easily generate as much lorem ipsum dummy text as one needs with a few keystrokes.

I have split my configuration into multiple files, which can help make a long configuration easier to deal with. I load each of these config files explicitly in my main .emacs file. In my bitbucket repository, the dbrenneman-init.el file is what I use as my ~/.emacs file. You can symlink it in if you like.

I turn tempbuf on for dired buffers, so that I don’t have to kill a bunch of dired buffers all the time.

;; turn on tempbuf mode for dired buffers
(require 'tempbuf)
(add-hook 'dired-mode-hook 'turn-on-tempbuf-mode)

I also run a nifty little function called uniquify that makes buffer names unique.

;; Unique Buffer Names - makes navigation of open buffers easier
(require 'uniquify)
(setq uniquify-buffer-name-style 'reverse)
(setq uniquify-separator "/")
(setq uniquify-after-kill-buffer-p t) ; rename after killing uniquified
(setq uniquify-ignore-buffers-re "^\\*") ; don't muck with special buffers)

I like TRAMP, it allows you to edit files on remote machines via ssh in a local Emacs buffer. Here, I set TRAMP to default to using ssh to connect to remote hosts.

;;TRAMP should default to ssh
(setq tramp-default-method "ssh")

Another function I find useful is the desktop. This function can be configured to save the buffers one has open in an Emacs session upon exiting the editor, and then reload the saved buffers when Emacs is started again.

;; load saved desktop on startup and save buffers to desktop on exit
(load "desktop")
(desktop-save-mode 1)
(desktop-read)
(setq desktop-load-locked-desktop t)

;; save a bunch of variables to the desktop file
;; for lists specify the len of the maximal saved data also
(setq desktop-globals-to-save
      (append '((extended-command-history . 30)
                (file-name-history        . 100)
                (grep-history             . 30)
                (compile-history          . 30)
                (minibuffer-history       . 50)
                (query-replace-history    . 60)
                (read-expression-history  . 60)
                (regexp-history           . 60)
                (regexp-search-ring       . 20)
                (search-ring              . 20)
                (shell-command-history    . 50)
                tags-file-name
                register-alist)))

A similar function is saveplace; when you enable it, it saves your place in each file you open when you close the editing buffer. When you re-open the same file in a new buffer, the cursor is placed in the same position it was in when the last buffer associated with that particular file was closed.

;; save our place when closing a file
(require 'saveplace)
(setq-default save-place t)

I also set global-auto-revert mode to true, so that any time a file connected to an open Emacs buffer changes on disk, the editor will attempt to reload the contents of the file from disk.

;; always revert buffers if their files change on disk to reflect new changes
(global-auto-revert-mode 1)

Next time I will discuss the configuration of modes such as ido and org modes. We’ll get to Python specific configuration in a future installment.

Cocoa Emacs 23 as a Python IDE on MacOS 10.6 Snow Leopard: Part One

Introduction

So, you want to use Emacs for programming in Python? With many features of an IDE? And you’re on a mac? And you have far too much time on your hands? You may have tried Aquamacs or Carbon Emacs, but you find that you really just want an Emacs that is true to its GNU self and retains sane key bindings instead of trying to be more mac-like. Perhaps you prefer to follow the development of Emacs more closely. Perhaps you are slightly masochistic.

You could simply download one of the binaries available from http://emacsformacosx.com/ and be done with it, but you yearn for the bleeding edge; you want to run Emacs 23 in Cocoa native mode from bzr trunk.

Well friend, you’ve come to the right place.

Welcome to my hand-crafted Emacs tutorial, focusing on how to get started using Emacs on MacOS 10.6 as an integrated development environment for coding Python based web applications.

In order for this to work, of course, you’re going to need a copy of the Developer Tools from Apple installed on your machine. This installation will also require bzr: http://bazaar.canonical.com . The installation of these components is beyond the scope of this tutorial.

While definitely an Emacs fanatic, this author does not claim the same functionality can not be configured using VIM, and freely admits to using VIM on a regular basis. The author is also a recovering TextMate user, the Duke Nukem Forever of editors: http://wiki.macromates.com/FAQ/TextMate2 .

Installation

First of all we are going to grab a copy of Emacs 23 from the official bzr repository and compile it. You said you wanted bleeding edge, right?:

bzr branch http://bzr.savannah.gnu.org/r/emacs/emacs-23 emacs
cd emacs

Now we’ll patch this emacs so that we can run in full-screen mode:

wget http://gist.github.com/raw/291150/59bbe67fe996ed4c2c71198a335f1a47a235dd3d/fullscreen.patch
patch -p 1 -i fullscreen.patch

If everything goes according to plan:

./configure --with-ns
make
make install
sudo cp -r nextstep/Emacs.app /Applications/

The ‘–with-ns’ flag tells the configure script that we want to build a ‘NextStep’ binary, which is what the Emacs project calls their Mac binary.

Congratulations, now you have compiled your very own bleeding edge Emacs. Careful, it’s sharp.

Let’s try starting it up to verify that it works.:

open /Applications/Emacs.app

Emacs should now be running happily on your Mac.

You can quit Emacs with the following command:

C-x C-c

Tune in next time for a discussion of how to configure Emacs now that it is installed. Until then, try running the built in tutorial: Invoke Emacs and then press:

C-h t

Tahoe Sprint 2010 Report Out

The view from the Tahoe Sprint 2010

The Tahoe Sprint 2010 is sadly over, but it was a great success! The team is already looking forward to next year’s sprint.

We have made significant progress on several Dexterity related fronts:

David Glick has done some cleanup and refactoring of the Dexterity code, in addition to providing invaluable assistance to the other sprinters.

Ross Patterson has implemented user entered single selection choice vocabularies and multi-choice vocabularies for through the web schema editing of content types. Ross has also made progress in his effort to make it easier to access the api docs of the zope component architecture from a running Zope 2 instance.

Alex Limi has implemented drag and drop reordering of items in the web based schema editor, in addition to other schema editor UI tweaks.

Joel Burton has written over 20 pages of integrator documentation for Dexterity.

The team has had some interesting and fruitful discussions regarding the future of Plone and Dexterity, in addition to discussions regarding the implementation details of the Dexterity back  end and user interface.

archetypes.z3crelationfield

With Ross Patterson’s patient assistance, I have successfully implemented an Archetypes field that can be used to relate Archetypes objects to Dexterity objects called archetypes.z3crelationfield. You can find the code for archetypes.z3crelationfield on bitbucket here.

archetypes.z3crelationfield uses the infrastructure provided by zc.relation and z3c.relationfield to allow relationships to be created from Archetypes content types to Dexterity types using an extension of the standard reference widget that ships with Archetypes. archetypes.z3crelationfield includes a demo profile that installs a custom AT content type that uses this extended reference field and widget.

Creating relationships from existing Archetypes objects to Dexterity objects will allow folks to get started with Dexterity without throwing out all of their existing Archetypes based content types, a huge win for both content type frameworks and for Plone as well.
This initial implementation is meant to be a reference for future functionality that we would like to see merged into Archetypes itself, but it can be used today as a standalone package.

Future plans for archetypes.z3crelationfield include:

1. An archetypes.referencebrowserwidget implementation
2. A content migrator to migrate existing AT content and references to use zc.relation style relationships.
3. An alternate install profile that runs the migration scripts for existing content to use the new relations and monkeypatches Archetypes using archetypes.schemaextender to use the new archetypes.z3crelationfield instead of the default AT reference field.
4. A release to pypi and plone.org

Stay tuned for further developments.

collective.pdfpeek 0.18: asynchronous processing of image preview generation

The other day I released a Plone 4 compatible version of collective.pdfpeek in which I’ve implemented an asynchronous processing queue for pdf to image conversions. In the initial versions of pdfpeek, a user would be forced to wait for their pdf file to be processed, one page at a time, by ghostscript, before Plone returned a response to their browser. Talk about a bad user experience. Many PDF files are so large that this time spent waiting by the user became unacceptably long. So I’ve tried to work around this by implementing a simple queue where a conversion job is added to the queue when a PDF file is uploaded to a file object in Plone. One can then either process the queue manually by visiting a certain url as admin, or preferably by automating this process with cron and curl or via a zope clock server process. When the user uploads a PDF file, a conversion job is added to the queue and the user can then go about their business, adding or editing other content and return to the file object later to see the image previews available once the conversion job in the queue has been processed. collective.pdfpeek 0.18 works under Plone 3, but the integration doctests currently need some attention in order to get them passing on both Plone 3 and 4, so that is why version 0.18 only “officially” supports Plone 4.

Tahoe Sprint 2010

A handful of Plone geeks and I are heading for the snow in a couple weeks to do some serious coding and boarding at South Lake Tahoe, California. We’ll be sprinting on the Dexterity next-generation content type framework for Plone in a cabin near the shore of Lake Tahoe and the Heavenly Mountain Resort Gondola. So far we’ve got five sprinters and we still have room for five more, so if you think you’d like to join us, add your name to the signup list on the sprint wiki. Our sprint is running parallel to the Cathedral Sprint in Köln, Germany and we’ll have a continuous video link between the two sprints to facilitate inter-sprint communication.

So far, these are the chosen sprint topics:

• Widget improvements
• Member content types with Dexterity
• Replacements for ATCT with Dexterity
• Image handling improvements
• Benchmarking
• Bugfixing – killing issues in the tracker

I’m looking forward to contributing to the Dexterity project, as it has the potential to make Plone easier to use not only for developers, but also for site managers and integrators, as well as improving Plone’s performance beyond the substantial gains we see in Plone 4.

Let’s sprint!

collective.pdfpeek

I recently committed a new package to the Plone collective called collective.pdfpeek. Pdfpeek is a quick and dirty implementation of a PDF to PNG transform independent of the portal_transforms framework and created with many aspects of the zope component architecture, such as interfaces, adapters, events, subscribers and annotation storage. Not only does pdfpeek transform the PDF to a list of images using GNU ghostscript, one per page of the PDF file, when a pdf file is uploaded to your Plone site; pdfpeek stores the list of images on the ATFile object in standard annotation storage and displays the images when viewed through the browser using the lightbox2 javascript library.

Pythonistas Unite!

I recently returned from PyCon ’09 in Chicago, and it was amazing! I was genuinely impressed by the quality of the tutorials and the presentations. I learned more than I had hoped or imagined during the first two days of tutorials only to have my brain stuffed full of more yummy knowledge during the following three days of listening to talks and geeking out with other Pythonistas. Unfortunately, I wasn’t able to attend all of the conference talks, but luckily they were all recorded and are now available here.

I was equally impressed by the welcoming nature of the Python community and by how easy it was to speak with some of the core developers of various python-based projects. I also learned a great deal from socializing with various Python geeks and from resting my finger on the pulse of this dynamic and interesting community. This was my first PyCon and from the feedback I read in the blogosphere, this year’s conference has received rave reviews. I consider myself fortunate to have attended such a strong year. If you are on the fence about attending PyCon 2010 in Atlanta, I say go for it, it will rock your world.

One thing that struck me about the Python community is the diversity of Python-based web frameworks and how very few Pythonistas are intimately familiar with Zope and related technologies such as Plone. It appears as though when some folks hear the word Zope, they turn around and run away screaming.

I think this fear of the “Z” word is a bit unfounded, or perhaps founded in ignorance (which breeds fear, I’m told). Perhaps some knowledge sharing amongst the various Python web frameworks would benefit the community as a whole? I think it would be healthy to have more cross-pollination of ideas from the Django, WSGI and Zope communities. I believe there are things that each framework/collection of libraries can learn from the others. Perhaps via more collaboration and idea-sharing efforts will not be duplicated within the Python community and superior software can be released as a result.

Some knowledge sharing is beginning to occur between the WSGI folks and the Zope crew, which is a good start, but more sharing is imperative and must include the Django / pinax communities as well.

Despite the differences in implementation and opinion, as Python programmers we have more in common than we have differences, regardless of which of the various independent web frameworks we use and love.

Pythonistas of the world unite! You have nothing to loose but your chains.