Never walk alone... http://gajon.org New blog posts on gajon.org en-us Sat, 08 Feb 2014 18:48:09 -0600 Sat, 08 Feb 2014 18:48:09 -0600 Common Lisp development with Vim gajon@gajon.org (Jorge Gajon) Tue, 07 Dec 2010 00:00:00 -0600 http://gajon.org/common-lisp-development-vim http://gajon.org/common-lisp-development-vim <p>You have used Vim for so long, and you have it so ingrained within you, that when somebody suggests you use other editor you instinctively reply <em>“over my dead body!”</em>. I know how you feel.</p> <p>And then one day you start programming in Common Lisp; everybody says that you must use Emacs with <a href="http://common-lisp.net/project/slime/">SLIME</a>; and after not so long you actually start considering switching to it!</p> <p>The horrors!</p> <p>I believe that for some long-time Vim users, switching to Emacs is not an easy option. Personally I’m afraid of immersing into Emacs, spending countless hours learning its ways and idiosyncrasies, without gaining any significant advantage over Vim. Those countless hours could be better spend immersing yourself into learning Common Lisp (or something else).</p> <p>In this article I’ll show you how I’m using Vim to develop applications with Common Lisp.</p> <p>There are a few plug-ins for Vim to do Common Lisp development, two of them are <a href="http://www.vim.org/scripts/script.php?script_id=2219">Limp</a> and <a href="http://www.vim.org/scripts/script.php?script_id=2531">slimv</a>. Of those two I liked Limp better, although it does some things in ways that I don’t agree with, therefore I modified it a little to meet my needs. Keep in mind that those are my personal preferences, and I don’t claim them to be the <em>correct</em> way of doing things.</p> <p><strong>But before continuing</strong>, I’d like to express my gratitude and admiration to Limp’s author, Mikael Jansson, for all the hard work put into it.</p> <p>Now I’m going to show you the modifications I did to <a href="http://www.vim.org/scripts/script.php?script_id=2219">Limp</a>, but first you need to download and install it; it shouldn’t be too difficult, follow the documentation.</p> <h2 id="1_disable_highlighting">1. Disable highlighting.</h2> <p>Limp does automatic highlighting of contents inside any pair or parenthesis, but I think there’s two problems with it. First, I find it distracting, I don’t like the constant changing of colors all over the place while moving the cursor through the code. Second, it slows down the rendering of the text significantly, and the whole thing feels unresponsive.</p> <p>I don’t see the benefit of this highlighting. I think that keeping a sane indentation of your code is enough to see the nested structures in it. I also think that if you don’t keep a logical indentation of the code at all times, you are doing something wrong.</p> <p>We will also see how to make Vim indent Lisp code correctly further below.</p> <p>Inside the main Limp folder there is a <code>vim</code> subfolder; open the file <code>limp.vim</code> and at the end you can see several <code>runtime</code> commands; comment out the one that loads the <code>highlight.vim</code> file.</p> <pre><code>&quot;runtime ftplugin/lisp/limp/highlight.vim</code></pre> <p>Then open the file <code>mode.vim</code> and around line number 58 you can see the call to initialize the highlighting mode; you need to comment it out too.</p> <pre><code>&quot;call LimpHighlight_start()</code></pre> <p>And the call to stop the mode, located around line number 68, comment it out too (if you are using an SVN version of the code, it may already be commented).</p> <pre><code>&quot;call LimpHighlight_stop()</code></pre> <h2 id="2_disable_autoclosing_of_parenthesis">2. Disable Autoclosing of parenthesis.</h2> <p>Another feature of Limp is an automatic insertion of closing parenthesis when you type the opening parenthesis. There’s a big problem with this feature in that it is buggy and leaves visual debris on the screen when undoing operations (at least with gvim).</p> <p>Even if it were not buggy, I think that far from helping it will only distract you unnecessarily. You will have a huge number of closing parenthesis after your cursor while in the middle of trying to figure out the shape of your code. And you <strong>still</strong> need to type the closing parenthesis on your keyboard to continue writing the next form.</p> <p>I believe that you only need the highlighting of matching pairs, automatic indentation, and a shortcut to automatically close any remaining open parenthesis once you are satisfied with your block of code</p> <p>We’ll see how to add such a shortcut further below.</p> <p>If you want something to manage parenthesis that is similar to Emacs’ <a href="http://www.emacswiki.org/ParEdit">ParEdit</a> you could probably find a plugin that does that (I didn’t look up).</p> <p>Like in the previous step, comment the instruction to load this feature inside the file <code>limp.vim</code>:</p> <pre><code>&quot;runtime ftplugin/lisp/limp/autoclose.vim</code></pre> <p>And inside <code>mode.vim</code> comment the function call to initialize the mode:</p> <pre><code>&quot;call AutoClose_start()</code></pre> <p>And of course the call to stop the mode needs to be commented out too:</p> <pre><code>&quot;call AutoClose_stop()</code></pre> <h2 id="3_restore_your_own_color_scheme">3. Restore your own color scheme.</h2> <p>I don’t like the colors that Limp sets up, and I don’t see why it should change them in the first place. If you want to continue using your own colorscheme, open the file <code>mode.vim</code> and around line number 30 comment these lines:</p> <pre><code>&quot;set t_Co=256 &quot;if !exists(&quot;g:colors_name&quot;) &quot;colorscheme desert256 &quot;endif</code></pre> <p>You can see that there’s a bunch of highlight group definitions there. These are definitions for when using Vim in a terminal, and since I mostly only use gvim I didn’t have to do anything with them. But if you do use Vim inside a terminal you may want to adjust these colors to match your colorscheme, or simply comment them out and use the defaults.</p> <h2 id="4_better_options">4. Better options</h2> <p>I didn’t like the set of options that Limp sets, especially the old Vi Lisp indentation mode. I also dislike folding so I disabled that too. My current set of options look like this (file <code>mode.vim</code> around line 82):</p> <pre><code>syntax on setlocal nocompatible nocursorline setlocal lisp syntax=lisp setlocal ls=2 bs=2 et sw=2 sts=2 ts=8 tw=80 setlocal statusline=%&lt;%f\ \(%{LimpBridge_connection_status()}\)\ %h%m%r%=%-14.(%l,%c%V%)\ %P &quot;setlocal iskeyword=&amp;,*,+,45,/,48-57,:,&lt;,=,&gt;,@,A-Z,a-z,_ &quot;setlocal cpoptions=-mp &quot;setlocal foldmethod=marker foldmarker=(,) foldminlines=1 setlocal foldcolumn=0 set lispwords+=defgeneric,block,catch,with-gensyms &quot;----------- &quot;Taken from the bundled lisp.vim file in VIM &quot;(/usr/share/vim/vim72/ftplugin/lisp.vim) setl comments=:; setl define=^\\s*(def\\k* setl formatoptions-=t setl iskeyword+=+,-,\*,/,%,&lt;,=,&gt;,:,$,?,!,@-@,94 setl comments^=:;;;,:;;,sr:#\|,mb:\|,ex:\|# setl formatoptions+=croql &quot;----------- &quot; This allows gf and :find to work. Fix path to your needs setlocal suffixesadd=.lisp,.cl path+=/home/gajon/Lisp/\*\*</code></pre> <p>Notice I set <code>tw=80</code> (you may want to disable it); modified the <code>statusline</code> to be less verbose; disabled folding; disabled the <code>cpoptions</code> line because I want Vim’s default; copied the options that come bundled with Vim (they are a lot better); added some symbols to <code>lispwords</code>; and added a missing dot to <code>suffixesadd</code> (cl extension was missing a dot).</p> <p>It is common to find files edited by Emacs (and other editors) that uses a mixture of spaces and tabs to indent code, and usually they use tabs for 8 spaces and hard spaces for the rest of the indenting if it is not a multiple of 8. That’s the reason why I set <code>ts=8</code>, so that you can see those files with appropriate indentation.</p> <h2 id="5_disabling_the_transposing_of_sexps">5. Disabling the transposing of sexps.</h2> <p>Limp binds they keys <code>{</code> and <code>}</code> to functions that transpose the current sexp with the previous and next sexp. But they don’t work reliably and I think they are unnecessary when you can just as easily use <code>dab</code> and <code>p</code> at the proper place. Besides, the default Vim <code>{}</code> bindings are quite useful to jump to other top-level forms.</p> <p>In file <code>keys.vim</code> comment these lines:</p> <pre><code>&quot;nmap &lt;buffer&gt; { &lt;Plug&gt;SexpMoveBack &quot;nmap &lt;buffer&gt; } &lt;Plug&gt;SexpMoveForward</code></pre> <h2 id="6_bug_when_connecting_to_a_running_repl">6. Bug when connecting to a running REPL</h2> <p>There’s a bug that prevents Limp from reconnecting to an already running REPL. In file <code>bridge.vim</code> inside the <code>vim</code> subfolder, around line number 13:</p> <pre><code>let cmd = s:Limp_location . &quot;/bin/lisp.sh &quot;.core_opt.&quot; -s &quot;.styfile.&quot; -b &quot;.name</code></pre> <p>A space was missing between <code>&quot;.core_opt.&quot;</code> and <code>-s</code>.</p> <p>If you obtained an SVN version of Limp, this have been fixed already.</p> <h2 id="7_lets_add_some_improvements_a_better_repl">7. Let’s add some improvements, a better REPL</h2> <p>The preceding changes were adjustments that I believe work best for me. You may want to follow them or do other modifications, it’s basically all a personal preference. In the rest of this article I’ll show you a few small additions that I made to Limp.</p> <p>When you press <code>&lt;F12&gt;</code> Limp will launch an xterm and start SBCL inside it; this is done from the <code>lisp.sh</code> file in the <code>bin</code> folder.</p> <p>Let’s make the SBCL REPL able to complete symbols from the thesaurus that Limp generated when you installed it (see Limp’s install docs). This is done by telling rlwrap where to look for a list of words to consider for auto completion.</p> <p>Around line 149 in file <code>lisp.sh</code> we can see the line that defines the parameters to rlwrap.</p> <pre><code>[[ `which rlwrap` ]] &amp;&amp; RLWRAP=&quot;rlwrap -b $BREAK_CHARS&quot;</code></pre> <p>We can modify that line like this:</p> <pre><code>[[ `which rlwrap` ]] &amp;&amp; RLWRAP=&quot;rlwrap -b $BREAK_CHARS -f $LIMPDIR/vim/thesaurus&quot;</code></pre> <p>Now when Limp starts an SBCL REPL, you can start typing a symbol and hit <code>&lt;Tab&gt;</code> to complete it; for example if you start typing <code>get-u</code> and press <code>&lt;Tab&gt;</code>, rlwrap will complete it to <code>get-universal-time</code>.</p> <p>But you can add more options to rlwrap, for example I have this:</p> <pre><code>[[ `which rlwrap` ]] &amp;&amp; RLWRAP=&quot;rlwrap -pgreen -r -s 2000 -m -i -c -b $BREAK_CHARS -f $LIMPDIR/vim/thesaurus&quot;</code></pre> <p>The <code>-pgreen</code> option makes rlwrap colorize the prompt to green (see <code>man rlwrap</code> for available colors); the <code>-s 2000</code> option is to make rlwrap remember 2000 lines of input; the <code>-m</code> option enables multiline editing (see below); the <code>-i</code> turns case sensitivity off, and the <code>-c</code> option enables you to complete filenames.</p> <p>The <code>-r</code> option makes rlwrap remember all words seen in the input and output streams; this means that, in addition to those symbols found in the thesaurus, it will also be able to complete symbols you typed in the REPL.</p> <p>You can edit the input by calling an external editor defined by the environment variable <code>$RLWRAP_EDITOR</code>, and that editor can of course be Vim; this allows us to do multi-line editing.</p> <p>Around line 154 in the same file, we can see the actual invocation to SBCL:</p> <pre><code>$RLWRAP $SBCL --noinform $core</code></pre> <p>We can modify that line to introduce the <code>$RLWRAP_EDITOR</code> environment variable:</p> <pre><code>RLWRAP_EDITOR=&quot;vim -c \&quot;set ft=lisp\&quot; +%L&quot; $RLWRAP $SBCL --noinform $core</code></pre> <p>To invoke the editor you can hit <code>CTRL-^</code>, which the man page of rlwrap says is the default binding but that didn’t work for me and I had to explicitly add it to my <code>~/.inputrc</code> file.</p> <p>My <code>~/.inputrc</code> (see <code>man rlwrap</code> and <code>man readline</code>):</p> <pre><code>$include /etc/inputrc set editing-mode vi tab: complete set completion-ignore-case on set blink-matching-paren on &quot;\C-^&quot;:rlwrap_call_editor $if sbcl set comment-begin ; $endif</code></pre> <p>I would also suggest that you load the <code>sb-aclrepl</code> module, which will give you a better REPL prompt and an inspector. To enable it edit your <code>~/.sbclrc</code> file and add this form:</p> <pre><code>(require :sb-aclrepl)</code></pre> <p>You can find more info here <a href="http://www.sbcl.org/manual/#sb_002daclrepl">http://www.sbcl.org/manual/#sb_002daclrepl</a>. After you have loaded it you can type <code>:help</code> at the prompt to see a list of available commands. You an also skim the following web page to understand some of the commands simulated by <code>sb-aclrepl</code> <a href="http://www.franz.com/support/documentation/6.2/doc/top-level.htm">http://www.franz.com/support/documentation/6.2/doc/top-level.htm</a></p> <h2 id="8_additional_mappings">8. Additional mappings</h2> <p>Now let’s add a few useful mappings, you shouldn’t have any problem figuring out how to use these mappings:</p> <p>In the file <code>bridge.vim</code> add the following lines after line number 265:</p> <pre><code>nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;EvalUndefine :call LimpBridge_send_to_lisp(&quot;(fmakunbound '&quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;EvalAddWord :let &amp;lispwords.=',' . expand(&quot;&lt;cword&gt;&quot;)&lt;cr&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugTrace :call LimpBridge_send_to_lisp(&quot;(trace &quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugUnTrace :call LimpBridge_send_to_lisp(&quot;(untrace &quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugInspectObject :call LimpBridge_inspect_expression()&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugInspectLast :call LimpBridge_send_to_lisp(&quot;(inspect *)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugDisassemble :call LimpBridge_send_to_lisp(&quot;(disassemble #'&quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugMacroExpand :call LimpBridge_macroexpand_current_form( &quot;macroexpand&quot; )&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;DebugMacroExpand1 :call LimpBridge_macroexpand_current_form( &quot;macroexpand-1&quot; )&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileSet :call LimpBridge_send_to_lisp(&quot;(sb-profile:profile &quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileUnSet :call LimpBridge_send_to_lisp(&quot;(sb-profile:unprofile &quot;.expand(&quot;&lt;cword&gt;&quot;).&quot;)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileShow :call LimpBridge_send_to_lisp(&quot;(sb-profile:profile)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileUnSetAll :call LimpBridge_send_to_lisp(&quot;(sb-profile:unprofile)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileReport :call LimpBridge_send_to_lisp(&quot;(sb-profile:report)&quot;)&lt;CR&gt; nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;ProfileReset :call LimpBridge_send_to_lisp(&quot;(sb-profile:reset)&quot;)&lt;CR&gt;</code></pre> <p>And at the end of the file we add these two functions:</p> <pre><code>function! LimpBridge_inspect_expression() let whatwhat = input(&quot;Inspect: &quot;) call LimpBridge_send_to_lisp( &quot;(inspect &quot; . whatwhat . &quot;)&quot; ) endfun function! LimpBridge_macroexpand_current_form(command) &quot; save position let pos = LimpBridge_get_pos() &quot; find &amp; yank current s-exp normal! [( let sexp = LimpBridge_yank( &quot;%&quot; ) call LimpBridge_send_to_lisp( &quot;(&quot; . a:command . &quot; '&quot; . sexp . &quot;)&quot; ) call LimpBridge_goto_pos( pos ) endfunction</code></pre> <p>As you can see we added a bunch of internal mappings which we will use to send forms to the REPL, by using the function <code>LimpBridge_send_to_lisp()</code>. We also added two new functions, <code>LimpBridge_inspect_expression()</code> which will prompt you for a form and inspect its result in the REPL. The other function <code>LimpBridge_macroexpand_current_form()</code> will call <code>macroexpand</code> or <code>macroexpand-1</code> on the form under the cursor.</p> <p>To actually use these new mappings we have to map them to some keys by adding the following lines inside the file <code>keys.vim</code>:</p> <pre><code>nmap &lt;buffer&gt; &lt;LocalLeader&gt;eu &lt;Plug&gt;EvalUndefine nmap &lt;buffer&gt; &lt;LocalLeader&gt;ea &lt;Plug&gt;EvalAddWord nmap &lt;buffer&gt; &lt;LocalLeader&gt;dt &lt;Plug&gt;DebugTrace nmap &lt;buffer&gt; &lt;LocalLeader&gt;du &lt;Plug&gt;DebugUnTrace nmap &lt;buffer&gt; &lt;LocalLeader&gt;di &lt;Plug&gt;DebugInspectObject nmap &lt;buffer&gt; &lt;LocalLeader&gt;dI &lt;Plug&gt;DebugInspectLast nmap &lt;buffer&gt; &lt;LocalLeader&gt;dd &lt;Plug&gt;DebugDisassemble nmap &lt;buffer&gt; &lt;LocalLeader&gt;ma &lt;Plug&gt;DebugMacroExpand nmap &lt;buffer&gt; &lt;LocalLeader&gt;m1 &lt;Plug&gt;DebugMacroExpand1 nmap &lt;buffer&gt; &lt;LocalLeader&gt;pr &lt;Plug&gt;ProfileSet nmap &lt;buffer&gt; &lt;LocalLeader&gt;pu &lt;Plug&gt;ProfileUnSet nmap &lt;buffer&gt; &lt;LocalLeader&gt;pp &lt;Plug&gt;ProfileShow nmap &lt;buffer&gt; &lt;LocalLeader&gt;pa &lt;Plug&gt;ProfileUnSetAll nmap &lt;buffer&gt; &lt;LocalLeader&gt;ps &lt;Plug&gt;ProfileReport nmap &lt;buffer&gt; &lt;LocalLeader&gt;p- &lt;Plug&gt;ProfileReset</code></pre> <p>Now, assuming that your <em>“leader”</em> key is the inverted slash you can type <code>\eu</code> when the cursor is over a symbol to undefine it, by calling <code>fmakunbound</code> on it. And it’s similar with the rest of the mappings we just defined; you can consult the <a href="http://www.sbcl.org/manual/">SBCL manual</a> to learn how to use the tracing functionally as well as the profiler.</p> <p>One mapping that is not connected to REPL functionality is <code>EvalAddWord</code> mapped to the keys <code>\ea</code>. I use this to add symbols to Vim’s <code>lispwords</code> option. When you write your own macros, unless you add its name to <code>lispwords</code>, Vim will indent all forms passed to it as if they were arguments to a regular function.</p> <p>As an example, suppose you wrote a macro:</p> <pre><code>(defmacro my-macro (foo &amp;body body) ... body of the macro ...)</code></pre> <p>Vim will indent code that calls this macro like this:</p> <pre><code>(my-macro hello (+ 10 20))</code></pre> <p>But after adding the macro name to <code>lispwords</code> by positioning the cursor over <code>my-macro</code> and hitting <code>\ea</code>, Vim will indent the code like this (hit <code>\ft</code> to indent the top form):</p> <pre><code>(my-macro hello (+ 10 20))</code></pre> <h2 id="9_closing_open_parenthesis">9. Closing open parenthesis</h2> <p>I told you at the beginning of this article that having a shortcut to auto-close open unbalanced parenthesis was desirable. Let’s add such shortcut. I extracted the following from the <a href="http://www.vim.org/scripts/script.php?script_id=2531">slimv</a> plugin, written by Tamas Kovacs.</p> <p>Let’s add a new mapping inside the file <code>sexp.vim</code>, around line number 53:</p> <pre><code>nnoremap &lt;silent&gt; &lt;buffer&gt; &lt;Plug&gt;SexpCloseParenthesis :call SlimvCloseForm()&lt;CR&gt;</code></pre> <p>and in the same file, at the very bottom, we’ll add these two functions:</p> <pre><code>&quot;------------------------------------------------------------------- &quot; Close open parenthesis &quot; Taken from the Slimv plugin by Tamas Kovacs. Released in the &quot; public domain by the original author. &quot;------------------------------------------------------------------- &quot; Count the opening and closing parens or brackets to determine if they match function! s:GetParenCount( lines ) let paren = 0 let inside_string = 0 let i = 0 while i &lt; len( a:lines ) let inside_comment = 0 let j = 0 while j &lt; len( a:lines[i] ) if inside_string &quot; We are inside a string, skip parens, wait for closing '&quot;' if a:lines[i][j] == '&quot;' let inside_string = 0 endif elseif inside_comment &quot; We are inside a comment, skip parens, wait for end of line else &quot; We are outside of strings and comments, now we shall count parens if a:lines[i][j] == '&quot;' let inside_string = 1 endif if a:lines[i][j] == ';' let inside_comment = 1 endif if a:lines[i][j] == '(' || a:lines[i][j] == '[' let paren = paren + 1 endif if a:lines[i][j] == ')' || a:lines[i][j] == ']' let paren = paren - 1 if paren &lt; 0 &quot; Oops, too many closing parens in the middle return paren endif endif endif let j = j + 1 endwhile let i = i + 1 endwhile return paren endfunction &quot; Close current top level form by adding the missing parens function! SlimvCloseForm() let l2 = line( '.' ) normal 99[( let l1 = line( '.' ) let form = [] let l = l1 while l &lt;= l2 call add( form, getline( l ) ) let l = l + 1 endwhile let paren = s:GetParenCount( form ) if paren &gt; 0 &quot; Add missing parens let lastline = getline( l2 ) while paren &gt; 0 let lastline = lastline . ')' let paren = paren - 1 endwhile call setline( l2, lastline ) endif normal % endfunction</code></pre> <p>Now we only need to add a key mapping inside <code>keys.vim</code> right after where we added the previous mappings</p> <pre><code>nmap &lt;buffer&gt; &lt;LocalLeader&gt;cp &lt;Plug&gt;SexpCloseParenthesis imap &lt;buffer&gt; &lt;C-X&gt;0 &lt;C-O&gt;&lt;LocalLeader&gt;cp</code></pre> <p>Now you can type <code>\cp</code> to close open parenthesis, but notice that I also added an <code>imap</code> to be able to type <code>CTRL-X 0</code> while in insert mode as well; that’s what I usually use.</p> <h2 id="10_thats_all">10. That’s all</h2> <p>I hope this helps you make a better use of Limp. When I started learning Common Lisp I didn’t want to be distracted by trying to simultaneously learn Emacs and SLIME. I used Vim and Limp exclusively during my first project with Common Lisp, and I didn’t have any major problems.</p> <p>For my second project - and already feeling comfortable with Common Lisp - I decided to use only Emacs and SLIME. And I have to say that yes, SLIME is a superior development environment for Common Lisp. It is not indispensable, you can develop software without it just fine, but it is indeed a useful tool.</p> <p>However, the editing model of Emacs is too limiting, and I think that if I want to continue using it with SLIME, I’ll have to explore options like viper or <a href="http://www.emacswiki.org/emacs/Vimpulse">vimpulse</a>; it’s just too painful not to have the Vi editing model.</p> <h2 id="addendum_my_window_environment">Addendum: My window environment</h2> <p>I use a window manager called Ion, which is designed to be used with the keyboard instead of the mouse. Another nice feature of Ion is that the windows never overlap (you could make them overlap though) and they are positioned in tiles that you define.</p> <p>This means that I can, for example, position my code in the left half of my screen and a REPL on the right half, without any effort. I don’t have to struggle with resizing and moving windows so that I can see them both at the same time, as would be the case with a traditional window manager.</p> <p>Here’s a screenshot of how my screen looks:</p> <p><a href='/media/images/vim-limp-screenshot.png' title='Screenshot, Common Lisp development with Vim and Limp, and the Ion3 Window Manager'><img alt='Screenshot, Common Lisp development with Vim and Limp, and the Ion3 Window Manager' height='400' src='/media/images/vim-limp-screenshot.png' width='500' /></a></p> <p>~<br />~<br />:wq</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Common Lisp presentation at the HackerRoom MX gajon@gajon.org (Jorge Gajon) Wed, 10 Nov 2010 00:00:00 -0600 http://gajon.org/common-lisp-presentation http://gajon.org/common-lisp-presentation <p>The <a href="http://hackerroom.mx/">HackerRoom MX</a> is a co-working space for hackers in Mexico City. I gave a Common Lisp talk this last September to a few of the inhabitants there.</p> <p>I’m putting the slides at <a href="http://gajon.org/otherstuff/lisp-slides/">http://gajon.org/otherstuff/lisp-slides/</a></p> <p>If you are creating a similar presentation feel free to take whatever you want from my slides. <strong>I hereby authorize You to do whatever the hell you want with the content referred to above</strong>. Except that I took some parts from other places (always showing a link to the original source). If you use something from those parts you should include the link to the original sources as well.</p> <p>Enjoy.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Review of "Eric Sink on the Business of Software" gajon@gajon.org (Jorge Gajon) Fri, 09 Apr 2010 00:00:00 -0500 http://gajon.org/review-eric-sink-business-software http://gajon.org/review-eric-sink-business-software <p><em><strong>Eric Sink on the Business of Software</strong><br /> Eric Sink<br /> Apress (March 20, 2006)</em></p> <p>This is a book targeted to people (more specifically, programmers) who are thinking about starting a software business, or are working in a startup or small software business.</p> <p>The author is the founder of SourceGear, an Illinois based company which develops and sells version control tools for developers, and through his own experiences he shows us the lessons he has learned; many times by making a <em>lot</em> of mistakes.</p> <p>The book is divided in four parts; (1) Entrepreneurship, (2) People, (3) Marketing, and (4) Sales. In each part the author answers the most common questions most developers have when thinking about becoming an entrepreneur.</p> <p>He also tries to demystify many topics that are nebulous and obscure to most programmers, especially the ones that relate to marketing, sales and finances. He shows us that these topics are not as complex as they may seem and he even provides us with <em>algorithms</em> that we can use to better understand them.</p> <p>The nice thing about this book is that the author, not being a <em>“business”</em> person, but a programmer at heart, writes from that perspective, making it easy to relate to him, and see that he had the same questions, self-doubts and fears that most programmers have.</p> <p>The content of this book is taken from blog articles the author has published on his own website, with some comments added at the beginning of each one. This means that (1) you could read everything on his web site (though I prefer reading from a book), and (2) there is some repetition of ideas that were said in a previous article.</p> <p>Highly recommended reading if you have dreams of starting your own software business someday.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Review of "ANSI Common Lisp" gajon@gajon.org (Jorge Gajon) Wed, 10 Mar 2010 00:00:00 -0600 http://gajon.org/review-ansi-common-lisp http://gajon.org/review-ansi-common-lisp <p><em><strong>ANSI Common Lisp</strong><br /> Paul Graham<br /> Prentice Hall (November 12, 1995)</em></p> <p>This is a really good book for newcomers to Common Lisp, however take note that this is not a book for beginning programmers. This book will not guide you on the details of how to get, install and launch a Common Lisp environment. If you don’t feel able to do that on your own then this book might not be for you.</p> <p>The pace of the book is fast, and the information is presented in a very compressed form, meaning that the author does not ramble unnecessarily on any topic. The concepts are introduced one after another, many times building new concepts on top of the ones that have been introduced before; in this sense this book is meant to be read in order.</p> <p>To give you an idea of the compactness of the book, the 17 chapters of the book (not counting the appendixes) add up to only 285 pages.</p> <p>There are exercises at the end of each chapter, with varying degrees of difficulty. Personally, I would’ve liked it better if there were more exercises that required the student to construct more levels of abstraction to find a solution.</p> <p>The topics of CLOS and Macros receive a fair treatment on this book, but not an exhaustive one. For further study into these two topics I would suggest the books <em>“Object-Oriented Programming in Common Lisp: A Programmer’s Guide to CLOS”</em> by Sonya E. Keene, and <em>“On LISP: Advanced Techniques for Common LISP”</em> also by Paul Graham.</p> <p>Although the latter book is out of print and very hard to find a copy of, a PDF version is offered by the author at his website <a href="http://www.paulgraham.com/onlisp.html">http://www.paulgraham.com/onlisp.html</a></p> <p>One thing I like about this book is its thorough treatment of <em>cons</em> objects and how they are used in Common Lisp, along with discussions on destructive and non-destructive <em>(side-effect free)</em> functions, and the issues of sharing structure.</p> <p>Other highlights are, a simple implementation of a Ray-tracer; an implementation of a custom Object-Oriented language within Common Lisp; an implementation of a program that makes inferences from a set of rules <em>(very similar to Prolog)</em>; and utilities to generate HTML.</p> <p>Finally, it’s got a reference of all the language at the end of the book, which makes it very convenient to take the book with you, away from the computer, and try understanding the topics and do the exercises using only pen and paper. In fact, I recommend you do that, and use the computer only to verify that the solutions you found are correct.</p> <p>It took me 48 hours over the space of 39 days (averaging nearly 1:15 hrs a day) to read this book and do the exercises.</p> <p>Conclusion: Highly recommended as an introductory book in Common Lisp.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Trees as Linked Lists in Common Lisp gajon@gajon.org (Jorge Gajon) Tue, 23 Feb 2010 00:00:00 -0600 http://gajon.org/trees-linked-lists-common-lisp http://gajon.org/trees-linked-lists-common-lisp <p>If you are starting to learn Common Lisp, and have already read the chapter about <em>“Cons”</em> from an introductory book (also called <em>cons cells</em>), then you could try the following exercise. How to represent a tree using nothing but <em>conses</em>; and abstract that implementation with functions to create and traverse the tree.</p> <p><strong>PLEASE NOTE</strong>, that if you need to represent trees in a production program you shouldn’t use lists as described here unless you have a good reason. This is only an exercise in understanding how <em>cons cells</em> work.</p> <h2 id="1_a_tree">1. A Tree</h2> <p>Suppose we would like to represent the following tree in memory, using only lists (which are created by chaining <em>conses</em>). You should already know how a <em>cons</em> is created and what parts constitute one. If not, go back to an introductory book on Common Lisp.</p> <p><img src="/media/trees-lists/Diagram1.png" alt="Figure 1: A simple tree" /></p> <p>This tree can be represented as a set of linked lists, like in the following diagram:</p> <p><img src="/media/trees-lists/Diagram2.png" alt="Figure 2: A tree based on lists" /></p> <p>The natural way to represent the previous tree as a Common Lisp list is like the following:</p> <pre><code>(1 (2 6 7 8) 3 (4 (9 12)) (5 10 11))</code></pre> <p>The first element of a list is the root of that tree (or a node) and the rest of the elements are the subtrees (more nodes). For example, the subtree <code>(2 6 7 8)</code> is a tree with “2” at its root and the elements “6 7 8” as its children.</p> <h3 id="exercise">Exercise</h3> <p>How would you draw a diagram of the <em>cons cells</em> of the previous list? Compare it to the figure below.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>The <em>cons cells</em> that build up the list above can be drawn like this:</p> <p><img src="/media/trees-lists/Diagram3.png" alt="Figure 3: Diagram of cons cells of the simple tree" /></p> <h2 id="2_another_representation_of_the_tree">2. Another representation of the Tree</h2> <p>The list representing the tree shown above is intuitive and easy to understand, however there’s a slight inconvenience. A subtree can be represented as a <em>cons</em> that directly contains the data in it’s <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> position, like in the case of “3”, or as a <em>cons</em> that refers to another <em>cons</em> that starts a proper list, like in the case of <code>(2 6 7 8)</code>.</p> <p>There’s no consistency, when you need to access the data contained in a node, you must first check if the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> position of the <em>cons</em> refers to a list, in which case you have to access that list to get the data.</p> <p>Also, in the example list above, we are storing numbers in the nodes of the tree. What if you wanted to reference any kind of data from a node?, including a list?</p> <h3 id="exercise_2">Exercise</h3> <p>How would you represent a single node of a tree, so that the methods to access the data, the children, and the next sibling are always the same?, how could the node reference any kind of data, including other lists?</p> <p>Compare it to the representation given below.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p><img src="/media/trees-lists/Diagram4.png" alt="Figure 4: A node of a tree" /></p> <p>There could be more than one way to represent a node, but in the rest of this small article we are going to represent them like in the figure above.</p> <p>A tree node consists of two <em>conses</em>, the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a> of the first <em>cons</em> refers to the siblings of the node, while the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a> of the second <em>cons</em> refers to the subtrees (children) of the node. The <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> of the first <em>cons</em> refers to the second <em>cons</em>, while the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> of the second <em>cons</em> refers to the data stored in this node, which can be any kind of object.</p> <p>Using this node representation, the example tree given in <em>Figure 1</em> would be displayed by Common Lisp as the following list. Can you see why?</p> <pre><code>((1 (2 (6) (7) (8)) (3) (4 (9 (12))) (5 (10) (11))))</code></pre> <h3 id="exercise_3">Exercise</h3> <p>How would you draw a diagram of the <em>cons cells</em> of the previous list? Compare it to the figure below.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p><img src="/media/trees-lists/Diagram5.png" alt="Figure 5: Diagram of the whole tree as cons cells" /></p> <h2 id="3_the_api">3. The API</h2> <p>These are the functions that we are going to define for creating and traversing the nodes of a tree.</p> <pre><code>(defun make-tree (data) &quot;Creates a new node that contains 'data' as its data.&quot; ...) (defun add-child (tree child) &quot;Takes two nodes created with 'make-tree' and adds the second node as a child of the first. Returns the first node, which will be modified.&quot; ...) (defun first-child (tree) &quot;Returns a reference to the first child of the node passed in, or nil if this node does not have children.&quot; ...) (defun next-sibling (tree) &quot;Returns a reference to the next sibling of the node passed in, or nil if this node does not have any siblings.&quot; ...) (defun data (tree) &quot;Returns the information contained in this node.&quot; ...)</code></pre> <p>Notice that the function <code>add-child</code> will modify the node passed as the first argument, it is a destructive function, it has side effects.</p> <p>If you haven’t read about <em>destructive</em> or <em>side-effecting</em> functions, then you should go back and continue reading your book on Common Lisp.</p> <h3 id="exercise_4">Exercise</h3> <p>Given the structure of a node as shown in <em>Figure 4</em> above, come up with the code for the functions outlined above.</p> <p>Compare your code to the solutions given below.</p> <p>Test your code with the following usage example:</p> <pre><code>(let ((one (make-tree 1))) (add-child one (make-tree 2)) (add-child one (make-tree 4)) (add-child one (make-tree 5)) (let ((two (first-child one))) (add-child two (make-tree 6)) (add-child two (make-tree 7)) (let ((four (next-sibling two))) (add-child four (make-tree 9)) (add-child (first-child four) (make-tree 12)))) one) =&gt; ((1 (2 (6) (7)) (4 (9 (12))) (5)))</code></pre> <p>.</p> <p>.</p> <p>.</p> <h3 id="code_for_the_functions_outlined_above">Code for the functions outlined above</h3> <p>I’m going to skip showing a code for the function <code>add-child</code> for now, we are going to discuss it in the next section.</p> <p>The first function <code>make-tree</code> just needs to create the two <em>conses</em> as depicted in <em>Figure 4</em>, and place the data in the correct place.</p> <pre><code>(defun make-tree (data) &quot;Creates a new node that contains 'data' as its data.&quot; (cons (cons data nil) nil))</code></pre> <p>Similarly, for the other functions, we just need to follow the correct cells as suggested in <em>Figure 4</em>.</p> <pre><code>(defun first-child (tree) &quot;Returns a reference to the first child of the node passed in, or nil if this node does not have children.&quot; (cdr (car tree))) (defun next-sibling (tree) &quot;Returns a reference to the next sibling of the node passed in, or nil if this node does not have any siblings.&quot; (cdr tree)) (defun data (tree) &quot;Returns the information contained in this node.&quot; (car (car tree)))</code></pre> <p>Easy, is it not?. Were you checking for null trees? That is, where you doing something like this?</p> <pre><code>(defun first-child (tree) (if (null tree) nil (cdr (car tree))))</code></pre> <p>or</p> <pre><code>(defun first-child (tree) (if (null (cdr (car tree))) nil (cdr (car tree))))</code></pre> <p>Well, don’t, that’s Java/C++ thinking. Notice that <code>(car nil)</code> evaluates to <code>nil</code>, and similarly, <code>(cdr nil)</code> evaluates to <code>nil</code> too.</p> <p><strong>However</strong>, trying to pass an atom or other object that is not a <em>cons</em> to either <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> or <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a> is an error. For example <code>(first-child 3)</code> would cause an error.</p> <p>You could try to guard against that redefining <code>first-child</code> like this:</p> <pre><code>(defun first-child (tree) (when (listp tree) (cdr (car tree))))</code></pre> <p>However, I wouldn’t do that, because I think that signalling an error on a function call like <code>(first-child 3)</code> is the correct thing to do. The user of the function is the one that is committing an error, not the implementor of the function.</p> <h2 id="4_first_approach_to_addchild">4. First approach to add-child</h2> <p>There are a few ways you could have implemented the function <code>add-child</code>, let’s consider an example that at first sight might look fine.</p> <pre><code>(defun add-child (tree child) (setf (car tree) (append (car tree) child)) tree)</code></pre> <p>Remember that the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a> of the second <em>cons</em> <code>(car tree)</code> refers to the children of the node, so that when we do <code>(append (car tree) child)</code> we are <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a>ing <code>child</code> to the end of the list of <em>conses</em> that start with the one in <code>(car tree)</code>. Since <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> does not modify any of its arguments and instead returns a fresh <em>“consed”</em> list (a new copy), we have to capture it again with <code>(setf (car tree) ...)</code>.</p> <p>By the way, the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_setf_.htm">setf</a> line can be replaced with this line,</p> <pre><code>(rplaca tree (append (car tree) child))</code></pre> <p><a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_rplaca.htm">rplaca</a> stands for <em>“replace car”</em>, and it replaces the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> of the first argument with whatever the second argument evaluates to.</p> <p>Now, let’s try building the whole tree depicted in <em>Figure 1</em>; the following code can accomplish that.</p> <pre><code>(let ((one (make-tree 1))) (add-child one (make-tree 2)) (add-child one (make-tree 3)) (add-child one (make-tree 4)) (add-child one (make-tree 5)) (let ((two (first-child one))) (add-child two (make-tree 6)) (add-child two (make-tree 7)) (add-child two (make-tree 8))) (let ((four (next-sibling (next-sibling (first-child one))))) (add-child four (add-child (make-tree 9) (make-tree 12))) (let ((five (next-sibling four))) (add-child five (make-tree 10)) (add-child five (make-tree 11)))) one) =&gt; ((1 (2 (6) (7) (8)) (3) (4 (9 (12))) (5 (10) (11))))</code></pre> <p>By visually inspecting the resulting list you can see that the tree is being built correctly. But you might wonder why juggle with <code>first-child</code> and <code>next-sibling</code> with nested <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_let_l.htm">let</a> bindings, why not create the nodes that we need to refer more than once in the first <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_let_l.htm">let</a> binding.</p> <pre><code>(let ((one (make-tree 1)) (two (make-tree 2)) (four (make-tree 4)) (five (make-tree 5))) (add-child one two) (add-child one (make-tree 3)) (add-child one four) (add-child one five) (add-child two (make-tree 6)) (add-child two (make-tree 7)) (add-child two (make-tree 8)) (add-child four (add-child (make-tree 9) (make-tree 12))) (add-child five (make-tree 10)) (add-child five (make-tree 11)) one) =&gt; ((1 (2) (3) (4) (5 (10) (11))))</code></pre> <p>What??, the children of nodes <em>“2”</em> and <em>“4”</em> were lost, but somehow the children of node <em>“5”</em> were added correctly. There’s no difference in the way we create nodes 2, 4 and 5, and there’s no difference in how we add children to them either. What’s going on?</p> <h3 id="exercise_5">Exercise</h3> <p>This is where you will really test your understanding of how <em>conses</em> work and how references to objects in Common Lisp work. Try to figure out why the function <code>add-child</code> as used above is failing to correctly build the tree that we want.</p> <p>Drawing the <em>cons cells</em> after each operation will be helpful.</p> <p>You will also need to be sure to understand how <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> works.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <h2 id="5_understanding_what_is_going_wrong">5. Understanding what is going wrong</h2> <p>To understand what is going wrong, let’s see what happens after <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_let_l.htm">let</a> creates the bindings, but before any form inside the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_let_l.htm">let</a> body is evaluated.</p> <p><img src="/media/trees-lists/Diagram6.png" alt="Figure 6: Variables created by let and the objects they bind to" /></p> <p>We have four variables bound to four tree nodes. Then after evaluating the form <code>(add-child one two)</code> we have the following objects.</p> <p><img src="/media/trees-lists/Diagram7.png" alt="Figure 7: State of cons cells in memory after appending TWO as a child of ONE" /></p> <p>The function <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> does not modify any of its arguments, and therefore to be able to build a new list it has to create new <em>conses</em> copying the top list structure of all the lists it has been passed in as arguments, except for the very last list, which will be just pointed to by the second to last list.</p> <p>After <code>(append (car tree) child)</code> has created the new list, we point to it by doing <code>(setf (car tree) ...)</code>, thereby losing the reference to the original <em>cons</em> that was there (shown in grey in the above figure).</p> <p><strong>Please Note,</strong> the figure above may create a confusion. I’ve been drawing the numbers inside the <em>cons</em> for simplicity, but this is not very accurate. The two components of a <em>cons</em> (the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> and the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a>) can refer to any type of object, and these objects may live in other parts of the memory.</p> <p>Sometimes an implementation may store integers and other data types directly inside the <em>cons</em>, but you shouldn’t care or depend on this.</p> <p>Functions like <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> only copy the <em>“list structure”</em>, it will never copy the data that a <em>cons</em> refers to. (But make sure to understand the difference between <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> and <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_cp_tre.htm">copy-tree</a>, refer to your Common Lisp book or the <a href="http://www.lispworks.com/documentation/HyperSpec/Front/index.htm">HyperSpec</a>.)</p> <p>A more accurate depiction of the <em>cons cells</em> and the objects they refer to would look like this.</p> <p><img src="/media/trees-lists/Diagram8.png" alt="Figure 8: append only copies conses, never the data they point to" /></p> <p>However, I’ll continue drawing the numbers inside the <em>cons</em> for simplicity.</p> <p>OK, so moving on, after the form <code>(add-child one (make-tree 3))</code> has been evaluated, we have the following situation.</p> <p><img src="/media/trees-lists/Diagram9.png" alt="Figure 9: State of cons cells in memory after appending (make-tree 3) as a child of ONE" /></p> <p><a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> copied the <em>top list structure</em> of the first list it had as an argument, this new list points to the last list, the node <em>“3”</em>. <em>“Top list structure”</em> means that <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> will only copy the <em>conses</em> that it reaches by following the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a>s of each <em>cons</em>; it will never go down the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">car</a> of any <em>cons</em> (that’s what <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_cp_tre.htm">copy-tree</a> would do).</p> <p>The <em>conses</em> of the new list are shown in blue, while the original <em>cons</em> that was holding the value <em>“1”</em> and pointing to TWO is shown in grey. Since nothing is pointing to this grey <em>cons</em>, it may be garbage collected.</p> <p>Now let’s see what happens after evaluating <code>(add-child one four)</code>.</p> <p><img src="/media/trees-lists/Diagram10.png" alt="Figure 10: State of cons cells in memory after appending FOUR as a child of ONE" /></p> <p>Again, we see the new copies of <em>conses</em> marked in blue, while the original <em>conses</em> are marked in grey. Since those grey <em>conses</em> can’t be reached anymore, they may eventually be garbage collected.</p> <p>And, after evaluating <code>(add-child one five)</code> we have the following.</p> <p><img src="/media/trees-lists/Diagram11.png" alt="Figure 11: State of cons cells in memory after appending FIVE as a child of ONE" /></p> <p>Again, the blue <em>conses</em> are the copies made by <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> and the grey <em>conses</em> here were the blue <em>conses</em> of <em>Figure 10</em>. Can you see why there’s only three grey <em>conses</em> (which can be GC) and four new blue <em>conses</em>?</p> <p>So far everything seems to be working ok. But now comes the interesting part.</p> <h3 id="exercise_6">Exercise</h3> <p>From the state of <em>cons cells</em> depicted in <em>Figure 11</em> above, can you see what will happen after the form <code>(add-child two (make-tree 6))</code> is evaluated?, can you see where lies the problem?. Drawing the <em>conses</em> could be helpful.</p> <p>Compare it to the figure below.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p><img src="/media/trees-lists/Diagram12.png" alt="Figure 12: State of cons cells in memory after appending (make-tree 6) as a child of TWO" /></p> <p>Let’s recall the definition of <code>add-child</code>:</p> <pre><code>(defun add-child (tree child) (setf (car tree) (append (car tree) child)) tree)</code></pre> <p>We can see that we are calling <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> on <code>(car tree)</code> and <code>child</code>, which correspond to <code>(car TWO)</code> and the value of <code>(make-tree 6)</code>. Therefore it’s easy to see that <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> is creating a copy of the <em>cons</em> in <code>(car TWO)</code>, returning a new list. Finally, <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_setf_.htm">setf</a> sets the <code>(car TWO)</code> to point to this new list, completely separating <code>TWO</code> from the original tree starting with <code>ONE</code>.</p> <p>Now we have two disjoint trees, one starting at the node pointed to by <code>ONE</code> and the other starting at the node pointed to by <code>TWO</code>. If we keep adding children to <code>TWO</code>, they will not be reachable from <code>ONE</code>. Likewise if we add children to the node <code>FOUR</code>.</p> <p>Can you see why there’s no problem in adding children to the node <code>FIVE</code>?</p> <h2 id="6_a_correct_version_of_addchild">6. A correct version of add-child</h2> <p>The problem with the definition of <code>add-child</code> above is that while it is defined to modify the argument <code>tree</code>, it’s actually only modifying part of it, and making new copies of other parts of <code>tree</code> (its children).</p> <p>In other words, we should consider that the argument <code>tree</code> refers not only to a single node, but to a whole <em>“tree”</em> with subtrees. Of course the fact that we named the argument <code>tree</code> is to remind us of that.</p> <p>We have to be always careful when designing our functions, and be clear if they will modify their arguments and how; or otherwise specify that our function will be completely side-effect free.</p> <p>When we say that the function <code>add-child</code> will modify the first argument to add the second argument as a child of it, then it must do that, but do it to the whole conceptual <em>tree</em>.</p> <p>The other alternative is to build a side-effect free <code>add-child</code>, which will create a new copy of the whole tree with the child appended, and leave the responsibility of capturing that new tree and updating any and all references to it or parts of it, to the user of our function.</p> <p>So how do we correct <code>add-child</code> to do the correct thing?. We know that we can’t use <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> because it creates new copies of the children that <code>tree</code> had. Knowing that, the solution is easy, we just need to use the destructive version of <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a>, which is called <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_nconc.htm">nconc</a>.</p> <p><a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_nconc.htm">nconc</a> won’t create any copy of anything, and instead will modify it’s arguments so that they are linked in a single list, much like what <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_append.htm">append</a> does.</p> <pre><code>(defun add-child (tree child) &quot;Takes two nodes created with 'make-tree' and adds the second node as a child of the first. Returns the first node, which will be modified.&quot; (nconc (car tree) child) tree)</code></pre> <p><a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_nconc.htm">nconc</a> will modify the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_car_c.htm">cdr</a> of the second <em>cons</em> of <code>tree</code> to add the <code>child</code> if it had not previous children; otherwise it will modify the last node in the list of children of <code>tree</code>, and link the <code>child</code> to it.</p> <p>Can you see why it’s just <code>(nconc (car tree) child)</code> and there’s no need to use <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_setf_.htm">setf</a>?</p> <p><strong>But be very careful</strong>, <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_nconc.htm">nconc</a> is the only destructive function for which you don’t need to capture it’s result with <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_setf_.htm">setf</a>. When you are using other destructive functions like <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_revers.htm">nreverse</a> you must always capture the result with <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_setf_.htm">setf</a>.</p> <h2 id="7_traversing_the_tree">7. Traversing the tree</h2> <p>Finally, it’s difficult to see the returned list and interpret it. Let’s try to create a function to traverse the tree as the last exercise.</p> <pre><code>(let ((one (make-tree 1)) (two (make-tree 2)) (four (make-tree 4)) (five (make-tree 5))) (add-child one two) (add-child one (make-tree 3)) (add-child one four) (add-child one five) (add-child two (make-tree 6)) (add-child two (make-tree 7)) (add-child two (make-tree 8)) (add-child four (add-child (make-tree 9) (make-tree 12))) (add-child five (make-tree 10)) (add-child five (make-tree 11)) ;; Print the contents of the tree, (traverse one) ;; and return it. one)</code></pre> <p>If we input the above into the <em>REPL</em>, we should get the following output.</p> <pre><code>Data: 1 Children: (2 3 4 5) Data: 2 Children: (6 7 8) Data: 6 Data: 7 Data: 8 Data: 3 Data: 4 Children: (9) Data: 9 Children: (12) Data: 12 Data: 5 Children: (10 11) Data: 10 Data: 11 ((1 (2 (6) (7) (8)) (3) (4 (9 (12))) (5 (10) (11))))</code></pre> <p>Notice that the list <code>((1 (2 ...)</code> is the value of <code>one</code> returned by the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/s_let_l.htm">let</a> form, while the information before it is being printed by <code>(traverse one)</code>.</p> <h3 id="exercise_7">Exercise</h3> <p>Write the function <code>traverse</code> so that it generates the output shown above. Compare it to the solution given below.</p> <p><strong>Tip:</strong> you can use the <code>&quot;~v@T&quot;</code> directive to the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_format.htm">format</a> function to move the cursor forward. For example:</p> <pre><code>(format t &quot;~v@TFoo&quot; 2) =&gt; Foo (format t &quot;~v@TBar&quot; 4) =&gt; Bar (format t &quot;~v@TFoo&quot; 6) =&gt; Foo</code></pre> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>.</p> <p>Let’s build the solution in steps. First we should print the node data:</p> <pre><code>(format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree))</code></pre> <p>The parameter <code>padding</code> will tell <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_format.htm">format</a> to indent the line a certain number of columns. We’ll see how to calculate that later. The <code>&quot;~&amp;&quot;</code> directive tells <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_format.htm">format</a> to print a newline only at the beginning of a line.</p> <p>We also need to print the children of the node, on the same line, but only if there are children to print.</p> <pre><code>(format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree)) (when (first-child tree) (format t &quot; Children: ~A&quot; (first-child tree)))</code></pre> <p>But wait, that’s not correct, it would print the whole tree of the children of node, like this:</p> <pre><code>Data: 1 Children: ((2 (6) (7) (8)) (3) (4 (9 (12))) (5 (10) (11)))</code></pre> <p>We only want the data in the nodes of the direct children of the node, as a list. We can use the <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_mapc_.htm">maplist</a> function for that. <a href="http://www.lispworks.com/documentation/HyperSpec/Body/f_mapc_.htm">maplist</a> applies a function to successive sublists of a list.</p> <pre><code>(format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree)) (when (first-child tree) (format t &quot; Children: ~A&quot; (maplist #'(lambda (x) (data x)) (first-child tree))))</code></pre> <p>Now let’s wrap this in a function.</p> <pre><code>(defun traverse (tree &amp;optional (padding 0)) (format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree)) (when (first-child tree) (format t &quot; Children: ~A&quot; (maplist #'(lambda (x) (data x)) (first-child tree)))))</code></pre> <p>OK, now that we have printed the data of the node and a list of the children, we have to repeat this process for each of the children of this node, and they should be printed indented below the current node.</p> <p>Since we already have the code to print the node and a list of its children, we will reuse this function and do a recursive call. One important thing to keep in mind when writing a recursive function is write a test for ending the recursion. In this case the recursion must end when the parameter <code>tree</code> is <code>nil</code>.</p> <pre><code>(defun traverse (tree &amp;optional (padding 0)) (when tree (format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree)) (when (first-child tree) (format t &quot; Children: ~A&quot; (maplist #'(lambda (x) (data x)) (first-child tree)))) (traverse (first-child tree) (+ padding 3))))</code></pre> <p>We can see above how the <code>padding</code> is being incremented, in this case by 3 columns. Since the parameter <code>padding</code> is optional, in the first invocation <code>(traverse one)</code> the <code>padding</code> will be zero, and then it will be incremented by 3 on each recursive invocation.</p> <p>But wait, the above code will only print the leftmost nodes in the tree, we still need to print the siblings of the nodes. That is very easy.</p> <pre><code>(defun traverse (tree &amp;optional (padding 0)) (when tree (format t &quot;~&amp;~v@TData: ~A&quot; padding (data tree)) (when (first-child tree) (format t &quot; Children: ~A&quot; (maplist #'(lambda (x) (data x)) (first-child tree)))) (traverse (first-child tree) (+ padding 3)) (traverse (next-sibling tree) padding)))</code></pre> <p>Since the siblings should be printed at the same level as the current node, the <code>padding</code> is left unmodified.</p> <h2 id="eof">:EOF</h2> <p>And that’s it. I hope this helped you get a better grasp of how <em>conses</em> work.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Other uses for OpenSSH gajon@gajon.org (Jorge Gajon) Sun, 31 May 2009 00:00:00 -0500 http://gajon.org/other-uses-openssh http://gajon.org/other-uses-openssh <p>Many people use <a href="http://www.openssh.com/">OpenSSH</a> to connect to a remote machine, but many don’t know that you can actually do other interesting things with it, other than just opening a remote login shell. In this article I’ll show you a few of them.</p> <p>We will see how to:</p> <ul> <li> <p>Copy files from one machine to another securely.</p> </li> <li> <p>Forward ports from your machine through the remote machine, creating a secure tunnel for your unsecured applications.</p> </li> <li> <p>Increase the security of your OpenSSH communications.</p> </li> </ul> <h2 id="make_sure_the_openssh_daemon_is_running">Make sure the OpenSSH daemon is running.</h2> <p>OpenSSH is the most widely used implementation of SSH, and you’ll probably find it already installed in most UNIX like systems.</p> <p>To be able to connect to the remote machine you’ll need to have the <code>sshd</code> daemon running in it. If this remote machine is actually physically remote then it must have it already running, otherwise how could you do anything at all?.</p> <p>But if your remote machine is sitting right next you, or you are running it as a virtual machine and you are experimenting with OpenSSH, then you can find out if the daemon is running by executing the following command:</p> <pre><code>$ ps -ef | grep sshd</code></pre> <p>You should see a line similar to this (the numbers will vary):</p> <pre><code>root 2859 1 0 13:29 ? 00:00:00 /usr/sbin/sshd</code></pre> <p>If it is not already running consult your distribution documentation to find out how to to start it. On an <a href="http://slackware.com/">Slackware</a> system you can do that by executing the following command as root:</p> <pre><code># /etc/rc.d/rc.sshd start</code></pre> <p>To have the <code>sshd</code> daemon run every time your Slackware system boots up, you’ll have to <code>chmod</code> the startup script so that it has execute permission.</p> <pre><code># chmod 755 /etc/rc.d/rc.sshd</code></pre> <p>On a <a href="http://debian.org/">Debian</a> system you can start the daemon by executing the following command:</p> <pre><code># /etc/init.d/ssh start</code></pre> <p>To find out how to make it start every time your Debian system boots up, read the <code>update-rc.d</code> man page to figure out how.</p> <pre><code>$ man update-rc.d</code></pre> <h2 id="1_copy_files_from_one_machine_to_another_securely">1. Copy files from one machine to another securely.</h2> <h3 id="11_using__for_simple_transfers">1.1. Using <code>scp</code> for simple transfers.</h3> <p>You can easily copy files from one machine to another through an encrypted connection with OpenSSH, and this will prevent anyone else from spying what you are sending over. The basic form is:</p> <pre><code>mymachine:~$ scp somefile.tar gajon@remote.com:</code></pre> <p>That will copy the file <code>somefile.tar</code> to the remote machine <code>remote.com</code> at the home folder of the user <code>gajon</code> (I’ll be using my own user name in all the examples here, but of course you should use your own instead).</p> <p>It is important to put a colon (<code>:</code>) after the remote machine name, otherwise <code>scp</code> will just rename your local file as <code>gajon@remote.com</code> in the same local folder in your machine.</p> <p>It’s also possible to specify the path and final name that you want the file to have in the remote server. For example, this will copy the file to the folder <code>backups</code> that is located in the user’s home.</p> <pre><code>mymachine:~$ scp somefile.tar gajon@remote.com:backups/</code></pre> <p>This will copy the file to the same folder as the previous example, but will save the file with a different name.</p> <pre><code>mymachine:~$ scp somefile.tar gajon@remote.com:backups/back02.tar</code></pre> <p>You can also copy files from the remote machine to your local machine, you just have to reverse the order of the arguments. For example, this will copy the file <code>myapp.log</code> that is located in the <code>/var/log</code> folder of the remote machine to your local machine, but with the name <code>remote.log</code> instead.</p> <pre><code>mymachine:~$ scp gajon@remote.com:/var/log/myapp.log remote.log</code></pre> <p>If you wanted to preserve the original name you could have used the following command:</p> <pre><code>mymachine:~$ scp gajon@remote.com:/var/log/myapp.log .</code></pre> <p>Notice that the remote path after the colon starts with a slash, that indicates an absolute path. If you don’t start the path with a slash then it’s a relative path, relative to the home of the user in the remote machine.</p> <h3 id="12_using__for_secure_ftp_transfers">1.2 Using <code>sftp</code> for secure FTP transfers.</h3> <p>Many people know how to use FTP, but some may not realize that everything they transfer during an FTP session, including their user names and passwords, are sent unencrypted or protected in any form. This opens up the possibility that someone with ill intentions might capture that information.</p> <p><code>sftp</code> solves that problem and it is used the same way as <code>ftp</code>, so that the user doesn’t need to learn anything new, and most graphical FTP clients can also use <code>sftp</code> – for example, <a href="http://filezilla-project.org/">FileZilla</a>.</p> <p>In this example we get the file <code>back02.tar</code> from the remote machine and save it in our local computer.</p> <pre><code>mymachine:~$ sftp gajon@remote.com Connecting to remote.com... gajon@remote.com's password: sftp&gt; cd backups sftp&gt; get back02.tar sftp&gt; bye</code></pre> <p>When you log on to a remote machine with the <code>sftp</code> client the prompt will change to <code>sftp&gt;</code>. The commands are almost the same as with a traditional <code>ftp</code>, you can enter <code>?</code> to see a list of available commands.</p> <h2 id="2_forwarding_ports">2. Forwarding ports.</h2> <h3 id="21_from_your_local_machine_to_a_remote_machine">2.1. From your local machine to a remote machine.</h3> <p>Port forwarding is useful for creating a secure data tunnel for an application that doesn’t support encryption or other form of secure communication out of the box. This is also useful to go through a firewall and skip any of its restrictions.</p> <p>As an example, lets suppose you want to access a POP3 server. You can redirect one of your local ports, lets say <code>9999</code>, to the port <code>110</code> (POP3) of the remote machine.</p> <pre><code>mymachine:~$ ssh -L 9999:localhost:110 gajon@remote.com</code></pre> <p>Next you need to configure your email client to use port <code>9999</code>. OpenSSH will take all traffic on port <code>9999</code> in your local machine and send it encrypted to the remote machine, where the data will continue through port <code>110</code> to its original destination.</p> <p>Notice that the remote machine will send the data to the final destination using port <code>110</code>, <strong>AND</strong> this data will <strong>NOT</strong> be encrypted anymore from that point on. You must be aware of that, if your remote machine and the destination machine (POP3 server) are the same then it’s OK, otherwise there’s still a risk.</p> <h3 id="22_from_the_remote_machine_to_your_local_machine">2.2. From the remote machine to your local machine.</h3> <p>It’s also possible to do a forwarding in the other direction, from the remote machine to your local machine. An example of when this could be useful is when you have a machine in your office that sits behind a firewall that blocks all inbound connections, but permits all outbound connections, and you want to be able to connect to your office machine from your home and open an <code>ssh</code> session.</p> <p>From the office machine you can establish the tunnel to the home machine:</p> <pre><code>office:~$ ssh -R 2222:localhost:22 gajon@home</code></pre> <p>This tunnel will listen on port <code>2222</code> of the home machine and will encrypt data and send it to port <code>22</code> of the machine at the office. Once you are at home you can connect to the office machine from home with OpenSSH, using this tunnel that you established from the office machine.</p> <pre><code>home:~$ ssh -p 2222 gajon@office</code></pre> <p>Notice that we are indicating to <code>ssh</code> that we want to connect with the port <code>2222</code> (<code>-p</code> option), this is because that’s the port that your local machine is listening to.</p> <h3 id="23_dynamic_port_forwarding">2.3. Dynamic port forwarding.</h3> <p>Dynamic port forwarding lets you forward any port on your local machine through a secure tunnel to a remote machine, by creating a SOCKS server.</p> <p>Let’s say that you have a restrictive firewall at the office that blocks web pages that have not been authorized. Imagine you need to find a solution to a programming problem you are having but can’t do that because the firewall is blocking all websites that contain anything similar to a forum, and we all know that forums are where we always find solutions to a technical, undocumented problem.</p> <p>Well, that has happened to me, and the solution is a SOCKS server that can proxy our data through a remote machine; and have it encrypted as an added bonus!.</p> <p>You can establish a dynamic port forwarding tunnel as:</p> <pre><code>mymachine:~$ ssh -D 12345 gajon@remote.com</code></pre> <p>With that command, you establish a connection to the remote machine, but OpenSSH will listen to any SOCKS communications on your local machine at port <code>12345</code>.</p> <p>To be able to use this tunnel, your application that needs to connect must know how to use the SOCKS protocol (either SOCKS4 or SOCKS5). On Firefox it’s easy to do that, we go to <em>Preferences</em> and in the <em>Advanced</em> group select the <em>Network</em> tab, click on the <em>Settings</em> button and select <em>Manual proxy settings</em> and in the <em>SOCKS Server</em> field put <code>localhost</code>, and <em>port</em> <code>12345</code>.</p> <p>And that’s it, when Firefox needs to load a website, it will communicate with the SOCKS server in your local machine with port <code>12345</code>; OpenSSH will encrypt your data and send it through a secure channel to the remote machine, where the data will continue its way to its destination and original port.</p> <p>But there are many applications that don’t implement a SOCKS protocol, one example is Opera. There are tools that can wrap a non SOCKS application and transparently send all its connections through a SOCKS server. One such application that I have used is <code>tsocks</code>.</p> <h2 id="3_increase_the_security_of_your_openssh_communications">3. Increase the security of your OpenSSH communications.</h2> <p>There are several things you can do to increase the security of your OpenSSH communications, and it’s a good idea to do them.</p> <h3 id="31_disable_root_logins">3.1. Disable root logins.</h3> <p>The most useful and easiest way to increase the security of your OpenSSH server is to disable <code>root</code> logins. To do this you must edit the <code>sshd_config</code> file in your server.</p> <pre><code>remote:~# vim /etc/ssh/sshd_config</code></pre> <p>Search for the <code>PermitRootLogin</code> attribute and change it to <code>no</code>. Then restart your OpenSSH server. On Slackware you use the following line:</p> <pre><code>remote:~# /etc/rc.d/rc.ssh restart</code></pre> <p>On a Debian system you use the following line:</p> <pre><code>remote:~# /etc/init.d/sshd restart</code></pre> <h3 id="32_change_the_port_where_openssh_listens">3.2. Change the port where OpenSSH listens.</h3> <p>There are many people with bad intentions that use automated tools that try to breach OpenSSH servers. If you have a server on the Internet, you may have noticed in your logs that there are lots and lots of attempts to login to your server using many different user names. What they are trying to do is to get to one server by applying a so called <em>dictionary attack</em>, that is, attempting a huge combination of common user names and passwords, with the hope that one of those combinations will actually succeed.</p> <p>A very easy way to stop that is to simply change the port of your OpenSSH daemon. If the attacker cannot find a server at the default port <code>22</code> he will probably just move to the next server that he can find. These kind of attackers don’t bother checking every port of the machine, that would take too much time, instead what they do is try all the ip addresses they can.</p> <p>To change the port number of your server, edit the <code>sshd_config</code> file again and change the <code>Port</code> attribute. For example:</p> <pre><code>Port 30234</code></pre> <p>Restart the OpenSSH daemon, and now to connect you have to specify the port to use. You can use the <code>-p</code> switch:</p> <pre><code>mymachine:~$ ssh -p 30234 gajon@remote.com</code></pre> <p>But having to type the <code>-p</code> switch and the port is tedious, so instead of that you can specify the port in a <code>config</code> file that your <code>ssh</code> client can use. This <code>config</code> file is usually found in the <code>$HOME/.ssh/</code> folder.</p> <p>Open up the <code>$HOME/.ssh/config</code> file in your editor and put these lines there:</p> <pre><code>Host remote.com Port 30234</code></pre> <p>Make sure the <code>Port</code> line below the <code>Host</code> is indented.</p> <p>This way you don’t have to specify the port on the command line.</p> <pre><code>mymachine:~$ ssh gajon@remote.com</code></pre> <p>You could also add the user name in the <code>config</code> file:</p> <pre><code>Host remote.com User gajon Port 30234</code></pre> <p>And then you can just type:</p> <pre><code>mymachine:~$ ssh remote.com</code></pre> <h3 id="33_specify_which_groups_can_connect">3.3. Specify which groups can connect.</h3> <p>You can tell OpenSSH that only those users belonging to a certain group can log in. For example, lets create a group <code>sshers</code>, and add your user to that group, you do that with <code>root</code>. In this example I’ll be adding the user <code>gajon</code> to the group.</p> <p>We have to do this on the remote server, and the first thing to do is create the group:</p> <pre><code>remote:~# groupadd sshers</code></pre> <p>Then check what groups the user is already a member of:</p> <pre><code>remote:~# groups gajon gajon : users wheel audio video cdrom games</code></pre> <p>Then make it a member of the <code>sshers</code> group:</p> <pre><code>remote:~# usermod -G wheel,audio,video,cdrom,games,sshers gajon remote:~# groups gajon gajon : users wheel audio video cdrom games sshers</code></pre> <p>Notice that the first group listed with <code>groups gajon</code> is <code>users</code>, that’s the primary group of the user, the rest are the secondary groups. You modify the list of secondary groups with <code>usermod -G</code>, separating each group name with a coma. See <code>man usermod</code> for more details.</p> <p>OK, now that you have your new group and your user is part of that group, let’s configure the OpenSSH daemon so that only members of that group can connect at the remote server. Edit the <code>sshd_config</code> file in the server:</p> <pre><code>remote:~# vim /etc/ssh/sshd_config</code></pre> <p>And add the <code>AllowGroups</code> property to to file:</p> <pre><code># Only members of group sshers will be able to connect AllowGroups sshers</code></pre> <p>And that’s it, any user that is not part of the <code>sshers</code> group will not be able to log in.</p> <h3 id="34_use_publickey_authentication">3.4. Use Public-Key Authentication.</h3> <p>It is not a good idea to type the password of your remote user while connecting to the remote machine. The problem is that if someones guesses your password, or looks over your shoulder while you are typing it, or somehow it is compromised, that person that has your password can easily connect to the remote machine from anywhere.</p> <p>To avoid this danger, you could instead authenticate by setting up a so called <em>“Public-Key Cryptography”</em> authentication.</p> <p>In Public-Key Cryptography, each party has a pair of keys, one key is known as the <em>“private key”</em> while the other is known as the <em>“public key”</em>. The idea is that when you want to communicate with another party you use their <em>public key</em> and only that who has the corresponding <em>private key</em> can decrypt the message.</p> <p>This topic of Public-Key Cryptography is quite interesting, if you want to read more you can read the <a href="http://en.wikipedia.org/wiki/Public-key_cryptography" title="Public-Key Cryptography - Wikipedia page">Wikipedia page</a> about it.</p> <p>The OpenSSH daemon can be configured to authenticate users with a Public-Key Cryptography method. The advantage of this scheme is that you will have your pair of keys, one public and one private, and if someone managed to obtain your user password it won’t matter, because no one will be able to log in to the remote system without the proper private key needed to establish the authentication.</p> <p>Also, for an attacker to be able to log in into your system, not only he’ll need to get your private key, but also the password you use to decrypt your private key, which is only inside your head.</p> <p>Let’s create your key pair in your local machine:</p> <pre><code>mymachine:~$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/gajon/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/gajon/.ssh/id_rsa. Your public key has been saved in /home/gajon/.ssh/id_rsa.pub. The key fingerprint is: fd:6b:f8:82:85:40:e2:8b:1b:31:47:c6:94:25:47:87 gajon@mymachine</code></pre> <p>Two keys have been generated, one private (<code>id_rsa</code>), and the other public (<code>id_rsa.pub</code>). The password that you entered is used to encrypt your private key.</p> <p>Notice that the files were placed in your <code>$HOME/.ssh/</code> folder in your computer. Make sure that’s the case, you don’t want them scattered on your file system.</p> <p>Next you have to copy your public key to the <code>$HOME/.ssh/authorized_keys</code> file in the remote server. The <code>authorized_keys</code> file can contain the information of several public keys, so to avoid overwriting an existing <code>authorized_keys</code> in your remote server, you are going to append the contents of your public key to the <code>authorized_keys</code> file. And if that file did not exist, it will be created anyway.</p> <pre><code>mymachine:~$ cat .ssh/id_rsa.pub | ssh gajon@remote.com &quot;cat &gt;&gt; .ssh/authorized_keys&quot;</code></pre> <p>Now to connect to the remote machine using Public-Key Cryptography authentication, you need to use the <code>-i</code> switch and specify the <strong>private</strong> key to use:</p> <pre><code>mymachine:~$ ssh -i ~/.ssh/id_rsa gajon@remote.com Enter passphrase for key '/home/gajon/.ssh/id_rsa': remote.com:~$</code></pre> <p>Notice that your <code>ssh</code> client is asking you for your private key password, which is NOT the same password for your system user, but the password you entered when you created this pair of keys.</p> <p>After you have seen that public key authentication works, it is a good idea to disable simple password logins on your remote machine, so that the only way of connecting to it is with the use of Public-Key Cryptography. To do that you’ll need to set the <code>PasswordAuthentication</code> property with the value of <code>no</code>, in the <code>sshd_config</code> file.</p> <pre><code>remote.com:~# vim /etc/ssh/sshd_config</code></pre> <p>Disable tunnelled clear text passwords:</p> <pre><code>PasswordAuthentication no</code></pre> <p>Restart your <code>sshd</code> daemon, and now you will only be able to log in to the remote server if the user in the remote server has the public key info in the <code>.ssh/authorized_keys</code> file, AND you have the corresponding private key in your own machine, and of course you know the password to that private key.</p> <p><strong>BE VERY CAREFUL</strong>, after you make the above change and restart the daemon, <strong>do not</strong> log out of your session just yet, open up another terminal and try to make a second connection to your remote machine. If you can’t, try to figure it out in the other session that you still have open. I’m telling you this because if you disable password authentication, and the public key authentication fails, you could easily lock yourself out of the remote machine, which would be really bad.</p> <p>Now, it is annoying having to specify the private key to use with the <code>-i</code> switch every time you want to connect to the remote machine. So instead of that, you can configure what private key to use in the <code>$HOME/.ssh/config</code> file in your local machine.</p> <p>Retaking the example we saw in section 3.2, you add the <code>IdentityFile</code> setting which specifies a private key to use when connecting to the host.</p> <pre><code>Host remote.com User gajon IdentityFile ~/.ssh/id_rsa Port 30234</code></pre> <p>This way it’s not necessary to specify the private key on the command line, nor the user or the port (if it were different from the default <code>22</code>):</p> <pre><code>mymachine:~$ ssh remote.com Enter passphrase for key '/home/gajon/.ssh/id_rsa': remote.com:~$</code></pre> <p>A final note; you can name your keys any way you want. If you have more than one remote machine you should give more meaningful names to your key pairs, so that you can more easily manage them. For example, if we had three servers, <em>“breakpoint”</em>, <em>“lucretia”</em> and <em>“she-wolf”</em>, we could create a <code>$HOME/.ssh/config</code> file with the following entries:</p> <pre><code>Host breakpoint HostName 11.22.33.44 User gajon IdentityFile ~/.ssh/id_rsa_breakpoint Port 30234 Host lucretia HostName lucretia.example.com User johnny IdentityFile ~/.ssh/id_rsa_lucretia Host she-wolf.remote.com User admin IdentityFile ~/.ssh/id_rsa_shewolf</code></pre> <p>Notice that I added another setting, <code>HostName</code>, which specifies the actual host to connect to. It could be an IP address or a fully qualified name of the server. That way the <code>Host</code> setting is actually an alias that you can freely define. With that example we can easily connect to the <em>“breakpoint”</em> server, with an IP address of <code>11.22.33.44</code>.</p> <pre><code>mymachine:~$ ssh breakpoint</code></pre> <h2 id="and_thats_all_for_now">And that’s all for now.</h2> <p>I hope these tips are useful to you, but remember that we have just seen a few uses of OpenSSH, and that there are a lot more. For example, in a next article I’ll write how to use Public-Key authentication to easily issue commands to remote servers and automate tasks.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Going to PyCon 2008 gajon@gajon.org (Jorge Gajon) Mon, 10 Mar 2008 00:00:00 -0600 http://gajon.org/going-pycon-2008 http://gajon.org/going-pycon-2008 <p>There’s only 4 days left before I fly to Chicago to attend the <a href="http://us.pycon.org/2008/about/">PyCon 2008</a> conference, the biggest event for the <a href="http://python.org">Python</a> community. This will be the second time that I go to PyCon, <a href="http://us.pycon.org/TX2007/HomePage">last year</a> the conference was held in Addison Texas and I really had a great time, I met cool people, visited a few restaurants and learned interesting things about Python.</p> <p>Here are some photos that I took last year, and you can see more on my <a href="http://flickr.com/photos/jorge_g/sets/72157604074072678/">flickr set</a>:</p> <p><a href='http://www.flickr.com/photos/13556513@N02/2319789671/' title='PyCon 07 - Addison, Texas by J Gajon, on Flickr'><img alt='PyCon 07 - Addison, Texas' height='180' src='http://farm3.static.flickr.com/2418/2319789671_e7e79d4263_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/13556513@N02/2319789675/' title='PyCon 07 - Addison, Texas by J Gajon, on Flickr'><img alt='PyCon 07 - Addison, Texas' height='180' src='http://farm3.static.flickr.com/2402/2319789675_2af946a248_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/13556513@N02/2319789673/' title='PyCon 07 - Addison, Texas by J Gajon, on Flickr'><img alt='PyCon 07 - Addison, Texas' height='180' src='http://farm3.static.flickr.com/2015/2319789673_7082da3859_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/13556513@N02/2320633978/' title='PyCon 07 - Addison, Texas by J Gajon, on Flickr'><img alt='PyCon 07 - Addison, Texas' height='180' src='http://farm4.static.flickr.com/3042/2320633978_d50da2e9a6_m.jpg' width='240' /></a></p> <p><span class='clear' /></p> <p>I’m sure this year’s PyCon will be great!</p> <p><em>Update</em>: PyCon 08 is over, <a href="http://flickr.com/photos/jorge_g/sets/72157604225881775/">here are</a> some of the photos that I took.</p> <p><a href='http://www.flickr.com/photos/jorge_g/2355693743/' title='PyCon 08 - Chicago, Illinois by J Gajon, on Flickr'><img alt='PyCon 08 - Chicago, Illinois' height='180' src='http://farm3.static.flickr.com/2171/2355693743_b1939ce197_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/jorge_g/2355921819/' title='PyCon 08 - Chicago, Illinois by J Gajon, on Flickr'><img alt='PyCon 08 - Chicago, Illinois' height='180' src='http://farm3.static.flickr.com/2199/2355921819_e2f31b795f_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/jorge_g/2356307568/' title='PyCon 08 - Chicago, Illinois by J Gajon, on Flickr'><img alt='PyCon 08 - Chicago, Illinois' height='160' src='http://farm3.static.flickr.com/2345/2356307568_7b1b59d6d7_m.jpg' width='240' /></a></p> <p><a href='http://www.flickr.com/photos/jorge_g/2355921883/' title='PyCon 08 - Chicago, Illinois by J Gajon, on Flickr'><img alt='PyCon 08 - Chicago, Illinois' height='180' src='http://farm4.static.flickr.com/3123/2355921883_b82a90186f_m.jpg' width='240' /></a></p> <p><span class='clear' /></p> <p>It’s nice to meet intelligent people and be surrounded by all this enthusiasm around this cool language and all the projects that are being built with it.</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: --> Installing a Realtek RTL8185 wireless card gajon@gajon.org (Jorge Gajon) Mon, 25 Feb 2008 00:00:00 -0600 http://gajon.org/installing-realtek-rtl8185-wireless-card http://gajon.org/installing-realtek-rtl8185-wireless-card <p>In this post I explain how I made my wireless card with a Realtek RTL8185 chipset work in <a href="http://www.slackware.com">Slackware GNU/Linux</a>.</p> <p>I recently moved to a new apartment and the Internet modem had to be placed in the bedroom, separate from the living room where I have my workstation, so I had to choose between buying a wireless card for my pc or drilling a hole through the wall to slip in an ethernet cable.</p> <p>Since I don’t have a drill, and my wife wouldn’t have liked a cable going through the middle of the makeup mirror, I got an Encore Electronics Wireless-G PCI Adapter (<a href="http://www.encore-usa.com/product_item.php?region=us&bid=2&pgid=81_4&pid=285">ENLWI-G2</a>). Actually this is just a card with a Realtek RTL8185 chipset on it, and you can get the Linux drivers from the <a href="http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=1&PFid=1&Level=6&Conn=5&DownTypeID=3&GetDown=false&Downloads=true#RTL8185L">Realtek website</a>.</p> <h2 id="building_the_modules_and_testing">Building the modules and testing</h2> <p>I have to warn you that I had my PC crash a few times while experimenting with the wireless settings, so be patient and be prepared to hit the power button.</p> <p>Download and extract the appropriate <em>*.tar.gz</em> file from the Realtek website, you will find instructions on how to build and load the modules in the <code>readme</code> file. Basically you have to:</p> <ol> <li> <p>Run the file <code>makedrv</code> to build the modules from the source code.</p> </li> <li> <p>Run the file <code>wlan0up</code> to load the modules into the running kernel.</p> <p>This is where I encountered the first problem, for some reason the last line in the <code>wlan0up</code> script (<code>ifconfig wlan0 up</code>) made my computer crash. After rebooting I commented out that line and tried again, no problem was found and I saw that the <code>wlan0</code> interface was already up anyway (run <code>ifconfig</code> to see the interfaces that are up).</p> <p>These modules are not going to be loaded automatically at boot up yet, we’ll see how to do that later.</p> </li> <li> <p>The third step after the wireless interface is up is to configure your wireless link settings and getting an IP address, see the <code>readme</code> file to get the details.</p> </li> </ol> <p>In my case I only needed these two commands to set up the wireless link to the router.</p> <pre><code>iwconfig wlan0 essid &quot;Megadeth&quot; iwconfig wlan0 key abcd123456</code></pre> <p>My wireless LAN name is <em>“Megadeth”</em> and it uses WEP encryption in Open security mode. Yes, I know that WEP is practically useless for security, but I have other devices that only talk WEP.</p> <p>After setting the wireless link I had to run this command to get an ip address from the router:</p> <pre><code>dhcpcd -t 20 wlan0</code></pre> <p>I did <em>NOT</em> run the <code>wlan0dhcp</code> file that was provided in the tarball as it appears to be RedHat specific, or something like that, I really don’t know, but I know for sure that it wouldn’t work in Slackware.</p> <h2 id="making_the_modules_load_at_boot_up">Making the modules load at boot up</h2> <p>The next step was to figure out how to have the modules loaded up when the computer starts.</p> <p>One option is to leave the compiled modules where they are (or move it wherever you want) and load them up by putting these instructions at the end of the <code>/etc/rc.d/rc.modules-XYZ</code> script where <code>XYZ</code> is the version of the kernel that the modules were built for:</p> <pre><code>/sbin/insmod /root/rtl8185/ieee80211/ieee80211_crypt-rtl.ko /sbin/insmod /root/rtl8185/ieee80211/ieee80211_crypt_wep-rtl.ko /sbin/insmod /root/rtl8185/ieee80211/ieee80211_crypt_tkip-rtl.ko /sbin/insmod /root/rtl8185/ieee80211/ieee80211_crypt_ccmp-rtl.ko /sbin/insmod /root/rtl8185/ieee80211/ieee80211-rtl.ko /sbin/insmod /root/rtl8185/rtl8185/r8180.ko</code></pre> <p>The <code>/root/rtl8185/</code> folder is where I unpacked and built the modules.</p> <p>The second option, which is the one I prefer, is to move the modules to the appropriate kernel directories so that you don’t clutter your root folder.</p> <p>But before that you should know that the ieee80211 modules that are built from this package are intended as a replacement for the ieee80211 stack that comes with the kernel. This means that you have to move out the original ieee80211 stack to avoid having them loaded into the kernel. This might not be strictly necessary since the modules have different names, but I wasn’t using the original stack and I didn’t see any need for it so I moved them out just as a precaution, as having both stacks loaded would have caused a conflict.</p> <pre><code># Back up the original stack. cd /lib/modules/2.6.21.5-smp/kernel/net mv ieee80211 /root/ieee80211_original_stack # Move the new modules there. # we are at /lib/modules/2.6.21.5-smp/kernel/net mkdir ieee80211 cp /root/rtl8185/ieee80211/*.ko ieee80211/ # Finally move the r8180.ko module too. cd /lib/modules/2.6.21.5-smp/kernel/drivers/net/wireless cp /root/rtl8185/rtl8185/r8180.ko . # Update module dependencies. depmod -ae</code></pre> <p>The last command, <code>depmod -ae</code>, updates a file <code>/lib/modules/2.6.21.5-smp/modules.dep</code> which indicates the dependencies for each module. If you inspect that file you’ll see that the <code>r8180</code> module depends on the <code>ieee80211_rtl</code> and <code>ieee80211_crypt_rtl</code> modules; indeed, if you reboot and type <code>lsmod</code> you’ll see that these modules are loaded.</p> <pre><code>r8180 ieee80211_rtl ieee80211_crypt_rtl</code></pre> <p>However, there are three additional modules that need to be loaded too:</p> <pre><code>ieee80211_crypt_wep-rtl ieee80211_crypt_tkip-rtl ieee80211_crypt_ccmp-rtl</code></pre> <p>Without these modules you won’t be able to set encryption keys for your wireless link. These are not being loaded because no module depends on them. To change that we have to edit the <code>modules.dep</code> file and edit the line that specifies the dependencies of the <code>r8180</code> module, the format of a <code>modules.dep</code> entry is:</p> <pre><code>/path/to/module_a.ko: /path/to/module2.ko /path/to/module1.ko</code></pre> <p>Which means that <code>module_a.ko</code> depends on <code>module1.ko</code> and <code>module2.ko</code>. The modules are loaded from right to left, which means that module1 is loaded first, followed by module2 and finally module_a last.</p> <p>So then, open up the <code>modules.dep</code> file and search for the line where the dependencies of the <code>r8180</code> are defined and change it so that all the necessary modules are loaded.</p> <p>Here I break the line into multiple lines so that you can read it easily, but make sure it is only <em>one whole line</em>, so please pay attention. Also, the order is important, so pay double attention.</p> <p>I repeat, this <em>must be one single line</em> in your <code>modules.dep</code> file:</p> <pre><code>/lib/modules/2.6.21.5-smp/kernel/drivers/net/wireless/r8180.ko: /lib/modules/2.6.21.5-smp/kernel/net/ieee80211/ieee80211-rtl.ko /lib/modules/2.6.21.5-smp/kernel/net/ieee80211/ieee80211_crypt_ccmp-rtl.ko /lib/modules/2.6.21.5-smp/kernel/net/ieee80211/ieee80211_crypt_tkip-rtl.ko /lib/modules/2.6.21.5-smp/kernel/net/ieee80211/ieee80211_crypt_wep-rtl.ko /lib/modules/2.6.21.5-smp/kernel/net/ieee80211/ieee80211_crypt-rtl.ko</code></pre> <p>After you reboot you’ll see that all the modules have been loaded up.</p> <h2 id="setting_the_wireless_link_and_getting_an_ip_at_boot_up">Setting the wireless link and getting an IP at boot up</h2> <p>The final step is to set your wireless <em>essid</em> and <em>key</em> and get an ip from the router when your computer boots up.</p> <p>Since I run Slackware, the place to define ip settings is the file <code>/etc/rc.d/rc.inet1.conf</code> - the relevant lines in my settings file are:</p> <pre><code>... IFNAME[4]=&quot;wlan0&quot; USE_DHCP[4]=&quot;yes&quot; DHCP_TIMEOUT[4]=30 WLAN_ESSID[4]=Megadeth WLAN_KEY[4]=&quot;abcd123456 open&quot; ...</code></pre> <p>There are a few things to notice here.</p> <p>I added the variable <code>DHCP_TIMEOUT</code>, this is because the <code>/etc/rc.d/rc.inet1</code> script executes an <code>ifconfig wlan0 up</code> if that variable is not defined (look around line 117), and I don’t know why, but for some reason sometimes that makes my computer freeze.</p> <p>This is strange because the <code>rc.wireless</code> script also executes this command but I have not found any problem with it, it seems that the card doesn’t like to be “upped” so many times.</p> <p>For the <code>WLAN_ESSID</code> variable you’ll type your wireless LAN name without quotes.</p> <p>You’ll notice that the <code>WLAN_KEY</code> contains the word <em>open</em>, this is because if you don’t specify either <em>open</em> or <em>restricted</em> the <code>/etc/rc.d/rc.wireless</code> script will set the key to <em>restricted</em> mode by default. In my case I wanted it to be in <em>open</em> security mode.</p> <p>One last thing that is important, the <code>/etc/rc.d/rc.wireless</code> script sets a <em>nick</em> option on the wireless settings, but this driver does not support that option and it will throw an error. Open that file and comment out that line, look around line 184, i.e.:</p> <pre><code>if [ ! -n &quot;$NICKNAME&quot; ] ; then NICKNAME=`/bin/hostname` fi if [ -n &quot;$ESSID&quot; -o -n &quot;$MODE&quot; ] ; then echo &quot;$0: $IWCOMMAND nick $NICKNAME&quot; | $LOGGER # $IWCOMMAND nick $NICKNAME &lt;-- this is not supported fi</code></pre> <p>And that’s it, try running <code>/etc/rc.d/rc.inet1 stop</code> and then <code>/etc/rc.d/rc.inet1 start</code>.</p> <p>You’ll notice that this line is printed on the screen:</p> <pre><code>./rc.wireless: wlan0 information: 'Any ESSID...'</code></pre> <p>If it bothers you, you can edit the <code>/etc/rc.d/rc.wireless.conf</code> file and comment out the lines from line number 38 to 41 and you won’t see that message anymore.</p> <p>Or, you can change the <code>INFO</code> variable to something else. I’m more nostalgic and so I changed it to a quote from an old song I like:</p> <pre><code>## --------- START SECTION TO REMOVE ----------- ## Pick up any Access Point, should work on most 802.11 cards *) INFO=&quot;The warheads will all rust in peace&quot; ESSID=&quot;Megadeth&quot; ;; ## ---------- END SECTION TO REMOVE ------------</code></pre> <p>It’s not necessary to put the <code>ESSID</code> here as it is overridden by what you set on the <code>rc.inet1.conf</code> file.</p> <h2 id="if_things_get_really_nasty">If things get really nasty….</h2> <p>If you screwed it with the wireless settings and you can’t get your pc to boot up normally because the driver crashes it, you can boot up in single user mode (also called emergency mode). When you boot up your computer and get the LILO prompt, select your Linux installation and type the word <em>single</em> after the label.</p> <p>In single user mode no networking configurations will be set up at all so that you can edit your settings and try again.</p> <p>Enjoy</p> <!-- vim: set tw=74 sw=4 ts=4 et spell filetype=mkd: -->