-----
title: "RawLine - a 100% Ruby solution for console inline editing"
content-type: article
timestamp: 1205128740
tags: "ruby|programming|opensource|rawline"
-----
<p>One of the many things I like about Ruby is its cross-platform nature: as a general rule, Ruby code runs on everything which supports Ruby, regardless of its architecture and platform (yes, there are quite a few exceptions, but let&#8217;s accept this generalization for now).</p>
<p>More specifically, I liked the fact that I could use the <a href="http://tiswww.case.edu/php/chet/readline/rltop.html"><span class="caps">GNU</span> Readline library</a> with Ruby seamlessly on both Windows and Linux.<br />
Readline offers quite a lot of features which are useful for those people like me who enjoy creating command-line scripts, in a nutshell, it provides:</p>
<ul>
	<li>File/Word completion</li>
	<li>History support</li>
	<li>Custom key bindings which can be modified via .inputrc</li>
	<li>Emacs and Vi edit modes</li>
</ul>
<p>Basically it makes your command-line interface fast and powerful, and that&#8217;s not an overstatement. Ruby&#8217;s own <span class="caps">IRB</span> can be enhanced by enabling readline and completion, and it works great &#8212; at least on *nix systems.</p>
<p>For some weird reason, some people had problems with Readline on Windows: in particular, things get nasty when you start editing long lines. Text gets garbled, the cursor goes up one or two lines and doesn&#8217;t come back, and other similar leprechaun&#8217;s tricks, which are not that funny after a while.</p>
<p>Apparently there&#8217;s no alternative to Readline in the Ruby world. If you wan&#8217;t tab completion that&#8217;s it, you&#8217;re stuck. Would it be difficult to implement <em>some</em> of Readline functionality natively in Ruby? Maybe, but the problem is that for some reason the Ruby Standard Library doesn&#8217;t have low level methods to operate on keystrokes&#8230;</p>
<p>&#8230;but luckily, the <a href="http://highline.rubyforge.org/">HighLine</a> gem does! James Edward Gray II keeps pointing out here and here that HighLine&#8217;s own <code>get_character</code> method does just that: it returns the corresponding character code(s) right when a key is pressed, unlike <code>IO#gets()</code> which waits for the user to press <span class="caps">ENTER</span>.</p>
<p>Believe it or not, that tiny method can do wonders&#8230;h2. Reverse-engineering escape codes</p>
<p>So here&#8217;s a little script which uses <code>get_character()</code> in an endless loop, diligently printing the character codes corresponding to a keystroke:</p>
<div class='ruby'><pre><code>#!/usr/local/bin/ruby -w

require 'rubygems'
require 'highline/system_extensions'

include HighLine::SystemExtensions

puts "Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit)."

loop do

	print "=&gt; "
	char = get_character
	case char
	when ?\C-x: print "Exiting..."; exit;
	else puts "#{char.chr} [#{char}] (hex: #{char.to_s(16)})";
	end
	
end</code></pre></div><p>A pretty harmless little thing. Try to run it and press some keys, and see what you get:</p>
<div style="font-family: Monospace">
<p>Press a key to view the corresponding <span class="caps">ASCII</span> code(s) (or <span class="caps">CTRL</span>-X to exit).</p>
<p>=&gt; a <sup class="footnote" id="fnr96"><a href="#fn96">96</a></sup> (hex: 61)</p>
<p>=&gt; 1 <sup class="footnote" id="fnr49"><a href="#fn49">49</a></sup> (hex: 31)</p>
<p>=&gt; Q <sup class="footnote" id="fnr81"><a href="#fn81">81</a></sup> (hex: 51)</p>
<p>=&gt; &alpha; <sup class="footnote" id="fnr224"><a href="#fn224">224</a></sup> (hex: e0)</p>
<p>=&gt; K <sup class="footnote" id="fnr75"><a href="#fn75">75</a></sup> (hex: 4b)</p>
</div>
<p>Hang on, what are the last two codes? <em>A left arrow key on Windows</em>, apparently.</p>
<p><strong>Welcome to the wonderful world of input escape sequences!</strong></p>
<p>To cut a long story short, both Windows and *nix system &#8220;terminals&#8221; translate special keystrokes into sequences of two or more codes. This applies to things like <span class="caps">DEL</span>, <span class="caps">INSERT</span>, arrows, etc. etc.<br />
For some ideas, check out:</p>
<ul>
	<li><a href="http://www.microsoft.com/whdc/device/input/Scancode.mspx">Windows Scancodes</a> (Thanks <a href="http://64.223.189.234/node/92">Huff</a>)</li>
	<li><a href="http://www.connectrf.com/Documents/vt220.html">VT220 Terminal Input Sequences</a> (Thanks <a href="http://www.grayproductions.net/">James</a>)</li>
</ul>
<p>Let&#8217;s now assume that we&#8217;re smart and we can write a program which can parse keystroke properly, including handling different input escape sequences according to the OS, what can it be used for?<br />
Well:</p>
<ul>
	<li>For normal characters, just print them back to the screen (<code>get_character</code> doesn&#8217;t print anything, it &#8220;steals&#8221; the keystroke)</li>
	<li>For special characters, do something nice!</li>
</ul>
<p>We could setup <span class="caps">TAB</span> to auto-complete the current word according to an array of matches, or bind the up arrow to load the last line typed in by the user, for example, that&#8217;s basically something Readline does, right?</p>
<h2>RawLine: how it works and what it does</h2>
<p>I created a small project on RubyForge called <a href="http://rubyforge.org/projects/rawline/">RawLine</a> (not to be confused with RubyInline, a completely different thing altogether, sorry about that) to play around with the possibilities offered by the <code>get_character</code> method. The library is just a preview of things which can be done, but it&#8217;s already usable, provided that you&#8217;re brave enough to try it out, that is.</p>
<p>The basic idea behind RawLine is to be able to parse keystrokes properly on different platforms and re-bind them to a set of predefined, cross-platform actions or a user-defined code block.</p>
<h3>Basic line-editing operations</h3>
<p>The first challenge was to re-invent the wheel, i.e. re-bind keystrokes to their typical actions: a left arrow moves the cursor left, a backspace deletes the character at the left of the cursor and so on. Yes, because <code>get_characters</code> gives you the right character codes at the price of <em>cancelling their normal effects</em>, which is a great thing, as you&#8217;ll soon find out.</p>
<p>Printing a character on the screen was one of the easiest tasks (at first). <code>IO#putc</code> does the job pretty well: it prints a character out.<br />
What about moving left? Easy: print a non-descructive backspace (\b) and hope it is really not destructive. I did some tests and it seems to do as it&#8217;s told and move the cursor back by one position.</p>
<p>Moving right was a little trickier: the easiest thing I found was to re-print the character under the cursor, which will then move the cursor forward (as naive as it may seem, it does the job!). If there&#8217;s nothing under the cursor, then we must be at the end of the line and it shouldn&#8217;t move anywhere, so there we go.</p>
<p>What if I move left a bit and then start typing normal characters? Well, everything is rewritten of course: this will be our &#8220;character replace mode&#8221;. Unfortunately users don&#8217;t like this behavior that much, so what I did was this:</p>
<ol>
	<li>Copy all characters from the one at the left of the cursor till the end of the line</li>
	<li>Print the character to be inserted</li>
	<li>Re-print the previously-copied characters</li>
	<li>Move the cursor back at the right place</li>
</ol>
<p>Again, a primitive solution which works seamlessly on all platforms, and yes, it&#8217;s fast enough that you don&#8217;t notice the difference.</p>
<p>As you may have guessed, this of course means that I always had to keep track of:</p>
<ul>
	<li>The cursor position within the line</li>
	<li>The text currently printed to the screen</li>
</ul>
<p>Backspace and delete were implemented in a similar way, you can figure it out yourself or look at the source code: I won&#8217;t bore you any further!</p>
<h3>History management</h3>
<p>The next step was to implement a history for both the characters inputted by the user (to allow undoing and redoing operations) and for the whole lines. This was just an ordinary programming exercise: a simple buffer with some extra controls here and there, nothing too scary.</p>
<p>So every &#8220;modification&#8221; to the current line being typed is saved in a line history buffer and all the lines entered are saved in another history buffer. All is left is to allow users to navigate through these buffers back and forth. <br />
Nothing impossible: all I had to do was keeping track of the current element of the history being retrieved and then overwrite the current line with a new line stored in the buffer? How&#8217;s this line overwriting done? Same old:</p>
<ol>
	<li>Move the cursor to the beginnig of the line</li>
	<li>Print X spaces, where X is the line length, so that the characters are no longer displayed in the console</li>
	<li>Move the cursor back to the beginning of the line</li>
	<li>Print the new line.</li>
</ol>
<p>Easy and naive, as usual. But again, it works well enough.</p>
<h3>Word completion</h3>
<p>The other challange was word completion. The current implementation can be summarized as follows:</p>
<ul>
	<li>If <span class="caps">TAB</span> (or another character, if you wish) is pressed, call a user-defined <code>completion_proc</code> method which returns an array and show the first element of the array (in this case I actually used a cyclic RawLine::HistoryBuffer, not an array)</li>
	<li>If the user presses <span class="caps">TAB</span> again, show another match, and so <em>ad infinitum</em> if the user keeps pressing <span class="caps">TAB</span>.</li>
	<li>If the user presses another key, accept the default completion and move on.</li>
</ul>
<p>Obviously this means that:</p>
<ul>
	<li>RawLine has to keep track of the current &#8220;word&#8221;. A word is everything separated by a user defined <code>word_separator</code>, which can obviously modified at runtime, with care.</li>
	<li>Regarding the <code>completion_proc</code>, typically you may want to return only the elements matching the word which is currently being written, so that&#8217;s given as default parameter for your proc. Exactly like with ReadLine, the only difference is that you can access other things like <em>the whole line</em> and <em>the whole history</em> in real time, which can be really handy at times!</li>
</ul>
<p>Here&#8217;s a simple example:</p>
<div class='ruby'><pre><code>editor.completion_proc = lambda do |word|
	if word
		['select', 'update', 'delete', 'debug', 'destroy'].find_all	{ |e| e.match(/^#{Regexp.escape(word)}/) }
	end
end</code></pre></div><h3>Custom key bindings</h3>
<p>All these pretty things are obviously bound to some keystrokes. If the key corresponds to only one code, everything is fine, but because special keys typically aren&#8217;t so it was necessary to implement a mechanism to track an escape key (e.g. 0xE0 and 0 on Windows and \e on Linux) and listen to further characters, in case a known sequence is found. Anyhow, the final result of the method used for character binding is the following:</p>
<p><code>bind(key, &amp;block)</code></p>
<p>Where key can be:</p>
<ul>
	<li>A <code>Fixnum</code> corresponding to a single character code</li>
	<li>An <code>Array</code> of one or more character codes</li>
	<li>A <code>String</code> corresponding to an escape sequence</li>
	<li>A <code>Symbol</code> corresponding to a known escape sequence or key</li>
	<li>A <code>Hash</code> to define a new key or escape sequences</li>
</ul>
<p>So, in the end you can do things like this:</p>
<div class='ruby'><pre><code>editor.bind(:left_arrow) { editor.move_left }
editor.bind("\etest") { editor.overwrite_line("Test!!") }
editor.bind(?\C-z) { editor.undo }
editor.bind([24]) { exit }</code></pre></div><p>Which, for Rubyists, it&#8217;s far sexier and more flexible than editing an .inputrc file.</p>
<h3>How do I use it, anyway?</h3>
<p>A code example is better than a thousand words, right? So here you are:</p>
<div class='ruby'><pre><code>#!/usr/local/bin/ruby -w

require 'rubygems'
require 'rawline'

puts "*** Inline Editor Test Shell ***"
puts " * Press CTRL+X to exit"
puts " * Press CTRL+C to clear command history"
puts " * Press CTRL+D for line-related information"
puts " * Press CTRL+E to view command history"

editor = RawLine::Editor.new

editor.bind(:ctrl_c) { editor.clear_history }
editor.bind(:ctrl_d) { editor.debug_line }
editor.bind(:ctrl_e) { editor.show_history }
editor.bind(:ctrl_x) { puts; puts "Exiting..."; exit }

editor.completion_proc = lambda do |word|
	if word
		['select', 'update', 'delete', 'debug', 'destroy'].find_all	{ |e| e.match(/^#{Regexp.escape(word)}/) }
	end
end

loop do
	puts "You typed: [#{editor.read("=&gt; ").chomp!}]"
end</code></pre></div><p>This example can be found in examples/rawline_shell.rb within the RawLine source code or gem package.</p>
<h2>Current status and availability</h2>
<p>I currently <a href="http://rubyforge.org/forum/forum.php?forum_id=22543">released</a> RawLine 0.1.0 on <a href="http://rubyforge.org/projects/rawline">SourceForge</a>, and it can be installed via:</p>
<p><code>gem install -r rawline</code></p>
<p>The RDoc documentation is available <a href="http://rawline.rubyforge.org/">here</a>.</p>
<p>Feel free to try it out. First of all try the <code>rawline_shell.rb</code> example, and see if it works on your machine. If it doesn&#8217;t than maybe you try re-binding some keys (use <code>key_tester.rb</code> to &#8220;reverse-engineer&#8221; your terminal&#8217;s input escape sequences), and let me know!</p>
<p>Status information and limitations:</p>
<ul>
	<li>It has been tested on Windows (XP, using the usual command prompt) and on Linux (ZenWalk, using <span class="caps">XFCE</span> Terminal).</li>
	<li>It can handle lines no longer than the maximum terminal width &#8211; 2. This is to ensure that the cursor never &#8220;falls down&#8221; to the next line.</li>
	<li>On Windows, the cursor doesn&#8217;t blink immedialy when moving left, but it moves, don&#8217;t worry.</li>
	<li>On Linux, you should really consider installing the <a href="http://raa.ruby-lang.org/project/ruby-termios/">Termios</a> library for a faster experience (otherwise <code>get_character</code> won&#8217;t parse characters correctly if you press and hold a key, and that, trust me, is a real mess!).</li>
	<li>RawLine is very far from being a complete replacement for the ReadLine library, and it is currently in alpha stage.</li>
	<li>Release 0.1.0 has been created after 2 weeks of sporadic coding during lunch breaks and week-ends.</li>
</ul>
<p>For any ideas on where to go from here, comments and feedback, just reply below or send an email to my usual email address.</p>