-----
title: "RawLine - a 100% Ruby solution for console inline editing"
content-type: article
timestamp: 1205128740
tags: "ruby|programming|opensource|rawline"
-----
<p>One of the many things I like about Ruby is its cross-platform nature: as a general rule, Ruby code runs on
	everything which supports Ruby, regardless of its architecture and platform (yes, there are quite a few exceptions,
	but let's accept this generalization for now).</p>
<p>More specifically, I liked the fact that I could use the <a
		href="http://tiswww.case.edu/php/chet/readline/rltop.html"><span class="caps">GNU</span> Readline library</a>
	with Ruby seamlessly on both Windows and Linux.<br />
	Readline offers quite a lot of features which are useful for those people like me who enjoy creating command-line
	scripts, in a nutshell, it provides:</p>
<ul>
	<li>File/Word completion</li>
	<li>History support</li>
	<li>Custom key bindings which can be modified via .inputrc</li>
	<li>Emacs and Vi edit modes</li>
</ul>
<p>Basically it makes your command-line interface fast and powerful, and that's not an overstatement. Ruby's own <span
		class="caps">IRB</span> can be enhanced by enabling readline and completion, and it works great &#8212; at least
	on *nix systems.</p>
<p>For some weird reason, some people had problems with Readline on Windows: in particular, things get nasty when you
	start editing long lines. Text gets garbled, the cursor goes up one or two lines and doesn't come back, and other
	similar leprechaun's tricks, which are not that funny after a while.</p>
<p>Apparently there's no alternative to Readline in the Ruby world. If you wan't tab completion that's it, you're stuck.
	Would it be difficult to implement <em>some</em> of Readline functionality natively in Ruby? Maybe, but the problem
	is that for some reason the Ruby Standard Library doesn't have low level methods to operate on keystrokes&#8230;</p>
<p>&#8230;but luckily, the <a href="http://highline.rubyforge.org/">HighLine</a> gem does! James Edward Gray II keeps
	pointing out here and here that HighLine's own <code>get_character</code> method does just that: it returns the
	corresponding character code(s) right when a key is pressed, unlike <code>IO#gets()</code> which waits for the user
	to press <span class="caps">ENTER</span>.</p>
<p>Believe it or not, that tiny method can do wonders&#8230;h2. Reverse-engineering escape codes</p>
<p>So here's a little script which uses <code>get_character()</code> in an endless loop, diligently printing the
	character codes corresponding to a keystroke:</p>
<div class='ruby'>
	<pre><code>#!/usr/local/bin/ruby -w

require 'rubygems'
require 'highline/system_extensions'

include HighLine::SystemExtensions

puts "Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit)."

loop do

	print "=&gt; "
	char = get_character
	case char
	when ?\C-x: print "Exiting..."; exit;
	else puts "#{char.chr} [#{char}] (hex: #{char.to_s(16)})";
	end
	
end</code></pre>
</div>
<p>A pretty harmless little thing. Try to run it and press some keys, and see what you get:</p>
<div style="font-family: Monospace">
	<p>Press a key to view the corresponding <span class="caps">ASCII</span> code(s) (or <span
			class="caps">CTRL</span>-X to exit).</p>
	<p>=&gt; a <sup class="footnote" id="fnr96"><a href="#fn96">96</a></sup> (hex: 61)</p>
	<p>=&gt; 1 <sup class="footnote" id="fnr49"><a href="#fn49">49</a></sup> (hex: 31)</p>
	<p>=&gt; Q <sup class="footnote" id="fnr81"><a href="#fn81">81</a></sup> (hex: 51)</p>
	<p>=&gt; &alpha; <sup class="footnote" id="fnr224"><a href="#fn224">224</a></sup> (hex: e0)</p>
	<p>=&gt; K <sup class="footnote" id="fnr75"><a href="#fn75">75</a></sup> (hex: 4b)</p>
</div>
<p>Hang on, what are the last two codes? <em>A left arrow key on Windows</em>, apparently.</p>
<p><strong>Welcome to the wonderful world of input escape sequences!</strong></p>
<p>To cut a long story short, both Windows and *nix system &#8220;terminals&#8221; translate special keystrokes into
	sequences of two or more codes. This applies to things like <span class="caps">DEL</span>, <span
		class="caps">INSERT</span>, arrows, etc. etc.<br />
	For some ideas, check out:</p>
<ul>
	<li><a href="http://www.microsoft.com/whdc/device/input/Scancode.mspx">Windows Scancodes</a> (Thanks <a
			href="http://64.223.189.234/node/92">Huff</a>)</li>
	<li><a href="http://www.connectrf.com/Documents/vt220.html">VT220 Terminal Input Sequences</a> (Thanks <a
			href="http://www.grayproductions.net/">James</a>)</li>
</ul>
<p>Let's now assume that we're smart and we can write a program which can parse keystroke properly, including handling
	different input escape sequences according to the OS, what can it be used for?<br />
	Well:</p>
<ul>
	<li>For normal characters, just print them back to the screen (<code>get_character</code> doesn't print anything, it
		&#8220;steals&#8221; the keystroke)</li>
	<li>For special characters, do something nice!</li>
</ul>
<p>We could setup <span class="caps">TAB</span> to auto-complete the current word according to an array of matches, or
	bind the up arrow to load the last line typed in by the user, for example, that's basically something Readline does,
	right?</p>
<h2>RawLine: how it works and what it does</h2>
<p>I created a small project on RubyForge called <a href="http://rubyforge.org/projects/rawline/">RawLine</a> (not to be
	confused with RubyInline, a completely different thing altogether, sorry about that) to play around with the
	possibilities offered by the <code>get_character</code> method. The library is just a preview of things which can be
	done, but it's already usable, provided that you're brave enough to try it out, that is.</p>
<p>The basic idea behind RawLine is to be able to parse keystrokes properly on different platforms and re-bind them to a
	set of predefined, cross-platform actions or a user-defined code block.</p>
<h3>Basic line-editing operations</h3>
<p>The first challenge was to re-invent the wheel, i.e. re-bind keystrokes to their typical actions: a left arrow moves
	the cursor left, a backspace deletes the character at the left of the cursor and so on. Yes, because
	<code>get_characters</code> gives you the right character codes at the price of <em>cancelling their normal
		effects</em>, which is a great thing, as you'll soon find out.</p>
<p>Printing a character on the screen was one of the easiest tasks (at first). <code>IO#putc</code> does the job pretty
	well: it prints a character out.<br />
	What about moving left? Easy: print a non-descructive backspace (\b) and hope it is really not destructive. I did
	some tests and it seems to do as it's told and move the cursor back by one position.</p>
<p>Moving right was a little trickier: the easiest thing I found was to re-print the character under the cursor, which
	will then move the cursor forward (as naive as it may seem, it does the job!). If there's nothing under the cursor,
	then we must be at the end of the line and it shouldn't move anywhere, so there we go.</p>
<p>What if I move left a bit and then start typing normal characters? Well, everything is rewritten of course: this will
	be our &#8220;character replace mode&#8221;. Unfortunately users don't like this behavior that much, so what I did
	was this:</p>
<ol>
	<li>Copy all characters from the one at the left of the cursor till the end of the line</li>
	<li>Print the character to be inserted</li>
	<li>Re-print the previously-copied characters</li>
	<li>Move the cursor back at the right place</li>
</ol>
<p>Again, a primitive solution which works seamlessly on all platforms, and yes, it's fast enough that you don't notice
	the difference.</p>
<p>As you may have guessed, this of course means that I always had to keep track of:</p>
<ul>
	<li>The cursor position within the line</li>
	<li>The text currently printed to the screen</li>
</ul>
<p>Backspace and delete were implemented in a similar way, you can figure it out yourself or look at the source code: I
	won't bore you any further!</p>
<h3>History management</h3>
<p>The next step was to implement a history for both the characters inputted by the user (to allow undoing and redoing
	operations) and for the whole lines. This was just an ordinary programming exercise: a simple buffer with some extra
	controls here and there, nothing too scary.</p>
<p>So every &#8220;modification&#8221; to the current line being typed is saved in a line history buffer and all the
	lines entered are saved in another history buffer. All is left is to allow users to navigate through these buffers
	back and forth. <br />
	Nothing impossible: all I had to do was keeping track of the current element of the history being retrieved and then
	overwrite the current line with a new line stored in the buffer? How's this line overwriting done? Same old:</p>
<ol>
	<li>Move the cursor to the beginnig of the line</li>
	<li>Print X spaces, where X is the line length, so that the characters are no longer displayed in the console</li>
	<li>Move the cursor back to the beginning of the line</li>
	<li>Print the new line.</li>
</ol>
<p>Easy and naive, as usual. But again, it works well enough.</p>
<h3>Word completion</h3>
<p>The other challange was word completion. The current implementation can be summarized as follows:</p>
<ul>
	<li>If <span class="caps">TAB</span> (or another character, if you wish) is pressed, call a user-defined
		<code>completion_proc</code> method which returns an array and show the first element of the array (in this case
		I actually used a cyclic RawLine::HistoryBuffer, not an array)</li>
	<li>If the user presses <span class="caps">TAB</span> again, show another match, and so <em>ad infinitum</em> if the
		user keeps pressing <span class="caps">TAB</span>.</li>
	<li>If the user presses another key, accept the default completion and move on.</li>
</ul>
<p>Obviously this means that:</p>
<ul>
	<li>RawLine has to keep track of the current &#8220;word&#8221;. A word is everything separated by a user defined
		<code>word_separator</code>, which can obviously modified at runtime, with care.</li>
	<li>Regarding the <code>completion_proc</code>, typically you may want to return only the elements matching the word
		which is currently being written, so that's given as default parameter for your proc. Exactly like with
		ReadLine, the only difference is that you can access other things like <em>the whole line</em> and <em>the whole
			history</em> in real time, which can be really handy at times!</li>
</ul>
<p>Here's a simple example:</p>
<div class='ruby'>
	<pre><code>editor.completion_proc = lambda do |word|
	if word
		['select', 'update', 'delete', 'debug', 'destroy'].find_all	{ |e| e.match(/^#{Regexp.escape(word)}/) }
	end
end</code></pre>
</div>
<h3>Custom key bindings</h3>
<p>All these pretty things are obviously bound to some keystrokes. If the key corresponds to only one code, everything
	is fine, but because special keys typically aren't so it was necessary to implement a mechanism to track an escape
	key (e.g. 0xE0 and 0 on Windows and \e on Linux) and listen to further characters, in case a known sequence is
	found. Anyhow, the final result of the method used for character binding is the following:</p>
<p><code>bind(key, &amp;block)</code></p>
<p>Where key can be:</p>
<ul>
	<li>A <code>Fixnum</code> corresponding to a single character code</li>
	<li>An <code>Array</code> of one or more character codes</li>
	<li>A <code>String</code> corresponding to an escape sequence</li>
	<li>A <code>Symbol</code> corresponding to a known escape sequence or key</li>
	<li>A <code>Hash</code> to define a new key or escape sequences</li>
</ul>
<p>So, in the end you can do things like this:</p>
<div class='ruby'>
	<pre><code>editor.bind(:left_arrow) { editor.move_left }
editor.bind("\etest") { editor.overwrite_line("Test!!") }
editor.bind(?\C-z) { editor.undo }
editor.bind([24]) { exit }</code></pre>
</div>
<p>Which, for Rubyists, it's far sexier and more flexible than editing an .inputrc file.</p>
<h3>How do I use it, anyway?</h3>
<p>A code example is better than a thousand words, right? So here you are:</p>
<div class='ruby'>
	<pre><code>#!/usr/local/bin/ruby -w

require 'rubygems'
require 'rawline'

puts "*** Inline Editor Test Shell ***"
puts " * Press CTRL+X to exit"
puts " * Press CTRL+C to clear command history"
puts " * Press CTRL+D for line-related information"
puts " * Press CTRL+E to view command history"

editor = RawLine::Editor.new

editor.bind(:ctrl_c) { editor.clear_history }
editor.bind(:ctrl_d) { editor.debug_line }
editor.bind(:ctrl_e) { editor.show_history }
editor.bind(:ctrl_x) { puts; puts "Exiting..."; exit }

editor.completion_proc = lambda do |word|
	if word
		['select', 'update', 'delete', 'debug', 'destroy'].find_all	{ |e| e.match(/^#{Regexp.escape(word)}/) }
	end
end

loop do
	puts "You typed: [#{editor.read("=&gt; ").chomp!}]"
end</code></pre>
</div>
<p>This example can be found in examples/rawline_shell.rb within the RawLine source code or gem package.</p>
<h2>Current status and availability</h2>
<p>I currently <a href="http://rubyforge.org/forum/forum.php?forum_id=22543">released</a> RawLine 0.1.0 on <a
		href="http://rubyforge.org/projects/rawline">SourceForge</a>, and it can be installed via:</p>
<p><code>gem install -r rawline</code></p>
<p>The RDoc documentation is available <a href="http://rawline.rubyforge.org/">here</a>.</p>
<p>Feel free to try it out. First of all try the <code>rawline_shell.rb</code> example, and see if it works on your
	machine. If it doesn't than maybe you try re-binding some keys (use <code>key_tester.rb</code> to
	&#8220;reverse-engineer&#8221; your terminal's input escape sequences), and let me know!</p>
<p>Status information and limitations:</p>
<ul>
	<li>It has been tested on Windows (XP, using the usual command prompt) and on Linux (ZenWalk, using <span
			class="caps">XFCE</span> Terminal).</li>
	<li>It can handle lines no longer than the maximum terminal width &#8211; 2. This is to ensure that the cursor never
		&#8220;falls down&#8221; to the next line.</li>
	<li>On Windows, the cursor doesn't blink immedialy when moving left, but it moves, don't worry.</li>
	<li>On Linux, you should really consider installing the <a
			href="http://raa.ruby-lang.org/project/ruby-termios/">Termios</a> library for a faster experience (otherwise
		<code>get_character</code> won't parse characters correctly if you press and hold a key, and that, trust me, is
		a real mess!).</li>
	<li>RawLine is very far from being a complete replacement for the ReadLine library, and it is currently in alpha
		stage.</li>
	<li>Release 0.1.0 has been created after 2 weeks of sporadic coding during lunch breaks and week-ends.</li>
</ul>
<p>For any ideas on where to go from here, comments and feedback, just reply below or send an email to my usual email
	address.</p>