Computers are pesky things. There's two major types of problems that computers and software can have: when they don't do what we tell them, and what we do. The other option is for software to do things without telling us at all: an increasingly popular option, for better and worse. This is the true essence of interface design or "human computer interaction": not whether your buttons will be "lickable" or which of "Yes", "No", "Cancel", or "OK" should be the default. These are real issues, and very important ones, but the most basic issues are about designing computer behaviour and when (and then how) it will interact with human behaviour.
Almost all programs are about manipulating data. Programs that are focused on information creation and management almost all share a set of related actions for doing this. You'll find them in almost every "File" and "Edit" menu, and they are "Save", "Save As", "Undo" and "Redo". They're the simplest and most common menu items that exist, and the ones that we have become most used to using.
They're also wrong, wrong, wrong, and wrong. Read on for my take on the situation.
The whole concept of "Save" and "Save As" is a classic example of humans pandering to computers. There's a good technical, historical reason for why we currently have to "save", but I am going to argue that those days are gone.
When you save a document (I will generally be talking about documents, but they could be of many types, such as letters, or essays, or pictures, or program source code, or accounting information), the current state of the document is stored with an associated file name. If you have previously saved something of the same file name (usually the same document in an earlier state), it is now gone.
The reason we save documents is because we would like to be able to work with them in the same state at some later time, at a later time or in a different program or on a different computer. We save because we want to give a copy to someone else, either to view or to edit, or because we want to create a snapshot of the document in a particular state, before we go on to make more copies.
We also save because we have been taught that computers are inherently unstable and unreliable, and that the changes we are making will be lost unless we choose to save.
Saving is something we do because the computers don't let us do the things we really want to. I'll come back to that.
(I'm not the first person by far to say these things: here (cached) is another rant about why save is dumb)
Undo is a pretty revolutionary concept, if you're used to, say, a typewriter. It lets you take back the last action you took, and return your document to an earlier state. (This is not quite true, depending on your definition of "document" and "state", but it is close enough.) When you have done something wrong, it can be un-done! Previously, the only recourse a user had would be to manually recover from the mistake (backspace, if they're lucky, or manual re-editing) or to load a previously-saved (you did save, didn't you?) version and start again from there.
You can think of "Undo" as a special type of "Save": it's like there's always a save made of the last document state, without you having to press "Save" yourself and keep naming files "mydoc one second ago". Except it's not quite the same: the "saves" that undo gives you aren't real saves: they don't last past when you close the document, and if anything unreliable or unstable happens to the computer, they vanish into the ether.
The concept of "undo" has grown over the years to include multiple undo and also "redo", completing the idea that there is a recent history of states that can be flipped forwards and backwards. The convention is still that this undo-history is temporary (per-session).
"Undo" and its related commands are kept completely separate from "Save" and its related commands: Undo lives in the Edit menu, and Save lives in the File menu. And never the twain shall meet.
Y'all know where I'm going by now, right?
Revision control (or version control) is a way to manage the complete history of a data that is changed over time. The revision control article on Wikipedia is decent but software-development-centric; this is only to be expected, since I would say that at least 90% of revision control is used to manage software.
Here's how revision control works. Every time you save (or check in) a new version of your document, it gets stored in a repository which also remembers every other version. This is usually done efficiently by only saving the differences from one version to the next. At any time, from a single repository, it is possible to get a copy of your document as it looked at any time you checked it in.
Undo and redo are operations that work backwards and forwards along a line of document-states. One of the great advantages of revision control is that it allows for a document-states to branch: this means that a document can start the same, and then two separate versions of it can be made and kept track of individually. Changes between branches might later be merged together, or might not.
Revision control is a great idea, and great in practice, and has become completely standard practice for software development. It is not without its own drawbacks, however. First, revision control systems are almost always format-agnostic, and only work well with text. This is good and bad: it means that as long as your program's output is text, you can use the same system to manage data from many programs. This means that text files, XML-formatted data (including documents saved in the Open Document Format), web pages and so on can be stored efficiently. On the other hand, it means that types of data that can't be stored easily as text (eg. Photoshop files) can't be managed at all. Additionally, the "diffs" between one version and the next are completely procedural: they will show you what text was changed, and how it can be changed back, but not what that means in the context of a specific program, or how the change was made in the first place.
One of the big problems with revision control is that it is since as a niche product: something that is only necessary to software developers and Wiki-makers. It seems that everyone else gets by with some combination of internal change-tracking (ala Word) or saving versions as files with different names.
OK. I'm almost there. Now, for a quick detour.
There's another problem with most document-editing software. Most of the time, we use the same document format for editing and for display. Whether we want someone to read or edit our document, when we give them a Word document we give them the ability to do both. Worse, document formats often contain a lot of information that is not necessary for viewing and shouldn't be available to anyone but the author(s). One classic is example is the "track changes" feature in Microsoft Word, which allows the changes made to the document to be highlighted, often along with accompanying notes. This has caused various public embarrassments, where track changes showed differences of opinion in the UK, put words into Mark Latham's mouth in Australia, cause some red faces at Microsoft, and so on and so on.
For almost every type of document, there are always two types of users: those who are producing it, and those who are consuming it. When a document is printed, the hard copy is (mostly) "view-only". These days, when sending electronic copies is the norm, people often send more than they think. Formats like Adobe's PDF make the separation much clearer: PDFs contain mostly static, locked content, for which no extra information (eg. prior changes) is available.
We have to save because otherwise we lose our changes. We need undo and redo to step backwards and forwards through a few of our most recent changes. We need "save as" to make multiple versions or backups of a document. We accidentally send to much information because editing and viewing use the same format.
We can do away with all of these actions and problems if we have automatic and ubiquitous program-aware revision control.
Automatic means that that the changes are saved by themselves, constantly. Just imagine it as an infinite undo (infinite undo has existed for a long time in the editor sam, by the way) that goes back between saves.
Ubiquitous means that it's there all the time, for every program. This is only feasible if you either force developers to conform to a standard, or provide easy standard data-management libraries for them. The former will never happen; the latter has already been done by Apple with Core Data, which also can save data in XML and would be a perfect basis for a system like this.
Program-aware means that the changes saved are saved in terms of actions in the program, not just text-string differences in the saved files (which is fine as a fallback). I'll talk about this a bit more in the case study below.
If you do nothing, and type and edit and type and edit, all of your changes will be saved, regardless. If you press "Save", you can give a nice name to the current snapshot for future reference, but that's all it does, nothing more.
One advantage of explicitly keeping a repository rather than a "document" is that people are much less likely to share it all. All that is needed is an "export" command that takes one particular snapshot of the document and exports it into some other format, such as PDF, to be shared and viewed etc. It's also possible to explicitly manage how much of the repository you'd like to send if, for example, you would like someone else to be able to make some changes.
So this type of automatic revision control would get you the sane, stable, and safe version of what we have learned to tolerate with Save, Save As, Undo, and so on. It also allows you to do a whole lot more.
Revision control lets you keep different branches of changes, and compare the differences between them. Most ways of viewing the differences are pretty techy: they highlight the changes in various ways and show how the branches could be merged, etc. In consumer (ie. non-programming) applications, there are many more possibilities for displaying and using versioning information.
Consider a word-processing example. You write some text, format it, then decide you hate it and delete it all. You write another paragraph, then realise that you do want the other paragraph, after all. You could have a history browser panel, showing miniature versions of what this page looked like before - just scroll back a few pages until you get your paragraph, highlight the text (with formatting - it is rendered as a document, not as plain text), and drag-and-drop it into your current version. Simple!
The same thing can be done with series of commands: since the commands themselves are stored in the history (bold, move down, highlight line, change font, etc), these two can be retrieved from previous work and used again.
Most software has the usual undo/redo functionality. One of the few that offers a significantly different type of functionality is Adobe Photoshop. It also has a few of the features that I've mentioned in this article.
Photoshop manages document history a lot more explicitly than most other software: there is a panel dedicated to showing all of the available undo-steps, together with the a description of exactly what was done at each step. Additionally, you can mark a "snapshot" at any time, and you can return to that state by clicking on the snapshot. If you click back to a snapshot, you can then go back again to the current state, so it is not just a simple reversion. Snapshots are like labelled versions, but it's not full branching: the connection between snapshots is lost. Also, undo only works within a session, and is limited in number and disk space.
Using the "history brush", information from a snapshot (how the image looked at some previous time) can be used to paint the current image. This is like copying-and-pasting from previous versions, except for pictures instead of words.
The chances are, when you write something, you don't want it to just disappear. Gmail knows that you probably don't want to delete emails (as long as you can find them again). Word takes automatic backups in an admission that computers are inherently flaky. Why not go the whole way, and just make sure that everything you ever do to a document is saved? As long as there is a clear line between what you keep (your whole history) and what you show people (only the version/s that you want to show, in the formats that you want), there is no problem with keeping the extra information. Creating a distinction between the creation process (save everything) and the finished document is necessary given the confusion and problems caused by the current mish-mash, such as Word's built-in "track changes" feature.
The only other argument against saving everything is storage space. There is a tradeoff between the storage efficiency of the changelog and the efficiency of reconstituting a particular state: with text, it's easy to store everything, but for other media like images, sound, or video, it might be necessary to store lists of commands and then re-apply them from earlier master sources. The concept doesn't change, though.
Next time you're doing the save-edit-undo-redo dance, have a think about it. Why the hell are you pressing Save? Why is the default behaviour, "Go ahead, lose all my changes, you bastard computer? You're right, obviously I didn't want it!"
Computers can do better, and revision control is one way they can be improved.
Posted by Casey at November 25, 2005 05:41 AMSounds great! I have a caveat and a suggestion of how to take it further.
The caveat is where your operations are not just editing a file, but have a side-effect on other computers or equipment. The Undo command can't unemail a document, unpurchase shares on eTrade, unformat your camera's memory card, etc.
The suggestion to take it further is to take the ubiquitious revision control and make it (somewhat) centralised. The data need not be stored in one location, but a reference to the undoable command that just occurred should be stored in one place.
The reason I propose this is to enable a computer-wide "rewind and replay" of what you did - a black-box recorder for the entire device.
When I wonder on Monday what happened to the missing file I was working on all of Friday, I want to be able to backtrack and see exactly what I did with it, and where exactly I put it.
Posted by: OddThinking at November 26, 2005 12:56 PMHey Casey,
Nice post. You make some good points. The truth is that managing documents and the myriad of changes we make is a pain in the ass.
I wonder how one could actually make this work, though. What sort of actions would cause a branching of the document? Deleting words or a paragraph? If the user wants to revert to a previous version of the document and there are conflicts between the branches, how is the user going to cope? As a developer, I have enough trouble merging conflicts between branches in CVS and Subversion. Perhaps with a kick-ass interface this could be achieved (and if so, can we get it for merging code as well?).
How often would a "commit" occur with the functionality you describe? For instance, if a user starts typing a word, deletes a few characters and types a few more, is every single key press recorded and written to disk as a separate "revision"?
Anyway, sounds like a great idea. Maybe there's an opportunity for someone to do an application like this, which can read Word documents, convert into OpenDocument format and maintain as a revision control repository. Is this something Google is working on? ;)
Posted by: Jeremy Higgs at November 26, 2005 06:25 PM
Julian,
You're jumping the gun again, damn you! I'll get to the black box soon. You are right, though: every action you take on your computer is information. It would be nice to be able to retrieve it and use it again, but what else could be done with it?
All of this is tied up with the notion of "state", which has become less implicit and more explicit over the years. Web browsers still struggle with state (Opera is the best), operating systems are getting better at hibernating and fast-booting (compressing and recovering state), but the problem with all of these types of state management is that they have limited reach: live network connections, or removable media, etc.
Jeremy, version control is a pain in the ass... but does it need to be? It is for us developers because of the way we use it - to do complex merging and so forth. I picture the consumer version of version control as more of a "history browser" that can show you differences but doesn't even need to be able to do merging. Snapshots and the ability to reconstitute any prior state, plus the sequence of semantics-bearing actions that brought it about, should be plenty to make people very happy.
With regard to the frequency of commits, it is again action-based. This means (shock!) that programs would need to understand what actions could take place within it, and "atomise" them. This is already the case in the vi editor, for example, since it is completely command-driven, and for Photoshop, which already labels actions. Most non-text-entry commands are already atomic. Something like Word is scriptable; an action could be defined as the largest single VBA command that could produce the change (implementation left as an exercise to the reader!).
Posted by: Casey at November 27, 2005 02:39 AMCasey,
Agree entirely, although I suspect that the transition to the document-as-repository is going to be too much of a usability problem to solve. People are so used to emailing Word documents about the place. Now you're telling them not to do that? You can bet that initially there will be lots of embarrassment caused by people mailing repositories containing all of their bad typing, false starts, accidental copy-n-pastes, etc etc.
I'm just not sure about the widespread social implications of the change, given that the current save/undo model is mostly accepted.
Alastair, the application that I haven't mentioned so far is Microsoft's OneNote, the uber-notepad. It does some very un-Office things: it saves constantly (there is no save command), and there are no files. You can copy parts of pages out, but from memory there's not an easy way to export (eg. to text) a lot of notes. OneNote's great, surprising for MS but with a few dumb proprietary quirks that smell a lot like Redmond. If Microsoft is willing to ship a product that does auto-save and hides individual files, I think that it's not such a jump to version control.
The easy way to stop people sending whole repositories to each other is just to change the way the applications interact with the filesystem, just like OneNote. You can't send your whole "notepad" at once from OneNote, it's just not an option. So in the same vein, version-controlled OpenOffice wouldn't prominently display anything about the doc-repository: it would have easy-to-access Export commands and the like, but it would have a new menu (or whatever) that dealt explicitly and separately with the repository and so on. Users would feel just fine as long as they could do what they wanted, and they would even find out that it was actually what they wanted, too!
Posted by: Casey at November 27, 2005 08:00 AMNow you've done it, Casey: you've written about a pet topic of mine. Now I've no choice but to ramble. You have been warned.
Points to ponder:
- cross file linkage
I'm sure there are very few authors who can get away with editing just one file at a time. When I'm wearing my document-editing hat, I know I often find myself editing some Visio (a curse on both its houses) diagrams, a spreadsheet and a word doc at the same time. If I were in a more graphically-oriented field, I'd also be editing icons and images. All of these edits are made in concert, in order for the document to be updated: eg the spreadsheet is an appendix, either embedded or weakly linked. The version control system should make it easy to find which spreadsheet corresponds to which word document, etc.
- multi-file export (eg "Export as website")
If everything is getting edited from the one Uber-editor (as Word at one stage was becoming. Remember OLE?), then there needs to be a way of generating the multiple distinct files required for display on a website, or other media I haven't thought of. Simply separating the stylesheet from the text should be straight forward in Word, but I've never seen it done.
- group editing
Many of the people who are emailing around word docs are doing so because they are expecting others to make changes (using track changes) or comments (using word's commenting features) and send them back for merging. Support for this in any new system would be ... tricky. Smalltalk's image file (and related changes file) could be used for this sort of sharing, but allowing users to copy these around is precisely what we don't want them to do when sending files out to their readers.
- backup and restore
Despite the wondrous improvements in storage capacity and software quality (hell, Windows is only blue screening on me once a month or so now. Remember Windows 95?), there's still a long way to go before file storage will be 100% reliable and corruption free. Any new system would need to support easy backup and not-too-difficult restore operations. As a side issue it will also need to detect as soon as possible when its store has been corrupted, so users don't get any nasty shocks later on.
- language and library support.
Til very recently, support for this style of environment has been a total train wreck. The number of hoops you need to jump through to save a file (serialization, ORM, or worse) meant that many programs made damn sure it was something that the user initiated, or at least didn't happen too often (Word's autosave still isn't very good, as it can lose much more than a few keystrokes of work). Improving library support so that it's easy for deltas to be appended to files (making each write as simple as possible, and only involve the data that changed), and each delta includes a corresponding UI action (so we can line up the Undo/Redo stacks in the UI with the file contents) would go a long way to making this more common.
- different editor tools
Subtly different to the different people case, we're now talking about trying to support using two different programs to edit the one file. Should they be able to browse the verison histories created by each other? If so, then there's a need for a standard way of describing a UI action associated with a change, and possibly even of translating that action into human readable text. Do I have to give up on this foolish notion of having different tools for different aspects of a task (For code, that's editing, lint checking, formatting. For images, that's photo retouching, free-hand drawing, compositing and captioning. For every document apart from a memo, that's editing, grammar, spelling and style checking, layout, review, rinse, repeat) and try to find another Uber-editor? I know that this is the way coding has gone, with the IDE now controlling all aspects of the code.
Possible helpers for this may come from thinking of edits as transactions and changesets.
When I say transactions, I mean ACID-like transactions, just like in databases. ACID-like in that before you share a change, they're only visible to you (isolation), they're easily persisted (durability), they are naturally consistent (consistency), but because we're allowing merges, they're not necessarily atomic. Also, because I'm not assuming that you want to share each and every change with the whole world, isolation in this context doesn't follow the strict database definition (which is essentially private while changing, world readable when commited). Figuring out how transactions can really work a) in distributed environments and b) while irreversable actions are going on (e.g. scheduling trains) is still AFAICT an open issue.
Changesets are things that all good version control systems have as a fundamental concept (that's right, I'm saying CVS is not a good version control system). Subversion even goes so far as to make the changeset the only form of version control by saying that every file gets the same version number for each commit. Having changesets makes finding corresponding changes across multiple files trivial: look up the changeset, then see what changed in that set.
Regarding when a branch would occur, I'd say it should happen after a user presses undo several times, and then starts typing again. This way, they can still access the changes they've undone: they're in the branch they just left.
Regarding presenting the user with these branches, so they can see and merge their changes, I don't think you can just say "it's the applications problem to figure it out". Some good support for merging in the VCS is vital. Providing standard hooks where an app can choose how to display a merge (with some of the choices being as text, as a tree (like some XML viewers do) or as rendered content), and to prompt the user to guide a merge (as automatic as possible, make everything manual, let me choose later). Photoshop's novel approach to version browsing may be the "right" way forward here.
I think the social implications are as widespread as you fear, Alastair: under this system, files either become an interchange artifact (generated on export), or simply become buried under the hood of the tools. I think that this might be a good thing. The file metaphor has had its 30+ years in the sun. Perhaps it's time for it to move on? But what metaphor can we use to explain this new system? Elephant memory?
Posted by: Richard at November 27, 2005 12:03 PM
Richard, excellent points and well made. There's enough here for plenty of articles, but for now:
- cross-file linkage: I think that this is an area that having better semantics will help. Presumably any automatic VCS will be able to label a document-state uniquely (if only by name and timestamp); as long as a copy-paste takes note of the source's information, there's no problem with linkage. This reminds me again of OneNote, and its best feature: when pasting text from a web page, it automatically annotates the pasted text with the original URL!
- multi-file export: I see the move to a repository as an advantage, here. Once the repository is conceived of as "more than the document", it is easier to think of creating multiple views (different states, or in different formats, or multiple files, etc) from the same data. Since "save" has been replaced in our heads with "export", it still makes sense.
- groupwork / collaboration: I envisaged sharing based upon snapshots or limited access to history. You share a snapshot, a new branch gets made, you patch it back in if you get it back.
- backup and restore: A very, very real issue. Hard drives still die. Repositories are no worse than files, and may be easier to backup well; their only disadvantage is their relatively larger size, but I will wave that away dismissively and claim that hardware improvements will save us all!
- languages and libraries: I think Apple's CoreData is pushing in this direction, but I say that as someone who has only skimmed the developer's notes and has never written anything for a Mac. XML, standardised libraries for data management, etc. They're doing it for searchability, but revision control could come along for the ride.
Hmm, perhaps an article will be necessary after all... regarding your last point: files aren't quite dead, but for version control to go mainstream we really need a better analogy for it - think on!
Posted by: Casey at November 27, 2005 12:57 PMCasey writes: "With regard to the frequency of commits, it is again action-based."
I once wrote a paper on this issue, a long time ago. I've was thinking recently about digging it up and using it as a blog article.
In brief, as you hinted, some editing environments, including modal editors like vi and Photoshop, have natural transaction units.
Modeless editors, such as Word or Emacs, do not have such clear distinctions, and various heuristics need to be applied. This can already be seen in action.
Try typing a sentence into Word, and then hit undo. The whole sentence disappears.
Try it again without capitalising the first letter. This time it removes all but the first word. A second undo decapitalises the first word. A third undo removes it.
I am not saying this is ideal behaviour, but just that it demonstrates that automatic detection of commit points is already known to those skilled in the art.
Posted by: OddThinking at November 27, 2005 04:52 PM