Language-independent programming (Part 1; the basic motivation)

I have been gradually coming to believe that one of the biggest problems with programming is that code lives in text files that are written in a particular programming language, and that programmers directly manipulate these text files.

The improved alternative that I would like to propose is for code to live in abstract trees with reverse-compilation hints that allow it to reverse-compile differently depending on context.

Let me give you a little bit of motivation for this idea.

Let's say I'm building a complicated object for an end-user to manipulate and interact with via a computer, and I want be able to improve my end users experience of interacting with this object over time. For example, let's say I've created a computer game. If I were to write a computer game, I wouldn't give my users tools to directly modify the files that their computer stores to keep track of their progress and let them quit the game and reload it later. Instead, I would give the user the ability to play the game, and I would write utilities to allow the user to save the game... i.e. to abstract from the game state sufficient information that the program I wrote could recreate the game-state that the user had reached the next time the user loads the game.

This process of saving an abstraction of what the user was interacting with and then converting that abstraction back into game state later allows the game creators a lot more freedom to update the game. Let's say a particular gun is overpowered and the game creators decide that they need to nurf that gun to make the game more fair. Since the saved game-state only has a reference to that gun rather than a full description of that gun, the programmers only need to update the game, and they don't have to update the saved file at all.

Now, switch over to programming languages. Let's say I wrote a programming language in which I had statements like 'print' or 'exec' that were a little bit different from functions, mostly with regard to their syntax but that otherwise behaved a lot like functions. Let's say that after twenty years of developing this language, I decided that having these things be statements instead of functions was a bad idea for several reasons, both in terms of the way the language internally distinguishes between functions and statements (statements cannot be overridden with an alternative implementation but functions can), and how the syntax works. It's a bit clunky to have these particular concepts be key words or to have them have different syntax rules from functions.

Now, I want to release an update to the programming language, but I really can't do it because doing so would involve requiring people who have been using the language for the past twenty years to change a whole bunch of files that use syntax that is no longer consistent with what the update would require, so instead I create a backwards-incompatible update and cease to actively update the old programming language and cause there to be a great schism between code compatible with the old way of doing things and code compatible with the new way of doing things.

It's a step forward, but it isn't a very tidy one.

It would be a lot nicer if instead of having to update these files, we could just have them reverse-compile to the appropriate syntax when we decided that the change needed to be made. (And forward compile to behaving like functions instead of behaving like statements. To satisfy both constraints we would need something that is not truly compiled, but capable of compiling in either direction.)

Now, let's consider a second point: programmer preference.

Let's say I really don't like programs to have more than four spaces worth of indentation per line, and in fact I prefer two. My coworker would rather have eight spaces worth of indentation, and he would rather have that spaces-worth of indentation be encoded in tabs than in spaces (we're working in a language that doesn't really care which one we use). So we pick one of our preferences and stick to it, but both of us are working on other projects that we code according to our own preferences, so we occasionally get confused and write code that mixes the two.

Again, it would be a lot nicer if somehow we had the code stored in a way that would cause it to be written back to us in a manner consistent with our preferences instead of a way that forced us to look at the same file of text even though we would both rather that it be formatted in different ways from each other.

For an even stronger disagreement of preferences, we use different editors. In my co-workers editor having 100 characters to a line makes a ton of sense, but in mine, it's best to keep it to 80. Again, the same thing.

We can go even further. After we start working on this project, I realize that having
3+5 * 4
represent
32
makes a whole lot more sense to me than having it represent
23
and that furthermore, I find this representation to be significantly more readable to me than
(3+5) * 4
or
(3 + 5) * 4
My co-worker is a purist who wants to keep the traditional interpretation of parentheses and thinks that that having whitespace sensitivity for that sort of thing in a programming language is a BAD idea.

So I install a module that lets me represent the code the way I want it represented, and he leaves it set up the default way.

And all of this is just the beginning of what you could begin to do differently (and better) if you found a way to store code in something abstract with guidance for how to be reverse-compile back to what the programmer wanted.

If you were going to build this system you would start by doing something that doesn't do too much in the way of producing abstractions, but you could systematically build out and maintain this sort of system much more easily than you can systematically build out and maintain the sorts of things that people currently write.

lang: Part 0, the motivation

[Another old post I didn't want to delete. It's mostly tongue-in-cheek. I don't stand by what it says.]

Let's design a programming language. Do let's!

Actually, let's not. Instead let's try to describe lang.

lang, as its name suggests, is a programming language that seeks to incorporate as little originality as possible. The goal is to avoid designing it so that no one will ever have to learn it.

The designers of Go said they would know they'd done something wrong if printf was magic. I think that that's admirable, but it's also original, so it's not what we're going for. In lang, we know we've messed up if we finish it and some piece of it seems brilliant, cute, or original.

Hmm... actually, in lang we've messed up if we have to write any official documentation. It should all be unofficial and casual. Besides, we are intending that no one will have to learn lang.

It's 2013, so lang will be object-oriented. At this point, what could possibly be less original than yet another object-oriented programming language?

So what sort of object-oriented language is lang? Obviously, it's inspired by C, really there aren't other options when we are trying to avoid being original, but explicitly typed languages are so disco-era. Retro is a form of self-expression, and we want lang to just blend in. So it shouldn't be explicitly typed, not in 2013. So let's plan, for now, to go with dynamic typing.

So now we're heading into the territory between Python and C. Do we go with elif or else if?

Going with either would be making a decision, so let's go with both. In lang, the statement else if is the same as the statement elif. Is that cute? Personally, I think it seems lazy and tacky, which is about as far from cute as you can possibly get. So it tentatively gets my stamp of approval. (Also, as someone who switches back and forth between Python and C family languages on a regular basis, I mess this up more than I should, and I always get worried that someone might be looking over my shoulder thinking, "Does this guy seriously not know how to write conditional statements in a programming language he uses pretty much every day?" So lang just won't have that problem.)

There are occasions when somebody just gets something right. Guido van Rossum got white space right, really right, except for allowing tabs. At least, he got it exactly right for functions.  I think the forced correlation between visual layout and logical layout is very nice especially given that the correlation is a matter of conventional redundancy for programming in most languages. I don't like writing redundantly when I'm programming. I've also quite fond of how it handles semicolons. Line breaks should mean something, but occasionally, there are reasons that line-breaking is stylistically undesirable -- or simply inconvenient. I mostly use semicolons in the interpreter when I'm editing a file. I type something like:

>>> reload(Module); c = Module.Class(stuff); c.dothathingimdebugging()

All in all, Python's handling of white space leaves almost nothing to be desired. I just occasionally wish that I didn't have to nest all of my sub-declarations, especially when I have a class with a function that produces classes, something I've only done once, because it made something that seemed complicated at first much easier. I forget what it was. So let's treat C-style brackets like C-style semicolons. Something like

class Class {
x = 30
y = 45

def function(self):
    if y > x:
        return 'well this is a dumb example'

}

is perfectly valid code in lang.

I also wouldn't mind at all if the colons became optional. Something primarily used like the semicolon for one liners, but that don't break anything when they are included. They simply are not strictly necessary. Getting a text editor to do smart auto-formatting might be slightly harder, but we do have a list of relevant keywords: if, for, else, elif, else if, def, class, while, try, except, switch, case, etc. (Yes, we are picking up case and switch from C.)

For the sake of consistency, let's revisit our type system. Dynamic typing is nice, but it shouldn't preempt the possibility of explicit static typing. So in lang, we permit

int x = 15

This creates x, a statically typed int. Obviously, the goal of static typing is not just to produce new compiler errors, and force you to recast your variables before passing them into functions. A better purpose might be for the static type to take the form of an inline unit test... but never mind that would be original. More compiler errors it is!

[Now, if I could remember what I was trying to parody (Was it Ruby?), I might be able to complete the post, but I don't remember that anymore well enough to maintain a consistent trail of sarcasm...]

Complicatedness

[This is another unpublished old post that I decided to edit, finish, and post instead of delete.]

Every project will rapidly build up complexity until it reaches the point where the most skilled developer working on the project routinely finds his work somewhat challenging.

Human's thrive when they are working on problems that are challenging but solvable. Read virtually any book about design from the last twenty years, and it will tell you that the flow state is real and that it matters. The flow state is the state in which people are most productive and happiest to be working. They achieve it only when they are working on something that challenges them.

What happens naturally when people are maintaining code is that the most adept programmer tends to have the biggest individual impact on the codebase. Others may have equally or more drastic impact on the overall project by introducing problems that everyone has to spend time cleaning up or by setting the tone for some piece of the project that ends up becoming a central piece, but in the average day, the smartest developer is probably the one writing the most lines that are being kept and the one doing most of the prototyping, factoring, and refactoring that create the context in which everyone else is doing their work. S/he won't slow down until s/he finds the complexity challenging, and in in fact her/his subjective evaluation of the quality of the codebase will be that it is improving with each increase in complexity until s/he begins to find managing the code somewhat challenging.

Not all challenges are the same. Witty Haskell-style one liners that showcase clever tricks are a very different kind of challenge than the kind of challenge that gets created by a nebulous distributed cloud of little intricacies and corrections that have to do pieces of code separated by 30 lines of other code interacting with each other. The witty one-liners are a lot less time-consuming to debug and maintain even though they pose a few skull scratching challenges when people look at them and try to figure out what makes them work.

If it is true that the complexity in the codebase is going to climb to some limit either way, it's much better to have complicated lines of code than it is to have complicated patterns of interaction between different lines of code. So you want to keep your codebase full of pithy one liners.

productive vs consumer ui

[This is an old unpublished post that I decided I would rather post than delete.]

tldr: The concept of "good UI" is too narrow. People want to develop optimal UI should distinguish between "good UI to optimize productivity for power users" and "good UI to maximize curbside appeal and minimize learning curve for entry users." Since these two concepts are inherently opposes, a product cannot be both. If you want to compete in a relatively empty space, design for power users, since most people design for curb-side appeal. (Designing for power users isn't creating feature bloat, it's minimizing the amount of time required to complete certain routine tasks for people who bother to learn the shortcuts. vim and emacs are the only two programs I know of that were truly built for power users.)


The greatest strides in technological improvement are those that improve productivity. Individual companies make and lose temporary fortunes with changes to consumer experience, but in the long run, these changes are inconsequential. Having the world's flashiest interface today does not necessarily make it any easier to have the world's flashiest interface tomorrow — especially for a company that consumes its own products.

Interfaces that look nice, frequently do so at the cost of efficiency. Animations between screens take time, not just the imaginary time that my computer uses, its clock cycles that are mostly spent idle anyways. But my time, the time you and I spend sitting at our desks waiting for our computer to let us implement the things we are thinking about implementing. The mouse was a great step forward for consumer UI. For many applications, it's point and click functionality was also a step forward in productivity. But for many other purposes it was a step backwards. It is simply true that you can work more quickly if you can leave your hands in place than if you have to move them between the keyboard and the mouse.

Hence, I take Vim as my example of the greatest achievement in productive UI. It has a huge learning curve, but it pays handsomely for the effort that the user takes to learn it. Firstly, it serves as practically the only example of a productive interface that has seen several radical improvements and rebirths. ed begat vi begat Vim. With each of these changes, the scope of the tool increased radically, and with the scope, so too did the amount that the user had to learn to take full advantage of the program.

Consumer UIs tend to move in the opposite direction. The manual shrinks away to nothing over time. Both trends make sense, and learning to contribute to the continuation of both trends is certainly worthwhile. They both make sense in their own context. Those who expect to use a program regularly for the next ten or twenty years can afford to spend a few hours a day for several months or even years learning how to take advantage of the features available in that program. Whereas, consumers typically want to pull up a product and have it work instantly. Very often, they have no plans to stick with that particular product long into the future. Its instantaneous accessibility is what brings in new users, and its inability to offer more efficient use with practice is not really a defect because nobody wants to put the practice into learning how to use the product efficiently.

Unfortunately (or fortunately depending on how well you can position yourself to seize the opportunity the situation presents), trends in optimizing for consumers tend to infect the practices of developing for productivity. People who aren't thinking attentively are likely to evaluate a productivity tool based on its consumer appeal even though that appeal is often inversely corilated with increased productivity. IDEs are almost always too GUI.

[When I originally wrote the post two years ago, I had rants about Xcode here. I haven't used Xcode in a long time, so I have no idea if those complaints still apply.]

Bad Code Considered Harmful

If you are not familiar with the very brief article, "with Statement Considered Harmful", this post will not make much sense to you. While not particularly well-known, it has been one of the more influential "Considered Harmful" articles. The JSLint tool forbids the use of with without giving any option to permit it as a direct result of this article.

For the most part, javascript conventions don't have much of an impact on how I write code. I have yet to encounter anything that can be written better in pure javascript than it can be in CoffeeScript, LiveScript, or clojurejs (or even TypeScript, though I am emphatically not a fan of explicitly statically typed languages -- implicit static typing like Haskell is great though). For the most part, I am convinced that you can get much cleaner, better code by precompiling something else to js than you can by actually writing code in js. In addition, every application I have spent much time working on sent a lot of data and did more processing on the server than in the application. The slightly larger files and slower performance from using these libraries is tiny compared to the differences that come from making sure that image resolution is no bigger than it needs to be, for example.

If you can't write code where you can keep track of what properties you know an object has, you probably don't have much of a future in programming. After all if ooo.eee.oo.ah_ah.ting.tang doesn't have a walla property, you're going to get an error when you try to access ooo.eee.oo.ah_ah.ting.tang.walla.walla because you can't access properties of undefined.

Secondly, one of the great misfortunes of javascript, in general, is that it does a very poor job of handling scope. This isn't a deficiency of  the with statement. It's a problem with javascript in general. Variable's default to global scope, when you practically always want them to be local; there are no nested scopes for conditions and loops when declaring variables, there's no one to isolate a namespace to a file, etc. (In browser javascript, Node has cleaned up the last one.)

There are two fundamental problems with javascript's scoping. One is that it enables you to easily make mistakes, the other is that it enables you to write bad code. A mistake is code you wrote that behaves differently from what you expected it to. If you forget to specify a local scope for your variable you will end up with a global variable, and possibly with bad behavior. This kind of thing is in general pretty easy to test for, pretty easy to catch, and pretty easy to correct. The ease of debugging becomes double plus true in this sort of situation when it is based on a known pitfall of the language that you routinely check for. Bad code is code from which it is very difficult to or impossible to discern from reading what you intended the code to do.

This is exactly the thing that the rant I linked to was complaining about. However, there is a key distinction between what I'm saying and what that rant was saying.

The key distinction is the word enables. Javascript does not force you to write bad code, any more than it forces you to misscope your variables. Some languages, do force you to write bad code. I don't want that last sentence to turn into a debate, so I'll use INTERCAL as an example of a programming language that forces the programmer to write exclusively bad code. It does so by design because it was created for entertainment value rather than usefulness. In my opinion, plenty of other languages accidentally  force you to write bad code as well but that's another rant for another day.

If you find yourself needing to set a global variable while programming in Python, you are forced to use a syntax that makes that intention clear. In javascript, you have the freedom to just, set it. However, you also have the freedom to clearly indicate that you mean to set a global variable.

If you are programming for a browser, you can use the following function, to clearly indicate when you intend to be accessing a global variable.

function SET__GLOBAL(var_name, value) {
    window[var_name] = value;
}

(Or you could use a comment!)

Is this something you should do regularly? Probably not. Are there cases when you should do it? Almost certainly.

There is one good reason to set global variables. There is also one good reason to use the with statement, and it's the same as the only good reason to use practically any feature of any language. It's also the same as the only consideration that actually matters 90+% of the time you are picking what language or toolset you should code something in. Choosing to use that language, that toolset, or that feature makes your code easier to write, easier to read, and easier to maintain.

For picking a language, the libraries and tools available do make a little difference but not much. You will be hard pressed to find a readable language in reasonably common use today that doesn't have good libraries for practically every task you want a good library for. If the syntax for a language is well-suited to your task, you can rest assured that somebody else has noticed the same thing, and coded a library to help with whatever you are trying to do. (This isn't to say that you should necessarily use that library, if the syntax is really well-suited to what you are trying to do, then sometimes it's a lot easier to build something up from scratch than it is to learn how to use somebody else's library, especially when that library approximates your goals but doesn't perfectly address them.)

When does the with statement make code better? It can in situations like the following. Suppose you are writing a lot of functions that make extensive use of trigonometry, you might want to do something like this:

var MathFunctions = {};
with (Math) {
    MathFunctions.should_I_trust_floating_point_equality = function(x) {
        var one = sin(x) * sin(x) + cos(x) * cos(x);
        return (one === 1);
    };
}

Incidentally, I've mentioned a way to clearly identify that you intend to set a global variable, but that's only half the problem that can arise when you are using the with statement. What if you need to set an attribute of Math? You can use Math.attribute. What if Math has a Math attribute. Then before your with statement you can do Math__Accessor = Math;. For the sake of consistency, the final option should probably be the default (or turning it into a function like the one that was previously mentioned). I can't necessarily think of a reason why you would ever want to do some of these things, but in general, I think having the freedom to them, along with the discipline to do them right results in much better code than proscribing their use altogether.



I cannot think of any reason you would ever want to write javascript without surrounding it with

with _

referring to either lodash or underscore). Of course, if you're going to do that, you might as well just go a step further and use CoffeeScript or LiveScript unless you have some performance-critical processing to do client-side. I do write javascript, and I don't do this. I do want to, at least assuming I can't use either of the languages I just mentioned. For a substantial part of every week, I am payed to write code that conforms to given instructions. Those given instructions have technical constraints due to "business considerations." I personally believe that good developers tend to be fluent enough in coding, in general, that they have an easier time working with a codebase that is halfway well written because half of it is written with modern tools and practices than a codebase that is entirely poorly written for the sake of maintaining consistency with the half that was written badly a few years ago.

I don't think I write lousy code for fun (when I'm writing code without being constrained by a lot of rules), but I do get payed to write lousy code. So I write a lot of lousy code. I go back to management routinely and say that given the spec, which always tells me what language I must use and frequently tells me what third party libraries I must use to accomplish the things specified, that the only conforming code I can write will be bad. Then I do as I'm told, and rant about it later.

Management isn't just making up rules for no reason. It is enforcing rules suggested from programming blogs about how to avoid common pitfalls of programming. I work for a Microsoft shop at the moment, and, as such, most of the rules and difficulties I'm dealing with come from what I will call the Microsoft philosophy.

My general understanding of the Microsoft philosophy is that it consists mainly of tools and guidelines intended to permit even poor programmers to slowly build out and maintain an application. These tools and philosophies come at the cost of forcing even good programmers to slowly build out and maintain an application.

This causes the enormous problem of creating a situation in which the optimal strategy for building out an application is to hire an army of mediocre or poor programmers, thus ensuring that rather than having a few problems created by a few bad practice, your codebase as a whole is plagued with issues arising from general incompetence. This problem arises even if the core development is carried out by a team of very good programmers and has been for years. Firstly, as I mentioned earlier, development is slow. I have yet to encounter a problem that can be solved more succinctly and quickly in C# than it can in Ruby or Python. As a result, mercenaries occasionally descend upon the project to fill it with thousands of lines of the most unreadable, unmaintainable garbage I have ever seen, so that some objective can be achieved a month sooner, at the expense of every other deadline that will arise in the future. The resulting vicious cycle is only checked by the fact that the code eventually becomes so bad that the product stops working. Then money dries up, and armies of mercenaries cannot be afforded until the code and the product are salvaged again.

There are a huge number of patterns that radically hurt the quality of code. Religious observation of rules that someone came up with due to the misuse of a code pattern in one place rank among the worst. Defensive programming also ransacks the readability of code. In my experience, explicit static typing is also something that tends to hurt badly.

People do all of these things intentionally. People advocate all of these patterns. The above are optimized for code safety, a concept I don't fully understand. What is code safety? It clearly doesn't mean maintainability. The aforementioned things all make the code less maintainable. It is also difficult to see how they reduce bugs. Indeed, defensive programming introduces a whole new category of bugs. The-overzealous-error-checking-resulting-in-errors-being-thrown-when-they-really-should-not-have-been-thrown-because-now-you've-got-edge-cases-in-both-directions-instead-of-just-one category of bugs. Good testing finds anything that you can find through defensive programming. In both cases you can find all of the errors that you expected. (Actually, in good testing, you find many errors you didn't anticipate in addition to the ones you expected because your old unit tests continually validate that you didn't actually change old behavior. Whereas with defensive programming you tend only to find the errors you expected. The old checks don't provide much value because they aren't set up to check for the ordinary case.)

There are two other patterns that, in my opinion are mostly harmful, that again, are explicitly aimed at improving the quality of the code. These two are especially pernicious because they aim to simplify. The first is the intention to prevent complicated lines. I won't discuss it now, because I plan to discuss it in depth as my next topic. The second is a foolish consistency.

Consistency is good at least in part because a deliberate consistency is usually based on a standard that has been well-thought-out, and in many times, incorporates things you had not yet planned to do. When I began using vim, I immediately regretted the fact that I had previously disregarded 79 character line length rules. I hadn't intended for my code to be edited on a computer so old that it lacked resizeable editors and had not yet come to appreciate the fact that their may be good reasons to prefer not to resize your editor even when you have a choice.

The very worst form of foolish consistency in my opinion is the restriction of third party libraries to a core few so that people can "learn it quickly" (as if well-documented third party libraries provide a steeper learning curve than in house code written without them, or worse, with a different third party library that does not actually suit the task at hand, and must therefore be continually hacked to accomplish its objectives.)

All of the things I'm mentioning are policies that are in place because some programmer advocated them to avoid some pitfall. My counterargument that I will be presenting throughout this blog is that these pitfalls don't produce problems. Bad code produces problems. Bad code is just code that isn't good. And good code only results from giving good programmers the freedom to program the way they think the should. It helps to have a few practices in place like unit testing (which doesn't interfere with how the programmer goes about writing code) and accountability via code reviews with other good programmers (which improves communication and general cohesiveness of the project without directly impacting the way code is being written while it's being written).

I realize my overall thesis has a major point of circular ambiguity. I haven't defined good programmers, but I do mean good not great. My general belief is that most programmers are good enough to be considered good programmers. However most code is bad, especially when developed by a team of people, partly because of poor communication (which has little to do with how programming itself is done) and largely because one or two bad programmers (and outside consultants who have no incentive to care about long term maintainability) can do damage much more quickly than five or six good programmers can fix it.