For last few days, I have been looking into the source code of Console 2 to understand what makes it tick. I went through the MSDN documentation and obligatory The old new thing to get an idea about how console operates in windows. Spurred on by the comments, I even read the Terminal HOWTO to get a wider picture. Eventually I understood how Console2 works, its ingenuity and sadly its limitations. So here is a blog post for consolidating whatever info I have gathered. Hopefully you will find a use for it or even teach me a thing or two. This part just doles out historical background you may want to skip it.

The console, terminal and shell

Any operating system (or a runtime library) usually provides a program with stdin, stdout and stderr files. Anything written to stdout is displayed to user and any input provided by user can be read from stdin. In the beginning, there were line printers for user output. So stdout can be considered as just a file stream. Then came mainframes & user terminals which were connected to them via serial bus. The terminals were no different than the printers. Communication was one way only. So anything sent to the terminal was considered final. Jump a little ahead in future and you could see so called smart terminals. These terminals had some kind of controller which allowed them to do fancy stuff. A good example is VT100 which was even able to display colors! But the interface to these terminals was still the serial bus. So escape codes were invented. By placing these escape codes in the data stream sent to the terminal, one could use the advanced facilities provided by these terminals; even display a simple GUI! However, the program running on the mainframe had to maintain a buffer in main memory to keep track of the display screen.

The OS used on these mainframes was either custom design or UNIX. So the terminals in UNIX are simple beasts. The OS writes to a device (usually /dev/tty) and the data stream was relayed over serial link to the terminal. Even after personal workstations became a norm; there was no reason to change this interface all you have to do is to emulate a terminal in software. To date all UNIX derivatives use this interface. Your terminal program (xterm or derivative) just listens on /dev/pty and emulates a long dead VT terminal. But this post is about Windows so lets move on.

In the beginning, there was DOS

As always many things in windows find their roots in the DOS. Now video adapter was always a part of PC for which the DOS was designed. So the video memory was always accessible. Therefore DOS programs never bothered with escape codes. Even though the support was there, it was always easy & rewarding to directly update the video RAM. So all programs on DOS (except your hello world variety) just took control of video RAM for rendering a GUI.

Welcome to Windows. Please carry your DOS baggage with you.

By reading the old new thing I have concluded that, Windows has just one design principle: Backward Compatibility. With windows, the programs started communicating with OS using messages & system dlls. By comparison, DOS programs just cleared DOS from RAM and handled hardware themselves. Up to Windows 98, DOS was real i.e. the command.com was your command interpreter which ran in Virtual mode or for playing DOS games, you could boot into real mode DOS with no windows baggage attached and your game could then run using unreal mode These were the simple old days. The obvious problem with this mode of emulating DOS is a DOS program could very well bring windows to its knees.

The rebel NT

The Windows NT line however was a completely different beast. As most of the people are using 2000, XP or Vista which inherit the traits of NT, we are going into the meat of the subject here. Windows NT was designed to be a container OS. An OS which could host multiple user environments which would share the kernel and device drivers. The only sore thumb was the DOS environment, which placed no restrictions on programs and had no native means of running a message loop. Windows programs communicate using messages for user interface, OS events and pretty much everything else. So a program must wait for a message from windows to do its work.
However, the DOS programs needed some alternate arrangement. They were using API designed around interrupts to perform user IO. So somebody must translate windows messages into the DOS jargon and simulate the DOS video RAM to display contents in a window. Meet csrss.exe, the program responsible for handling DOS programs in windows NT.

What you (or maybe its just me) didn’t knew about the console

Every windows program can have a console window! By default windows creates a console window only for programs having main function as entry point. The programs with WinMain function as entry point do not get a default console window. However, such a program can get its own console by calling AllocConsole. Now you could build a better console by either replacing csrss.exe or kernel32.dll. While you are at it you may also want to use your time machine to alter course of history Clearly there is a third way, one used by Console2 which does not involve modifying kernel though it uses scary techniques. We will discuss it in the next post.