IBM’s paranoid defense of its mainframe monopoly is going to be tested again but this time if it blows the start-up out of the water, or denies it clearance to compete, it’ll be messing with an open source operation, and a European open source start-up to boot.

The European Commission, an admitted open source advocate, is already entertaining at least one known complaint about IBM’s allegedly anti-competitive behavior in the mainframe market.

The new would-be mainframe player is a Franco-American start-up called TurboHercules SAS.

It wants to commercialize the Hercules open source mainframe emulator that’s been in development by a community of IBM mainframers for the last 10 years, and sell it to mainframe shops so they can use x86- or Itanium-based PCs ostensibly for the test and development of mainframe applications.

Hercules is a software implementation of IBM’s System/370, ESA/390 and z/Architecture instruction set, but not IBM’s companion operating systems, and although Hercules has never been commercialized before, it is, for all intents and purposes, the only existing alternative to IBM.

But right away TurboHercules has got a problem because it needs an IBM operating system and IBM ties its mainframe operating systems to its hardware, a fact that should make it the subject of an antitrust investigation but hasn’t lately.

See, IBM is specifically forbidden to tie under the lingering terms of its now-dissolved 1956 consent decree with the United States government.

But IBM has so far evaded capture. Last year it bought Platform Solutions, one of the last companies to sue it for antitrust for tying the OS to the mainframe, to hush the complaint and get PSI’s cheaper, z/OS-capable Itanium boxes off the market.

TurboHercules president Bill Miller, an American who used to work for Softway Systems, the Interix Unix-to-NT interoperability shop that Microsoft bought way back when, says that, being incorporated in France, TurboHercules wrote to the head of IBM France in July asking that IBM loosen up its license enough to let the start-up function.

It wants IBM to let its mainframe customers use their 64-bit z/OS, z/VM and z/VSE – or even older 31-bit and 24-bit – operating systems on a TurboHercules configuration as IBM used to permit before its change of policy in 2000. They would pay IBM a reasonable licensing fee for the privilege.

TurboHercules hasn’t heard back yet.

Miller says there are mainframe shops and schools that would kill for such a device. Universities that can’t afford a mainframe could start teaching mainframe programming again. Mainframe sites could use it for ancillary workloads, training, demonstrations, pre- and post-processing, data preparation and archiving.

Anyway, in the absence of an answer from IBM TurboHercules turned up at Intel’s Developer Forum this week claiming to have found a niche it can occupy in mainframe disaster recovery/business continuity without violating IBM’s myriad proscriptions.

It says there’s a provision in the IBM license that allows IBM operating system software to run temporarily on an alternate machine in the event the customer’s mainframe is inoperable.

TurboHercules doesn’t need to touch the IBM OS to back up a system, merely to restore it.

TurboHercules probably can’t make a living at such a thing, which would likely be popular only with mainframes that had gone off maintenance, or sites that can’t afford a new mainframe, and none of its people have given up their day jobs yet but it’s hoping IBM will either voluntarily relent or be finally pushed to it by the regulators.

The start-up says it a press release that it “hopes to benefit from IBM’s long-standing support of open source software.” In other words it’s trying to stick IBM none too subtly between a rock and a hard place.

It wouldn’t take much to banter about the conclusion – voiced by T3 Technologies (T3T), once the second-largest mainframe systems integrator in the world, in a still-pending antitrust suit lodged against IBM at the end of 2007 – that Big Blue is nothing but a big fraud and hypocrite and “postures itself as a champion of open systems and standards,” demanding that competitors like Microsoft provide reasonable and non-discriminatory access to interoperability information, but won’t do it itself.

IBM is said to be sensitive about its image.

Anyway, TurboHercules co-founder Tom Lehmann observes that “In these tough economic times smaller mainframe shops, as well as state and local governments, are looking for ways to reduce costs and still meet their obligations in the event of a disaster.”

An unidentified mainframe site has got a prototype TurboHercules backup system installed (see http://www.youtube.com/watch?v=q8km-xOD2vc).

Due to IBM’s licensing restrictions, the demo TurboHercules ran at IDF was reportedly limited to IBM’s older MVS 3.8j mainframe operating system and SUSE Linux Enterprise Server for System z. There’s no problem running z/Linux on the thing. The demo was hosted on Windows Server 2008 on Nehalem and Itanium processors.

Hercules is reportedly spiffier in some respects than its mighty clone.

The widgetry, which can run under Linux, Windows (98, NT, 2000 and XP), Solaris, FreeBSD and Mac OS X (10.3 and later), is supposed to be able to run mainframe apps with the performance of a multi-hundred MIPS mainframe on a single Intel-based server.

Otherwise, TurboHercules means to provide corporate users with commercial services, and provides free downloads and documentation for Hercules installed on Windows and Linux host systems. Its voyage to a recent Share user group meeting suggested that there may be more use of TurboHercules in the corporate world than anybody suspects.

On its web site the start-up says that in the next few months it will be releasing “fully tested commercial distributions to match several real-world scenarios. Our preliminary assessment is that there are several important – albeit not necessarily mission-critical – ways in which Hercules can be deployed in mainframe shops.” It will be offering SLAs.

Hercules is available under the Q Public License to avoid what its community called the “political baggage” of the GPL.

Roger Bowler, the creator of Hercules, is a TurboHercules co-founder.

By the way, IBM filed two motions for summary judgment against T3T on August 30. T3T has until October 14 to respond. That’s likely the last obstacle IBM can throw at it. If it survives, its antitrust case could get to trial sometime in Q1. T3T is also waiting to hear whether the European Commission will open a formal investigation of IBM on its complaint.