Mac@Work : The Ultimate Business Machine

I'm done comparing the SQL calls I need to do across MySQL and PostgreSQL in order to support transaction processing (i.e., Locks, Begin, Commit or Rollback) in a multi-user version of Luca that I'm working on.

I'm also going to do versions of Luca and Liya for the iPad. I've got a new iPad 2 that I'm looking forward to working on.

Fortunately, the SQL calls for transaction processing are consistent across MySQL and PostgreSQL. They're the same - BEGIN, COMMIT or ROLLBACK - for each platform.

There is a more Oracle-like START TRANSACTION in place of BEGIN in MySQL but BEGIN will do. Why add complexity? Always simplify.

While PostgreSQL has always supported transactions for as long as I can remember, MySQL with MyISAM tables do not. We'll need Innodb, which is the default setting for newly created tables in MySQL 5.5.

Once we can get our Lion installer for MySQL 5.5 done, Hai Hwee has a functionality built into her installer such that it'll migrate forward all the data in your current databases to the new version of MySQL that you are installing. I'm hoping that when that kicks in, all the data in the MySQL tables will then automatically support transaction processing, and then we'll be able to do BEGIN, COMMIT or ROLLBACK from applications like Luca, knowing that we'll be accessing databases that support it.

We also need advisory locks - putting a lock on database updates among consenting apps (I almost said adults). This is a simple form of record locking, to prevent the database from being written to the same place at the same time by different people, using Luca on different client machines. I don't think Luca needs the more sophisticated form of record-locking at the moment, so advisory locks will do for now.

On MySQL it's very easy. SELECT GET_LOCK('Lock1', 10); To test for the lock, it is SELECT IS_FREE_LOCK('lock1'); To release the lock, just do SELECT RELEASE_LOCK('lock1'); I just tested it, accessing MySQL from two client instances. It works great.

On Postgres, the equivalents are SELECT pg_advisory_lock(5), SELECT pg_try_advisory_lock(5), and SELECT pg_advisory_unlock(5).

So two differences - one is the name. The other is that MySQL uses a string for the semaphore while Postgres uses a big integer.

I can hide these differences in my database frameworks so that an app like Luca calls the same code, no matter which type of database it is accessing. So we're good to go.

When I had the idea for Liya and my database access frameworks, I never expected to be able to go so far creating a unifying API that will work for both MySQL and PostgreSQL - mainly because there are so many differences between the two dialects. It's been a long slog. But I'll happily take what I can get.

But just why are we mucking around with this MySQL 5.5 and Lion stuff?

That I think is a good question because that's the nature of things when we're doing development on an Apple platform - it's the "let's get going and take no prisoners" way of thinking.

If the only way I can do these custom builts of MySQL is on Snow Leopard, then I need to keep one of my machines running Snow Leopard. If I can get this problem solved, then I can envisage that every one of my Macs (including the live server) will be running Lion within a month or two. Then I've simplified things. I've got only one platform to think of, going forward. We mustn't forget - we usually only have a two-year window to build on top of all this stability, to get all the new features that we've been dreaming of done, before Apple, in their eternal wisdom, starts to change things all over again.

But why does Apple get away with this? I think it's because the result to the consumer is an endlessly fresh, exciting computing/media tool. But what it costs the developer is this never-ending race to keep up.

Otherwise we'll be like Adobe - trying to keep time in a bottle. Now who wants to be like Adobe. Except for Photoshop, I'm almost all Adobe-free.

Now MySQL 5.5? On 5.5, Innodb replaces MyISAM as the default database engine on MySQL. With Innodb comes support for transaction processing, and commits and rollbacks - things we need when we're working on more conceptually demanding database systems.

Of course, we can already use Innodb even now on MySQL 5.1 and it was so for a long time. But with this as the default, it's easier to just move on and build on it.

I'll hold up on MySQL for the moment and see how we can match up the similar features on PostgreSQL. The way I've been building our database frameworks is that I want to write one piece of code for the customer-facing app, like Luca and Liya, on the Mac, iPhone and iPad, and the database frameworks will take care of the platform differences when they access the relevant databases.

And I'd want to get back to my server apps, like MailServe and DNS Enabler and WebMon. And I need to take a look at Lion server, too. So, lots of work ahead.

More on MySQL 5.5 on Lion

Okay, I'm sure by now that we can't build MySQL 5.5.15 from source on OS X Lion. I'll wait for 5.5.16 to be released and try again.

Am I the only one trying to do this? Nothing else turns up on Google. You can build MySQL 5.5.15 from source on Snow Leopard (32/64 bit fat binary) and that will run on Lion but what i was going for was getting everything built and run on Lion.

I also tried building MySQL Connector/C 6.0.2 on Lion (which is needed to build our database access frameworks at the client ends, like on Mac, iPhone and iPad). That will build OK and I can get a fat 32/64 bit Intel binary. But when I run it, I get lots of NSAutoReleasePools errors. Seems like something to do with threads, and if it were so, then that's like the problem we're having building the whole MySQL 5.5 server on Lion (which failed at doing something like pthreads).

Anyway, I only have a vague understanding of what I'm doing here, at least right now. Will need to do more digging. But I'm sure we'll sort this out. They always get sorted out. Eventually. I'm the rational optimist.

MySQL 5.5 on Lion

I've been working on this the whole weekend but I still can't build MySQL 5.5 from source on Lion. I keep getting errors with the pthread and, after I've found a way around that, there were errors with my_atomics or something like that.

Will leave this for now. Seems to be fixed in the coming 5.5.16 version of MySQL. We'll see then.

Luca on Lion - A Descent into the Walled Garden

I was having problems even opening the Luca project's nib (user interface) files on Xcode 4.1 on Lion. Xcode, moving forward on Lion, has dropped support for custom Interface Builder palette objects. But that was one of the most powerful ideas from NeXT's Objective-C Project Builder/Interface Builder environment - that developers can build palettes of custom user-interface objects and drag and drop them or connect them, just like Apple's own built-in objects (like NSTextFields, NSTables, NSPanels, etc).

So, if I want to change anything on Luca's interface while developing on Lion, I sinply can't because I can't open any of my nib files due to their dependencies on my custom interface objects (in Luca's case, these objects dealt with the formatting of currencies whenever Luca presents money values in its windows). In Apple's scheme of things, I had to find a machine that is still running Snow Leopard, and run the previous Xcode 3.2.x on it and then open Luca and cut off all dependencies on the custom nib objects.

Theoretically, I could open Xcode 3.2 on Lion but, in reality, installing Xcode 3.2 on top of Xcode 4.1 would mangle Xcode 4.1. I took a long time to settle in on my Lion development - I didn't want to start all over again. So, fortunately, my wife's MacBook Pro was still running Snow Leopard. So I re-installed Xcode 3.2.6 on it. And killed off the dependencies and rewrote some code to replace the lost functionality.

This is really like taking a few steps back. But developing for Apple is often like that. You have to run just to keep in place.

But then, I've always been a Mac fan. I can buy in to what Apple would like to do to keep their platform alive going forward. But I wonder. Would we think back, years later, to Mac OS X Snow Leopard as being the zenith of Mac development, where we can make the Mac do almost absolutely whatever we can dream of, before it descended into this walled (curated?) garden that is the iOS?

MailServe and DNS Enabler for Lion Updates

With MailServe for Lion 5.0.1, I updated Dovecot to its current latest version, 2.0.13. I also fixed two problems with Dovecot. Dovecot can now allow SSL ports (995 and 993) to be opened without requiring the corresponding default ports (110 and 143) to be opened, too (so that you can force a user to connect to Dovecot only via SSL).

Also, this update restores Dovecot's ability to support custom port numbers for POP and IMAP. Please remember to save your config, do a de-install from MailServe's Help menu, re-open the saved config, and then restart all the services, in order to get these latest Dovecot binaries.

It's currently very hard to work with Dovecot 2.0.x because there is just no documentation to be found. I found the workarounds to these problems (plus that about how to use an Intermediate CA SSL cert) by Googling for the answer and not via the Dovecot Wiki. But I don't want to knock Dovecot, which is really a very good system.

For DNS Enabler for Lion 5.0.1, I fixed a problem whereby the dyndnsupdate.key used to support dynamic updates cannot be found on a Lion system with a case-sensitive file system.

So, it's really not that easy to build and support these systems because there are so many variations to test and consider.

MySQL & PostgreSQL Database Installers for Lion

I've uploaded new Lion versions of our database installers - for MySQL and PostgreSQL. They come with their own Startup Preference Panes that you can install into System Preferences. I've tested them. They work great.

The MySQL version this installs is 5.1.57. We'll want to build a version for MySQL 5.5.x and get everything we do aligned with that.

The PostgreSQL version is the current latest, 9.0.4. There's a PostgreSQL bundled within Lion but I haven't figured how to turn that on. But there's an advantage to using our own because I think we can keep up with the changes at Postgres faster than if we depend on Apple and our installer upgrades our database data automatically with each update. And, finally, we can get rid of our own PostgreSQL setup cleanly when we don't need them.

OS X Lion's 3000 new APIs

If you get to see Apple's Developer pages, you will see this page about the 3000 new APIs in Lion :

But the other side of the story, about the APIs that are lost or being pruned away (being deprecated in developer-speak) - that is not so well known.

Among those being deprecated are two specific API calls that I've used to build all these server setup apps. They're the calls that allow my system configuration tools to acquire admin privileges from the user so that they can move files into protected directories like /Library and /System/Library and /etc. These calls are going away and they don't look like being replaced. So if I don't find a way to get around this, I won't be able to build any more server setup apps going forward.

I would leave the conspiracy theories to others, like Apple wants to own everything (well, I'm not even allowed into the App Store with these apps - Apple specifically forbids apps that require escalation to admin privileges).

But I think I can build pretty good server administration tools, if at least only for my own use. So I will need to find another way. And soon.

But before that, I need to get Luca, my accounting app, working in Lion. It's not just a simple re-compile for Lion because Luca needs to go through database frameworks (like my version of Microsoft's ODBC calls) to get to the database servers and these frameworks need to be built from source for Lion and often they can fail to build when the Mac OS changes or the MySQL or PostgreSQL versions change. "So you want to charge even though you have no new features?" It's work, man, and I'll probably build this version of Luca only for the people who really need it.

(And I've got nib user-interface files that I can't now open in Lion because the system for building Interface Builder plug-ins change in Lion. There isn't even an Interface Builder.app in Lion. It's all built into Xcode which has changed, drastically.)

OS X Lion's AFP

OS X Lion doesn't support FTP, i.e., you can't share files from one machine to another using FTP. It's all AFP (Apple Filing Protocol) now.

I've been building my own Adobe GoLive/Dreamweaver replacement because I want to be Adobe-free on my Macs. (I'm now totally free of any Microsoft product). It's called Duomo. But when I try to connect to my server, so I can upload this blog entry (I built a blog editor into Duomo so I can see what the blog entry looks like, exactly, as I am typing it), I got an error with AFP. But I'm sure I tested this with the developers' GM version of Lion and it was OK. So this is a mystery and it makes me worry about what else might break.

P.S. : I've been installing and re-installing Lion today. I thought I had finally settled on a fully equipped system on my iMac today, with the developer resources all set up properly so I can finally dive deeper (and much deeper) into Lion and Lion Server. But I found I made the partition too small for what I planned to do and, for some reason, I couldn't use Disk Utilities to increase the partition size. So back to doing another installation. Hope this will be really it, so I can get back to work.

OS X Lion

And I'm glad to say, I'm now also able to make these new OS X Lion versions of all my apps - MailServe, DNS Enabler, DNS Agent and WebMon, etc. - available here, right on time. There were times, during Lion's gestation, when I felt that my stuff were all breaking apart so much that I couldn't put them back together again, Apple being in the habit of changing its innards each time they upgrade their OS.

Even now, I fear things might still break with the shipping version of Lion. I've done the testing against the developers' Golden Master relaese of Lion. The shipping versin from the App Store appears to have the same build number. But once, when Apple released Leopard, the final shipping version was different from the GM release and my apps broke and I had to pull them from my site. That taught me never to take things for granted.

Am also really glad that OS X Lion is now upon us. I haven't slept much the last two days. I hope that the changes Apple has been making to the system's innards can now stabilise so that I can finally start to build on top of it.

Lion Developer Preview 4

I've installed OS X Lion Developer Preview 4. It restores the stability that was lost with release 3, especially with Safari. So it feels good to be working with Lion again. Not sure about the GUI changes though, especially the scroll bars. The iOS scroll bars feel clumsy on the Mac.

New Lion, Dev Preview 3, Worst of the Lot

Developer Preview 3 of Lion doesn't work on the Thunderbolt MacBook Pro. Fortunately I backed up all my Lion-related work before I did the Software Update. The MacBook Pro refuses to boot up after the update.

I dropped the Lion partition, re-created it, and then installed Developer Preview 3 from scratch. Again, the MBP hung on startup and I gave up at this point.

I do have Preview 3 running on an older iMac Core 2 Duo. But Safari keeps getting stuck when loading web pages.

I hate the look of Lion. Will this be Apple's Vista? That's a distinct possibility. Has Apple been hiring a lot of refugees from Microsoft? Use Lion and you'll feel that thought creeping up on you.

I've no doubt that Apple has done a lot of performance tweaking in the innards of Lion. Lion does feel faster than Snow Leopard for tasks like mounting a remote shared volume. But the user interface really sucks, I finally decided, and that's not what you want to hear about Apple. (NeXT's grey interface has a dignified, understated elegance - if that's what they wanted to bring back, they failed).

Transactions and Locks in SQL