Sunday, March 20, 2011

alchemy 0.95beta Released to SourceForge

i just released version 0.95 of alchemy to SourceForge. Below are the readme.txt notes for this release.

About alchemy
The alchemy framework has been used to create two production systems. One which host a database with over 150 tables and 1 billion rows of data. Another hosting a database with over 225 tables, 15,000 fields, and 4.3 billion rows of data.

New Stuff
Enhanced login module with encryption and multi-salts
  • Document Management Module.
  • Lots of cool new JQuery integration and examples.
  • Enhanced context management.
  • New DataMiner tri-frame module.
  • Hooked up examples in the framework demonstrating functionality.
  • UI Template patterns.
Still Needs Work
  • Documentation needs some TLC and some more depth.
  • Integration of DataMiner login into alchemy login.
  • More examples demonstrating functionality added to framework.

Friday, February 25, 2011

Perl For Windows Admin and Software Build Management

i tend to only do one Perl project a year. This has been the trend for the past few years. But they all have filled some pretty big holes and remind me what a cool and utilitarian language Perl still is. Currently i am working on a tool similar to ant. i just finished coding out the functions that allow me to check a source and target build, doing a two way compare that produces three lists (for analysis and build):
1) files on source that are not on target
2) files on target with newer timestamp than source
3) files on target that are not on the source
Perl is still my #1 language for work like this.

I also pulled two great Perl resources from my bookcase. Win32 Perl Programming - The Standard Extensions by Dave Roth. If you are looking to use Perl with Windows sysadmin this is THE BOOK for the job. Also Network Programming with Perl by Lincoln Stein is a killer reference.

While i spend a great deal of my current life in Php i do enjoy these times when i get to do a Perl gig.

Wednesday, January 12, 2011

In Search of a New UI

As I have been in software engineering over the years I have seen a lot of technologies and technical fads come and go. I have been witness to some pretty incredible growth in bandwidth, capacity, and hardware technology. I have been able to watch open source software, which I first regarded as communism masked as technology change the technical landscape. I have also had the good fortune to become a convert and to be a part of this movement that still has not been fully adopted and embraced by much of the business community.

I have also watched certain hardware and mobile devices, and Apple steer technology back into the land of the proprietary. I think the forces of open source are too strong to let such a shift occur en masse.
What I have not seen in either the internet or the intranet are applications that truly put the technology stacks to their fullest capability. Sorry, but the Facebook UI and the apps leaves me wanting. In the space I live, the electronic medical record (aka: EMR) there is just to lack of imagination as to what is possible in the user experience (UI) and in defining workflows that addresses real business needs, that places healtcare providers more with their patients and less with the computer. In the business application space I cannot say I am all that impressed with ERP and other types of business apps. This criticism is as much leveled at myself as it is at the industry at large. For I fervently believe that real growth and innovation will seed and flourish when you are willing to hold up the mirror in front of you prior to pointing fingers.

Now that I am working in the front-end space I have come to realize just how much norms have stagnated our notion of what an application needs to be. As I am becoming more adept with these technologies I am coming to understand what an application and a workflow can be.

In concert with that last thought it has been my observation over the years that we (myself included) like to blame the shortcomings of technology for our own lack of a) knowledge; b) technical design; and c) functional design. Back in the 90's when I was making low level hWnd calls from VB to do what I thought should be standard UI fare there may have been some truth to these arguments. Though if you asked me back in the day I really did hold a similar position on such matters... just not as strongly formed as my experience then was far less.

As I look over the landscape of the technology stack I currently work with (Php, Javascript, JQuery, CSS, Html) I realize that there really is nothing lacking other than my own ability and imagination. Having prior experience in the Java world I can say the same about that technology stack as well.
Such a realization is empowering as it allows you to really dream and contemplate what is possible. It emboldens me as for every thing I wish to do, I realize that the only impediment I face is my own lack of knowledge and understanding as to how to go about solving the problem. This in itself is very cool as I have always loved to learn.

The freedom of the open source movement and the internet has created an asynchronous development environment and an informal community of developers that are driven by their passion for software and their love of sharing the fruits of their labor with others. This community has created more jobs and revenue than the world of proprietary software. It has created opportunity for those that are economically challenged. It has created a space for learning and sharing that is unprecedented in the history of ones and zeros. If it is comminism, then I, gladly am a commie.
I know firsthand that a real developer would practice their craft for free if they could not do it as a profession. Money is always tangential to the labor of love one gets in being able to play with technology and build things others find value and merit with. Sure, there is some hubris in what we do, but for the most part the business of technology becomes our rubics cube. There are so many of these proverbial cubes in the technology space that we have great choice as to where we want to play... Configuration Management, Database Development, Database Management, Application Development (both backend and frontend), Process Automation, Testing, Scalability. The list goes on and on and on.

Today it is time for me personally to re-think and dream about what an application can be, and to attempt and put those dreams into practice. Whether successful or not the most fun is always to be had in the journey itself.

Tuesday, October 26, 2010

Art and Programming

This has been a subject I have thought about for a very long time. Actually it is much more than thinking. It is also doing. In the past fifteen years I suspect I have written at least 1/2 million lines of code across a number of different languages. Everything from simple toycode to gnarley programs that do cross tabulation, or programs that convert and flatten proprietary hierarchical dictionary data [convert and flatten proprietary hierarchical dictionary]


There is a definite element of art to programming. If you do not wish to call it art, then one needs to accept that it as at least craftsmanship. Writing a quality program, or abstracting and developing a killer library takes a great deal of intellectual capital. There is enormous amounts of creativity involved in looking at and solving problems in code. For me not to consider this discipline as art is up there with stating that accounting and taxes are nothing more than problems we solve with math.

Ha! Have you ever known math that mostly boils down to simple arithmetic to be so mind bending?

Our buds at Wikipedia say that "Art is the product or process of deliberately arranging symbolic elements in a way that influences and affects the senses, emotions, and/or intellect."

They go on to describe traditional things like music, painting, photography as art. I own a bunch of different types of cameras and dabble as a rank amateur in photography.  I see many parallels between the two disciplines. The process involved in setting up a shot. Is it a worthy subject? Is the light right? How do you set everything up so the film records the same thing, the same experience you are witnessing? So many elements of design to consider.


If you are developing an application, and you are working with more than a narrow set of design inputs you are dealing with a similar canvas. There are numerous ways in which to can go about solving the problem. The technologies (the medium) you have to work with dictate how you will work. As you explore the possibilities and endless alternatives you begin to see patterns emerge. Will you, or how will you deconstruct and reuse these patterns? Again, so many elements of design to consider.

So we don't have galleries for our work. So much of the thought and design is only seen and appreciated by our peers and contemporaries. So what?

Art does not always have to be overt. Art does not always need to be for the masses. Art does not always have to be 100% pure. We call music art. Yet there are styles and genres (Jazz, Reggae, Rock, Country, Hip-Hop) that are nothing more than patterns we arrange. These higher order patterns are based upon a finite set of chords and notes and tones and symbols that we arrange and rearrange. Is programming that much different?

Okay, our net result is not something we can construe as art. But the process we go through to design and create can be. Meaning, it not always is.

I know many programmers that would not consider what they do to be art. And, I would concur that most coding is not art, and most people that code are not artists. This is reasonable. Not everyone that owns a camera and takes pictures is an artist either.

In the end I am not sure it really matters whether what we do is considered art or not. What matters most is that it is something we enjoy doing.

Wednesday, August 25, 2010

How Hardware Storage Increases Have Changed Software

Three weeks ago I bought a 750gb external drive for around $100, including tax. Three days ago I saw an even better deal for the same size. The cheap cost of space is really quite astounding if you look back in time.

5 years ago... Well, my current computer, I am about to swap out this friday has 80gb

15 years ago... My work issued computer was a 486dx-32mhz with a 120mb drive. Whohooo, or as they say in the Bronx, "fuuugettaboutit"

I remember back in the late nineties building web content and worrying about what impact adding a 3k or 4k graphic would do. Today I am having a hard time keeping up with the fact I can add video, graphics, build gnarley dynamic apps and not have to sweat a major load about disk space. We can thank the invention of
perpendicular recording back in 2005 for these advances in disk space.

Further, memory is cheap as well. We can quadruple memory today for the same price we paid 3-4 years ago.

And, I do not want to forget the low level infrastructure developers for their advances and optimizations in webserver and rdbms software.

Development has always been a field dominated by young bucks. But unlike us old geezers, they can't remember back to the prehistoric era when we were dealing with portable storage media that maxed out at 1.44mb, A hard drive of 40mb was the norm. You were envied by others if you had 80mb. And 512kb of memory was standard (like 2gb is today).

I am not just waxing nostalgic here. What I am trying to say is that back in that era, the hardware we had forced us to make technical design decisions that were germane to eaking out performance in the applications we developed. And, as a footnote to this, my technological forefathers would chuckle at me and say "sonny, you don't know what lack of resources are. We programmed on punch cards and had to run and debug our programs when we could actually get a time slot on the mainframe that took up an area the size of your house and had 1/1000 of the resources that you get today on that there iPod of yers".

My point is that I grew up in an era when resources were at such a premium we had to seriously consider optimization in our application development. Even today I still develop with this mindset.
Today we can throw hardware at a problem.

But, I do think performance is a key issue an application developer needs to factor into their code. Okay, programming has changed for many of us. I remember having to design databases where a table packed everything into a row that was 2k or less. Writing apps in C one would need to write queues and stacks and ponder using the verb register for which variable. Declaring a pointer against a pool of memory that was the size of a thimble of water compared to todays 4-person hot tub.

But, still there are many things todays developer can do to optimize business apps. For one, minimize disk access. I see programs all the time that invoke multiple queries or stored procedures when only one is needed. Also, there is also nothing wrong with using integers as foreign keys in database design. Using quit loop functions once you have what you need can be a time saver.

A lot of optimization of course depends on matters like how many concurrent users your app will have to contend with. And, what is the intended use of the app is another prime consideration. There are many more.

I would be the first to admit that if you are writing an app that will only have a few users saving 10ms access time through varying your coding practices is not worth anyones time.

I remember many years ago a peer looking at a Perl program I had written and giving me an obtuse (in my opinion) way of doing the same thing. Well, first off, my program was a command line batch program that was going to be run maybe once a week at the most by one programmer on their pc (it was a code generator. I won't bore you with the details). But I was curious. So, I went back to my desk and created two scenarios. One coding my way, one the way I thought was obtuse, I ran them in loops executing 1 million times using a hi-res timer to benchmark performance. And, the dude was right-on, his was faster. By about 15 milliseconds total under my test scenario.

For my intended use it made zero sense to change the code to a coding convention that seemed very foreign to my way of thinking. Plus, I thought, hey the single end user of my app may not have time to go out and have a beer and get back fast enough with time savings like that. We must always be considering the needs of our customer.

On the other end of the spectrum, during the era when client server apps were in-vogue I saw some fat client apps written that would create and dump (errr...I mean deliver) 3-5mb of data to the client in one fell swoop, taking a couple of minutes to load. Hmmmm, I remember thinking the customer will really dig that. And, this story is from a time frame when 16mb was a standard physical memory footprint.

Today is such a cool time to be computing in. One can consider true multimedia solutions in their design and not have to worry about needing to sell the customer a cray to host the app. The emergence of mobile devices (even with their evil proprietary languages) is way cool as well. Streaming video still impresses me.

And, knowing within ten years that someone will figure out how to mainstream a monitor you can see in your glasses, and a virtual full sized keyboard to type on that can be tied to your handheld. I can just picture someone on the bus coming home from work (that would not BE ME!!!) madly typing away some flame email on a keyboard none of us can see. Man, I can't wait for that day :-)

Saturday, August 21, 2010

Developing a Web Development Framework - Walking the Walk

(Actual Written on 7/31/2010)

Software architecture and design is and has always been important to me. A big part of of this is implementation. It is easy to code a hack. We all do it. Time constraints, money, whatever the reason making a hack is easy to do.

After one does enough gigs they should come to a realization that doing the same work over and over again can be simplified or organized in a way that makes life easy, shortens the development cycle, is easy to manage and maintain. With my alchemy web development framework this is a realization I am trying to achieve.
My main design goal regarding implementation is really founded on two main objectives.

1. Be able to quickly implement alchemy anywhere.
2. Easily update an existing alchemy implementation without breaking anything.

Alchemy is in alpha, but is already becoming a monster. The application framework alone is seventy directories and over 550 files. To makes matters just as interesting I host the framework on different posix servers, but my development environment is Windows.

I have just created a Sourceforge project to manage the builds. Right now I have started to move the content and documentation into my developergeekresources.com site. I am having to implement alchemy there as well in order to offer up the variousd templates and toycode examples.

Ultimately I want the alchemy framework to replace my four active websites. Replacing an existing legacy site is easier said than done. I want to make adding new content to them even easier than it currently is.

If you cannot effectively use your own stuff why would you expect anyone else to use it? Right now what I have is still rugged in certain areas. The only way to really see if you are on the right path is to use your own stuff. Too often software has become a modern factory where people that are producing the software never have to use it. Most do not even have to implement it, so they can be totaly insensitive to that discipline as well. A buddy of mine likes to say "fish or cut bait". I like to rephrase it to say "cut bait, then go fishing".

I must admit I am so busy my inclination and nature is to just slam something in and get back to work. Not such a good strategy if you are trying to build an entire framework. You gotta always be thinking about shit like "what will that change do to all the existing stuff?" and "how is that going to be able to implemented in a way that is extensible a year from now to include all the cool new stuff myself or someone else has conjured up that I have yet to even think about or consider.

Thursday, August 12, 2010

Open Source and Google

The other day I was building a portion of an app that dynamically generated charts based on the results of a query. Worked like a champ in my local development environment where the server and the browsers I use both reside on my laptop.

Then I uploaded it to my test server and the chart did not update without having to refresh the browser. Of course, the browser was caching the image and even though the data was changing, the file name remained the same. What to do?

Get the end users to lower there expectations?
Call it a WAD and modify the requirements?
Read a manual until I went into a coma?

No

I Googled: php image not refreshing   ...

... resulting in a link on page one that took me to the solution. (and for that, they deserve a link.)
http://forums.powweb.com/showthread.php?t=60177


Generally, I can find an answer to my questions through Google spending only five or ten minutes. One of the most profound and unheralded innovations in the discipline of computer science is the marriage of search engines and open source software.

Sure beats the days of being dead in the water for days or weeks, getting bounced around and listening to muzak on support lines, and then having to come up with convoluted workarounds because you could not find a solution.