Illusory Follies Andrew Flanagan's Blog

19Mar/150

On Mountains and Software

Way up in the hinterlands, miles from any other humans, two men faced each other on a rocky ridge covered in lava flows and avalanche lilies.

“It was clearly a suboptimal algorithm!” the taller man yelled, with his arms outraised.

“Everyone knows that premature optimization is the root of all evil” the short, stouter man parried. “If we had known all the details of the operational environment at the time that we had implemented the original code, we might have made different decisions, but ultimately, it doesn’t matter. The code worked and the solution was a good one -- at least for the first iteration.”

“The first iteration?! What is with you and iterations of software releases? Why not spend the time to do it right on the first try. Now your code is out there and it’s ridiculously slow -- the whole company looks bad when you deliver crap like this.”

The short man leaned against a large basalt pillar that rose almost unnaturally out of the mountain, like an ancient stella, erected to point out the glory of this lofty pinnacle. “First, make sure it works, then make it fast.” he said slowly. “Let me ask you this, do you think that the company would have looked better or worse if we had spent an extra 2 or 3 weeks on the initial release to make it faster but the whole thing had a fundamental logic error?”

The tall man writhed, “Why do you assume that we would have introduced logic errors? The code was pretty straightforward, the algorithm wasn’t that unique or novel. It was probably new in some sense, but really just a collection of existing algos that were assembled in a new way. What makes you think it was such a hard problem to begin with?”

“Because humans are imperfect. I’m not even saying that there aren’t any logic errors in the first iteration -- err, the initial release, or whatever you want to call what I just wrapped up. What I am saying is that we’re building from a more structured base. We have good confidence that it works now due to the unit tests that we put in place. Sure, it’s slow, but we can always work to improve that speed; then we can regress over the unit tests to ensure that the underlying foundation is solid.”

The taller fellow shifted his feet on the hillside, lava rock crunching like broken glass under his boots. “I think those unit tests are a waste of time. Out of the two weeks you spent building the solution, you spent almost a full week on those. Some testing is great and all, but come on! You spent thousands of dollars on code our customer will never see!” He shifted again and this time a clatter of jagged rocks slid down the hillside and off the edge of a cliff about 30 feet below.

“Our customer doesn’t see the code for the unit tests, but it’s there for us to ensure that everything’s working. Tell me this, Bob: how would you have shown the customer that the code worked as they expected?”

“I would have let them watch it run -- watch it digest data and then view the results! That seems kind of obvious… how else would they accept it. Sometimes I don’t get you’re thinking, Fred”.

Fred kicked at the base of the column he was leaning against, flattening out a section of gritty soil, flecked with chunks of lichens that had fallen from the monument he now rested against.

“The tests are nothing more than the reflection of the requirements. The function that the customer described in their requirements is accomplished and realized by the collection of the algorithms and code segments that we built and assembled. If each of the blocks of code that we assembled works perfectly with regards to its own inputs and outputs, and if the code segments are assembled in a way that logically generates the expected output from the input to the entire routine, we can safely say that we’ve tested the whole thing when all unit tests are green. Sure, it won’t take into account the details of how much they like it overall, the architectural “goodness” of the solution, or the way it looks, the fonts, the UI, or even the speed, as you’re harping about. But if you remember, the customer never gave us any non-functional requirements like what I’m talking about. What they asked for was an algorithm and that’s what we built.”

Bob put his hands on his hips. “So you’re saying that the customer isn’t going to like what you delivered, and you anticipated that, but you still delivered it?”

Fred sighed. “No, but I am saying that the customer maybe doesn’t know exactly what they wanted when we started the project.”

“Oh, so now you’re smarter than the customer? Sounds like typical engineer ego.”

Fred squatted in the hollow that he had shovelled out with his foot and leaned back against the softer, mossy base of the rock, facing north. “I may be smarter or I may not be. That’s not the issue. The issue is that my job is not just to give the customer exactly what they say or to try to tease every possible requirement out of them at the beginning. My objective is to make a customer delighted with the software that I’ve written for them. It should meet their business needs and it should be useful, and it should be pleasant to use. That’s true of all software -- that’s true of all products. What my customer wants is to solve problems, and that’s why they agreed to pay me money.What I am saying is that right now, my customer has a piece of software in front of them that works. I haven’t said I’m done, I haven’t said that the software is perfect, I’ve merely presented them with a quick release and asked them to take a look and then talk with me about it.”

Bob dropped his arms from his sides and attempted to squat on the hillside, resulting in more lava rocks tumbling down into the ravine below. He finally settled with one foot pointed down the hill, and the other uncomfortably folded under him and pressed up against some unpleasantly sharp looking chunks of obsidian. “I just don’t get it. You want to talk with your customer about slow software that doesn’t look pretty and only runs in an ugly console. Don’t you think they’ll be pissed with what they see?”

“It depends. We talked before I started work on the project. I explained what I was planning to do. I explained that I write software in iterations. They may not be used to doing business this way, but I think that their expectations are exactly in line with what I’m giving them. They may doubt in the approach I’m taking, but I didn’t lie to them and I’m not delivering anything other than what I said I was planning to deliver. They know I’m a professional, and they expect me to do my job.”

Bob raised his eyebrows. “So you think they’ll be OK with this? Aren’t they just going to add a whole bunch of those requirements that you were talking about? Speed? How it looks? How easy it is to use?”

“Probably,” Fred shrugged. “It doesn’t really matter if they add those things now. Like I said, we have a solid base. If they want a different user interface, we’ll add that. If they want it faster, we’ll optimize it. What I’m happy about is that the code that I’ve just delivered actually will work. If they wanted to, they could start using it now.”

Bob started to raise a hand, but the whole swath of rocks below him began to slide. Fred grabbed a branch of a small and unhappy looking Douglas fir that had managed to sprout in the crack between the basalt column and the hillside, and hauled Bob up closer to him. “Watch your step there! That hillside isn’t stable at all!”

Bob muttered a thank you under his breath and sat down as close to Fred as he could comfortable position himself. Both men sat for a while, staring at across the hillside and down into the seemingly endless valleys and terrain beyond. The sun was beginning to set, and off to the west the sky was lit up with a brilliant gradient of reds, oranges, and pinks.

“Did you know that sunsets are pretty mostly because of the pollutants in the air?” Bob asked. “If it wasn’t for humans burning things, the sunsets wouldn’t be nearly as dramatic. I’ve heard that it’s the imperfections in the atmosphere that give sunsets like this so much colors and variance.”

Fred nodded. “I’ve heard that too -- not sure if it’s true, but it sounds reasonable.”

The men fell silent again.

“It’s like switchbacks” Fred said suddenly. Bob looked up, uncertainty on his face. “What I mean,” continued Fred, “is that the iterative process I’m describing is like switchbacks on a mountain. You don’t just charge up or down the mountain. It’s too steep,. the footing isn’t sure. Better to progress from east to west, then west to east, then back again as you slowly gain elevation like we did coming up. It’s slower, but it’s more steady. Plus, if you’re talking about a path that you’ve never been on before, or that hasn’t been blazed by someone in the past at all, the movement back and forth gives additional perspective. As you traverse back and forth you get a better idea of the mountain you’re ascending. You can see whether your path is doomed to failure or if you can realistically reach the summit. Each switchback can feel slow and painful but you’re safe and you’re constantly able to see what the next steps are going to be. You’re constantly analyzing and re-analyzing while still making progress towards the top.”

Bob looked down. “I guess I can see something in what you’re saying. So you’re saying that in some ways, software engineers are like sherpas, leading their customers to the summit?”

“I like that,” Fred smiled. “It’s a lot like that. We know that our customers want a great experience, and we’re the professionals that know about how to get there. We don’t want to promise them a spectacular view only to find that they were hoping to see Mt. Rainier and we we’re leading them up a slope to see across the Puget Sound. We could ask them a million questions about every last detail before they begin in a questionnaire. Things like ‘do you like flora more than fauna?’ or ‘do you prefer trees to rocks?’. But for a lot of the people that climb these hills, they wouldn’t even be sure what you mean. What types of flowers are there? What types of trees? They need to experience a part before they know what they want in full. Best to give them something good soon and then tailor the rest of the hike based off of how they react to what you’ve given them.”

Bob laughed. “I never really thought of it like that. I guess I can buy that. What about all that stuff you were talking about unit tests in code? Does that have a parallel in your analogy?”

“Well, I guess all analogies are somewhat imperfect, but I see things like tests in this case to be the safety that you build in. You may end up crossing a particularly treacherous avalanche chute, like this one, you almost slipped down. It’s better to spend a few minutes either picking a really good path and marking it with a cairn. Or maybe you’re crossing a creek and you spend the time to stretch a rope across to make it a bit easier to pass through. You expect in all likelihood you’ll be back on this path at some point and you want to be sure that the path is safe and trustworthy. Unit tests are like that in a way. It’s not really wasted time. You end up safer the first time around, and also, you have more confidence when you return.”

Bob sighed. “So you think the customer will be happy?”

Fred picked himself up off the edge of the dry, dusty basalt block and stretched. “I think so, I really do.” A crepuscular pika poked his head out of a hole about 20 feet away for a few seconds, saw the men, and decided to call it a night. The sun was nearly down now and the two men had a bit of a trek back to their base camp in the valley below. “You never know though, some customers are never happy. Some people will never be content. Ultimately, you can’t please everyone.”

Bob scrambled up, brushing dirt and shards of rocks from his pants. “I guess not.” he said.

Fred turned and starting walking back down the rocky path that they had come up. He paused, as they passed the first switchback on their trip down. “You know, Bob, even though hikes aren’t always as much fun as I thought they might be, and even though you end up with blisters and scratches and maybe the mosquitoes are biting, I still enjoy them.”

Bob just smiled.

12Mar/112

Church Software, First Steps

Trying to do a little brainstorming here for the church software. I did just a little research and found that sure enough there are tons of products out there. The problem I'm seeing is that the interfaces seem a little less than useful and many of them are somewhat pricey. What I'd like is a "roll your own" option for any of these. Don't sell me the service or the product, sell me hosting and management if I can't do it myself.

Some new ideas based on what I found:

Facilities Management: would be nice to add. Isn't really fundamentally different than event planning and scheduling but could provide checklists, reminders, inventory (for consumable products), that sort of thing.

Childcare: Could easily integrate a check-in system for the nursery with all the data that we have.

Background, user meta-information: Useful for programs like Reducing the Risk. Basically, the ability to look at things like when a member has completed some necessary training or certification. Sad that this is needed, but it is.

Financials: After working on the budget committee this year and looking at other software, I believe that this is pretty straightforward. The big deal would be incorporating a tracker for two primary things:

1) Incoming checks and cash. Allow a weekly input where a user's check is recorded (possibly even scanned), the amount is recorded even before it's deposited with the bank. (With newer banks, it might be possible to simply upload check images). It would be very nice if users could check their own contributions online by logging in. Alternatively, any church administrator would have the information available without moving/copying Quickbooks files.

2) Outgoing checks and expense reimbursements. At least at our church we have a fairly small amount of accounts for expense tracking -- this would be easy to import and manage. Granted, this information wouldn't be available immediately in a full-fledged piece of software like Quickbooks, but what we need week-to-week and month-to-month would be adequate. Would provide immediate feedback to accountable parties for expenses that are being made rather than waiting for an end-of-month report.

3) Both of the above lead to budget tracking. With each account, if we have a weekly, monthly, or yearly budget, we can easily track actuals against budgeted amounts.

Anyway -- this is all well and good. It definitely, hugely increases the scope of work.

As a first step, I'm looking at trying to start with

1) Authentication system

2) User management

3) Scheduling system (rotating assignments with exceptions, email reminders, rescheduling/adjusting by admin only)

For our church, this is the most critical need. I'm going to be working on some models and some basic system requirements and we'll see how far I get. There's nothing there yet, but I've set up a new website here: http://steepleproject.org/ The plan will be to start formal documentation there, code repositories, etc.

If you have any thoughts or suggestions, please leave a comment!

20Jan/110

Church Software

I begin this post having spent almost no time looking at what options may already exist. However, I see the need for a piece of software (preferrably open-source) that could accomplish some of these tasks:

  • Manage church events and calendar items (ideally through interaction with a solid interface like Google Calendar, 30 boxes, Airset, etc.). Would be nice if event planning could include RSVP-type capabilities to alleviate manual planning (e.g. "Are you attending this Saturday? Click here...").
  • Management of scheduling/duties at a church (Similar to many calendaring options but perhaps a little more "duty roster" focused). Some features might include automatic rescheduling via email (replying with a "cancel" to an event would trigger a notification to admins to find a replacement -- that sort of thing)
  • "CRM"-like capabilities for sending emails, announcements, etc. to the church body while providing church leaders with the ability to make notes, add  prayer requests and other very church-specific type features.
  • Some sort of document management (possibly again through simple integration with an existing, reliable document management source) for things like procedures, newsletters, meetings, etc. as well as every-week things like bulletins.
  • Newsletter options -- email and web version of all "newsletters". Providing users with the ability to opt-out of email listings but also view old newsletters archived on the website
  • Audio handling. Some interface that would streamline the ability to upload and manage sermon/lecture/class recordings that doesn't require too much technical knowledge.
  • Budget-related financials -- not a full-blown piece of accounting software, but that something that could provide instant feedback to leadership on details of the budget and perhaps summary information to members.
  • Online church directory (not public)
  • Private portal (I know "portal" is such archaic jargon) -- but some sort of interface that would allow more insight to member-only items (like the church directory) as well as details like Annual Reports, etc. that may not be intended for the public.

Some of the design goals would include:

  • Lightweight -- keep it simple and straightforward
  • No "registration" required -- most features would be available to members even if they don't want to register on the site.
  • Solid and reliable authentication
  • Extensible -- I'd be building this for our church, but it would be nice if the concerns of both larger and smaller congregations with different leadership styles could be taken into account.
  • Possibly integrated into an existing CRM (like WordPress) as a plugin.
18Oct/080

Coding Well

I don't usually do this, but here's a verbatim excerpt from a post on LifeDev that I just love. Having goals in the pursuit of software engineering and design is a good thing. I've heard most of these expressed before but not as well as this. (original post by glen on July 23rd, 2008)

Programmers are a great example of thinking outside the box because, well… programmers can actually create the box. Here are some of the ways that we can learn from programmers on how to boost our creativity.

1. Learn a new language

Programmers are constantly learning new languages, either for fun or necessity. Don’t limit yourself to what just what you know and are comfortable with. Branch out and learn a new skill.

2. Start from the ground up

If you’re going to write software, you can’t just start halfway through the project. You have to start at square one. Sometimes this is the best way to find a creative solution for a problem is to go back to the beginning and work forward.

3. Question everything

Questioning everything means taking every assumption and making sure it’s correct. All programming starts with making the most basic assumptions, and then building on those basic assumptions. If something is wrong with the code at the base, then the software isn’t going to work well at all.

Sometimes creativity is limited by assumptions. New solutions arrive when we tear down assumptions and start with fresh perspectives.

4. Do it for fun

If you know any programmers, they’re constantly building something. Even when they’re done for the day on work-related projects, they’ll spend hours of time working on fun projects for themselves. Their work is also their hobby.

Continually mulling over new ideas and solutions is something that shouldn’t be a chore. It should be something that you find yourself doing constantly, like a reflex. And it should excite you.

5. Never stop testing ideas

Programmers are constantly benchmarking code to make sure that it’s as efficient as possible. Even the smallest change can bring a program or Web site to it’s knees, so constant testing and improvement is important to any bit of software.

Ideas should be tested rigorously and refined on a consistent basis. Your ideas will change over time, it just depends how much. Constantly evaluating them and just plain thinking them through is a great way to “benchmark” your idea.

6. Find a passion

If you’ve ever spent more than two minutes talking with a programmer about his work, you’ll find out very quickly that programmers have a passion for what they do. They eat, sleep and breathe programming.

Do you have a passion for your ideas and projects?

7. Master your tools

Programmers constantly improve their knowledge and usage of their tools. A great coder keeps tabs on software and is constantly finding ways to improve his usage of them. You’ll seldom find a programmer who doesn’t tweak his toolbox regularly.

No matter what your skill set, you’re limited to your skill with the tools you use to create. The more of an expert you are with your tools, the more you’ll be able to create.

8. Start making abstract associations

The people behind projects like Skype, Google Docs and Twitter all have one thing in common: They fused seemingly abstract concepts together. Taking what-ifs and testing them is a great way to start thinking of things in a different, more creative light.

9. Think of structure as a tool, not a limitation

People associate creativity with taking a giant, blank canvas and letting our ideas flow without any sort of limiting structures. However, there’s a huge problem with this type of thinking: It’s a great big creativity myth.

See, limitations are everywhere. We can’t avoid them, we can only hope to work with them. A programmer embraces the limitations of his programming language or tools and works around them. These limitations help him as they make a foundation to work from. Sometimes discovering a new workaround will lead to an even bigger idea. Necessity is the mother of invention.

10. Don’t rule anything out until you try it.

Your kindergarten teacher was right: There is no such thing as a stupid question. If you’re adhering to #3 and dismissing all assumptions, you can’t be certain it’s not going to work until you’ve tested it. How do you know it won’t work unless you try it? You might be surprised. Even if the proposed solution doesn’t work, it may help you find a solution.

Sometimes it’s just best to start with a prototype and try it out. If your prototype doesn’t work, then trash it. If it does, you’ll have stumbled upon something that just might work.

11. Always look for a simpler and more elegant solution.

A good programmer is one that understands that finding the simplest solution is always going to be better. Complicated solutions lead to… complications. A practical approach to programming always works best in the long run.

Our ideas sometimes become too complicated. We get caught up in the novelty of the idea that we ignore how practical it really is. The simplest way to solve a problem is often the best way to solve a problem.

12. Don’t be afraid to build off the code of others.

The beauty of the Internet is that the solution your looking for has probably already been done by someone else. When building a new site I almost always use pre-existing open source code. Why recreate the wheel?

Putting a great idea into motion doesn’t mean you have to start from scratch to create it. Use existing ideas and turn them in to something better. Sometimes a great idea is only modifying something that’s already been done. Gmail is a great example. They “reinvented” email by adding useful features to traditional email.

13. Don’t be afraid to collaborate.

Some of the best coding — or any creative projects for that matter — are done not just by one coder but by many excellent people inspired to work toward the same goal. Assemble a great team, use the most brilliant ideas no matter who they come from, and let everyone contribute.

14. From the very basic, create the beautiful.

Programmers often use some very basic code over and over, and while those small bits of programming language aren’t necessarily beautiful in and of themselves, they can come together to create a final product that is amazing. No matter what creative project you’re working on, pay attention to the details, but most especially pay attention to the effect those details have on the overall picture.

26May/080

Papaya

Well, I don't normally function as an advertising agency, but I thought I'd mention to anyone using OS X that there's a splendid new application out by Lighthead Software. It's call Papaya and it's purpose in life is to make sharing files easy and effortless.

Sharing can be such a pain because of problems between Windows and Mac or even just silly things like firewall rules, security, etc. It would be nice if there was a reasonable way to share files that was:

  1. Straightforward -- don't make me click around a lot
  2. Easy to share -- let me IM or email something that people can easily reference in order to access the shared material
  3. Used standards that work regardless of your friend's operating system

Papaya seems to be the answer.

Sharing files locally is automatic. Depending on your router/firewall, sharing files on the Internet may be just a tiny bit complicated but you only have to configure it once. For my own needs, this is beautiful. I can drag and drop files into Papaya, instantly get a link to share with friends and be on my way. No need to email large files, worry about acceptable formats or whatever else.

Papaya is priced at €20( $31 USD as of 5.26.08). If you have a Mac and need to share files, I definitely recommend this.

Also, Lighthead Software also makes the extremely handy Caffeine application that will keep your laptop from sleeping, having the screen dim, etc. It's a free program and it's operated with just one click on turn on, one to turn off. I use it often.

To Lighthead Software, thanks for some excellent programs!

22Jan/082

Computer Science vs. Programming

There have been a few Slashdot submissions here and here. They're concerned with an article published by two professors from NYU that assert that Java (and similar high-level languages) are damaging to teach as the "first language" of a Computer Science education. Since I wasn't a real CS major, I'm perhaps a little outside of this discussion. However, I cut my teeth on C/C++ at school before moving on to the high-level languages (really just .NET and some very high-level languages like Ruby and Python).

I tend to agree with the conclusions. It's not that there's no place for Java. It's just that without the fundamentals of pointers, memory management, and basic understanding of the construction of complex data structures which are just handed to you with Java or .NET, it's very difficult to fully comprehend what you're doing

I had a very good professor that taught algorithms and data structures at school and although at the time, the experience was painful, I'm sure it has helped immensely. Despite my affection for things like Ruby on Rails which is extremely high level, I'm annoyed sometimes because of the indeterminacy of functions and the vagueness of the specifications. When you write a language that can do powerful things in one line of code, you're taking a lot of shortcuts and it can be surprising when a function returns something very unlike what you expected do the complexity of the underlying code. Basically, you ignore things like sorting algorithms entirely in favor of the "built-in" sort routine. How does it work? Well, you can dig it up in the code, but most people will simply use it and assume that it's the fastest for all of their needs. What happens is that writing code becomes an assembling of pre-built components. It reminds me of "building" Ikea furniture. Granted it takes a certain amount of handiness to put together your new desk but you're not gaining skills that you can use to build anything yourself without first being handed the pre-built pieces.

I tend to think of myself as primarily a Software Engineer. I'm not just a programmer because I do a lot more than write code. But I'm also not much of a Computer Scientist because I spend very little time actually attempting to improve upon techniques and mechanics of processing information. These definitions are a little vague, but I feel that Software Engineering is more what I do because I apply creativity to the process. I think one can be a Computer Scientist and a Software Engineer but I don't think my work normally falls into both categories. I've always found the role of a traditional Architect to be similar to Software Engineering. It's an application of creativity (design, color, texture, material, etc.) to a field of science (physics) that results in [hopefully] useful buildings. There are some "cutting edge" Architects that attempt new and innovative projects but most Architects are working with existing ideas and applying them creatively.

I've heard that Frank Lloyd Wright's buildings although amazing in appearance and remarkable in their artistic qualities are often problematic in simple ways. Flooding basements, leaking roofs, etc. were the result of a poor implementation of a great and artistic idea. It's not enough to be artistic and creative; a good system like a good building works and functions as it should in addition to its aesthetic qualities (which make it unique).

I've always seen this distinction between implementation-focused approaches and theory-focused approaches. Implementation is desirable for the production of new applications and system but will always be held back by advances in theory. It seems that Computer Science has largely lost its way in North American schools by focusing too much on implementation without teaching theory. Programmers are cheap. It doesn't take a lot of brains to assembly code from pre-built components and creativity often is the only difference between a good programmer and a mediocre or poor programmer. Without new advances in theory, applications and systems will simply have to stand on their desirability of implementation (i.e. how easy is to use?). New ideas must be infused into the process for real advances to be made.

The use of so-called AJAX seems an interesting example. The ability to use things like the XMLHttpRequest object were available for quite some time before companies like Google began using it to do amazing things. This is entirely focused on implementation. Web 2.0 applications (whose primary distinction seems to be AJAX technology) are an innovation in implementation only. Many "hard-core" programmers find the terribly sloppy and inefficient results that often result less than satisfying. It does cool things but isn't there a Better Way? I use AJAX quite a bit these days and it's handy. However, I have only a bare understanding of how it works and what might be a better design. I don't tend to concern myself with the next evolution of the Internet-- I focus on building things that work with the technologies that now exist. But AJAX really isn't a huge advance -- in fact its "magic" often results in massive security holes, odd and unpredictable behavior, and hugely increased server overhead.

At the same time, a Software Engineer who truly understands the science of the code that he writes is likely to make far fewer mistakes and write much more efficient code. Even without much creativity, a programmer who can optimize code is a desirable catch for any software company. I think that everyone should understand the underlying details of code even if some end up focusing on the creative, implementation focused approach or the theoretically, algorithmic approach.

I mentioned security in regards to AJAX and this seems important. It's well and good to provide applications that do the same things in easier ways but without a strong cadre of Computer Scientists, developing faster, more secure, and more reliable ways of doing business, we end up with applications that are never properly tested (it's difficult to test code that just does magical things!) and never adequately secured.

A little rambling of a post -- hopefully I've managed to convey something. Your comments welcome.