RSS

First Impressions Matter

When it comes to researching vendors, first impressions matter so much.  I tend to judge any potential vendor by its sales apparatus, not just because it is the first impression, but because that positioning and interaction will tell you so much more than any press release, executive ‘corporate culture’ communication, or other third-party source of information on financial or industry strength.  Things I notice right off the bat that influence my decision to continue engagement or build trust:

Is the sales channel optimized?

Building great companies and great products is all about optimization at a later stage of an organization’s maturation life cycle.  Idea-driven founding staff are joined or replaced by data-driven staff as a company’s offering is validated and it grows to benefit from economies of scale and to show profitability to patient investors and equity holders.  The distance between my interest and the vendor’s name recognition is a marketing issue, but the distance between my identification of a vendor and getting a meaningful response from their sales organization is a sales/company issue.  If I’m clicking through a brochure-ware website to find the place to start engagement, filling out a general ‘Contact Us’ form, navigating a tedious phone tree, or heaven-forbid, clicking a ‘mailto:’ link to type my interest, then I’ve already learned a lot about your company.  I’ve learned one of the following statements is true:

  1. The number of client contacts you deal with through this channel is relatively small: you are new or slow to acquire customers through it
  2. Your company is too focused on the ideation and ‘fun’ phase of the business to optimize your sales channel – your company may not be mature enough for my needs
  3. Your company is too focused serving existing customers (keeping the wheels on) to work on growing your business by optimizing sales channels – your company may not be ready for my needs
  4. Your company is mature but not thinking about data-driven results, which tells me your product probably isn’t either.

What is the quality of the first contact?

Did the person who responds to my inquiry bother to look up the domain of my e-mail address to check out what my company does?  Does that sales executive reference recent PR releases we made?  This is a high-quality contact and this action shows me your sales executives aren’t quote-monkeys or order-takers, they are relationship-builders.  Or did I just get a form letter thank me for my form entry and letting me know someone may get back to me about whatever my interest might be?  If it is the latter, this tells me:

  1. Your company will require me to tell, and you probably won’t ask.  I’ll need to know what I want and be prepared to demand.  Since from the start of the relationship, there was little concern for finding a good fit, I will have extra heavy lifting to do.
  2. If you are asking what my interest is and you don’t already know, then that probably means you haven’t placed me in any segment or internal classification that represents the nature of my potential demand.  That tells me the out-of-the-box customization of the solution may be low, or if not, you are not capitalizing on the specialized needs of different classes of customers.
  3. If I get an “I don’t know” in the first conversation, that is okay, but it tells me I’m either working with someone that does not know their product well (new or inexperienced), or the sales group is not connected to the product group, which is a more fundamental problem.  The most important communication line is (in my view) between sales and product, and secondly between sales and operations to ensure in order that: (1) pre-sales the right solution is sold to a customer … if that doesn’t happen everything else will fail … and (2) post-sales the requirements are appropriately communicated to deliver a synchronized expectation and final result.

What is the speed of the first quality contact?

  1. If I get a poor-quality first contact very fast, I presume I’m talking to someone young and hungry.  This can be a good sign if I need a lot of attention or customization and you’re not a large player.  This is a very bad sign if you have a signature single product and are an established company, as I assume there’s inadequate sales training or high sales churn, both of which send a negative signal about your company’s position and our potential together.
  2. If I get a high-quality first contact very slowly, I’m not thrilled, but I’m willing to wait and pay for quality.  Not everyone is, but that’s how I do business.
  3. If I get a poor-quality contact very slowly, you really shouldn’t be in business, and you probably won’t be anymore very soon.

 

 
Leave a comment

Posted by on August 25, 2015 in User Experience

 

Alkami: Genesis

In the summer of 2008, I was preparing a large strategic product shift within Myriad Systems, Inc. to unify a suite of ancillary banking productions I had built and managed: remote deposit capture, merchant capture, expedited payments, e-Statements, e-Notices, check imaging, and a one-to-one marketing solution among many others. A key opportunity presented itself in that we had several large and progressive financial institution clients that were interested in what an MSI online banking offering could look like, particularly given the relatively poor user experience in the online banking offerings at the time.  This would have completed a big piece of the end-user product portfolio for MSI, and while as daunting as online banking from the ground-up is, it stood to provide substantial strategic value to our whole suite.

Computer Services, Inc. began courting MSI and started a full acquisition in August of 2009. It was clear CSI’s intent was to maximize the value of the print and mail operational assets of MSI, but it had little interest in its online banking products other than to preserve existing revenue streams. This disinterest in the strategic vision of the online web applications as a product portfolio was the impetus for me to pursue my personal career interests of building a best-in-breed online banking solution outside of the MSI umbrella.

Jeff Vetterick and Richard Owens, two industry colleagues that had previously had stints at MSI, reached out when they heard of my desire to continue to build online banking and move on and encouraged me to reach out to Gary Nelson, an acquaintance who was part of the very successful build and sale of Advanced Financial Services to Metavente (an interesting and great story in of itself), who had interest in this as well. After AFS, Gary had many interests and projects, a significant one being part of an idea to build a learning management system that provided tools for schools to impart educational content in an online tool where students would have a fictitious bank account balance and through different learning modules, understand concepts of spending, budgeting, and the time-value of money.

When I spoke to Gary in September, I found this initiative was in wind-down: the project had exceeded its funding, and only an IT manager had been retained as a temporary contractor to document and turn over all the company’s assets. Gary engaged me as a consultant to perform an analysis of the source code developed by that team to determine if there was any value in it as an asset for sale as the company was closed up. I reviewed the company’s source and patents, but when I started looking at the few cloud VM’s and pulled open the Subversion repository where the source code was to be, I found a shocking lack of value: what did exist were some architectural documents and some demoware in the form of static screens coded into a .NET MVC ‘shell project’ that had no actual implementation or integration of the key concepts around educational content delivery and assessment. Looking back at the Finnovate presentation the team from this company did, I found only that minimal proof of concept presented on stage, but little more complete.

The internal company documentation in the form of ‘wikis’, agile storyboards, and some unorganized developer notes showed no cohesive technical direction or architectural plan. When I began reviewing invoices for consultants and local contractors, a sad picture materialized: I felt Gary and other investors had been somewhat duped by a mixture of technical ineptitude and probably some overbilling greed by people and local development ‘firms’. I delivered the news that what assets I could find and review had little fire-sale value, other than perhaps one patent that had some intrinsic value, but no implementation. I exemplified this situation by opening the source code for the portion of the system that purported to calculate a ‘relationship score’ about how much an end-user understood financial literacy content and how their behavior in their accounts, transactions, and progress in meeting their financial goals; the source code simply ran in an endless empty loop, doing nothing. Demoware.

After delivering the news to Gary and preparing for whatever my next endeavor would end up being, Gary suggested I reach out to Stephen Bohanon, a consultant with Catalyst Consulting Group who had previous been a high-performing salesperson with AFS. After several discussions, it became clear Gary had an appetite to try a pivot in the financial technology web application space, and both Stephen and I were interested in building a world-class online banking solution – he as a formidably talented sales executive to build relationships and grow the organization, and myself to grow a technical team that would architect and build our next-generation online banking user experience.

And with no pre-existing source code, and only great ideas, tremendous perseverance, and some money (thanks, Gary!), we founded Alkami.

 
Leave a comment

Posted by on June 25, 2015 in Uncategorized

 

Security Advisory for Financial Institutions: POODLE

Yesterday evening, Google made public a new form of attack on encrypted connections between end-users and secure web servers using an old form of encryption technology called SSL 3.0.  This attack could permit an attacker who has the ability to physically disrupt or intercept an end-user’s browser communications to execute a “downgrade attack” that would could cause an end-user’s web browser to attempt to use the older SSL 3.0 encryption protocol rather than the newer TLS 1.0 or higher protocols.  Once an attacker successfully executed a downgrade attack on an end-user, a “padded oracle” attack could then be attempted to steal user session information such as cookies or security tokens, which could be further used to gain illicit access to an active secure website sessions.  This particular flaw is termed the POODLE (Padding Oracle On Downgraded Legacy Encryption) attack.  At this time this advisory was authored, US-CERT had not yet published a vulnerability document for release yet, but has reserved advisory number CVE-2014-3566 for its publication, expected today.

It is important to know this is not an attack on the secure server environments that host online banking and other end-user services, but is a form of attack on end-users themselves who are using web browsers that support the older SSL 3.0 encryption protocol.  For an attacker to target an end-user, they would need to be able to capture or reliably disrupt the end-user’s web browser connection in specific ways, which would generally limit the scope of this capability to end-user malware or attackers on the user’s local network or that controlled significant portions of the networking infrastructure an end-user was using.  Unlike previous security scares in 2014 such as Heartbleed or Shellshock, this attack targets the technology and connection of end-users.  The nature of this attack is one of many classes of attacks that exist that target end-users, and is not the only such risk posed to end-users who have an active network attacker specifically targeting them from their local network.

The proper resolution for end-users will be to update their web browsers to versions that have not yet been released that completely disable this older and susceptible SSL 3.0 technology.  In the interim, service providers can disable SSL 3.0 support, with the caveat that IE 6 users will no longer be able to access sites with SSL 3.0 without making special settings adjustments in their browser configuration.  (But honestly, if you are keeping IE 6 a viable option for your end-users, this is one of many security flaws those issues are subject to).  Institutions that run on-premises software systems for their end-users may wish to perform their own analysis of the POODLE SSL 3.0 security advisory and evaluate what, if any, server-side mitigations are available to them as part of their respective network technology stacks.

Defense-in-depth is the key to a comprehensive security strategy in today’s fast-developing threat environment.  Because of the targeted nature of this type of attack, and its prerequisites for a privileged vantage point to interact with an end-user’s network connection, it does not appear to be a significant threat to online banking and other end-user services, and this information is therefore provided as a precaution and for informational purposes only.

All financial institutions should subscribe to US-CERT security advisories and to monitor the publication of CVE-2014-3566 once released for any further recommendations and best practices.  The resolution for end-users of updated versions of Chrome, Firefox, Internet Explorer, and Safari which remove all support for the older SSL 3.0 protocol will be made through their respective vendor release notification channels.  For more information from US-CERT once published, refer to the Google whitepaper directly at https://www.openssl.org/~bodo/ssl-poodle.pdf

 
Leave a comment

Posted by on October 15, 2014 in Security

 

Alkami: A Retrospective

What a wild and crazy journey the last five years have been.

When I started this blog in 2009, it was shortly after I had inked a deal with an angel investor and journeyed down the road with him and my other co-founder and established Alkami Technology.  Against significant odds, this October marks the five year anniversary of a roller-coaster ride on up, which galvanized Alkami as the clear leader in the online banking space.  Before jumping into this endeavor, I was no stranger to walking products from idealization to realization or running enterprise services in a SaaS model.  But doing all that against the tremendous downside risks of the start-up world, as the new kid on the block among a world of established, very-well funded competitors has been challenging. Actually, it’s been brutal.

Reflecting on the past sixty months, I’ve started to pull together my notes from the early days, both before and after founding Alkami, and I will be commemorating this milestone with a series of blog posts on some company history – the why and how, as well as some valuable and hard-learned lessons along the way.  No one, no company finds tremendous success spontaneously.  While a Inc 500 splash piece on a company might portray success like a serendipitous fairy tale, only through a voracious appetite for risk, an iron stomach for failure, and a committed and skilled team does any great company find its footing.  It’s a great feeling to walk into the office every week and see new, fantastic talent we’ve added to our team and forward-leaning designs and concepts in our flagship solution.  It’s also a very satisfying one to know your personal efforts and sacrifices made that team and that company possible.

This series of posts will not be a beating of the chest or self-congratulatory account of our accolades.  Our work is far from over, and I judge success on a much longer time horizon.  But it will be a real account of our origin story, entrepreneurship, missteps and course correction, and moving from start-up to scale-out in a slow sales cycle, highly-regulated industry.  It’s one thing to have a hip product idea you incubate through an accelerator and debut on a demo day. It’s a very different thing to bootstrap a firm and an entire platform where you have to answer a few hundred RFP questions to get a prospect to even talk with you, many other steps to get just one sale, and many sales to get that kind of investor attention.

Those pieces are now in place and solidifying every day as we take an aggressive product and technical vision to its successful conclusion.  I’m honored to have found great working partners, worked (and still mostly continue to work) with some of the most committed and skilled people across a variety of disciplines along the way.  As we look back in retrospect at five formative years, I’m eager to chronicle our story and to add others who will extend and craft our bright future. Stay tuned.

 
Leave a comment

Posted by on October 1, 2014 in Uncategorized

 

Security Advisory for Financial Institutions: Shell Shock

“Shell Shock” Remote Code Execution and Compromise Vulnerability

Yesterday evening, DHS National Cyber Security Division/US-CERT published CVE-2014-6271 and CVE-2014-7169, outlining a serious vulnerability in a widely used command line interface (or shell) for the Linux operating system and many other *nix variants.  This software bug in the Bash shell allows files to be written on remote devices or remote code to be executed on remote systems by unauthenticated, unauthorized malicious users.  Because the vulnerability involves the Bash shell, some media outlets are referring to this vulnerability as Shell Shock.

Nature of Risk

By exploiting this parsing bug in the Bash shell, other software on a vulnerable system, including operating system components, can be compromised, including the OpenSSH server process and the Apache web server process. Because this attack vector allows an attacker to potentially compromise any element of a vulnerable system, effects from website defacement to password collection, malware distribution, and retrieval of protected system components such as private keys stored on servers are possible, and the US-CERT team has rated this it’s highest impact CVSS rating of 10.0.

Please be specifically aware that a patch was provided to close the issue for the original CVE-2014-6271; however, this patch did not sufficiently close the vulnerability.  The current iteration of the vulnerability is CVE-2014-7169, and any patches applied to resolve the issue should specifically state they close the issue for CVE-2014-7169.  Any devices that are vulnerable and exposed to any untrusted network, such as a vendor-accessible extranet or the public Internet should be considered suspect and isolated and reviewed by a security team due to the ability for “worms”, or automated infect-and-spread scripts that exploit this vulnerability, to quickly affect vulnerable systems in an unattended manner.  Any affected devices that contain private keys should have those keys treated as compromised and have those keys reissued per your company’s information security policies regarding key management procedures.

Next Steps

All financial institutions should immediately review their own environments to determine that no other third-party systems that are involved in serving or securing the online banking experience, or any other publicly-available services, are running vulnerable versions of the Bash shell.  Any financial institution that provides any secure services with Linux or *nix variants running a vulnerable version of the Bash shell could be at risk no matter what their vendor mix. If any vulnerable devices are found, they should be treated as suspect and isolated per your incident response procedures until they are validated as not affected or remediated.  All financial institutions should immediately and thoroughly review their systems and be prepared to change passwords on and revoke and reissue certificates with private key components stored on any compromised devices.

For further reading on this issue:

 
Leave a comment

Posted by on September 25, 2014 in Security

 

End-User Credential Security

This week’s announcement that a Russian crime syndicate has amassed 1.2 billion unique usernames and passwords across 420,000 websites would seem like startling news in 72-point font on the front of major newspapers, if it wasn’t sad it was such a commonplace announcement these days.  With four more months to go and still higher than the estimated 823 million compromised credentials part of 2013 breaches affecting Adobe to Target, it’s from Black Hat 2014 I find myself thinking about what we as ISV’s, SaaS providers, and security professionals can do to protect users in the wake of advanced persistent threats and organized, well-funded thieves wreaking havoc on the digital identities and real assets of our clients and customers.

Unlike Heartbleed or other server-side vulnerabilities, this particular credential siphoning technique obviously targeted users themselves to affect so many sites and at least 542 unique addresses affecting at least half that many unique users.  Why are users so vulnerable to credential-stealing malware?  To explore this issue, let’s immediately discard a tired refrain inside software houses everywhere: users aren’t dumb.  All too often, good application security is watered down to its least secure but most useful denominator for an overabundance of concern that secure applications may frustrate users, lower adoption, and reduce retention and usage.  While it is true that the more accessible the Internet becomes, the wider the spectrum the audience that uses it, from the most expertly capable to the ‘last-mile’ of great grandparents, young children, and the technologically unsophisticated.  However, this should neither be grounds to dismiss end-user credential security as a concern squarely in service provider’s court to address nor should it be an excuse to fail to provide adequately secure systems.  End-user education is our mutual responsibility, even if that means three more screens, additional prompts to confirm identity or action, or an out-of-band verification process.  Keeping processes as stupefying simple as possible because our SEO metrics show that’s the way to marginally improve adoption, reduce cart abandonment, or improve site usage times breeds complacency that ends up hurting us all in the long-run.

Can we agree that 1FA needs to end?  In an isolated world of controlled systems, a username and password combination might have been a fair assertion of identity.  Today’s systems, however, are neither controlled or isolated – the same tablets that log into online banking also run Fruit Ninja for our children, and we pass them over without switching out any concept of identity on a device that can save our passwords and represent them without any authentication.  Small-business laptops often run without real-time malware scanning software, easily harvesting credentials through keystroke logging, MitM attacks, cookie stealing, and a variety of other commonplace techniques.  Username and passwords fail us because they can be saved and cached just as easily as they can be collected and forwarded to command and control servers is Russia or elsewhere.  I’m not one of those anarchists advocating ‘death to the password’ (remember Vidoop?), but using knowledge-based challenges (password, out-of-wallet questions, or otherwise) as the sole factor of authentication needs to end.  And it needs to end smartly: sending an e-mail ‘out of band’ to an inbox loaded in another tab on the same machine, or an SMS message read by Google Voice in another tab means your ‘2FA’ is really just one factor layered twice instead of two-factor authentication.  A few more calls into the call center to help users cope with 2FA will be far cheaper in the long-run than the fallout of a major credential breach that affects your sites users.

We need to also discourage poor password management: allowing users to choose short or non-complex passwords and warning them about their poor choices is no excuse – we should just flatly reject them.  At the same time, we need to recognize that forcing users to establish too complex of a password will encourage them to establish a small number of complex passwords and reuse them across more sites.  This is one of the largest Achilles’s Heels for end-users: when a compromise of one site does occur, and especially if you have removed the option for users to establish a username not tied to their identity (name, e-mail address, or otherwise), you have made it tremendously easier for those who have gathered credentials from one site to have a much higher likelihood of exploiting them on your site.  Instead, we should consider nuances to each of complexity requirements that would make it likely a user would have to generate a different knowledge-based credential for each site.  While that in of itself may increase the chance a user would ‘write a password down’, a user who stores all their passwords in a password manager is still arguably more secure than the user who users one password for all websites and never writes it anywhere.

Finally, when lists of affected user accounts become available in uploaded databases of raw credentials that are leaked or testable on sites such as https://haveibeenpwned.com/ – ACT.  Find out your users that have overlap with compromised credentials on other sites, and proactively flag or lock their accounts or at least message to them to educate and encourage good end-user credential security.  We cannot unilaterally force users to improve the security of their credentials, but we can educate them, and we can make certain their eventual folly through our inaction.

 
 

When to Ride the Service Bus

One of the great things about adding new, senior talent to a storied team working on a large, complex, and successful enterprise application solution is the critical technical review that results in a lot of “why did/didn’t you do it this way?” questions.  You have two options for responding to those questions – ignore or passively dismissing them, or taking the questions seriously as a challenge to prove out why you would make a decision you and your team made 5 years ago the same if you had to consider for the first time today, in today’s frameworks, development methodologies, and the current team makeup and skills inventory.  If you choose to dismiss these opportunities to critically review your prior decisions, it says a lot about your management style, general appreciation of technology and response to its change, and positions your team to take a reactionary, defensive posture to architecture rather than create a team that honors a proactive, continuous improvement perspective.  Far more interesting too are those questions that ask why the system is architected in a general way, rather than a theological debate on whether a particular technology component choice is superior to all over or one’s preferred/familiar choice.

The particular question the new engineer asked was, “Why aren’t we using a service bus?”  Instead of answering him directly, I figured this as a good opportunity to explore the previous decision we made that not only did not include an enterprise service bus (ESB) in the original design, but rejected its inclusion when it was strongly suggested by our first customer because they were standardizing on a service bus-centric architecture themselves.  The primary advantage of a service bus is to layer an abstraction across heterogeneous systems by implementing a centralized communication mechanism across components.  By applying this architectural model, you can get some key benefits including orchestration, queuing to handle intermittent component availability, and extensibility points for message routing to alter dispatch logic or transform messages.  Implementing the service bus pattern requires some kind of adapter to be written for each component of the system, either as a local modification to each component or by choosing to standardize on a communication channel provided by the ESB.  Even in the latter, usually some minor accommodation is required to allow the ESB to receive and encapsulate the native message for delivery to the destination component. Our first customer was a notable player in the community banking market, and was productizing multiple new SaaS-based web applications that depended on data feeds coming from many different customers.  In their scenarios, data was consumed by one application, parsed, and delivered to other applications, which in turn may have created additional data feeds for other products, in a cyclic communication/dependency non-directed graph.  Each application was developed by different teams, and there was no unified technology stack adoption – some teams were developing on EJB and Flex, others were pure .NET, and teams generally had the discretion to choose whatever they could argue would solve the job, without a strong technology leader looking to unify the stack for similar applications that delivered CMS and pseudo-online banking functionality using a common input data set.

For this customer, ESB was a solution to a problem – their choices lead to a highly concurrent development process with multiple independent teams – but also supported connecting a heterogeneous environment of interdependent components, each of which accomplished limited objectives.  This organization was running red-hot – developing ancillary products to a highly engaged and fanatic client base of community banks, where their limiting factor was their speed of innovation and delivery.  By agreeing on a common communication mechanism that ESB could provide, there was something, albeit low-level, to which all teams agreed.  In the ‘controlled agile chaos’ they found themselves in, the abstraction bought them flexibility to adapt changing business requirements using orchestration.  In theory anyway – they ended up moving much slower than they anticipated, but this wasn’t the fault of ESB. ESB solves two classes of problems.  The first is the common use case of large, disparate enterprises looking to marry systems established from the dawn of client-server architectures to the newest Node.js hotness, without having to bend the will of any particular system to the communication conventions of any other, which may prove impossible if both systems are proprietary.  This is a common use case for BizTalk, especially in the financial sector.  All the other benefits you can name off from a service bus architecture are really secondary advantages to this key objective.  The second is the use case that any layer of indirection provides: an abstraction you can use to increase the speed of development when requirements are incomplete or prone to pivot.  In each case, you invest in a layer to reduce the cost of future change. This particular customer chose NServiceBus as their message-oriented middleware.  We seriously evaluated both the general architectural concepts ESB as well as the particular technology they suggested and came up with a definitive ‘no’ to that choice.  While it made a lot of sense for our customer, it did not make sense for us because:

  1. We did not require guaranteed event handling.  Our system connected to a system of record that provided transactional consistency, and virtually all state changes were initiated by users through a web browser.  A timeout was preferable to queued command handling system because of the possibility of duplicate transactions that frustrated users may initiate, not realizing their requests were queued.  Second, our interconnected systems did not provide guaranteed event handling, so the guaranteed provided by the ESB would now be honored end-to-end.  Third, we are using the Windows Identity Foundation with sliding time expirations end-to-end from the user’s browser through the lowest layer of service components, which doesn’t bode well for delayed delivery situations, even if the user was willing to wait.
  2. We do require transformation, but not orchestration between our components.  Our system features adapter-based design to allow multiple types of endpoints to be serviced by a single service implementation for those portions that may need to connect to a different type of third-party system through a provider model implementation loaded by dependency injection.  We could have chosen to use ESB for this piece, however, we perceived the long-term maintenance cost of multiple providers with the party-specific transformation logic to be lower than maintaining those transforms in ESB scripting or adapters.  In reviewing this perception today, I believe it was still the right decision because is allowed for us to unit-test our transformation logic without including the ESB.
  3. An ESB is a single point of failure that would independently need to scale for load exponentially proportional to the number of service interconnects in our solution, and would add some amount of latency between each. Because online banking is a mission-critical, customer-facing solution, it cannot have SPOF’s in any portion of the architectural design.  The SPOF nature of an ESB can be mitigated in multiple ways, but we felt that was at least two layers of complexity we could solve in other, simpler ways.
  4. All middleware increases the Mean Time Between Failures (MTBF).  This is not a risk specific to ESB, but of any layer added to a system.  If you add an ORM, IOC, ESB, or even a logging aspect, something can go wrong with them.  Each component has some small, but measurable failure rate, and when inserted into the communication chain between all components, its reliability of 99.999% still contributes to a reduction in the overall reliability of a serial system.  This is where the KISS principle shines – complexity creates unreliability, so all complexity must generate a compelling benefit in excess of its potential to fail.
  5. We wanted our application layer to be the platform, we did not want ESB to be the platform.  This was a business case / competitive advantage decision that we wanted to build as a feature of our system that the same services layer that supported our front-end user interfaces was also an open and extensible platform upon which our clients could integrate to, which would increase the overall value proposition of online banking not only as a sticky end-user experience, but also as a value proposition to capitalize on our solution as the middleware that marries together all the disparate systems within a financial institution, which ultimately online banking does like no other piece of technology within a bank or credit union.  We felt that by positioning everything behind an ESB, the perceived value of our technology piece would be lessened without additional client education.
  6. MSMQ made us feel dirty enough, and we did not want to mandate it for each component because it was in 2009 and still is relatively difficult to debug, and lately we have learned, queues do not work well with used with Layer 7 network load balancing.  The new hotness of 0MQ wasn’t around then, and while RabbitMQ was, it was arguably not production ready by that time.  For us, production-ready isn’t just whether a component is capable, but whether it will have general acceptance from the IT departments of our large clients – many newer technologies that are FOSS or from vendors without an establish track record require a ‘sale’ and buy-in during due diligence, long before ink is applied to a contract.  Even if they were options for the ESB queuing mechanism, they would not resolve the larger aforementioned concerns.
  7. At the time we made this choice, AMQP was an amorphous draft that did not solidify until later.  The lack of a vendor-independent protocol between components and an ESB made the choice to utilize an ESB subject to vendor lock-in, which we were not willing to tolerate for such a critical component.
  8. Because our product was both the end-user experience and the middleware we were writing, we felt strongly that the application protocol should provide descriptive metadata and support fast client proxy generation using .NET-based tools.  REST support was archaic at best (HttpRequest anyone?) in .NET 3.5, and to this day, consuming SOAP services is intrinsically more verbose in C# and VB.NET (HttpClient) than consuming REST or AMQP services due to a lack of better library and integrated language support for it.  Looking back on this, with a large amount of iterative change we went through from ideation to Version 1.0 of our solution, we could not have moved as fast without a fast way to regenerate proxies that would cause build failures to alert us of service operation signature changes — tracking these down at runtime (REST) or having to debug a secondary system (ESB) to find these would have bogged down our delivery timelines.
  9. A lesser concern was we felt that tracing SOAP messages, while definitely more difficult than REST, would be more difficult debug any issues in AMQP or other ESB encapsulation protocols than inspecting SOAP envelopes with built-in WCF tools already present in the .NET development stack.

So, that’s quite a case against an ESB, but they do have compelling uses for certain environments – just not ours.  Like all technology selection decisions, it’s important to pick the right tool for the job, and improve your tools as needed.  A standalone ESB would have provided significant benefits if we were developing with proprietary/closed third-party systems that were part of a call chain that required orchestration, or if we were developing with a heterogeneous mix of technologies.  In our case, we had a predictable homogeneous .NET environment based on web services, consumers of our API are our own technologies or a limited number of customers who were also using .NET, and we had no legacy baggage.  With the widespread adoption of WS-* standards, we have chosen to obtain some of the benefits, such as federation, from those standards rather than an ESB feature, which ultimately we believe makes our platform easier to support and distribute for our future API consumers.  Other side benefits such as logging are handled as separated concerns through dependency injection rather than external interceptors in a communication channel, a possibility for us only because we control the portion of the stack that requires orchestration.  And finally, by keeping all communication as SOAP over HTTP/HTTPS, we gain features like load balancing from Layer 7 network devices instead of an ESB process, which are much easier to switch out and upgrade.

The central design decision we made was that ESB’s provide some great features and that ties you into an ESB, but if we could get those features another way that was just as convenient or more so, we’d prefer the plug-and-play flexibility of leveraging existing solutions for components such as caching and load balancing in the environment our solution operates, or pick those pieces ad-hoc for those concerns rather than pick the best omnibus solution and work around any specific shortcomings for any one of them. In reviewing the current industry literature and blog posts and looking at general trends, it would seem our decision not to marry our solution is generally the path many take when not required to integrate legacy systems as part of an orchestration chain or when using non-HTTP based transport mechanisms.  If you’re using one, hopefully it’s for a good and necessary reason!  For us, though, we decided not to hop on a service bus that could take us somewhere we already arrived.

* As an aside, we actually did end up rolling our own small “ESB” as a TCP port multiplexer that queues and portions out connectivity to a socket-based, legacy third-party component that has no listener back-queue and no port concurrency, highly unusual for a server process.  Each connection consumes the port fully for the duration of the short transaction, and we had to write a way to buffer M number of requests and hand them off to (M-N) number of available ports as they became available,in a specialized type of producer-consumer problem. In hindsight, this was an opportunity to use an ESB, but in our case, we only required message routing and load leveling, and in a few hundred lines of code, we implemented what we needed for this particular third-party system what would have taken us far longer to do as our first time using an ESB. That being said, should we encounter this with another vendor, it would make sense to review using an ESB for this type of functionality in the future.
 
Leave a comment

Posted by on April 1, 2014 in Programming

 

Tags: , ,

 
Follow

Get every new post delivered to your Inbox.

Join 52 other followers