Why do tech projects fail and can we learn from past mistakes?

Two-thirds of large-scale technology programs still fail to deliver on their promises. Decades of lessons, billions spent, and yet the same mistakes repeat. Why? The Economist (May 2025) recently highlighted that IT initiatives continue to suffer from scope creep, poor stakeholder alignment, and weak execution discipline—patterns that repeat across industries regardless of scale . At the same time, newer trends such as AI bring their own challenges. A recent MIT study (August 2025) found that 95% of corporate generative AI pilots fail to deliver measurable returns, often because organizations chase hype rather than embedding AI into well-defined business processes. Together, these examples remind us that while technologies evolve, the underlying reasons for failure—unclear goals, lack of governance, and misalignment between strategy and execution—remain stubbornly persistent.

The Persistent Problem of Project Failure

Research consistently shows that large-scale programs such as ERP transformations, cloud migrations, or AI deployments are especially vulnerable. According to BCG, more than two-thirds of large-scale tech programs fail to meet timing, budget, or scope expectations. Panorama Consulting data, cited by BCG, shows that roughly a third of projects run over budget, with nearly 20% taking longer than expected . The impact goes beyond financial loss—when projects stall, they often create toxic environments marked by stress, misaligned priorities, and finger-pointing.

Lessons from the Trenches: Why Some Projects Beat the Odds

If failure is so common, the natural question becomes—why do some projects manage to succeed against the odds? The answer rarely lies in better technology alone. Instead, successful initiatives share a set of consistent practices:

• Clarity of Purpose – Projects that deliver measurable value start with clearly defined business goals, not aspirational slogans. Success depends on translating the “why” into practical guidance that every level of the team understands — from executives shaping direction to developers making daily trade-offs. Without that shared understanding, even strong execution loses direction.

• Disciplined Governance – Governance only works when it has teeth. Accountability must be supported at two levels: management backing to enforce priorities, and delivery leads enforcing checkpoints on the ground. Without both, misalignments grow quietly until they surface as budget overruns or scope chaos.

• Stakeholder Engagement – Engagement is not the responsibility of a single role — and pretending it is, guarantees failure. When executives set direction without owning trade-offs, or when architects aren’t asked to validate feasibility, no amount of goodwill can save delivery.

• Pragmatic Execution – Execution is where ambition meets reality. Teams that succeed align around achievable timelines, incremental wins, and open communication. True alignment across business, delivery, and technology prevents the endless cycle of adjustment and rework that derails so many programs.

Closing Thoughts

Project failure is not inevitable, but avoiding it requires discipline beyond technical expertise. The lessons are clear: define goals that matter, enforce governance that actually holds, engage stakeholders at every level, and execute with realism—not wishful thinking.

In my 30 years of building and leading large-scale programs for banks and insurance companies, I have seen how quickly good intentions collapse without these fundamentals. I’ve also seen how applying them consistently can turn even struggling initiatives into success stories.

I help organizations not only avoid these pitfalls but also establish repeatable practices that turn technology into a measurable driver of growth. 

What do you see as the biggest cause of project delays—unclear goals, weak governance, or something else?

Mountebank testing

 

I recently switched to do some testing from Wiremock to Mountebank. I found that Mountebank had far more flexibility and simplicity when doing complex data setups to support your test cases. I ran both on the docker and kubernetes and they work perfectly fine in both engines.

Sample docker  mountebank run:

docker run --name mountebank -p 2525:2525 -p 11000-11010:11000-11010 \\
 -v /home/user/mountebank-data/:/mb bbyars/mountebank --configfile /mb/templates/imposters.ejs \\
 --allowInjection --ipWhitelist "::ffff:172.17.0.1|172.17.0.1"

Sample wiremock run:

docker run --name wiremock \\
  -p 9999:8080 \\
  -v /home/bcavlin/IdeaProjects/moleculer/mock:/home/wiremock \\
  wiremock/wiremock --verbose --global-response-templating --max-template-cache-entries 0
 

Starting MEGA instances

 

MEGA is proving to be a solid provider for the data storage with encryption enabled. In case you wanted to start multiple instances in Linux (e.g. work and personal), here is the script I picked up and modified a bit from the web:

Sample:

Turning keyboard light on Ubuntu after sleep

 

One of the things that I like on the Mac I use for work is that everything seems to be well integrated, however the keyboard is pretty bad though in my opinion, at least on the model that I use. Nevertheless, the back-light feature seems to be working really nice and I wanted to model this partially on the Ubuntu. One of the things that was missing on my Ubuntu laptop is that if I turn off my keyboard back-light, it stays off even if I resume working after dark or if it was on during the night and I resume it during the day, it stays on and consumes a lot of battery. It kind of seemed a shortfall on Ubuntu side, so I decided to write a little program to start and check if I needed a back-light after sleep or not. This is based on my settings for the night mode and hours entered in the setting. The idea is that the laptop comes from the sleep and if it is in the night hours, it will automatically light my keyboard, or otherwise shut off my back-light during the day. This is only after sleep mode and it gives nice feeling to start the work. Script was placed in 

/usr/lib/systemd/system-sleep/

Since it runs under root user, you will need to replace <USER> to put in your user where night mode is configured.

Interviews from the aspect of a tech guy

Hi,

For the last 12 years I had the opportunity to interview quite a few people for various positions mostly related to Node.js, Java, Spring, Cloud and SQL processing. I would like to take this opportunity to write about what I was looking for in the candidates resumes, recruiters relationships and some common mistakes that people make during the interviews or writing resumes. Please note that this is strictly my opinion and that you can take this with a grain of salt.

Pre-interview, gathering requirements

On the several projects where I worked or where I am still working, most job openings will come from business, financing the needs of a project that is, where there is a specific business requirement that needs to be fulfilled. These requirements are created from ad-hoc need or opportunity, or strategic planning. 

There can be two forms of financing in general that I observed, one where we are looking to initialize the platform or novel way of processing data, and one where we are utilizing existing or expanding platform for the business need or competitive advantage. The difference is that first one is developing supporting technology and second one is developing the business. This is where some ideas of what kind of experience and profiles we would require from the candidates come to mind. The other important factor is the company and team culture. Some teams are conservative some more open, some follow strict structure and the others have more free form. These are all important factors for the success of the team. Remember, most of the time, the goal is usually to establish an effective team not exceptional individuals.

Thoughts about candidates

One of the things with agencies is that they use automated search engines when looking for the candidates most of the time and some people tend to add things into the keywords or their job description that they perhaps only heard of or just have bare understanding about them. We get candidates of different backgrounds, both cultural and technical and our goal should always be to see and understand how they can fit into our team and business culture and whether they will bring something valuable in terms of their personal and professional skills and their knowledge of technology. One of the most important aspect for the candidate is to show their enthusiasm and professionalism during this process, regardless on how they actually feel towards the process or the interviewer. This proves couple of points:

1. That they are capable of desired behavior when needed (professional and courteous)
2. That they understand the rules of engagement and what is expected of them
3. That they are not shy to enter into the new situation and do their best to fit in

Anything else leaves employer side at the mercy of the moment and unnecessary training where we should be focusing on bringing candidates up to the speed of our business and not basic understanding of the general work ethics and their professional skills.

Unfortunately, many times candidates will come unprepared or uninterested to the interview. There is a difference when someone is a bit scared or overwhelmed and when someone is unprepared. It is fully expected that candidate will know their resume in detail (what is written in the resume). This is why writing resumes that are more than 2 pages long is very wrong in my opinion. I would usually not go over second page for several reasons:

1. It is out of date (I am only interested in last 5 years at most)
2. Technology used may be outdated
3. Candidate may not have relevant knowledge to apply in the current scenario
4. I have many resumes to review and only limited time to do this
5. Many times candidates would say that they do not remember details of the implementation past 5 years and this does not help in this line of business where technology changes all the time 

It is, of course, good that candidate can reference their past experience and what they learned from it and how they evolved professionally, but details of the discussion should be kept at their most recent experience. There are many ifs and buts here, however, I need to make sure we establish a reasonable expectation of what is needed to pass the interview during the one hour process. This is a very limited time to get to know someone you have met for the first time and establish both objective and subjective opinion of the candidate. 

 

Reviewing resumes

As I have mentioned before, it is well advised to contact some of the agencies that professionally craft resumes towards the job. We all have various experiences and they do not always apply to every company or every job. The idea of a resume is to get invited to the interview in most focused and honest way and it is not the point to write you life story in there. Professional agents and tools that they use have a very limited time to go over the resumes especially for the job where hundreds of applicants apply. You must point out in your resume why you are the right person for the job requested even though in many cases you will not be a 100% match. You need to single out point from your career and experience that puts you on top over other candidates, or in other words, how to be better than the second last candidate. Once you get an interview, resume is not that much important and serves only as a reference for the conversation. As far as cover letters goes, I honestly do not remember when I last read the cover letter, so I would generally drop these. 

When reviewing resume, it always depends if I am reviewing for position of full time job or a contractor. The difference is that first position will have time to build on a career and experience and the later needs to perform immediately. Criteria may be a bit different for two types but it entirely depends on hiring manager and company that is hiring.  

Unless you are not the person that graduated recently, I would generally put very little emphasis on the education other than passing requirement for the job, but more one experience and enthusiasm to do the job. I would prefer to see more resumes where candidate has the experience in the open source community or had some recent (relevant) courses completed. In my opinion this would give an edge to this candidate over the rest. Continuous education and participation in your own or open source project in this line of work is extremely important. If candidate has a blog or example of work, even better. Those are the points that would put candidates way above over the rest that do not have this, providing that what is displayed demonstrates capability and is in line with the resume and job requirements.

The interview

On the interview day, whether it is online or in person, please do not be late. This is not a cliche but shows some responsibility and respect towards the people involved. In case you are being late for objective reasons, please pick up a phone and give interviewer a call, it will be appreciated. Later on this also shows that candidate understands importance of not being late to the many meetings that we may have. 

During the interviews, sometimes I interview alone and sometimes there is hiring manager or another senior resource present, but it depends on the role we are interviewing and our availability. In any case there is going to be different sets of questions designed to:

1. Validate candidate resume and expertise
2. Establish effective communication
3. Validate behavior under stress
4. Establish problem solving capacity

When interview starts, we always give a chance to candidates to introduce themselves. This can take anywhere from 5-10 minutes after we give a short introduction of our environment and job requirement. Expectation is that candidate already understands something about company and tools/processes that we are using, or how would they get the interview in the first place, right? This also shows that candidate is interested in this company/position and that they did some research prior to the interview. 

All my interviews are technical, and that means that we will talk about technology and processes that candidate used and that are relevant to us. I definitely do not know and can discuss about all the items on everyone's resumes, but based on my previous experience, there will be a lot of them where we can find common ground for the discussion. 

Regardless of the candidate I would always go through basic questions to get an understanding if the candidate thought about technologies they are using and not just copy pasted from the Internet. This would involve simple questions like: Why do you think there is an abstract class and interface in Java, for example, or tell me a difference between Java and JEE? You would be amazed how many people would struggle with this question even though this is a very foundation of how design patterns and Java coding is performed in general. After we cover basics just to get the idea where candidate stands in regards to what is written in the resume, we would get on relevant technologies that are listed as most recent or marked as expert or experienced. I would establish a and ask about scenario with the expectation from candidate to:

1. Demonstrate expertise in the area
2. Establish communication and problem solving capability
3. Provide a suggestion and see how candidate can follow a given direction

Depending on a candidate and their previous responses I may provide false leads or conflicting narrative and ask the candidate to proceed in that way to establish the possibility of a future conflict or behavior problem. This is all done in respectable way, of course, with the expectation from the candidate to politely and argumentatively point to the error. In several cases this proved to be impossible obstacle for few candidates... In general, and depending on a job, there are 3 major layers in the enterprise application. Front end, middle tier and database. There are hundreds of technologies that can go in between, like caching, routing, messaging, ETL, etc., but the important factor is that every candidate needs to be aware of these differences and how applications perform in general. When we discuss the problem, awareness of what is being applied as a solution needs to be present at all times. 

One of the important suggestions during the interview is that you should not start on a subject that you do not fully understand when giving examples. This will lead to more in depth questions and possible dead end. If a question pops up that you are not familiar with, do try to supplement with closest example from your experience. This will set you on familiar ground where we can make a constructive conversation. Having a good exchange of ideas is considered a good interview in my opinion. And remember, interview is not an interrogation, look on it more as a  focused conversation.

Post-interview thoughts

Once the interview has been completed, normally we would gather to discuss the candidates and their performance. I would generally present a written or verbal report and provide my opinion, but as a general rule, decision on hiring the candidates usually rests with the Director or VP or whoever owns the budget for the project.  

In any case, I have been in various interviews during my life where some were good and the others not so much. Not everything turns always as you expected, but in the end, you only need one good interview, and with some luck, knowledge and positive attitude, you will get your next job, or at least, a valuable experience be it positive or negative.

As always, please share your thoughts and opinions about this matter. All the best!

Proxy in Microsoft proprietary world

In a world where we have a corporate proxy server requiring NTLM authentication, having Mac/Linux may prove to be difficult choice if all data needs to be routed though the said proxy. Talk about Brew, Git, Node, etc., and all of them will require authenticated access. When you try to use these apps, you may get 407 error saying that you require authentication. In this case, we would need to authenticate using NTLM and this can easily be done with cntlm. We can leave proxy running and connect to its default port 3128 on localhost for most applications requiring it, be it through environment export and http or https proxy or config in e.g. gitconfig file. To start, we need to generate hash for our password:

cntlm -u myusername -d mydomain -H

After this, you will get a password hash that you need to copy in cntlm.conf file in /etc and this will be used to start your server and authenticate you. Server is started with cntlm command specifying configuration file. A good resource to look at is cntlm documentation,  or I found that this post also helps.

Thanks

MSSQL to MongoDB

There have been a lot of discussion in recent years regarding NoSQL databases and when would they be preferable to SQL databases. There are a lot of articles written on this subject, but I wanted to give some insight to one of my past projects that I have been part of, and provide my perspective on the subject.

When choosing topology for the system we should take in consideration all of the factors and use technology that is most suitable for the given task. IMHO, there is no technology which would be optimal for all kinds of scenarios in a complex enterprise system. I have heard arguments such as, that if there is a limited support in one database, for e.g. NoSQL processing, then it should be a viable option to consider just for the sake of it being already present in the ecosystem and for the purpose cutting down the costs of the initial deployment. I would rather disagree with this as I believe that we need to expand our thinking and use technologies that are built specifically for the task at hand as cost savings in speed and development further down the road should not be overlooked. Benefits of this approach are many and cannot only be calculated by initial deployment costs. There are also good articles on the somewhat opposing views [link1] if you would like an honest debate on this matter.

In one of our use cases, MSSQL was already present and deployed in the cloud and initial decision was made to use this to manipulate and store NoSQL data. Even though MSSQL has support to deal with NoSQL structures, they are stored as string that had to be continuously converted to table format (or use special functions) to have a full range of capabilities for e.g. PSF (paging, sorting and filtering) and any serious and frequent data updates. (Here is the guide from Microsoft regarding SQL/NoSQL on Azure in that matter).

A better choice, IMHO, for JSON structures would be e.g. MongoDB or Cosmos DB, depending what is available in your current infrastructure. MongoDB was choice for database due to more familiarity in the development team and the fact that we could deploy our instances to both public and private cloud relatively easily with open source version of the database.

What was gained is that MongoDB is already optimized to deal with JSON structures, fully supports PSF on driver level and it is extremely easy to setup and maintain. We decided to start with SSL connection on 3 replication nodes. We also decided to save on development environment and deploy 3 nodes on the same server (for prod this should be distributed).

In our case MongoDB was being used as a cache database to a secondary layer of APIs that were supported with Oracle database in business API layer. Since we were looking for more flexibility and increased performance, this was a good choice. Data arriving from business layer was already well structured as one JSON but due to size and GUI editing capabilities, we needed to break it down to offer a more flexible usage based on the given business requirements for sample GUI.

Our API for Mongo layer was written using Spring Boot and was previously designed to work with Hibernate and MSSQL. There was a lot of business logic generated and it was handled with Maps and Strings without explicit Java mappings in many cases. Yes, there were some usages to map certain objects using JSON parser but it was all done manually. To proceed, we needed to remove Hibernate, generate 2 additional sets of domain objects (VO->Mongo->Business API), write converters (e.g. Orika) and enhance business logic to avoid parsing to HashMaps but using MongoDB drivers to map directly to Java objects. We also gained ability to use projections, aggregations and MongoDB views. There was a portion of data in MSSQL that was extensively designed to use relations (and this was on top of cached API data) that we needed to convert to NoSQL and integrate into new collections. Removing relations and designing collections proved to be a tricky part as we did not want to change our Angular application extensively. Business requirement was that other than paging changes no other visible functionality can be upgraded, and performance had to significantly improve. MongoDB 4 came with support for transactions between collections and even though ideal usage of NoSQL is to contain everything within one collection, we could not afford to do this due to several factors. One was changing GUI extensively, second one is that we still could not lose concept of relation that was introduced (to some extent), third one was size of the payload if we kept everything under same JSON and the last is performance issues on Angular side due to parsing speed. Perspective of SQL is speed between tables, correctness of data and data safety and the second is ease of use and practicality. Setting up NoSQL vs SQL database engine if also secondary benefit of NoSQL as it is much easier to tune it up (at least from the aspect of what we were doing). Lastly, scaling is much easier to accomplish with NoSQL.

Creating collections, from SQL to NoSQL

One of the challenging aspects of moving from SQL to NoSQL is to design appropriate data storage considering everything already implemented and respect the best practices of the underlying technology. In SQL we have relations and normalization and NoSQL is quite opposite where we would ideally want to contain as many aspects of a request in a single collection. The thing is that we also need to consider how much code is there using already confirmed contracts and APIs. If we have ESB layer or gateway, we may use this to bridge some of the gaps, but for some APIs, to fully gain better performance, smaller corrections may be needed on both server and the client side. In our case, client was missing pagination, consistent contract definition and sort and filtering capabilities were inconsistent. One of the first things we did was to collaborate with the team to understand the benefits regarding performance and page navigation by looking into information being queried and paginated. Since pagination can be done on a driver level, this was our initial goal. There is a secondary option of slicing arrays within one document, but this was not preferred approach. Next problem was dealing with huge payload and frequent updates to API. Older browsers had difficulties parsing this content continuously. Situation was that we have huge and diverse user base with different technical capabilities. Payload delivery needed to be carefully calculated to provide business value with already crafted GUI components, but also to keep in mind performance. We absolutely could not count on fact that the customers will always work with up to date computers and browsers and that they will all have high speed network access. 

As I mentioned earlier, MongoDB 4 included transaction between collections so this significantly helped with our restructuring. We, of course, tried not to misuse this, as NoSQL philosophy is not to build relational collections. Reference data, cached data, configuration data all found its way to new collections and as they were separately accessed, so this was not the issue. Main data got separated into three main collections keeping in mind business flows and GUI design. Looking back at the end goal, I believe that it all worked out well. 

Last thing to do was to create indexes based on frequently accessed data elements, add few aggregations for different views and create several views to serve data for the future use.

Changes to API contracts

Changes to API contract were done to standardize API development and exposure of the data to the client, introduce paging, sorting and filtering in consistent way and to reorganize some of the APIs to better serve NoSQL data (all based on our changes to the collections). These changes were organized with client usage in mind. The question that we continuously asked ourselves was, how will the client get appropriate amount of data with good flexibility and least amount of interaction with the APIs. Network quality that is used varies through the world and the network is also quite huge. This all plays into performance enhancements that we were bringing to the solution. One of the main things was to restructure models used to have server side models separated from client side models. We also introduced abstraction layer in the business service layer to help with future changes.

Updating Angular side

Apart for the various performance changes, modifications for API changes were relatively straight forward (and time consuming). API was split into generic configuration and task engine, and paged data was added where we used native MongoDB paging functionality. We also added sorting and filtering as opposed to SQL string store. MongoDB was able to process this without any performance hits. Instead of one API call to back-end service we chained  two or more services, usually one un-paged for header data and paged services for business data. This worked well and gave us much smaller objects to deal with and also improved performance by a long shot. Ultimately, changes to Angular that were required were as listed (this included performance updates and MongoDB updates for the API):

• Introduction of modular children routes
• Introduction of lazy loading modules
• Introduction of PSF functionality based on native database support to execute such queries
• Reduction of the payload size by remodeling data as a cache and introducing concept that we should process and load only data that user can see + 1 page of buffered data
• Moving cross field validation logic to the API since only validation logic on the client side should be applicable to fields that are visible to the user.

The end result of all of these operations was vastly improved performance, simplified development, maintenance and solid platform for future operations.

Deploying Angular 6 + Spring Boot + Postgres to AWS

You have completed your Angular application and now you are looking for some deployment options... Among many in existence today, you can find choices like AWS, GCP, Azure, PCF, etc. and few other in the cloud. In this post, I will explain what I needed to do to deploy my services to AWS and keep the cost low (or non existent with AWS free tier). There is always an option to get full 'by the book' services and pay for those, but it is better to understand what your options are when deploying the application and then understand how your revenue is going to go against your expenses. As it is, My Open Invoice application is now designed to operate for a single contractor. To add another company a new URL and setup is needed. This can, of course, be upgraded with a few tweaks in the data model and the application itself. We can leave that to some other blog, though. For now, let me introduce you to my simple architecture:

1. Front end of the application is Angular 6 built on Google Material design capable of rendering on both desktop and mobile devices
2. Middle tier is Spring Boot 2
3. Database is Postgres (H2 as test database)
4. Application has cache that is currently used only for RSS feed (EH Cache)
5. Authentication and authorization are done through JWT with Bearer token.

Amazon offers several options for hosting your web application, like EC2 instance and S3 bucket. There is then CloudFront which is used to cache your content (dist folder) and front is with HTTPS access. S3 has durability and redundancy, Route53 has DNS covered and RDS has your database. Elastic Beanstalk is used for EC2 instance generation, auto scaling and Load Balancing setup. CloudWatch is used for log tracing and then there are several other options that you can turn on for multiple instances, load balancing, reporting, caching etc. 

My goal here is to create something that won't drain your money away but still give you a decent application with the ability to have backups. I will also mention the option that would be needed for more robust solution.

This is my current setup:

Let us start by explaining components that I have used:

• Angular 6
• This is a client facing application that is using a template that is both desktop and mobile app friendly. This app is compiled into /dist directory with production setting and code optimization, and uploaded to S3 bucket.
• S3 Bucket
• This is standard AWS S3 bucket in one zone. You can create zone replication but Amazon already guarantees 99.999999999% durability on single zone in case you worry about your data. You could even go to S3-IA for lower cost, but storage is very low for hosting one small web app anyway. This is going to host two items for us:
• Angular 6 code (you can keep this private with access only to CloudFront)
• Beanstalk repo for WAR files 
• CloudFront
• This service is our fronting service to enable routing to /index.html, content distribution closest to your customer location and HTTPS certificate termination and routing for our Angular app. There are few important items here worth mentioning:
• you need 400 and 404 (at least) routing enabled to /index.html 200 HTTP code or your app will not work properly
• you can provide your certificate here if you have your own domain registered (you can use Amazon ACM to generate and provide certificate here)
• CloudWatch
• This service is enabled by default and will track and report your usage for your AWS components. Max resolution is 1 minute. Depending on configuration and logging amount different charges may apply. Modest logging should remain relatively cheap or non existent. Here is the pricing.
• Elastic Beanstalk
• This is your PaaS. You will use this as an entry point to create your configuration, environment, VPC, load balancing, auto scaling groups, RDS, etc. This can be done in an easy and convenient way. You can save hours of work by deploying this way. There are a few important items to consider:
• I have created my database separately from this config. There is a difference if you want to use Tomcat (e.g.) to manage your DB connection or your application. There is also more flexibility to configure RDS individually. 
• I am using Nginx as a reverse proxy and Tomcat 8 to serve content (WAR). One important item is that since I am using Spring Boot application, you need to pass in -D style properties to overwrite Spring ones and NOT environment ${} variables. Took me good 1 hour to figure what 'environment' means in Beanstalk.
• I did not turn on load balancing as this costs money even in free tier. You can alternatively load balance with Route53, but you will need to connect directly to EC2 instances and this limits auto-scaling option.
• If you want to increase instance numbers, I did not find an option to change auto-scale Beanstalk other than enable LB or Time-based Scaling. But, you can also log into auto-scaling directly and increase Max instances to your desired number and also configure trigger for activation. This, however, will not help a lot as you would need load balancer to connect to these. The only other option would be to have ElasicIP address on EC2 instances and DNS balance on those, but I honestly did not try this.
• When deploying WAR file, you need to create custom 443 port nginx.conf in addition with uploaded certificates (I got mine from this SSL site for free). You will need .ebextensions in WAR file with all configurations and certificates. EC2 config is rebuilt on every restart so you will lose 443 port if you do not have this enabled. This is ONLY needed if you do not have LB. Otherwise, LB will take care of 443 port for you (after you configure it).
• You need to open 443 port to your EC2 security group (one will be created by EBS if you already do not have one). This needs to be accessible from 0.0.0.0/0. Your Angular app will connect directly to servers using this rule.
• RDS
• My choice of RDS is Postgres for prod and H2 for dev. On the side note, I was amazed how fast and compatible H2 is with SQL standards in terms of functions (I notice this every time :)). Postgres was closest for some custom queries for capability that I needed (compared with e.g. MySQL). RDS was created in only 1 zone with minimal sized instance, and security group was opened from EC2 to this group by name for desired port. RDS access was limited only to EC2 and if needed access can be done directly for DB management through port forwarding.
• Route 53
• I have registered domain through Amazon and AWS created a Hosted Zone for me. Inside, I have created ALIAS record  pointing to Beanstalk for both www., api. and naked domain name. If you choose option to point directly to EC2, you can do that, and you can always choose load balancing using DNS (though this is not primary balancing method on AWS).
• Auto Scaling Group
• This is created by Beanstalk and in general, you have two options. One is for the single instance and one for the load balanced. Again, load balanced instance will cost you so use only if you need it.

This is just one way of setting up you environment and for a small application it works for me. I have all the backups so I am not too worried about my downtime if any.

As suggested, there are a few items to change if you need more robust environment. Couple of them would be:

1. Enable LB on Elastic Beanstalk and have your system balance across at least 2 zones.
2. Have at least 2 zone deployments, but depends on your clientele, hosting in each major zone would be beneficial (and then use Geolocation/proximity to optimize content delivery)
3. Right now, I have everything deployed in one VPC, but depending on your needs and layout, you may want more than one and then deciding which would be private and public and where to use Gateways, and how to connect VPCs.
4. API Gateway is always and option for more complex environments where HTTPS also can terminate (with your certificate). This adds a layer of abstraction if you are using microservices from multiple points and with various infrastructure.
5. RDS needs to be deployed on Multi-AZ with read replicas enabled. Backup is already enabled even for 1 instance. A possibility exists to use NoSQL database that is automatically deployed into Multi-AZ (like DynamoDB).

There are many ways things can be configured on Amazon, so it is worth a while to investigate and try it out. It is not expensive to test configurations to figure out what is the right one for you, but it may be expensive to try to correct everything down the road once you realize that it was not set up properly. Amazon offers option to pay for only what you use, so why not try it?

So far, my costs were for registering my domain and a small Route53 charge ($0.50 per hosted zone).

If you have a better setup in mind, please let me know. I am always trying to learn new and better ways of optimizing data, infrastructure and cost.

Number formatting in Angular 6 (multirow form)

Working as a contractor made me build an application that I use to generate my invoices and track my activity. But why build an application where there are other commercial solutions, you ask? Well, I needed a way to track my activity other then Excel like tools (which most of my colleagues use), I did not want to purchase other tools that may be available as they did not have format and reports that I needed (although some of them are really good), and most importantly, it is fun to build stuff. As developer, I created my own solution that you can get from my GitHub. Part of the solution is in the BitBucket as I needed to develop Angular based solution on a private repo. This is because I purchased an excellent Angular starter template called Egret that has some license limitations regarding sharing. It offered me a framework to work with with features such as sample forms, basic security, routing, animations and etc. I modified all of these to suit my needs.

One of the issues I had is that I needed to format numbers in multi column/row for all edit boxes for timesheet entry. I had to spend a bit of time to research how this is best accomplished but landed on a specific solution with a few modifications of my own.
I needed to:

• Format on blur and focus events
• Format on form init without blur and focus events
• Format and calculate fields like totals on every field update with these fields being read only

Here is the number-format.directive.ts that will take care of the bluer and focus events:

The second part is that will take care of the form init and totals:

Here is the final look and feel:

A minor item to add to this would be to only allow increments of 0.25, or to round up number to quarters.

Spring Boot (1.5) OAuth2 Server in Enterprise environment

The problem

If we want to have an array of microservices and support user interaction through delegated authorization, this implementation would be one of the options to consider or at least review. We have to first understand the differences between OAuth2 and e.g. OIDC before we continue explaining how to achieve OAuth2 implementation in Spring Boot in a way that is stateless and integrated so many microservices can rely on this implementation.  OAuth 2.0 is the industry-standard protocol for authorization where OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner. This does not mean that OAuth2 cannot do authorization, it is just that OIDC is better suited for this work.

We are going to focus on OAuth2 implementation in this article. There are several good references for reading that you can look for before venturing to create your own implementation (one, two, three, four, five, six). My reasons to do this were several, like not having OAuth2 service available, need for microservices in our architecture, uncertain structure of the client-server architecture and several other. Since we already worked with Spring Boot implementing this solution was the next best thing we could do to move our project further towards our goals.

There are two ways we can handle tokens in OAuth2 and those are plain token and JWT token. For our purposes we chose plain token implementation. The difference was that plain token needs to be verified by OAuth2 service every time it is accessed and JWT can be stored in the resource and verified by the public keys provided. Either way we can have a working solution but implementing plain tokens was a simpler and faster way to go.

Our requirements were to provide solution for the stateless authentication/authorization for the we client, server had to have a small footprint, had to be scalable, we had to see the tokens/users that we generated, we had to have revoke token capability, we had to provide automated solution to obtain the token to integration testing, microservices had to be able to both authenticate themselves and take user authenticated tokens, we had to have ability to connect to LDAP or database and ability to later support SSO from third party provider. There is always an opportunity to implement different solutions but this was something that could potentially play well into the future banking architecture and plans.

Server configuration

To start off we chose Spring Boot OAuth2 and created spring boot application. There were several configurations that we needed to implement to make our server to perform authorization and authentication.

• WebSecurityConfigurerAdapter (to define /login, /logout, swagger, filters - we also can use @EnableOAuth2Client to additionally configure SSO client)
• GlobalAuthenticationConfigurerAdapter (to define user details service, BCrypt password encoder and init method to distinguish between LDAP and database source). This adapter was needed as there are several filters to read users depending on the flow invoked. 
• auth.ldapAuthentication() was starting point for LDAP
• auth.userDetailsService(...) was starting point for user details service and password encoding
• LdapAuthoritiesPopulator bean for LDAP custom authorities (used repository to load authorities based on user authentication)
• AuthorizationServerConfigurerAdapter (to define OAuth2 server infrastrusture, including custom SQL queries (as we needed DB access for stateless solution across servers). This included tables like oauth_access_token, oauth_refresh_token, oauth_code and oauth_client_details. Tables are used depending on the flow invoked. Action involved overriding TokenStore, ClientDetailsService, AuthorizationCodeServices, configure(AuthorizationServerEndpointsConfigurer endpoints), configure(AuthorizationServerSecurityConfigurer security), DefaultTokenServices and configure(ClientDetailsServiceConfigurer clients) - with @EnableAuthorizationServer.
• ResourceServerConfigurerAdapter (to define adapter that will server as entry point and configuration for any custom APIs) - with @EnableResourceServer.
• We also needed to expose API for the user verification where we publish Principal object (this will be used by the microservices to obtain user details)

It is very important to note that adapter ordering is extremely important and that you may loose a lot of time investigating why something is not working just because of this. The order (lowest to the highest) should be Web (needed for authorization_code and implicit and stateful - because of the login page and authentication) -> OAuth2 (needed for all grants but stateless for password, refresh_token and client_credentials grants) -> Resource (needed for APIs).

We opted to use authorization_code without secret and refresh token for user authentication, client_credentials for the server and microservices that needed to authenticate themselves and password grant for the integration test cases (as this is the easiest way to obtain the token for specific user). Our client_credentials added a default role for the client and the rest added a default role for the user. This way, every authenticated client/user will have a default role to start with. This was a sure way to distinguish between human user and server API. The one problem that we still needed to solve for is propagation of the tokens in the layers of services. It is not a good practice to propagate same token between different horizontal layers due to the loss of the identification of the service doing the authorization.

Client configuration

Before we start with the microservices it is important to say that all services are defined as resources (using @EnableResourceServer annotation). This automatically means that we can for one identify a resource and enable its usage to the client for the OAuth2 configuration, and second, we can setup verification URL for the token. In order for any microservice to identify itself, we have two options for the configuration. First in the application.yml and the other programmatic (in case we need to obtain the token itself). For the first option it is useful to use it on any service that needs to implement verify_token, that is, whenever we receive the token our API will send the request to validate the token and populate user details in the SecurityContext in the spring. This is achieved in the security.oauth2.client and security.oauth2.resource entries. There we have to specify our given client id, secret, verify_token URL, resource id, user details URL and a few other parameters. The second option is to obtain token programatically, for example in the spring integration layer, where declarative approach might be difficult, non existent or depending on the extensive logic. In this case the approach is to create a client code that is annotated by the @EnableOAuth2Client. In our case this was done in the ClientHttpRequestFactory. Obtaining the token is achieved by the OAuthClient and OAuthClientRequest to our authorization server using client_credentials grant. 

In the end the goal we had was achieved and all flows are now functional and serving the purpose. The next thing we need to worry about is switching the framework to the OIDC coupled with OAuth2. This will, perhaps, be a good topic for one of the next blogs.

MyBatis Paging, Sorting and Filtering

When considering which database persistence framework to use to complete the goal of development, we must take in consideration several factors:

• Knowledge of your team and ability of the team leads to help out with problems
• Available documentation
• Maturity of the framework
• Availability of the helper classes or supporting frameworks (e.g. how much custom functionality we need to create)
• Underlying structure of the database and it's complexity (if exists)
• Portability (if required)
• "Top down" vs "bottom up" approach driven by either business requirement or existing technology
• Whether we want or have to write SQL statements and how complex they need to be to fulfill the business goals
• Do we "own" the data model or is this vendor maintained data model

As you can see, reaching the decision of what to use can be based on the  experience or trial and run errors.

In one of my recent projects, we reached the decision to use MyBatis as the framework that will enable us to fulfill most of the goals set upon us by the business and existing applications and database topology. In the enterprise environment, you may be faced with the decisions that span not only to your immediate application, but that may involve several others and their data models. We needed to do just that. Read data from the various data sources, integrate with several different web services (both REST and SOAP) and provide unified DB and API interface (facade). This interface, needed to bring web services and various databases together to work as one and with improved performance. Introducing ESB layer was needed but also we needed to create uniform data model (that we can model our facade objects on). MyBatis was perfect tool for this. The only problem we faced was the lack of dynamic paging, sorting and filtering (PSF) functionality, given that we needed to combine results from the different databases (where some of them had awkward design, to say the least). Hibernate was dead in the water here. We ended up using PL/SQL and SQL from various sources, using pipelines and extensive logic to bring order into the data models. This solution worked very well, and in the end, we only needed to implement and expose the Search + PSF to the API and clients. We chose this supporting framework to help us build dynamic additions to the query: squiggle-sql. In their own words:

Squiggle is a little Java library for dynamically generating SQL SELECT statements. It's sweet spot is for applications that need to build up complicated queries with criteria that changes at runtime. Ordinarily it can be quite painful to figure out how to build this string. Squiggle takes much of this pain away.

 This worked perfectly as we could expose REST API JSON with PSF parameters and reuse this through various interfaces. To avoid SQL Injection (as we were building these custom), we needed to use enumerations to match exact constructs, avoid certain risky operations like OR 1=1 or comments in filters, limit the field types and lengths. Overall, we achieved a good mix of security and usability with flexible interface.

We started first by defining the interface in JSON that must have Paging (or streaming), Sorting and Filtering functionality. Again, it is important to limit any functionality with Constants or Enums to make sure all constructs can exactly be matched to operations or underlying supporting Beans. This is important security feature.

Executing Paging in database is relatively straight forward in Oracle by adding this OFFSET <x> ROWS FETCH FIRST <y> ROWS ONLY. The other thing that is needed is to get total number of rows returned from database in order for GUI to calculate how many rows are present. This generally requires executing the statement second time without the row limiting clause or we could write a WITH statement in Oracle and execute once and link into broader query that can execute count on first occurrence and limit rows on second. Important constraint is that the page can start with 0 and up and you should not allow returning of huge pages. If you require a single result that has all records, this may be achieved by secondary API that is limited in use and users that can access it. The reason is that if parallel multiple request are executed on a huge underlying data set, it may lead on DoS type of attack.

Sorting can have multiple columns, so this means that we must match any columns for the sort with ascending or descending parameters and participating Bean properties. Sorting is added as ORDER BY clause. Important feature is to match ORDER BY clause by Enum and columns by underlying Bean. Bean that is exposed to the API should only carry fields that are necessary for API to function properly and to satisfy a business need. Any other database functionality regarding the tables should be hidden behind a services and transfer objects (ref). Order clause can be applied to the complex queries, e.g. multiple set joined by creating an encapsulating select statement around original request.

Filtering may be the trickiest one to implement due to wide range of criteria that can be applied. Same as before, using Enums to define operations and limiting search capability to the filter to use only single fields names and only AND operation is important for the aspect of security and speed. Allowing OR or free statement entry may be dangerous option for execution.

Overall, what we achieved is that now MyBatis has a potential to achieve some of the things that Hibernate has out-of-the-box.

Spring Boot with JSF

I have open sourced an application, recently, that I created for my own time tracking and invoicing and published it on GitHub under the Apache 2.0 license. When I started working on it, I thought about which technology should I use and decided on Spring, more precisely Spring Boot, because that was something that is natural to me as I have used this for many years. But this would only serve a purpose for the middle tier and I still needed to think about what to use for the front end. Trend for the  Spring Boot is to use AngularJS or similar frameworks that would tie in it's capabilities and potentially be deployable on the platforms like Cloud Foundry or similar cloud based solutions. This is due to the fact that these are the stateless solutions, and due to their scalability and ease of deployment they are a good fit for the Spring Boot. However, once we start distinguishing between web applications and enterprise applications, depending on what they need to achieve, I believe that  JSF still plays a big role due to its robustness and number of out of the box components that we can use. PrimeFaces framework is something I have used extensively in the past years and it played well with older Spring implementations, but Spring Boot was not meant to work with this out of the box. I have found a project on the GitHub called JoinFaces to help me with this. JoinFaces incorporate best practices and frameworks from JSF world and allow it to work together with the Spring Boot. In an environment where we will need scalability and multiple server deployments, we would still need a solution to share sessions or create sticky client connections, but for my purpose, this was ideal. So here is the stack that I used:

• JoinFaces
• Spring Boot
• Hibernate (with QueryDSL)
• H2 database
• BIRT reporting engine

These are the basic libraries that allowed me to create an easy deployable solution for the local computer or cloud based application depending on your need. This application still has only one deployable file, but we can easily abstract and create mid tier with business services if we needed micro services type of an app. So why use these components? Let's address my thinking behind each one of these:

JoinFaces

This is the project that ties JSF to Spring Boot and it can be found on this location. In their own words: This project enables JSF usage inside JAR packaged Spring Boot Application. It autoconfigures PrimeFaces, PrimeFaces Extensions, BootsFaces, ButterFaces, RichFaces, OmniFaces, AngularFaces, Mojarra and MyFaces libraries to run at embedded Tomcat, Jetty or Undertow servlet containers. It also aims to solve JSF and Spring Boot integration features. Current version includes JSF and CDI annotations support and Spring Security JSF Facelet Tag support.

Since it includes PrimeFaces and this is my favourite JSF engine, it was perfect for what I was trying to do. It is also up to date and well maintained.

Spring Boot

Spring Boot is the new flavor and trend for the development if you prefer this framework. This is what Maven was to Ant when it came out. The goal here is to pre-configure as many parameters as possible and autodetect dependencies. The whole Cloud Foundry was created to make development and deployment as easy as possible in the enterprise environment. In their own words: Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can "just run". We take an opinionated view of the Spring platform and third-party libraries so you can get started with minimum fuss. Most Spring Boot applications need very little Spring configuration.

Spring in general was a revelation when it came out. It made developing in Java something that you can actually like. Using this as a wiring framework for the libraries is a really good fit in my opinion.

Hibernate (with QueryDSL)

Hibernate was always one of my favourite ORM frameworks. In the world where good programmers are difficult to find and where Java developers do not know or understand how to write a proper SQL, Hibernate offers to bridge this gap. Another good thing is that it will adapt to any supported database you can use. The bad thing is that you may not be able to use database specific things (e.g. hierarchy queries) or where you have complex and demanding queries. Hibernate also may not be a good option for optimisation of the complex queries. If your existing database model is also not 'by the book', you may experience additional problems. In their own words, Hibernate is: Hibernate ORM enables developers to more easily write applications whose data outlives the application process. As an Object/Relational Mapping (ORM) framework, Hibernate is concerned with data persistence as it applies to relational databases (via JDBC). 

You may fall back to JDBC from Hibernate in case you need to write database specific queries or write native type of queries.

QueryDSL is a good addition to JPA as it enables for an easy filtering and query structuring where this may be required. I found it to be very helpful. In their own words: Querydsl is a framework which enables the construction of type-safe SQL-like queries for multiple backends including JPA, MongoDB and SQL in Java. Instead of writing queries as inline strings or externalizing them into XML files they are constructed via a fluent API.

H2 database

I have gone through several databases (Derby, H2 and HyperSQL) for this project and H2 for me was easiest to setup, backup, run in shared mode and just support a lot from the SQL standard. Very fast to initialise and super easy to understand. It integrated well into the application and so far I have not seen any issues with it. In their own words: Welcome to H2, the Java SQL database. The main features of H2 are: Very fast, open source, JDBC API, Embedded and server modes; in-memory databases, Browser based Console application, Small footprint: around 1.5 MB jar file size.

BIRT reporting engine

For invoices, I required an open source reporting engine that was free. I looked into two: Jasper Report and BIRT. I started with Jasper as it seemed easier to integrate but once I started doing  development for the subqueries is when the things got 'interested'. They both have a good UI for designing reports but I found BIRT to be easier and faster to work with. The main difference is that Jasper enables absolute positioning of the elements and BIRT does not. With some planning, this really is not relevant. In their own words: BIRT is an open source technology platform used to create data visualizations and reports that can be embedded into rich client and web applications.


In the next few blogs, I will try to explain what I did, how and why. You may find the application that I called My Open Invoice on GitHub.

Thanks

Books 2

Here is the new batch of the books that I have read and liked.

[Homeland] Cory Doctorow wrote a sequel to Little Brother. His protagonist, Marcus, continues the fight against the police state. If you enjoyed the first book, this is certainly going to entertain you. You can get the book here.

[Lost Symbol] The Lost Symbol is a masterstroke of storytelling that finds famed symbologist Robert Langdon in a deadly race through a real-world labyrinth of codes, secrets, and unseen truths . . . all under the watchful eye of Brown’s most terrifying villain to date. Set within the hidden chambers, tunnels, and temples of Washington, D.C., The Lost Symbol is an intelligent, lightning-paced story with surprises at every turn. You can get the book here.

[Inferno] Harvard professor of symbology Robert Langdon awakens in a hospital in the middle of the night. Disoriented and suffering from a head wound, he recalls nothing of the last thirty-six hours, including how he got there . . . or the origin of the macabre object that his doctors discover hidden in his belongings. Langdon's world soon erupts into chaos, and he finds himself on the run in Florence with a stoic young woman, Sienna Brooks, whose clever maneuvering saves his life. Langdon quickly realizes that he is in possession of a series of disturbing codes created by a brilliant scientist-a genius whose obsession with the end of the world is matched only by his passion for one of the most influential masterpieces ever written-Dante Alighieri's dark epic poem The Inferno. Racing through such timeless locations as the Palazzo Vecchio, the Boboli Gardens, and the Duomo, Langdon and Brooks discover a network of hidden passageways and ancient secrets, as well as a terrifying new scientific paradigm that will be used either to vastly improve the quality of life on earth . . . or to devastate it. You can get the book here.

[A.I. Apocalypse] Leon Tsarev is a high school student set on getting into a great college program, until his uncle, a member of the Russian mob, coerces him into developing a new computer virus for the mob’s botnet - the slave army of computers they used to commit digital crimes.

The evolutionary virus Leon creates, based on biological principles, is successful -- too successful. All the world’s computers are infected. Everything from cars to payment systems and, of course, computers and smart phones stop functioning, and with them go essential functions including emergency services, transportation, and the food supply. Billions may die.

But evolution never stops. The virus continues to evolve, developing intelligence, communication, and finally an entire civilization. Some may be friendly to humans, but others are not.

Leon and his companions must race against time and the military to find a way to either befriend or eliminate the virus race and restore the world’s computer infrastructure. You can get the book here.

[Influx] Are smartphones really humanity’s most significant innovation since the moon landings? Or can something else explain why the bold visions of the 20th century—fusion power, genetic enhancements, artificial intelligence, cures for common diseases, extended human life, and a host of other world-changing advances—have remained beyond our grasp? Why has the high-tech future that seemed imminent in the 1960s failed to arrive?

Perhaps it did arrive…but only for a select few. You can get the book here.

[Flash Boys] In Michael Lewis's game-changing bestseller, a small group of Wall Street iconoclasts realize that the U.S. stock market has been rigged for the benefit of insiders. They band together—some of them walking away from seven-figure salaries—to investigate, expose, and reform the insidious new ways that Wall Street generates profits. If you have any contact with the market, even a retirement account, this story is happening to you. You can get the book here.

[The Money Bubble] The US, Europe and Japan are making financial mistakes that will soon cause a crisis of historic proportions. This book explains those mistakes and the likely shape of the crisis, and offers advice to those hoping to protect themselves and profit from what's coming. You can get the book here.

[Beginning Python] is not a bad book to remind yourself of the basics. It is written for Python 2.4.

This tutorial offers readers a thorough introduction to programming in Python 2.4, the portable, interpreted, object-oriented programming language that combines power with clear syntax
Beginning programmers will quickly learn to develop robust, reliable, and reusable Python applications for Web development, scientific applications, and system tasks for users or administrators
Discusses the basics of installing Python as well as the new features of Python release 2.4, which make it easier for users to create scientific and Web applications
Features examples of various operating systems throughout the book, including Linux, Mac OS X/BSD, and Windows XP




This is it for now and if you liked my selection, please drop me a note or recommend another book.

Aspose document generation

Hello,

In today's business world reporting comes out as the end result of the functional application and it serves the purpose of giving introspection into the system, functionality, current and future needs and much more. Having a good system to produce reports is a challenging requirement as one has to balance functionality, requirements, price, potential support needs, easiness of use and widespread acceptance in development community, documentation, performance, interoperability etc. There is a huge potential of various products on the Internet when searching for a suitable framework to satisfy such needs.

During one of my last projects, we needed to accommodate several requirements in finding a product that can do it all :). I believe that we have come across a product that in our testing and POC proved to be a right choice. That product is Aspose.

Aspose is a leading vendor of .NET, Java, Cloud and Android APIs, SharePoint components and rendering extensions for Microsoft SQL Server Reporting Services and JasperReports. They provide various products for working with Word, Excel, Images etc. To make the process easier, Aspose offers a license for all of the products called Total license. The pricing is very competitive too.

The benefit of using the Aspose is that end reports or document templates can be designed by business users in form of Excel of Word document and Aspose can be used to programmatically populate these documents. Documentation is very good with a lot of examples. Support is done primarily through the forums. The response times for support requests at the time I was writing this were very good even for unpaid support.

Usefulness of the Aspose comes from the versatility of the product, in a way that we can for example, start working with Word and add Excel details or pull parts of it, convert and standardize both and produce combined PDF, and then insert images or graphs on the go. Product comes in a form of library and supports both Java and .NET worlds. Library use is straight forward and license is loaded through the code before invocation of the function calls. Documents can be processed from the files or from the streams (if you keep them in the database). Library supports multithreading and in our tests proved to be very consistent and without apparent memory leaks. Data population uses Mail Merge functionality from Word, and supports both plain fields and repeating fields (like tables). Nesting tables require some playing around design in Word documents but it is doable and result looks very good. The product can run in both Windows and *NIX world and having Word or Excel installation in not needed.

I also have to add that libraries can be converted into OSGi friendly libraries and they work quite well in OSGi container too, whether you package them as JAR dependency or expose them as an OSGi bundle.

When working with Aspose and Mail Merge, data stream is established through the java.sql.Statement and has to be passed like this. We have tested functionality on both Oracle and DB2 and both work really well. Minor overriding needs to be done in case of BLOB fields to accommodate output based on the client library that you want to use.

A sample generic application (you can find this code on Aspose site in various sections):

ThinkPad t440s + Ubuntu

Several weeks ago I have purchased a ThinkPad t440s as I needed a new laptop for development. I was looking at several options including Carbon X1 and MacBook Pro. Having been using Ubuntu for quite some time I decided that I do not want to go with Mac. I also recently switched from iPhone to Nexus 5 so it did not seem like a good choice to me. And it was expensive. Between X1 and t440s, after some research, I found that t440s will suit my needs much better. I also use Lenovo at my work, but unlike that one (W520) which looks and feels like a big brick, t440s is a really elegant laptop. I was worried a bit that there might be some incompatibility with Ubuntu based on some reviews, but this was not the case. The main issue everyone was complaining about was the new touchpad. It supports multi gesture and it has entire area clickable. Ubuntu works almost perfectly with it (few glitches here and there but nothing major). In fact, after switching back to traditional one, it seems very unnatural to use the old one. You can do almost everything with one hand easily with the new touchpad.

I also upgraded my t440s with SSD hard drive and additional 8GB memory and added better wireless network card, but other than that, it was good to go. I opted to go with lower resolution monitor because I just like bigger letters on my screen as some of the applications do not scale well on high resolution monitors. Default Ubuntu installation took some time to organise to my liking, but after that it was just fantastic! Right now, startup time is ~10sec and sleep and hibernate work great! Two batteries last forever (6-7 hours) and they are very light. I also found this cool wallpaper to use from http://wallpaperswide.com/.

Cheers!

Note: Now after 2 years of usage, I can say that everything stands for this laptop that I wrote except for the battery. Unfortunately, its capacity is down to 50% for both batteries and barely last 2.5 hrs.

OLAP with Mondrian

If you ever wanted to do OLAP analysis, first thing that would needed to be done is selection of the tools. There are several good tools out there but I would like to write a little bit about Mondrian OLAP as it comes in form of open source community edition and commercially supported project.

Pentaho Analysis Services Community Edition also known as Mondrian, is an Online Analytical Processing (OLAP) server that enables business users to analyze large quantities of data in real-time. The need to analyze the data comes from data feeds that are detrimental for successful planning and strategies when making decisions about past, present and future sales/expenditures. A great introduction to Mondrian can be found on Pentaho's site so I will skip some of the explanation and get to the point of describing why I liked this product.

To begin with, Mondrian concept is based on ROLAP which means that it draws its data from relational database and it uses relational model for its operations. A good comparison between different concepts can be found here (MOLAP, ROLAP, And HOLAP). To draw on one segment of description, ROLAP methodology relies on manipulating the data stored in the relational database to give the appearance of traditional OLAP's slicing and dicing functionality. In essence, each action of slicing and dicing is equivalent to adding a "WHERE" clause in the SQL statement.

A few other concepts that you might want to understand before using Mondrian are:

1. You will use MDX query language to work with your data. Multidimensional Expressions (MDX) is a query language for OLAP databases, much like SQL is a query language for relational databases. It is also a calculation language, with syntax similar to spreadsheet formulas. You can find a few online tutorials to learn about it and if you have tools like JPivot or Pivot4J, you will only need a starting MDX query where olap4j and JPivot will build up queries based on your navigation through UI.
2. olap4j is a common API for any OLAP server, so you can write an analytic application on one server and easily switch it to another. Built on that API, there is a growing collection of tools and components. If you ever wanted to write a custom API do work directly with Mondrian, you would use this library.
3. Mondrian schema is a single XML file where you will create OLAP constructs like Cubes or Dimensions. You will load the schema when connecting to Mondrian and you will write your MDX queries to be executed against this schema.
4. Mondrian comes with two tools, Mondrian Schema Workbench, that you will use to build schema, and Aggregation Designer, that you will use to create aggregate tables based on your MDX query.
5. There are two front ends for Mondrian, JPivot (older, fast and more stable) and Pivot4J (newer, with JSF, but slower).

Edit: I did view Pivot4J and Analytics UI as one and that would not be completely fair to say that Pivot4J core is slower than JPivot as it is comprised from two parts. Anyway, many thanks to Xavier who is lead on the Pivot4J project for explaining the difference and for quickly addressing some of the issues that I had. I feel that Pivot4J will be great product considering that it started less than a year ago.

Now that we understand some basic concepts, here are a couple of pointers on how to start building your OLAP solution.

First thing is that you will probably want to create a separate web project just for OLAP purpose as depending on your facts table size, you might need a lot of memory. Mixing Mondrian with your application can unnecessarily slow things down. Mondrian (used with Java 5 and up) has the ability to report high memory consumption and gracefully shut down the query before exception is thrown (well, exception is still thrown but not OutOfMemoryError exception).

You can create link back to the OLAP servers and integrate them using IFrames or popups, for example (making sure that you have SSO, of course). If you are using JPivot I would not worry too much about integration with JSF as it comes with WCF (Web Component Framework). I found it a bit difficult to integrate with JSF anyway and hence the need for new project Pivot4j with full JSF support.

Next item on the list is creating a solid fact table. Mondrian schema will use one fact table per Cube. Designing a good fact table means creating proper foreign keys and measure columns and assigning adequate indexes. It also means thinking about implementation and implementing, for example, Star or Snowflake schema. Generally, having too many branches of tables will be more complex to work with (two tables per dimension should be good enough, and if you have more, just create views as needed) as modeling in current workbench will require you to work directly in XML (more than one join). Fact table should have basic dimensions where you will then create hierarchies and levels. Mondrian schema is pretty flexible, so it can work with tables and views, and allows for many complex constructs, but for maintenance purpose, aggregating data to appropriate level will prove easier to design and faster to use. Aggregation Designer is used for that purpose. For OLAP, speed and flexibility are important concepts and we should try to optimize things as much as possible.

If we take FoodMart example, we can analyze a schema and potential usage of data. So, lets examine Sales Cube for example. If we open FoodMart.xml in Schema Workbench we can then expand the Cube Sales node.

We will find following sub-nodes:

1. Fact Table - with several aggregate commands defined. This will shape the fact table to desired cube. 
2. Dimension Usage - these are dimensions that you will share between cubes and this tag allows you to construct dimensions outside of the cube and reference it in the cube. This increases re-usability.
3. Measures - like sales, cost etc. These are going to be numbers that you will work with. This will give you different perspective on performance using different hierarchies/dimensions combinations.
4. Calculated Memebers - these do not exist in the fact table but are calculated from existing measures.

If we get into one of the dimensions (e.g. Promotions), we will find that each dimension has link to fact table using a foreign key column. Usually, dimension will have only one hierarchy unless we can link directly to two or more tables from the fact table foreign key columns. Please note that we can also create a dimension from the fact table if more data is stored in the fact table (other than measures and keys). Each hierarchy has a primary key that links this hierarchy with a dimension. Dimension foreign key matches hierarchy table primary key. Each hierarchy must have a table, view or a join. Each join can be nested e.g. (T-J(T-J(T-T))). Each hierarchy has to have one or more levels. Each level can have property (like employee id, has name which is property). Each level represents one column (unless it has properties which can expand this). You will need to specify a column type. When using joins, you can create aliases for easier navigation. For usage on many other XML elements, you can check this page.

Sample MDX:

When starting to work with a schema, you will be able to put dimensions on columns and rows in addition to the measures. Once you start drilling into dimensions, a different breakdowns will appear as a combination of different levels and hierarchies. With each level you can get different measures. You will be able to change positions on rows and columns until you are happy with the data presented.

Designing meaningful information for your business is a key to success. You should be able to use this information to better understand current sales and predict future trends. Once you are at acceptable level you can drill down into data where Mondrian will display appropriate detailed information based on selected levels and slices (time in this example). This would be a report level. To achieve this level of flexibility without an OLAP, you would need to build many different reports.

Mondrian also has the ability to cache the data and this cache is maintained per connection. There are many options that you can set using mondrian.properties file which can greatly improve the performance.

I hope that this gives you a brief overview of what you can accomplish with the Mondrian. If you are looking for a more detailed explanation, please reference Pentaho documentation as I found it very extensive and helpful.

If you have any suggestions, please send me a note.

Regards