When we’re approached by a customer to perform or support a Cloud Proof of Concept (PoC), regardless of its conceptual state, we have a number of key steps and initial questions in mind.
The most obvious is to state the exact problem which needs to be solved. Since any proof of concept is almost always part of a wider programme of work the actual scope of the PoC is vitally important.
Defining and Running Proof of Concept Projects
There are many reasons as to why a proof of concept (PoC) should be undertaken but principally it is to prove the feasibility of a solution for a particular problem. There are important questions to answer prior to the inception of any PoC. Predictably there is no one size fits all approach to defining a proof of concept but in order to have a reasonable chance of success, it must include the following broad steps:
Understand the Business Benefit
If you can’t articulate this at the beginning of a project, it is probably a waste of time. The link between this activity and benefits to the business must be conceptually understood, even if the magnitude is not yet known.
It’s also important to understand how the results of any PoC are likely to be acted upon. Also, if there is any formal process to move from a PoC, to a pilot and then to a production deployment. Depending on the answers to these questions, decisions taken during the definition and engineering phase of a project may come back to haunt you.
Understand the Success Criteria
A clear definition of success criteria is crucial. It must include views from an appropriate range of stakeholders including: business, architecture, IT, support, and end-users.
There are two key types of success criteria; technical and business. Each of the chosen success criteria will in turn have two components: the criterion itself and a test case, with a pass or fail outcome. Some criteria, particularly technical ones, will quickly suggest a test case, while others, especially business criteria, may require further discussions (and often preparatory work) and validation.
Along with any business-specific success criteria, it is likely that the PoC will also need to address some or all of the following:
• Will this technology or technologies meet our needs?
Answering this question requires a clear understanding of workflows and (future) requirements.
• Will this product perform as advertised?
Does it solve some or all of a problem? Is customisation required, who does it, pays for it and supports it?
• Will the end user communities be productive with a new workflow?
Accurate requirement capture is critical as well as buy-in from senior stakeholders and users.
• Will the ultimate solution be feasible and supportable?
Answers must be based on sound technical and financial metrics to avoid nasty surprises later.
• Can we work with the vendor?
If communication is difficult during a PoC it’s unlikely to improve after.
• Can we prove the business case?
Ties into how you track return on investment which in turn needs a good total cost of ownership (TCO) model or a whole life costing (WLC) model.
Engineering of Proposed Solution(s)
Once the success criteria and associated test cases have been defined, the PoC solution(s) can be engineered. The test environment should be as close to a production system as practicable. This has interesting implications for Cloud PoCs, especially if you do not already have a current Cloud presence. Understanding how this is configured and any data required for the PoC is moved to and made available in the Cloud can be a key technical challenge and potential blocker to the PoC.
Stakeholders and managers should give the team(s) sufficient time to complete the task, allowing for sufficient interaction with the vendors (and potentially multiple iterations) in case of hiccups. Note, however, that a badly defined PoC means it is easy to get distracted; actively manage the process to maintain focus on the key question(s) and do not deviate. Focus on timeliness, minimal effort with maximum effectiveness.
Evaluation of Solution Against Success Criteria
Once a solution is engineered, evaluate against the test cases defined by the success criteria. The net result is a collection of pass or fail results for the test cases. Ensuring that all the key stakeholders are aware of the evaluation and engage with the process is important to secure buy-in. The old adage goes “You can please some of the people all of the time, you can please all of the people some of the time, but you can’t please all of the people all of the time”’ – this usually applies here.
Everyone will have reasons to prefer one solution over another. People should be encouraged to explore their preference, but often they are not well encapsulated by the adopted success criteria. Usually more than one possible solution to a problem exists, so multiple iterations of ‘engineer and evaluate’ may be necessary depending on the use-case being evaluated.
Decision to Move Forward
The bigger the impact of a proof of concept, the wider the decision-making tree usually becomes. Implications of future solutions will often mean that making a decision to move forward or not is not just a technical one – it’s rare for a PoC to be conducted in a vacuum. Even if all the success criteria for an individual PoC were met, if the technical solution means a complete re-architecting of the rest of a system then the outcome may not provide enough business benefit to move forward. Ideally these sorts of decision points should be captured early by the business case for the PoC.
A strict methodology focused on the success criteria is important to achieving a useful result from a PoC. If the PoC can answer all of the questions set out by the success criteria (all pass, all fail, or a mix of pass and fail), then it is successful. If the end result is a decision to not move forward, the PoC can still be considered successful. A PoC is not successful, however, when the success criteria were not appropriate or the test cases not detailed enough where it mattered.
Some Slightly Cloudier Questions
It won’t be a surprise that many of the questions you should know the answers to before engaging in a PoC for on-premises evaluations have analogues in the Cloud. They include:
Know thine workflow/code/performance and capacity requirements/TCO before you start.
This is often surprisingly challenging and requires some level of ongoing architecture function and business analysis. Capacity and capability forecasting is hard and prone to errors which is one reason why Cloud may be a good option.
Is this Cloud PoC part of a wider business strategy?
Perhaps not owning or operating your own DCs is the eventual goal – have stakeholders understood whether this PoC is leading towards a “Cloud-burst”, “Cloud-first”, or “Cloud-only” future?
What sort of Cloud do you want?
IaaS vs PaaS vs HPCaaS? In other words, how much control and additional money are you willing to sacrifice to make the administration more hands-off for your internal staff?
Where is your data? How much is there?
Depending on the problem, the ratio of compute to data can vary significantly, leading to data storage and egress charges dwarfing cost savings for elastic compute.
Is Cloud agnosticism important?
How do you quantify the business risk/value of single sourcing? What levels of abstraction are you willing to pay for and why? Is it even worth front loading the costs of moving Cloud vendor?
Is your Cloud solution to meet baseline requirements or handle emergent and peak/periodic demands?
Also consider – are you replacing existing capability or building new services and systems?
What’s it worth to the business?
A loaded question as it also covers what level of resiliency and redundancy you may want to provision.
Is it possible to forecast demand so you can buy cheaper cycles and storage?
Depending on the demand and timeliness requirements of your workload you can potentially reduce the costs of running in the Cloud by buying reserved instances or working the spot markets. Some level of platform agnosticism helps here but again the engineering costs have to be balanced against the WLC.
When benchmarking, ensure that you are tracking the right metrics.
For example, is time to solution the most important metric, or is it cost to solution?
The correct answer is likely to be a balance between the two, which must be understood. Where you are on these scales affects how you can configure and run your problems on available platforms.
Do your development methodologies and your maintenance/support methodologies mesh well with the fast-evolving Cloud platforms?
The on-going costs to support and maintain a solution in an on-premises setting are relatively predictable, but Cloud platforms are evolving rapidly so it probably pays to approach this differently in the Cloud.
As you can see, there is a lot to consider! Planning and undertaking a PoC is often the best way to make sure all stakeholders understand both the risks and benefits of a shift towards the Cloud prior to taking the leap.
If you have any questions regarding running a PoC in the Cloud or any HPC related questions then please get in touch, we would be delighted to hear from you.
sales@redoakconsulting.co.uk
