Vendor ranking/Criteria

We are trying to develop a tool that will help us subjectively compare vendors. We already have an RFI that ranks the functionality of the software. But there is more to consider than just what it does. You need to consider things like support, cost, stability of company, references, ongoing development, etc.

Does anyone use a tool that takes responses to these criteria and puts them in a spreadsheet or other tool that ranks vendors?

I work at the national headquarters for the YMCAs in the USA and local YMCAs look to us for technology advice and we want to give them a ranking of software vendors just as a starting point.

I would think any tool would produce supect results as who defines what it good and bad ?? pro and cons etc ..
However knowing what the " Y " does in general it seems to me you should be looking at a national html based what ever.

maybe a php myslq based app that can link to a national DB then local " clients " that can serve locally, then feed the national data if needed as required..
tahst the direction I would be looking at

I am not if I was clear in my first posting. We already have several vendors selected that offer packages that handle membership, AR, financial development, program, etc.

Now what we are trying to do is compare the different companies selling this software. But we dont want to only compare functionality. We want to compare cost and the vendor also.

Each local YMCA makes its own decision what software to purchase and use. As a department at the national headquarters we are just trying to provide a way for the local YMCAs to compare these vendors we have information on.

Thanks for the reply though.

Hi Steve,

I developed a spreadsheet for evaluating some vendors. I've just made it generic by wiping out the names, comments, etc. from the sheet but left the structure in place. It's set up for 4 possible vendors and 5 evaluators with 8 criteria but you can take the structure and adjust it to your needs. The rows are the criteria and the columns are the vendors. The cells where the two intersect are for comments to justify the evaluators' ratings. Off to the right is where the evaluator data is filled in.

---

VendorComparison.xls

---

I'm interested to know your opinion of the tool. If you have some suggestions for improvement, please let me know.

---

Regards,

Hi Steve,
Our company is a reseller of accounting software and represent 4 different vendors. We have been asked often to fill out the type of forms you are referring to as part of an RFP (request for proposal).
Each time they are a little different. However, they always want to know the following points at a minumum;

-resource bandwidth. How many consultants do you have available for the implementation and post implementation support

-How long will the implementation take

-experience of the company as well as the project team (in length of time and number of clients) in the organizations specific industry.

-How long has the company been around, and what is their financial viability. If the company is new, or struggling, they will often offer some sort fo support guarantee or source code to ensure the purchase is protected.

-install base - how many clients are actively using the product (and current on maintenance)

-they ask for 2-3 references to talk to.

-Support plan - what is included with your annual maintenance. Do you get unlimited support? What about software upgrades? Is that included or does it cost extra? This helps you determine your recurring costs.

-Local presence. Are there local resources to implement the product and support it afterward, or will any on site visits cost travel expenses as well.

If you would like a copy of a generic RFP, vendor section, I'm sure I can supply one to you. You could use the RFP questionaire to elecit the responses, then plug them into the ranking spreadsheet to make your determination.

Let me know if you want the document - it's kind of large, but can be parced out for your specific questions.

I hope this helps.

Jill McCormick
jill@LibrisSolutions.com
510-915-2334

Hi Steve,

I can honestly say that your question has tipped me over the edge; at least over the edge of making your question the focus of my next software development project :-)

Probably the most common question posed to forums like TechSoup is: “Which software program is best for my needs? - this, that, or something else?” – The answer to this question is subject to a host of client specific variables mixed with more than a bit of marketing hype and personal preference (the specific skills or affiliations of the referrer in one specific platform or program; commercial or ‘community spirit’ marketing campaigns such as those used by commercial vendors or open source developers etc). Add to the mix TCO (Total Cost of Ownership), suitability of purpose and the hugely varying features of modern software, and its no wonder people increasingly find it difficult to make the ‘right’ purchasing decision.

What I intend to do (with hopefully some help from members of these forums) is to develop a program capable of easily measuring various applications (and possibly different brands of computers), against each other. Interestingly I have not been able to find any software of this type freely available.

I envisage the quality (or qualifying) variables would be highly specific to each end-user, however that a generic set of measures could be incorporated so long as they are malleable, easily modified and subject themselves to a user-defined priority schema (i.e. this variable is very important to us; this variable is of lesser importance). Analysis would take the form of a recognised statistical correlation process to ensure that bias was precluded from any findings (excepting in cases where bias is a suitable selection criteria – i.e. I want the software to run on Debian Linux simply because I prefer to use Debian Linux etc.)

After about ten minutes of thinking on this I have already managed to come up with 34 generic selection criteria :-( - each with a number of subsets – i.e. “Installation Costs” would incorporate a number of factors such as the availability of internal expertise; the provision of quality installation manuals; actual dollars charged; time and downtime etc. etc. - and I’m sure we could very easily add to this list!

Yet I don’t think the project is insurmountable, and I have in fact constructed a draft database to see how easily the concept can be mapped to an actual piece of software, and perhaps more importantly, how easy such a system would be to use – to this end I will be asking if anyone wants to act as a beta-tester in the next week or so to see how well the concept evolves.

PS – The software (in whatever form it finally takes) will be provided as freeware.

Rgds, Don

Thanks astever for the reply. This looks like a good sheet for an end comparison. This is the simplistic view one would need to see to start the decision making. Whatever process or tool I use would need to come back to an overview like that.

The suggestion I would add is the ability to weigh the factors. So the more important items make a bigger impact on the final score. This would make the tool more flexible to be used by different people or for different things.

However for our purposes I need something much more robust. The RFP we used had about 10 worksheets each with between 100 and 200 functionality points. So we can see the level of functionality pretty well. But now I want to rate the vendor beyond software functionality. Because if the software can do everything, but the company goes bankrupt right after purchase and you were the only customer, you are in trouble.

The tool I was hoping to find would handle weighing factors, allow for many criteria and would sub-group criteria into a final easy to read overview.

Thank you for the reply though. I am gathering each different tool I find together to help with my final design.

Thanks jmccormick for the reply. Those are exactly the type of criteria that I want to rank the vendors on. Trying to find a way to do that subjectively so that local YMCAs dont think we are just sharing our person opinion. Like we have a favorite or are getting something out of it. Thanks

DONC, I didnt mean to push you over the edge. Here at the YMCA we try to build strong kids, families and communities. I would hate to think I went against that. JK Ha, Ha.

I would be very interested to see this in a working prototype. I have done my share of software testing. I was on a major development project for about 4 years as a tester and business analyst. I would be more than happy to test for you. I dont have development resources available any longer though. Thanks.

Great Steve,

I have put together a rudimentary back-end database structure to hold info and variables and a fairly basic GUI - over the next few days I'll add some generic queries and reports hopefully suitable for later modification and/or suitable for template re-works. For analysis I was just going to use an arithmetic mean (thoughts?) using a median as the point of recommendation (highest score wins) - we could easily add other statistical formulae if the system proves successful / usefull.

The development-tool is Access 2000 (only because it's so easy to use and I can develop a draft in less time than it takes to write down what I want the project to do :-) - as soon as it becomes 'useable' (beta 0.1) I'll send a copy through.

Cheers, Don

Sounds good, Don. I'd like a copy too, please. My suggestion would be to ad a field for "weight" or "importance" and factor that into the ranking. For example, if you specify a weight of 5 (higher is better) for a given criterion, then it multiplies the score for that criterion by 5.

---

Regards,

Whilst I understand the "don't want to be big brother mentality ". Would not a solution that meets all needs under one platform which only needs a browser to use be a better solution .
otherwise you end up with 20 apps and 20 differnet reporting methods, 20 service contracts and 20 to the power of who knows headaches..
Given the tight economic climate it would seem to me to be prudent to look at an across the board solution which in the long run would be far cheaper to setup and maintain.

Sure you dont have to stuff a solution down anyone's throat but given the scope of you org you could proably save everyone a bunch of money and have a portable app that can be easily ported to ,ost languages etc and have one support function that will save everyone time and money..

oz
snip
Each local YMCA makes its own decision what software to purchase and use. As a department at the national headquarters we are just trying to provide a way for the local YMCAs to compare these vendors we have information on.

Hi Steve and all on this thread,

OK - A rough draft has been completed... it's not suitable to deploy on the 'net for download yet but if anyone wants a copy of the draft please drop a note to my Email contact addy (next to this message). An overview of developments to date:

In the first requirements analysis I considered building an app to help people evaluate the usefulness and/or suitability of software, as well as to measure the ability of a certain piece of software to meet your projected requirements and budget. The concept being that you get a piece of software and subject it to a variety of suitability, functionailty and economic viability tests. The test processes are determined by your own requirements analysis (the app helps you to create this) and to apply weightings to your requirements thereby providing a perfomance benchmark. Any subsequent pieces of software falling within the same category (accounting programs etc.) can be measured agaisnt the same benchmark to form a useful comparison.

That was the first go...

On constructing the draft, it seemed obvious to include computing hardware and IT consultancy services to the analysis (the same type of measurement criteria apply) - so the first major "rebuild" was to incorporate software, hardware, IT Consultancies and computer peripherals to the programs measuring paramaters. Taking the example of "suitability of purpose", a benchmark of this type applies across the board of computing providers and vendors - so let's be able to measure it all.

The second major reconstruction involved another conceptual expansion... if we are going to benchmark goods and services, why not be pre-emptive and help our suppliers and vendors meet our needs in the first instance?

This involved matching our newly created measurement criteria to product categories and producing an RFP based on our category-specific criteria (rather than what the vendors might think we want or should have). This futher evolved into producing a "vendor declaration" listing our requirements and requirement weightings (what is most important to us) on a document that a vendor can assess, sign and return based on their determinations of how well products or services "match" our requirements. This is all linked to improving accountability in the IT industry (back in 2001 the OEC determined that 73% of all IT projects in the US during 2000 failed or ran well over budget). The IT industry has to get better, but they (we) can only do this if our end-users likewise improve the methods of identifying exactly what it is they (we) want.

Anyway enough rambling (back to developing formulae and reports) - more on this as it progresses.

Rgds, Don

Sorry - I posted to techsoup before reading some of the recent posts. I have placed the beta database (very raw at this stage) on the web together with a spiel at: http://www.users.on.net/mra/itempower/ITEmpower.htm

Please let me know what you think... ? (lots of logic flow and program bugs in this innitial release however these should be sorted out soon - normal rules of "no guarantees for suitability of purpose etc. apply)

Rgds, Don