TechSoup.org The place for nonprofits, charities, and libraries

Tips for Moving Data

  • Has your nonprofit migrated data from one database to another? How did you negotiate the move, and what advice would you offer other nonprofits?

    If you are planning a data move and aren't sure where to begin, check out the article How to Plan for a Data Move on TechSoup, which show you what data conversion entails, how to prepare for a data move, and weigh the pros and cons of various data-migration techniques.

    Questions or comments about the article? Share them here.
    senior editor, TechSoup
  • Not being a database person, I only recently came to understand some of the complexities (e.g. moving data about multiple donations from a single donor). I attended a user's group meeting for salesforce.com users today. One of the experienced users said she has used products from crmfusion.com (demand tools) to both move data files and manage them, independent of the application in question. It could be that the author would categorize this as a professional assist, but I found the tip useful.
  • This article makes some excellent points. I would add that there is an additional option. During due diligence, it's also important to consider whether you can move the data to a central location and format that BOTH the source and the target programs can understand.

    Moving the data to a common standard, non-proprietary format (e.g. SQL) should help to ensure that you will be able to move between vendors easily and/or use tools like SharePoint to snap together your own CMS.

    Moreover, you might then be able to use in house tools like Access to slice and dice your data beyond the built-in capability of whatever system you happen to be using.
  • I already knew about the issues in the article. I was hoping for something in between built-in export/import tools and professional data conversion.

    I think that the manual conversion would not usually be technically difficult (though in some cases it could never be perfect), but, for a complex relational database, mapping of the data fields from one to the other would be very tedious.

    Are there any tools that minimize the need for basic command-line skills in manual conversion of data?
  • For anyone actually needing to convert data from one format to another, you might want to look at this Parse o Matic
    I have not personally used it but it looks functional. And it is free for personal and not-for-profit use.

    Jesse:

    Regarding your question:
    "Are there any tools that minimize the need for basic command-line skills in manual conversion of data? "
    Could you please clarify as I don't understand your question.

    Bob

    Free case management software for food pantries/emergency services 501(c)(3) organizations. See: http://webpages.charter.net/bobalston/bob1.htm

  • Regarding tools to minimize command-line skills, my question meant to incorporate my earlier statement that, "I was hoping for something in between built-in export/import tools and professional data conversion."

    I think this means something like a GUI for SQL processing of data. Parse-O-Matic is a partial answer, although it does require the conversion of data to flat files and (apparently) in limited formats. Built-in tools, in my experience, are typically limited to flat files, too.

    If you had a badly normalized database (which I have) and wanted to draw the phone numbers (just for a small example) from several different tables, the Parse-O-Matic would leave you with a ton of extra work to create new tables with the primary keys in each table, so that, after you extracted all those fractional flat files, you could combine the data again under the same keys.
  • Jesse,

      If you had a badly normalized database (which I have) and wanted to draw the phone numbers (just for a small example)...

    I asked Joe, our data conversion expert, if he had any suggestions along the line of tools for you.

    What he said is that badly normalized files cause a lot of problems with conversions and that so far he has not found any real tools that a help. Right now he is working with his own tools that to do what you are talking about and it is proving to be difficult. So far, he finds the easiest thing to do with badly normalized files is to write a custom program.

    Best wishes,

    ________________________
    Sasha Daucus
    www.FundRaiserSoftware.com
    www.FundRaiserBasic.com
    Volunteer TechSoup Moderator

  • Thanks for the reply. The programming hurdle is a big one for me -- I have not been able to find the time to make myself a competent programmer.
  • Jesse,

    I asked Joe if he had any pointers that might help you. He looked at the tools mentioned here, and said that from what he can see, they help, but do not provide a complete solution.

    One suggestion, not posted here yet is Jitterbit. It's a partially open source, enterprise class, integration, conversion and mapping package. It has a fairly steep learning curve, but appears to have a lot of potential.

    I am not sure if you are trying to make a career out of data manipulation. If you are, then there is no way around putting in lots of time and effort. If you aren't, then another tool you might consider is finding a college student taking an Access course to do it. :-)

    Or resign yourself to doing the manual work of breaking the data into flat files and going through the manual steps. If it is a one-off project, this might be the best, quickest and easiest way out.

    Hope that helps!

    ________________________
    Sasha Daucus
    www.FundRaiserSoftware.com
    www.FundRaiserBasic.com
    Volunteer TechSoup Moderator

  • Jesse -

    I would echo the prior comment about finding someone knowledgeable in Access. with Access and its built in SQL, you can access data in native format if there is an ODBC connection for the data or if there is a direct driver for Access. That makes Access pretty useful as a conversion tool.

    Since this thread was started as a generic one, without a specific problem any user has, if YOU actually have a need for such, please provide more specifics about what is the source data you need to convert and what is the target environment. Then more specific answers would be possible.

    Bob
    Bob

    Free case management software for food pantries/emergency services 501(c)(3) organizations. See: http://webpages.charter.net/bobalston/bob1.htm

  • Hi All,

    I know this thread is now quite old but in case other people are looking for information on similar data moving tips, I would suggest the use of a tool called Data Moving Tool. It's a very easy to learn "Do It Yourself" application that allows you to move data around from and to pretty much any data base.

    Discounts are available for non profit organizations.

    JB