TPP update - more on using node_import

By peterm, 4 February, 2010

Tags

I came back to work on the TPP project over the past few days. A couple of problems needed resolving. First, I'd used the CCK float type instead of using Decimal type to hold salary data. While it worked for hourly and monthly salaries, annual salaries were longer than the column could allow. So, I started my test by deleting the 37 float fields and recreating them as decimal 10,2.

Then I had to reimport the Titles, Hourly, Monthly and Annual data for the step based titles. I had an epiphany with the CSV files and how node_import was working. My Excel sheets to CSV had lots of trailing columns and Mac based carriage returns instead of line feeds. This caused node_import to choke.

We also have some other problems in the source data that I ran into again while reimporting the data. Technician, Nursery, Sr seemed to have an extra column (extra comma) that would allow 198 rows of 229 to be imported, then it would freeze. Once, I pulled the trailing commas off the last chunk of the CSV file, I re-ran the import with success. So, the troubleshooting tip is remove trailing commas. Probably easiest to do this with a Perl pre-process step or a regex that can iterate over 2 or more commas, deleting them as necessary.