How Did the Spot-Growth Database Come to Be?

Memory limitations and no settled upon standard for data resulted in fractured sets of information. Data from field visits and experiments takes different froms both hardcopy and electronic. Result is fragmented dataset, with no relational information, and a complexity whichs ends up hiding relevant information.

An All-Encompassing Structure for Field Data Was Required

One common electronic file format was labeled with an "spb" extension. In addition to information included in the file, valid, important information was also included in the filename (and only in the filename). A file named ARK1V1.SPB would contain information about the first visit to the spot with an id, "ARK1". The file contents:

AR

12S31W, PLOT 8

  1.DATE GROUND CHECKED         :   71276.0        10170.0     123199.0       01

  2.SHORTLEAF PERCENTAGE (%)    :     54.0        0.000000     100.000000     11

  3.LOBLOLLY PERCENTAGE (%)     :     46.0        0.000000     100.000000     11

  4.DBH <4 INCHES               :        0        0.00000      99.00000       03

  5.DBH 4 - <5 INCHES           :      2.0        0.00000      99.00000       03

  6.DBH 5 - <6 INCHES           :        0        0.00000      99.00000       03

  7.DBH 6 - <7 INCHES           :        0        0.00000      99.00000       03

  8.DBH 7 - <8 INCHES           :      1.0        0.00000      99.00000       03

  9.DBH 8 - <9 INCHES           :      2.0        0.00000      99.00000       03

 10.DBH 9 - <10 INCHES          :      3.0        0.00000      99.00000       03

 11.DBH 10 - <11 INCHES         :        0        0.00000      99.00000       03

 12.DBH 11 - <12 INCHES         :      1.0        0.00000      99.00000       03

 13.DBH 12 - <13 INCHES         :      3.0        0.00000      99.00000       03

 14.DBH 13 - <14 INCHES         :      6.0        0.00000      99.00000       03

 15.DBH 14 - <15 INCHES         :      2.0        0.00000      99.00000       03

 16.DBH 15 - <16 INCHES         :      3.0        0.00000      99.00000       03

 17.DBH 16 - <17 INCHES         :      1.0        0.00000      99.00000       03

 18.DBH 17 - <18 INCHES         :        0        0.00000      99.00000       03

 19.DBH 18 - <19 INCHES         :        0        0.00000      99.00000       03

 20.DBH 19 - <20 INCHES         :        0        0.00000      99.00000       03

 21.DBH 20 - <21 INCHES         :        0        0.00000      99.00000       03

 22.DBH 21 - <22 INCHES         :        0        0.00000      99.00000       03

 23.DBH 22 - <23 INCHES         :        0        0.00000      99.00000       03

 24.DBH 23 - <24 INCHES         :        0        0.00000      99.00000       03

 25.DBH 24 - <25 INCHES         :        0        0.00000      99.00000       03

 26.DBH 25 - <26 INCHES         :        0        0.00000      99.00000       03

 27.DBH 26 - <27 INCHES         :        0        0.00000      99.00000       03

 28.DBH 27 OR MORE INCHES       :        0        0.00000      99.00000       03

 29.# PREVIOUSLY INFESTED TREES :      2.0        0.00000      9000.00000     01

 30.# CURRENTLY INFESTED TREES  :     22.0        1.00000      9000.00000     01

 31.# TREES UNDER ATTACK        :      6.0        0.00000      9000.00000     03

 32.# TREES CONTAINING EGGS     :      2.0        0.00000      9000.00000     03

 33.# TREES WITH LARVAE & PUPAE :     13.0        0.00000      9000.00000     03

 34.# TREES WITH BROOD ADULTS   :      1.0        0.00000      9000.00000     03

 35.MEAN DBH OF PINES IN INCHES :     12.1        5.00000      30.00000       1

 36.MEAN AGE OF PINES IN YEARS  :     56.0        1.00000      150.00000      1

 37.5-YR RADIAL GROWTH IN INCHES:     -1.0        0.00001      10.00000       12

 38.PINE BASAL AREA (SQ FT/A)   :    125.0        10.00000     300.00000      11

 39.HARDWOOD BASAL AREA(SQ FT/A):     43.0        0.00000      300.00000      11

   92.60000        33.30000       24.00000

Another file format was called "asp", and its structure was a series of line-separated values. Which value meant what was established by an additional list of field names. Hundreds of individual asp files, for an example, would set up certain information about a large set of Spot-Visits, and this is then supplemented with further data, this time from a single file of comma-separated-values.

Obs,SpotVisitNameShort,SpotNameShort ,SimulationDay,TreesPreviouslyInfested,TreesCurrentlyInfested,OrdinalDayOfVisit,Year,SourceOfDataFile,SourceOfSurveyData,StudyID,LocationIndicator,Ownership

1,ARK73V1,ARK73,0,70,215,160,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

2,ARK73V2,ARK73,34,298,572,194,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

3,ARK73V3,ARK73,69,891,722,229,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

4,ARK73V4,ARK73,111,1637,651,271,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

5,ARK4V1,ARK4,0,0,39,160,76,University of Arkansas,University of Arkansas,DEN2b,12S31W,Weyerhaeuser

6,ARK4V2,ARK4,34,40,73,194,76,University of Arkansas,University of Arkansas,DEN2b,12S31W,Weyerhaeuser

7,ARK4V3,ARK4,69,136,137,229,76,University of Arkansas,University of Arkansas,DEN2b,12S31W,Weyerhaeuser

8,ARK4V4,ARK4,111,277,441,271,76,University of Arkansas,University of Arkansas,DEN2b,12S31W,Weyerhaeuser

9,ARK10V1,ARK10,0,0,46,194,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

10,ARK10V2,ARK10,35,52,61,229,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

11,ARK8V1,ARK8,0,2,22,194,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

12,ARK8V2,ARK8,35,24,37,229,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

13,ARK8V3,ARK8,77,61,118,271,76,University of Arkansas,University of Arkansas,DEN2,12S31W,Weyerhaeuser

A dataset of field visits created from asp files and supplemental csv data still would have a different set of values than a dataset of field visits created from spb files, or other means such as direct entry. An encompassing data-structure which could assemble these datasets as one, once and for all, was called for.

<SPBField>

  <Spot>

    <SpotName>ALB001</SpotName>

    <SpotDescription>ALB001 in Alabama</SpotDescription>

    <Visit>

      <VisitName>ALB001V1</VisitName>

      <VisitDescription>Visit 1 to spot ALB001 From File: ALB001V1.ASP</VisitDescription>

      <VisitNumber>1</VisitNumber>

      <CanSimulate>true</CanSimulate>

      <SpotName>ALB001</SpotName>

      <Fingerprint>

        <DateOfVisit>1993-01-12T02:00:00-08:00</DateOfVisit>

        <OrdinalDayOfVisit>12</OrdinalDayOfVisit>

        <StateName>Alabama</StateName>

        <Latitude>32.7</Latitude>

        <Longitude>-86.5</Longitude>

        <FirstSeenDataFormat>.ASP</FirstSeenDataFormat>

        <RangerDistrict>unset</RangerDistrict>

        <NatForWildernessName>unset</NatForWildernessName>

        <Ownership>unset</Ownership>

        <SPBISName>unset</SPBISName>

        <StudyID>unset</StudyID>

        <LocationIndicator>unset</LocationIndicator>

        <SpotVisitDescriptionText>Alabama - Winston - Plot 1 - Visit 1</SpotVisitDescriptionText>

        <Crew>unset</Crew>

        <GPSSystem>unset</GPSSystem>

        <GPSLatitude>-1</GPSLatitude>

        <GPSLongitude>-1</GPSLongitude>

        <SourceOfDataFile>University of Arkansas</SourceOfDataFile>

        <SourceOfSurveyData>unset</SourceOfSurveyData>

        <OriginalUnitsImperial>false</OriginalUnitsImperial>

        <StateAbbreviation>AL</StateAbbreviation>

        <SpotName>ALB001</SpotName>

        <VisitName>ALB001V1</VisitName>

      </Fingerprint>

      <ForestStatus>

        <MeanPineDBH_cm>26.7142857</MeanPineDBH_cm>

        <MeanPineAge>32</MeanPineAge>

        <FiveYearRadialGrowth_mm>-1</FiveYearRadialGrowth_mm>

        <PineBasalArea_SqMperHa>20</PineBasalArea_SqMperHa>

        <HardwoodBasalArea_SqMperHa>9</HardwoodBasalArea_SqMperHa>

        <TotalBasalArea_SqMperHa>29</TotalBasalArea_SqMperHa>

        <PinesPerHectare>356.823099609344</PinesPerHectare>

        <AzimuthSpotGrowth>-1</AzimuthSpotGrowth>

        <NaturalBarriers>unset</NaturalBarriers>

        <CerambycidIndex>unset</CerambycidIndex>

        <BlueStainIndex>unset</BlueStainIndex>

        <StandManagementID>unset</StandManagementID>

        <SpotName>ALB001</SpotName>

        <VisitName>ALB001V1</VisitName>

      </ForestStatus>

      <BroodStageDistribution>

        <TreesWithAttackers>3</TreesWithAttackers>

        <TreesWithEggs>3</TreesWithEggs>

        <TreesWithImmature>1</TreesWithImmature>

        <TreesWithBroodAdults>0</TreesWithBroodAdults>

        <TreesWithSpbTotal>7</TreesWithSpbTotal>

        <TreesCurrentlyInfested>7</TreesCurrentlyInfested>

        <TreesPreviouslyInfested>0</TreesPreviouslyInfested>

        <TreesDeadTotal>7</TreesDeadTotal>

        <SpotName>ALB001</SpotName>

        <VisitName>ALB001V1</VisitName>

      </BroodStageDistribution>

      <TreeDBHDistribution>

        <DBHGroup1>0</DBHGroup1>

        <DBHGroup2>0</DBHGroup2>

        <DBHGroup3>0</DBHGroup3>

        <DBHGroup4>0</DBHGroup4>

        <DBHGroup5>1</DBHGroup5>

        <DBHGroup6>2</DBHGroup6>

        <DBHGroup7>1</DBHGroup7>

        <DBHGroup8>1</DBHGroup8>

        <DBHGroup9>1</DBHGroup9>

        <DBHGroup10>0</DBHGroup10>

        <DBHGroup11>0</DBHGroup11>

        <DBHGroup12>0</DBHGroup12>

        <DBHGroup13>0</DBHGroup13>

        <DBHGroup14>0</DBHGroup14>

        <DBHGroup15>1</DBHGroup15>

        <DBHGroup16>0</DBHGroup16>

        <DBHGroup17>0</DBHGroup17>

        <DBHGroup18>0</DBHGroup18>

        <DBHGroup19>0</DBHGroup19>

        <DBHGroup20>0</DBHGroup20>

        <DBHGroup21>0</DBHGroup21>

        <DBHGroup22>0</DBHGroup22>

        <DBHGroup23>0</DBHGroup23>

        <DBHGroup24>0</DBHGroup24>

        <DBHGroup25>0</DBHGroup25>

        <TotalTreesInDBHDistribution>7</TotalTreesInDBHDistribution>

        <SpotName>ALB001</SpotName>

        <VisitName>ALB001V1</VisitName>

      </TreeDBHDistribution>

      <PineSpeciesDistribution>

        <LoblollyRatio>1</LoblollyRatio>

        <ShortleafRatio>0</ShortleafRatio>

        <LongleafRatio>-1</LongleafRatio>

        <SlashRatio>-1</SlashRatio>

        <VirginiaRatio>-1</VirginiaRatio>

        <OtherOrUnknownRatio>-1</OtherOrUnknownRatio>

        <PineSpeciesRatioTotal>1</PineSpeciesRatioTotal>

        <SpotName>ALB001</SpotName>

        <VisitName>ALB001V1</VisitName>

      </PineSpeciesDistribution>

    </Visit>

</Spot>

</SPBField>

Simply by marking up the values and structuring the data according to Xml rules has immensly improved matters. The data is Humanly Readable. The data is structured heirachically, and can be represented as a tree. Many, many tools exist for working with Xml--just getting the data in this format makes in much easier to import to Excel, or to SQL, for example.

The Spot-Growth Database will eventually be a single administrated set of validated Southern Pine Beetle infestation spot visit data.