Package emissary.output

Class DropOffUtil

  • public class DropOffUtil
extends Object

    • Field Detail

      • logger

        protected static final org.slf4j.Logger logger

      • SEPARATOR

        protected static final String SEPARATOR

      • OS_NAME

        protected static final String OS_NAME

      • unixRoot

        protected String unixRoot

      • placeOutputData

        protected String placeOutputData

      • idTokens

        protected List<String> idTokens
        Sources for building an ID for an item

      • dateTokens

        protected List<String> dateTokens
        Sources for building a date string for an item

      • parentParams

        protected List<String> parentParams
        params to get from parent and save as PARENT_param

      • outputDirectories

        protected Map<String,​File> outputDirectories
        Output directories

      • ALPHABET

        protected static final byte[] ALPHABET

      • prefix

        protected String prefix

      • uuidInOutputFilenames

        protected boolean uuidInOutputFilenames

      • maxFilextLen

        protected int maxFilextLen

      • defaultEventDateToNow

        protected boolean defaultEventDateToNow

    • Constructor Detail

      • DropOffUtil

        public DropOffUtil()
        Create with the default configuration

      • DropOffUtil

        public DropOffUtil​(Configurator configG)
        Create with the specified configuration

    • Method Detail

      • configure

        protected void configure​(@Nullable
                         Configurator configG)
        Set up config for this class
        • ID_PARAMETER : multiple parameter values, ordered list of how to build a EMISSARY_ID
        • ID : backwards compatibility for ID_PARAMETER only used if ID_PARAMETER does not exist
        • DATE_PARAMETER : multiple parameter values, ordered list of how to build a date path
        • OUTPUT_FILE_PREFIX: string to use when generating random filenames, default: TXT
        • UUID_IN_OUTPUT_FILENAMES: boolean [true]
        • AUTO_GENERATED_ID_PREFIX: prefix to use for an auto-generated id

      • generateBuildFileName

        public String generateBuildFileName()
        Generate a new random build file name using the configured prefix and strategy

      • makeWayForIncomingFile

        public String makeWayForIncomingFile​(IBaseDataObject ibdo,
                                     String suffix)
        Drop off often needs to make way for a file it wants to write Do so and return the name of the file that can now be written

      • removeExistingFile

        public boolean removeExistingFile​(String fileName)
        Remove a file if it exists

      • setupPath

        public boolean setupPath​(String fileName)
        mkdir -p to a path and make sure it is writable
        Parameters:
        fileName - the file name, including directory and filename parts
        Returns:
        true iff it works

      • getOutputDirectory

        public String getOutputDirectory()

      • getPathFromSpec

        public String getPathFromSpec​(String specArg,
                              @Nullable
                              IBaseDataObject d,
                              @Nullable
                              IBaseDataObject tld)
        Create a path name from the spec for the specified spec Specs understand the following 'language' of replacement stuff. Anything not understood is a literal 
         @TLD{'KEY'} is to pull the named KEY from the top level document Metadata
 @META{'KEY'} is to pull the named KEY from the MetaData
 %U% = USER
 %I% = INPUT_FILE_NAME (whole thing)
 %S% = INPUT_FILE SHORT NAME
 %P% = INPUT_FILE PATH (all but short name)
 %i% = INPUT_FILE_NAME with slashes converted to underscores
 %p% = INPUT_FILE PATH with slashes converted to underscores
 %F% = FILETYPE
 %L% = LANGUAGE
 %G% = DTG multi directory layout yyyy-mm-dd/hh/mi(div)10
 %R% = ROOT (Unix or Win depending on OS)
 %B% = ID for the payload depending on type (no -att-)
 %b% = ID for the payload depending on type (with -att-)
 %Y% = Four digit year
 %M% = Two digit month
 %D% = Two digit day of month
 %J% = Three digit ordinal day of the year
        Parameters:
        specArg - the incoming specification
        d - the payload we are making a path for
        tld - the top level document in the d family, possibly null
        Returns:
        string path name with correct separators for this OS

      • cleanSpecPath

        @Nullable
protected String cleanSpecPath​(@Nullable
                               String token)

      • getBestIdFrom

        public String getBestIdFrom​(IBaseDataObject d)
        Extract the ID from the payload. The ID from the payload is specified in the cfg file. An ID = SHORTNAME will use the shortname. An ID = AUTO_GENERATED_ID will use an auto gen uuid. If no id value is found, defaults to using an auto generated id.
        Returns:
        id

      • getExistingIds

        @Deprecated
public String[] getExistingIds​(IBaseDataObject d)
        Deprecated.
        use getExistingIdsList(IBaseDataObject)
        Extract the ID from the payload. The ID from the payload is specified in the cfg file. An ID = SHORTNAME will use the shortname. An ID = AUTO_GENERATED_ID will be ignored. If no id value is found, returns empty array.
        Returns:
        id

      • getExistingIdsList

        public List<String> getExistingIdsList​(IBaseDataObject d)
        Extract the ID from the payload. The ID from the payload is specified in the cfg file. An ID = SHORTNAME will use the shortname. An ID = AUTO_GENERATED_ID will be ignored. If no id value is found, returns empty list.
        Returns:
        a list of id values

      • getRootPath

        public String getRootPath()

      • fixFileNameSeparators

        public String fixFileNameSeparators​(String s)
        Replace any file separators that are not for this platform with the correct one

      • datePath

        protected String datePath​(@Nullable
                          String dtg)
        Format string to date path (yyyy-mm-dd/hh/(mm%10))
        Parameters:
        dtg - expected format yyyymmddhhmmss
        Returns:
        yyyy-mm-dd/hh/(mm%10)

      • makeDotFile

        public String makeDotFile​(String fullName)
        Make a dot file name from the supplied path Ex: /path/to/file.txt --> /path/to/.file.txt
        Parameters:
        fullName - the full path and filename of the file
        Returns:
        the path as is but with a leading dot in the filename

      • getFileType

        public static String getFileType​(IBaseDataObject bdo)
        Get the file type from the metadata or the form string passed in
        Parameters:
        bdo - IBaseDataObject
        Returns:
        the file type

      • getAndPutFileType

        public static String getAndPutFileType​(IBaseDataObject bdo,
                                       @Nullable
                                       Map<String,​String> metaData,
                                       @Nullable
                                       String formsArg)
        Get the file type from the IBaseDataObject or the form string passed in
        Parameters:
        bdo - IBaseDataObject
        metaData - Optional map of metadata that might be modified.
        formsArg - Optional space separated string of current forms
        Returns:
        the file type

      • getBestId

        public String getBestId​(IBaseDataObject d,
                        @Nullable
                        IBaseDataObject tld)
        Extract the ID from the payload. The ID from the payload is specified in the cfg file. An ID = SHORTNAME will use the shortname. An ID = AUTO_GENERATED_ID will use an auto gen uuid. If no id value is found, defaults to using an auto generated id.
        Parameters:
        d - the payload
        tld - if a param is specified and cannot be found in the d method parameter, tries to get param from tld.
        Returns:
        the id based the best ID available, shortname or auto gen id. If no id is found, defaults to auto gen.

      • getLanguage

        public String getLanguage​(IBaseDataObject d)
        Extract language or a default value
        Parameters:
        d - the payload
        Returns:
        the language or the default value (never null)

      • getEventDate

        public Date getEventDate​(IBaseDataObject d,
                         @Nullable
                         IBaseDataObject tld)
        Handle data
        Parameters:
        d - the data object in question
        Returns:
        the Date when the event occurred

      • extractEventDateFrom

        @Nullable
public Date extractEventDateFrom​(IBaseDataObject d,
                                 boolean lastResortDefault)

      • getFileExtensions

        public static Set<String> getFileExtensions​(List<String> filenames,
                                            int maxFilextLen)
        Given a list of filenames, extract and return a set of non-blank file extensions converted to lowercase.
        Parameters:
        filenames - The list of filenames to examine
        maxFilextLen - The maximum size we want a file extension to be
        Returns:
        A set of unique file extensions from the filename list

      • getFullFilepathsFromParams

        public static List<String> getFullFilepathsFromParams​(IBaseDataObject d)
        Checks the Original-Filename and FILE_ABSOLUTEPATH for the filename of the object. Returns a list with the non-empty strings found in these fields. If nothing is found in either field, return an empty list.
        Parameters:
        d - The IBDO
        Returns:
        The list of filenames found in the field Original-Filename or FILE_ABSOLUTEPATH

      • getFullFilepathsFromParams

        @Deprecated
public static List<String> getFullFilepathsFromParams​(IBaseDataObject d,
                                                      String[] filenameFields)
        Deprecated.
        use getFullFilepathsFromParams(IBaseDataObject, List)
        Uses the specified list of fields to check for filenames of the object. Returns a list with the non-empty strings found in these fields. If nothing is found in either field, return an empty list.
        Parameters:
        d - The IBDO
        filenameFields - The list of fields on the IBDO to check
        Returns:
        The list of filenames found in the list of fields on the IBDO

      • getFullFilepathsFromParams

        public static List<String> getFullFilepathsFromParams​(IBaseDataObject d,
                                                      List<String> filenameFields)
        Uses the specified list of fields to check for filenames of the object. Returns a list with the non-empty strings found in these fields. If nothing is found in either field, return an empty list.
        Parameters:
        d - The IBDO
        filenameFields - The list of fields on the IBDO to check
        Returns:
        The list of filenames found in the list of fields on the IBDO

      • processMetadata

        public void processMetadata​(List<IBaseDataObject> payloadList)
        Process metadata before doing any output
        Parameters:
        payloadList - list of items being dropped off that may need initial metadata computations