Primary Data Table
The primary data table uses a sharded approach and can be described as an intra-day hash partitioned table where fields in a data object are stored collocated in a single partition. The Shard ID is a function of the UID and therefore should be reproducible given the same object ingested at different points in time. This enables de-duplication of objects when they are re-ingested. The Data Type is a user defined category of the data that will typically be used at query time. The Data Type allows for further reduction in the amount of data to be searched.
The primary data table also contains an in-partition index, which we call the Field Index, and we denote the K,V pairs that are in the field index with a leading ‘fi’ in the column family. The field index is used by custom Accumulo iterators at query time to find data objects in the partition.
Optionally, if the table is used to store documents, then the original document or different views of the document can be stored in the ‘d’ column family. Typically this column family would be set up as its own locality group. An example of different views of a document could be .txt and .html versions of the original document.
To enable phrase queries on documents, the ‘tf’ column family contains a protocol buffer (PB) in the value that is a list of word offsets for the term in the document. This too could also be stored in a separate locality group.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| SHARD | DT + NB + UID | NFN + NB + FV | NULL | Represents a name/value pair for a given field within the data object |
| SHARD | FI + NB + NFN | NFV + NB + DT + NB + UID | NULL | Provides an in-partition index for objects within the given shard |
| SHARD | D | DT + NB + UID + NB + DVN | DC | (Optional) Provides a named 'view' of raw content from a given data object |
| SHARD | TF | DT + NB + UID + NT + NFN | TFPB | (Optional) To enable phrase queries on terms within the given document |
Global Index Tables
The forward and reverse index tables serve as global indexes mapping terms to partitions. The index maps a NFN:NFV pair to a category of data within the partitions of the primary data table. The Uid.List Protocol Buffer (ULPB) object contains the number of occurrences of the NFN:NFV pair in a category of data in the partition. Additionally, the ULPB may contain the UIDs of the objects that contain the NFN:NFV. We say “may contain” because there is an upper limit on the number of UIDs in the ULPB.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| NFV | NFN | SHARD + NB + DT | ULPB | Maps a NFN:NFV pair to a category of data within the given partitions of the primary data table |
NFV’s that are indexed within the global reverse index table can be searched using leading wildcards. Thus, the index is created by simply reversing the characters in the NFV…
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| RNFV | NFN | SHARD + NB + DT | ULPB | Maps a NFN:RNFV pair to a category of data within the given partitions of the primary data table |
Data Dictionary Table
The data dictionary table contains information about the data stored in the other tables, to include whether or not a field is indexed, the normalizer that is used for the field, etc. The structure of the table is as follows.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| NFN | e | DT | NULL | The e column family denotes that NFN is a field that is contained within the given DT. Note: The absence of an e entry for a given NFN and the presence of i and/or ri entries denotes that the NFN is an index-only field |
| NFN | i | DT + NB + NCN | NULL | The i column family denotes that NFN is in the Global Forward Index table and that it is normalized using NCN, for the given DT |
| NFN | ri | DT + NB + NCN | NULL | The ri column family denotes that NFN is in the Global Reverse Index table and that it is normalized using NCN, for the given DT |
| NFN | f | DT + NB + YYYYMMDD | Integer Count | The f column family conveys the integer count number of times that NFN has been seen on the given date, within the given DT |
| NFN | m | DT | NULL | The m column family denotes that NFN within type DT can be modified |
| NFN | desc | DT | Text Description | The desc column family may be used to supply a text description of the NFN within type DT |
| NFN | tf | DT | NULL | The tf column family denotes that NFN within type DT has term frequency CF entries, which enables phrase queries, etc |
| NFN | edge | CF from an Edge Table entry | EPB0 | The edge column family denotes that the given edge CF was formed by the list of field names in the PB object, allowing an Edge Table entry to be mapped back to the original Data Object from which it was derived (e.g., via the EdgeEventQuery query logic) |
Edge Table
The edge table may represent one or more graphs, any of which may be comprised of unidirectional and bidirectional edges depending on the user’s needs. A single edge key represents a unidirectional pair of Normalized Field Value vertices, which may be thought of as a source vertex and a sink vertex respectively. Thus, bidirectional edges are generated by simply creating a second key having the original source, sink, and other attributes reversed. Additional information may be encoded into an edge key as well, such as the relationship between the two vertices, the type of the edge, and others.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| ENFV1 + NB + ENFV2 | ETYPE + / + EREL | YYYYMMDD + NB + EAV | EPB1 | A unidirectional pair of vertices that expresses some relationship between a pair of Field Value attributes, which originated from a Data Object (DO) |
| ENFV1 | STATS + NB + ACTIVITY + ETYPE | YYYYMMDD + NB + EAV | EPB2 | Provides activity stats for ENFV1 for the given day |
| ENFV1 | STATS + NB + DURATION + ETYPE | YYYYMMDD + NB + EAV | EPB3 | Provides duration stats for ENFV1 for the given day |
Date Index Table
By design, the primary data table permits at most one YYYYMMDD value to be encoded within the assigned row partition (i.e., Shard ID) of a given data object, and, by default, this date will serve as the basis for the date range criteria of any query that targets the object. However, a given data object may contain any number date-related fields, any of which may be important to a user at query time for filtering purposes.
Indexing may be configured for such fields at ingest time. If date indexing is configured for a field, then its values along with the field name itself will be mapped by entries within this table to the partitions in the primary data table where the source objects are stored. Query clients can then leverage this table to enable date range filtering based on these dates, rather than on the dates encoded within the Shard IDs of the stored objects.
Note that the Date Type Name (DTN) column family identifier here is typically leveraged to provide semantic grouping of distinct field names from disparate datasets. For example, a DTN of “SALE_DATE” might be used to group the values of semantically equivalent fields such as “PURCHASE_DATE”, “RECEIPT_DATE”, “DATE_OF_SALE”, “DATE_PURCHASED”, etc.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| YYYYMMDD1 | DTN | YYYYMMDD2 + NB + DT + NB + NFN | BitSet denoting the date's presence within specific shard partitions. E.g., bit 0 is YYYYMMDD_0, and so on | Maps the date field (given by NFN) and its value (given by YYYYMMDD1) to the SHARD partition(s) specified by the YYYYMMDD2 and BitSet values |
Load Dates Table
The load dates table tracks the dates on which specific field names were loaded into specific tables via DataWave Ingest. This information may be leveraged internally for the purposes of query optimization, load date-based filtering for queries, etc.
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| NFN | FIELD_NAME + NB + Table Name | YYYYMMDD + NB + DT | Long integer encoded via Accumulo SummingCombiner (VARLEN) | Denotes that NFN from the given DT was loaded into the specified table on the given date |
Other Tables
Ingest Error Tables
The layouts associated with the four ingest error tables are identical to those listed above for the Primary Data Table, Global Index Tables, and Data Dictionary Table. The only difference is that the respective error tables here are meant to capture Data Objects that failed to be fully loaded during ingest due to one or more processing errors.
That is, these tables are intended to capture all successfully-processed NFN:FV pairs from their respective Data Objects, just as they would have appeared in the normal schema, including supplemental key/value pairs related to the errors themselves. Since schema descriptions for the four primary data tables apply here as well, we describe below only the specific entries used to convey information about the error(s)
| Row | Column Family | Column Qualifier | Value | Purpose |
|---|---|---|---|---|
| SHARD | DT + NB + UID | ERROR + NB + FV | NULL | Denotes the error category, where NFN = ERROR and FV is one of datawave.ingest.data.RawDataErrorNames |
| SHARD | DT + NB + UID | JOB_ID + NB + FV | NULL | Identifies the job, where NFN = JOB_ID and FV is the MapReduce Job ID |
| SHARD | DT + NB + UID | JOB_NAME + NB + FV | NULL | Identifies the job, where NFN = JOB_NAME and FV is the MapReduce Job Name |
| SHARD | D | DT + NB + UID + NB + EVENT | DC | Conveys the raw content of the actual data object that failed via DC, using EVENT as the DVN |
Query Metrics Tables
The layouts associated with the four query metrics tables are identical to those listed above for the Primary Data Table, Global Index Tables, and Data Dictionary Table. The only difference here is that the respective query metrics tables are intended to persist information associated with user queries exclusively. They can be leveraged by users to gain insight into their own queries, and by administrators to gain insight into active and historical queries. Since schema descriptions for the primary data tables apply here as well, we describe below only the specific NFN and FV components that are used to represent a query metrics Data Object.
| NFN | FV |
|---|---|
| AUTHORIZATIONS | User-specified list of Accumulo auths |
| BEGIN_DATE | User-specified date range start value |
| END_DATE | User-specified date range end value |
| PARAMETERS | User-specified query parameters |
| QUERY | User-specified query expression |
| QUERY_LOGIC | User-specified query logic name |
| QUERY_NAME | User-specified name assigned to the query |
| CREATE_DATE | Query creation date/time YYYYMMDD hhmmss |
| DOC_RANGES | Integer count of document-based table ranges generated by the query |
| FI_RANGES | Integer count of Field Index based table ranges generated by the query |
| HOST | Host name from which the query originated |
| LAST_UPDATED | Date/time of most recent update to the query (YYYYMMDD hhmmss) |
| LIFECYCLE | One of datawave.webservice.query.metric.BaseQueryMetric.Lifecycle |
| NEXT_COUNT | Number of times user requested a page of results, or invoked the /Query/{query id}/next endpoint, for the query |
| NUM_UPDATES | Integer count denoting the number of updates that occurred to the query |
| PAGE_METRICS.[n] | Metrics metadata for a single page of results, where N is an integer denoting the Nth page |
| PLAN | The final (actual) query expression to be evaluated by DataWave |
| QUERY_ID | UID generated by the system upon query creation, used in subsequent API calls |
| QUERY_TYPE | Name of the internal Java class that was employed to encapsulate the query and its state |
| CREATE_TIME | Amount of time elapsed (in ms) for the creation phase of the query |
| ELAPSED_TIME | Total amount of time elapsed (in ms) between creation and the query's current state |
| SEEK_COUNT | Number of Accumulo Iterator seeks required by the query |
| SOURCE_COUNT | Number of Accumulo Iterator sources required by the query |
| USER | User name associated with the query |
| USER_DN | User's distinguished name from client certificate |
Terms and Definitions
| Name | Definition | ID |
|---|---|---|
| Column Family | The portion of the Accumulo key representing the column family | CF |
| Column Qualifier | The portion of the Accumulo key representing the column qualifier | CQ |
| Document Column Family | Column family for document content. Literal string with the value of ' d ' | D |
| Document Content | Raw document content for the ' d ' column entry. Gzip compressed, then base64 encoded | DC |
| Data Object | A real-life object for which field names and field values can be derived. Typically, use of this term will denote an object stored within DataWave's primary data table | DO |
| Data Type | An identifier that describes some category or facet of the Data Object | DT |
| Date Type Name | A user-specified identifier used as the CF within the Date Index table. E.g., this value may be used as a query parameter to indicate to the Query API that it should leverage the Date Index for date range filtering rather than the default, shard-based date range | DTN |
| Document View Name | Identifier within the ' d ' column qualifier, which categorizes the content | DVN |
| Edge Attribute Vector | Three optional, user-supplied attributes which can be appended to the edge key's CQ. These attributes are typically delimited by ' / ' | EAV |
| Source Edge Vertex (NFV) | A Field Value processed by a Normalizer that represents the SOURCE vertex in a unidirectional edge key | ENFV1 |
| Sink Edge Vertex (NFV) | A Field Value processed by a Normalizer that represents the SINK vertex in a unidirectional edge key | ENFV2 |
| Edge Metadata Protocol Buffer | Protocol Buffer in the Data Dictionary Table that contains a list of Data Object field names, which are mapped to a specific edge table CF | EPB0 |
| Edge Table Protocol Buffer | Protocol Buffer containing one LONG which is the count of the edge for the day and an INT32 bitmask for hour of the day | EPB1 |
| Edge STATS-ACTIVITY Protocol Buffer | Protocol Buffer containing one LONG[24] which is the count per hour for each our of the day | EPB2 |
| Edge STATS-DURATION Protocol Buffer | Protocol Buffer containing histogram as LONG[7]: [ <10 sec, 10-30 sec, 30-60 sec, 1-5 min, 5-10 min, 10-30 min, >30 min ] | EPB3 |
| Edge Relationship | Two user-provided string values delimited by ' - ' used as part of the edge key CF, for describing the relationship between the respective vertices | EREL |
| Edge Type | User-provided string value used as part of the edge key CF, for categorizing edges | ETYPE |
| Field Index Column Family | A column family prefix for field index keys. Literal string with the value of ' fi ' | FI |
| Field Name | The name of a field in a Data Object | FN |
| Field Value | The raw value associated with a particular Field Name | FV |
| Normalizer | An object that transforms data so that it can be sorted lexicographically | N |
| Null Byte String | A string with the value of ' \0 ', used as a delimiter | NB |
| Normalizer Class Name | The fully qualified name of the Normalizer (N) Java class | NCN |
| Normalized Field Name | A Field Name (FN) transformed to be compatible with the query code | NFN |
| Normalized Field Value | A Field Value (FV) that has been processed/transformed by a Normalizer (N) | NFV |
| Normalized Term | An individual term within a document transformed to be compatible with the query code | NT |
| Null Value | No value is assigned for keys of this type | NULL |
| Protocol Buffer | Google Protocol Buffer | PB |
| Reversed Normalized Field Value | A Field Value (FV) that has been processed/transformed by a Normalizer (N) and then reversed | RNFV |
| Shard ID | The partition identifier in the form of YYYYMMDD_N, where N is an integer denoting a specific sub-partition within the given day | SHARD |
| Term Frequency Column Family | Column family for term frequency keys. Literal string with the value of ' tf ' | TF |
| Term Frequency Protocol Buffer | Google Protocol Buffer containing a list of word offsets for the term in the document | TFPB |
| Unique Identifier | An internal object identifier generated from the data object, should be deterministic and unique within a partition and Data Type | UID |
| Uid.List Protocol Buffer | A protocol buffer object that contains the number of occurrences of a Field Name / Field Value pair, along with a bounded list of UIDs | ULPB |
| Vertex | A single node within a graph | VERT |