Fix Conflicts by Looking at Suggested Features and Add Them to the Model as well as Reviewing the Conflicts¶
As you are updating the models by editing the schema or adding labels, conflicts between the label and the system’s prediction will arise.
As you label documents, the system will suggest features to remedy any existing conflicts. To resolve specific system conflicts in a document, click on the conflict count to the right to navigate to the documents that have conflicts.
Clicking on the conflicts will open up a page showing the labeled documents for a particular entity. Some of the labeled documents have conflicts. They are marked at the top of the listing with a red outline and a Conflict tag. You can click on each document to address the conflicts one-by-one.
Suggested features are also shown at the bottom right of the page. The feature suggestion pane is closed when there are no suggestions. Feature suggestions are associated with specific schema entities. Duet suggests the features to help the system discriminate between entities and resolve the conflicts but it needs help from the user to verify the suggestion. Hovering over a suggested feature will show the entity for which it is suggested. You need to review the content of the feature before adding it. You can also edit the content of the feature before adding it.
By default, suggested features are not shown in the document. By pressing "Show Features" in the upper right of the document widget, you can turn on feature triggers. Feature triggers are shown in the document to show the suggested feature in context. You can click on the tooltip shown over the word to initiate feature creation.
When you label and the system suggests features, it will sometimes ask you to tell it why. It will phrase the feature suggestion as an English question to make it a bit clearer what the system is trying to figure out. In the example below, we see that the system is wondering if "StreetNumber" is found at the end of an "Address". The system hasn't yet been taught that those numbers at the end of the address as a "Zip Code". You would not want to add this feature. Meanwhile, the context feature suggestion, where "State" precedes "Zip Code" is a useful prediction. Framing features in standard English questions will help you to discern whether the suggestion matches what you know as a subject matter expert.
Clicking a suggestion will open a popup where you can add it as a new feature or add the suggested phrases to an existing feature. Click to read more about adding dictionary features and context features.
Pay attention to fixing conflicts before pursuing labeling. Resolve conflicts as they arise and assess feature suggestions as they appear. There are two forms of feature suggestion:
- Dictionary phrases, which can represent concepts like "city", and
- Context features, which composes two features to represent a linguistic pattern where word order matters, where "city" may precede "state". Such context features will capture something like "Bellevue, Washington".
The teaching loop is fully interactive where the model gets updated with every schema edit, labeling a new entity, or adding a new feature suggested by the system when there are conflicts. See machine learning features to learn about resolving conflicts in more detail.
When you feel confident in your model, you should stop teaching and move onto testing.