Teach the Model¶
Teaching a model is an iterative process until you obtain the quality you desire for each category in your schema. Teaching happens interactively. By labeling documents shown to you, you are telling the system what vocabulary in these reviews you see that made you label them the way you do. This is what we call "teaching through features". Features will be suggested by the system when your labels don't match the system prediction. The two forms of feature suggestions are:
- Dictionary phrases, which can represent concepts like "positive word", and
- Context features, which uses dictionary phrases as tokens in a linguistic pattern, where "positive word" may precede "miles per gallon". Such context feature will capture something like "the car has great gas mileage".
The teaching loop is fully interactive where the model gets updated with every schema edit, labeling a new document, or adding a new feature suggested by the system when there are conflicts. To start the iterative process, you first need to provide enough labels for each leaf cateogry. The system suggests that you provide 5 labels per leaf node before publishing your model and starting to consume it. The categories that begin with "Other" do not require positive labels. For this model, you have to classify at least 25 documents to publish your first version.
To obtain customer support tickets from your dataset to label, you can press “Next Sample” to move onto the next document in the dataset. Alternatively, you can search for a keyword that you think will be positively associated with a category in the schema; searching will query documents that contain your search keywords. Read the document and choose which leaf category on the right best describes the classification for the ticket. Select the appropriate category.
Once you move onto the next document, the top right of the page will update to reflect the total number of labels you added so far. The counters next to each category on the right will also update after the model automatically trains with the new label you just submitted. With every model update, the system also updates the quality indicator of each category with at least 5 labels to give you a sense of the model quality. If you try to move onto publishing prior to providing 5 labels for each schema category, the system will give you a warning to provide more labels.
Quality metrics will not show on the schema categories until you have provided five labels for the category. When you have, the system will let you know with a pop-up.
Whereas feature suggestions and conflicts will begin showing as soon as the system finds them appropriate, predictions will not show in the document widget until you have labeled 10 total documents. When you have, the system will let you know with a pop-up.
Once predictions are active, they will show both on the schema and on the document widget. In the example below, the predicted schema category is "Payment Issues", as indicated by the blue highlight.
You may see conflicts between what was labeled and what the system predicts. In case of conflicts, you will need to look at the suggested features at the bottom right corner of the page and review them to select the ones relevant to each category. This has the effect of fixing the conflicts and improving the overall quality of the model.
To edit the schema:
Plan the new categories that you'd like to add, delete, rename or move to another parent node.
Select the pencil icon next to Categories on the right.
Make desired changes. Press "Done" to finalize.
As you are updating the models by editing the schema or adding labels, conflicts between the label and the system’s prediction will arise. The system will suggest features to remedy these conflicts. Feature suggestions are shown at the bottom right of the page. Our system will only show you the five best feature suggestions at a time. It is closed when there are no suggestions.
Hovering over a suggested feature will show the category it is suggested for. You need to review the content of the feature before adding it. Please make sure it is relevant to the category and will help the system distinguish this category from other categories.
Clicking a suggestion will open a popup where you can add it as a new feature or add the suggested phrases to an existing feature.