Assisted Labeling with Amazon Comprehend

Introduction

How does Datasaur help?
Datasaur enables you to automate the labeling process, reducing the need for human intervention and increasing the speed and accuracy of your results. With our assisted labeling feature, you can rest assured that your labeling process leverages the latest developments to automate the trivial work, so your human annotation team can focus on the truly interesting tasks.

Prepare the model endpoint using Amazon Comprehend

We will show you how to create a ready-to-use machine learning model through Amazon Comprehend. Before we're starting, make sure you already have the trained model from your session before. (You can also see our other articles to build the model with the Amazon Comprehend integration)

Create Endpoint

We will use the model in Datasaur, so we need to create endpoints for it.

To do this, go to Amazon Comprehend > Endpoints > Create endpoint.

Below are the configurations that we need to fill out:

1. Write the endpoint name.
2. Set up a custom model type. We recommend using the “Custom classification” option.
3. Select our trained model, which we named “AGNews” with the “v0” version.

4. Set up inference billing. We should use one inference unit.
5. Select “Create endpoint”.

Endpoint Status

After creating the endpoint, we need to check its status.

1. Go to Amazon Comprehend > Endpoints. On this page, we can see the endpoint details.

2. Click “Use in real-time analysis” to test the model’s logic on custom text. This will be important for our next step.

‍

Testing Endpoint in Real-time analysis

After creating the endpoint, we need to see the result for a specific text. If it works as expected, we can proceed to the next step.

‍

1. Go to Amazon Comprehend > Real-time analysis

1. We will have a choice: Choose the analysis type as “Custom” to predict a custom model or “Built-in” to predict the default model from Amazon. 
2. Select the “Custom” model type and endpoint. Choose “Classification” and select our created endpoint, which we named “AGNEWS01”.
3. Upload or write input text. We can select “Input text” to make it easier.
4. Insert this input text:

1. “NEWCASTLE, England(AP) – Striker Emile Hersey has pulled out of the England squad ahead of Wednesday #39 friendly against Ukraine because of a tight hamstring, the Football Association said Tuesday.”

5. Click “Analyze”.
6. We can see the analyzed text, followed by the model performance, in the insights section.

API Call and API response

We need to make some adjustments to enable the integration between Amazon Comprehend and Datasaur’s custom API. We can create the custom function using Amazon Lambda and orchestrate the custom endpoint API using Amazon API Gateway.

Port a custom API function in AWS Lambda

1. Go to Lambda > Functions.

1. Click “Create Function”.

2. Fill out the form for “Basic Information” and “Advanced Settings.” Set the function name as “getDocumentClassifier-point”, the runtime setting as “Python 3.9”, and instruction set for architecture as “x86_64”.
3. Now, select “Create function”.

4. Write the code in the editor.

1. Below is the code source that you can use:

import json

import boto3

def lambda_handler(event, context):

    try:

        body = json.loads(event["body"])

        final_resp_list = []

        for data in body:

            id_ = data['id']

            text = data["text"]

            #Get Comprehend client

            comprehend = boto3.client("comprehend")

            #Use detect entities function to get a list of entities

            resp = comprehend.classify_document(Text = text, EndpointArn = "{COMPREHEND_ENDPOINT_ARN}")

            conf_score = [v['Score'] for v in resp['Classes']]

            conf_score_max = max(conf_score)

            index_max_score = conf_score.index(conf_score_max)

            pred_label = resp['Classes'][index_max_score]['Name']

            final_resp = {"id": id_, "label": pred_label}

            final_resp_list.append(final_resp)

        return {

            "isBase64Encoded": True,

            "statusCode": "200",

            "headers": {

                "Content-Type": "text/html"

            },

            "body": json.dumps(final_resp_list)

        }

    except Exception as e:

        print(e.__class__, " occurred :", e)

        resp = json.dumps({"error": str(e)}, sort_keys=True, indent=4)

        return {

            "statusCode": "500",

            "body": resp

        }

‍

5. Click “Deploy”.
6. Select “Configure test event”.
   
   

7. Write dummy input to test lambda function and save it.

1. Select “Create new event” in the “Test event action”.
2. Write an event name. In this case, we will use “comprehend-point”.
3. Write dummy input for our lambda function in “Event JSON”.
4. Click “Save”.

8. After the “Configure test event” dialog box is closed, select Test. We will receive a 200 status code as a successful indication of testing.

Now that the model has obtained endpoints, let's attach it to Datasaur via API.

Orchestrate Amazon Comprehend endpoint and custom function in AWS Lambda using the AWS Gateway API

In this next section, we will create the API to enable Datasaur to communicate with the trained model.

1. Go to API gateway > APIs

1. Select Create API

2. Click Build button in REST API

3. Fill in the initial setting 

1. Choose REST as the protocol.
2. Since we are creating a new API, we can select “New API”.
3. Set the name as “new-comprehend-hit”. The other settings are optional, so we can leave it empty.

3. Select Create API.

5. Navigate to the newly created API (named “new-comprehend-hit”) and select it.

6.  Now, we can begin creating the method and resource.

7. We will start by creating the resource. Please refer to the configurations below:

1. Click “Actions”.
2. Select “Create Resource”.
3. Fill in the “Resource name” field. We will name it “points.”
4. The “Resource Path” will be generated automatically.
5. Select “Create Resource.”

8. Click on the created resource named “points”.

1. Click on “Actions”.
2. Select “Create Method”, and in the dropdown, select POST. 
3. When the page loads, select “Lambda Function” on “Integration type”, check “Use lambda proxy integration” and write the lambda function name as “getDocumentClassificationPoint”.

9. Check the method execution diagram, then click “Actions” and “Deploy API”.

10. In this window, select the Deployment stage. We need to use the event we created earlier (named “comprehend-point”).

11. Click Deploy.
12. The following page will that contain a ready-to-use API URL

1. The endpoint will be ready in the form of '{INVOKE_URL}/points'.

Now that we have the API URL ready for us to call, let's go back to Datasaur and test our model in the labeling interface!

Test out your model performance with Datasaur

By using your ML model, you can efficiently obtain labeled data, which can save you time and effort in the data preparation process. Below are the steps:

1. Open your existing project and enable the ML-Assisted Labeling extension.

1. 
   
2. Next, select “Custom” as the provider and use the configuration below.

• Target questions: Select the question set that you want to answer. In this case, we are using the “Category” question set from “Labeling with Datasaur” step 6, which contains “Business”, “Technology”, “Politics”, “Sports”, and “Entertainment”. 
• Custom API URL: Paste your endpoint URL from the previous step.

‍

3. To execute the model’s predictions, select the “Predict labels” button. The labels will be applied automatically in your project workspace.

We did it! We have now successfully built a model using Comprehend, obtained an API URL to call the model, and used that API to automatically return labels in Datasaur. 

‍

Saving your Time and Resources

In addition to updating labels in the cycle, the Datasaur - Amazon Comprehend integration can provide a range of other benefits to data scientists. For example, it can help fine-tune datasets, which can be a time-consuming and challenging task for many researchers. Furthermore, the integration can provide valuable experience to data scientists, which can be applied to a wide range of projects, whether small-scale or large-scale. With its powerful tools and features, the Datasaur - Amazon Comprehend integration can enable you to leverage your model and reuse it in Datasaur within minutes.

‍