Hello from the Product Development Department at Qualiteg Inc.

In this article, we explain how to implement ChatStream as a FastAPI web API.

Implementing the Endpoint

To create a web endpoint for streaming chat at the URL path /chat_stream,
call handle_chat_stream_request as shown below.

That alone completes a streaming chat implementation that handles user requests while controlling the number of concurrent text generations.

@app.post("/chat_stream")
async def stream_api(request: Request):
    # handling FastAPI/Starlette's Request
    response = await chat_stream.handle_chat_stream_request(request)
    return response

Intercepting Messages

When using FastAPI/Starlette, calling await request.body() or await request.json() in an endpoint consumes the request stream.
So if you want to intercept the request before delegating it to ChatStream, implement it as follows.

import json
from fastapi import FastAPI, Request

@app.post("/chat_stream")
async def stream_api(request: Request):

    # When intercepting the Request
    request_body = await request.body()
    data = json.loads(request_body)
    
    user_input = data["user_input"]
    regenerate = data["regenerate"]

    print(f"user_input:{user_input} regenerate:{regenerate}")
    
    # If you intercepted the request, pass `request_body`
    response = await chat_stream.handle_chat_stream_request(request, request_body)

    return response

Receiving a Callback When the Chat Stream Has Finished Sending

Because ChatStream sends a streaming response, the moment the endpoint executes return response is not the end of the text generation process.

If you want to catch the moment text generation completes,
specify a callback function in the callback argument of handle_chat_stream_request in your endpoint implementation.

When text generation completes, the specified callback function is invoked.

@app.post("/chat_stream")
async def stream_api(request: Request):

    def callback_func(request, message):
        # Called when text generation has finished
        
        # Example: retrieve the ChatPrompt stored in the session and generate a prompt from the conversation history so far
        session_mgr = getattr(request.state, "session", None)
        session = session_mgr.get_session()
        chat_prompt = session.get("chat_prompt")
        print(chat_prompt.create_prompt())

    pass

    response = await chat_stream.handle_chat_stream_request(request, callback=callback_func)

    return response

Possible values and meanings of the message parameter in the generation-complete callback