[ChatStream] Pin the Revision to Guard Against Unexpected Changes in Input/Output Prompts

Hello, this is the Product Development Department at Qualiteg Inc.

During Japan's Golden Week holidays, the tokenizer.json of microsoft/Phi-3-mini-128k-instruct was changed. As a result, prompt parsing failed and chat streaming stopped working.

Specifically, the following change was made:

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/commit/8a362e755d2faf8cec2bf98850ce2216023d178a

Originally, the prompt format described in Microsoft's article differed from the prompt format actually used by the model, so we had implemented a heuristic workaround to match the actual model. It appears, however, that the model (and its tokenizer) has now been revised to something closer to the original specification.

As a consequence, the prompt converter that had been working up to that point stopped working.

In the world of LLMs, speed is everything, so it is a given that models are not always released in a fully tested state, tokenizer included. To avoid prompt conversion errors caused by model updates, we believe the preferable practice is to pin a revision whenever a product loads a model or tokenizer via the Auto classes, and then run a proper test cycle whenever a new version of the model is released.

Example of loading with a pinned revision

MODEL_REVISION = "8a362e755d2faf8cec2bf98850ce2216023d178a"
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-128k-instruct",
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=True,
    revision=MODEL_REVISION,
)
tokenizer = AutoTokenizer.from_pretrained(model_path, revision=MODEL_REVISION)

(The revision can be specified as a commit ID, a tag name, and so on.)