Getting Started Using Phi-3-mini-4k-instruct-onnx for Text Generation with NLP Techniques (2024)

cd path/to/your/project

Create a new virtual environment by running the following command:

python -m venv .venv

.venv\Scripts\activate

On macOS/Linux:

source .venv/bin/activate

You’ll see the virtual environment name in your command prompt (e.g., (venv)). Now you can install Python packages specific to your project without affecting the global Python installation.
Remember to replace <virtual-environment-name> with your preferred name for the virtual environment.

winget install -e --id GitHub.GitLFS

For MacOS

brew install git-lfs

For Linux

apt-get install git-lfs​

We now need to run the Gif-Lfs

git lfs install​

pip install huggingface-hub[cli]​

@echo offsetlocalREM Select which model to downloadecho.echo Choose an option:echo 1. Download the Phi-3 Model for CPUecho 2. Download the Phi-3 Model for Nvidia Cudaecho 3. Download the Phi-3 Model for DirectMLset /p option=Enter the option number: if "%option%"=="1" ( huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir .) else if "%option%"=="2" ( huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cuda/cuda-int4-rtn-block-32/* --local-dir .) else if "%option%"=="3" ( huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include directml/* --local-dir .) else ( echo Invalid option. Please choose 1, 2, or 3.)endlocal

This command downloads the model into a folder calledcpu_and_mobile


Below is a batch script that allows the user to select the ONNX runtime installation option. . Save this script with a.batextension (e.g.,install_onnx_runtime.bat) and run it:

@echo offsetlocalREM Install numpy libarypip install numpyREM Pick which ONNX runtime to installecho.echo Choose an option:echo 1. For CPU (onnxruntime-genai)echo 2. For GPU (onnxruntime-genai-cuda)echo 3. For DirectML (onnxruntime-genai-directml)set /p option=Enter the option number: if "%option%"=="1" ( pip install --pre onnxruntime-genai) else if "%option%"=="2" ( pip install --pre onnxruntime-genai-cuda --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/) else if "%option%"=="3" ( pip install --pre onnxruntime-genai-directml) else ( echo Invalid option. Please choose 1, 2, or 3.)endlocal

Run the model using a Python Script and switch command for model selection

import onnxruntime_genai as og import argparseimport timedef main(args): # If verbose mode is on, print loading model message if args.verbose: print("Loading model...") # If timings mode is on, initialize timing variables if args.timings: started_timestamp = 0 first_token_timestamp = 0 # Load the model model = og.Model(f'{args.model}') if args.verbose: print("Model loaded") # Initialize the tokenizer with the model tokenizer = og.Tokenizer(model) tokenizer_stream = tokenizer.create_stream() if args.verbose: print("Tokenizer created") # Print a newline for readability if verbose mode is on if args.verbose: print() # Create a dictionary of search options from the command line arguments search_options = {name:getattr(args, name) for name in ['do_sample', 'max_length', 'min_length', 'top_p', 'top_k', 'temperature', 'repetition_penalty'] if name in args} # Set a default max length if one is not provided if 'max_length' not in search_options: search_options['max_length'] = 2048 # Define a template for the chat input chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>' # Main loop: ask for input and generate responses while True: # Get user input text = input("Input: ") # If the input is empty, print an error message and continue to the next iteration if not text: print("Error, input cannot be empty") continue # If timings mode is on, record the start time if args.timings: started_timestamp = time.time() # Format the input with the chat template prompt = f'{chat_template.format(input=text)}' # Tokenize the input input_tokens = tokenizer.encode(prompt) # Set up the generator parameters params = og.GeneratorParams(model) params.try_use_cuda_graph_with_max_batch_size(1) params.set_search_options(**search_options) params.input_ids = input_tokens # Create the generator generator = og.Generator(model, params) if args.verbose: print("Generator created") # Print a message if verbose mode is on if args.verbose: print("Running generation loop ...") # If timings mode is on, initialize variables for the generation loop if args.timings: first = True new_tokens = [] # Print the output prompt print() print("Output: ", end='', flush=True)​

If you do install the requirements for DirectML, Cuda and CPU support you can run the Python file above with the following switch

For CPU

python filename.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4

For DirectML

python filename.py -m directml\directml-int4-awq-block-128

For Cuda

python filename.py -m cuda/cuda-int4-rtn-block-32 

Running this a simple batch file

Below is the runnable Python script based on your provided code. You can save this script to a.pyfile and execute it. Make sure to replace--modelwith the actual path to your ONNX model file. You can run this script usingpython your_script_name.py

import onnxruntime_genai as ogimport argparseimport timedef main(args): # If verbose mode is on, print loading model message if args.verbose: print("Loading model...") # If timings mode is on, initialize timing variables if args.timings: started_timestamp = 0 first_token_timestamp = 0 # Load the model model = og.Model(f'{args.model}') if args.verbose: print("Model loaded") # Initialize the tokenizer with the model tokenizer = og.Tokenizer(model) tokenizer_stream = tokenizer.create_stream() if args.verbose: print("Tokenizer created") # Print a newline for readability if verbose mode is on if args.verbose: print() # Create a dictionary of search options from the command line arguments search_options = {name:getattr(args, name) for name in ['do_sample', 'max_length', 'min_length', 'top_p', 'top_k', 'temperature', 'repetition_penalty'] if name in args} # Set a default max length if one is not provided if 'max_length' not in search_options: search_options['max_length'] = 2048 # Define a template for the chat input chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>' # Main loop: ask for input and generate responses while True: # Get user input text = input("Input: ") # If the input is empty, print an error message and continue to the next iteration if not text: print("Error, input cannot be empty") continue # If timings mode is on, record the start time if args.timings: started_timestamp = time.time() # Format the input with the chat template prompt = f'{chat_template.format(input=text)}' # Tokenize the input input_tokens = tokenizer.encode(prompt) # Set up the generator parameters params = og.GeneratorParams(model) params.try_use_cuda_graph_with_max_batch_size(1) params.set_search_options(**search_options) params.input_ids = input_tokens # Create the generator generator = og.Generator(model, params) if args.verbose: print("Generator created") # Print a message if verbose mode is on if args.verbose: print("Running generation loop ...") # If timings mode is on, initialize variables for the generation loop if args.timings: first = True new_tokens = [] # Print the output prompt print() print("Output: ", end='', flush=True)if __name__ == "__main__": parser = argparse.ArgumentParser(description="Run the chatbot script") parser.add_argument("--model", type=str, required=True, help="Path to the ONNX model file") parser.add_argument("--verbose", action="store_true", help="Enable verbose mode") parser.add_argument("--timings", action="store_true", help="Enable timings mode") args = parser.parse_args() main(args)

In conclusion, the Phi-3 mini models are powerful AI tools for text generation using NLP techniques. These models can be run on a variety of devices, including GPUs and CPUs. By following the instructions in this tutorial, you can easily download and run these models on your own computer.