Skip to main content

ChatGroq

This will help you getting started with ChatGroq chat models. For detailed documentation of all ChatGroq features and configurations head to the API reference.

Overview​

Integration details​

ClassPackageLocalSerializablePY supportPackage downloadsPackage latest
ChatGroq@langchain/groq❌❌✅NPM - DownloadsNPM - Version

Model features​

Tool callingStructured outputJSON modeImage inputAudio inputVideo inputToken-level streamingToken usageLogprobs
✅✅✅❌❌❌✅✅✅

Setup​

To access ChatGroq models you’ll need to create a ChatGroq account, get an API key, and install the @langchain/groq integration package.

Credentials​

In order to use the Groq API you’ll need an API key. You can sign up for a Groq account and create an API key here. Then, you can set the API key as an environment variable in your terminal:

export GROQ_API_KEY="your-api-key"


If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:

```{=mdx}

```bash
# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

### Installation

The LangChain ChatGroq integration lives in the `@langchain/groq` package:

```{=mdx}

```bash npm2yarn
npm i @langchain/groq

## Instantiation

Now we can instantiate our model object and generate chat completions:

::: {.cell execution_count=1}
``` {.typescript .cell-code}
import { ChatGroq } from "@langchain/groq"

const llm = new ChatGroq({
    model: "mixtral-8x7b-32768",
    temperature: 0,
    maxTokens: undefined,
    maxRetries: 2,
    // other params...
})

:::

Invocation​

const aiMsg = await llm.invoke([
  [
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
  ],
  ["human", "I love programming."],
]);
aiMsg;
AIMessage {
  "content": "I enjoy programming. (The French translation is: \"J'aime programmer.\")\n\nNote: I chose to translate \"I love programming\" as \"J'aime programmer\" instead of \"Je suis amoureux de programmer\" because the latter has a romantic connotation that is not present in the original English sentence.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 73,
      "promptTokens": 31,
      "totalTokens": 104
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": []
}
console.log(aiMsg.content);
I enjoy programming. (The French translation is: "J'aime programmer.")

Note: I chose to translate "I love programming" as "J'aime programmer" instead of "Je suis amoureux de programmer" because the latter has a romantic connotation that is not present in the original English sentence.

Chaining​

We can chain our model with a prompt template like so:

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});
AIMessage {
  "content": "That's great! I can help you translate English phrases related to programming into German.\n\n\"I love programming\" can be translated to German as \"Ich liebe Programmieren\".\n\nHere are some more programming-related phrases translated into German:\n\n* \"Programming language\" = \"Programmiersprache\"\n* \"Code\" = \"Code\"\n* \"Variable\" = \"Variable\"\n* \"Function\" = \"Funktion\"\n* \"Array\" = \"Array\"\n* \"Object-oriented programming\" = \"Objektorientierte Programmierung\"\n* \"Algorithm\" = \"Algorithmus\"\n* \"Data structure\" = \"Datenstruktur\"\n* \"Debugging\" = \"Debuggen\"\n* \"Compile\" = \"Kompilieren\"\n* \"Link\" = \"Verknüpfen\"\n* \"Run\" = \"Ausführen\"\n* \"Test\" = \"Testen\"\n* \"Deploy\" = \"Bereitstellen\"\n* \"Version control\" = \"Versionskontrolle\"\n* \"Open source\" = \"Open Source\"\n* \"Software development\" = \"Softwareentwicklung\"\n* \"Agile methodology\" = \"Agile Methodik\"\n* \"DevOps\" = \"DevOps\"\n* \"Cloud computing\" = \"Cloud Computing\"\n\nI hope this helps! Let me know if you have any other questions or if you need further translations.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 327,
      "promptTokens": 25,
      "totalTokens": 352
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": []
}

Tool calling​

Groq chat models support calling multiple functions to get all required data to answer a question. Here’s an example:

import { ChatGroq } from "@langchain/groq";

// Mocked out function, could be a database/API call in production
function getCurrentWeather(location: string, _unit?: string) {
  if (location.toLowerCase().includes("tokyo")) {
    return JSON.stringify({ location, temperature: "10", unit: "celsius" });
  } else if (location.toLowerCase().includes("san francisco")) {
    return JSON.stringify({
      location,
      temperature: "72",
      unit: "fahrenheit",
    });
  } else {
    return JSON.stringify({ location, temperature: "22", unit: "celsius" });
  }
}

// Bind function to the model as a tool
const llm2 = new ChatGroq({
  model: "mixtral-8x7b-32768",
  maxTokens: 128,
}).bind({
  tools: [
    {
      type: "function",
      function: {
        name: "get_current_weather",
        description: "Get the current weather in a given location",
        parameters: {
          type: "object",
          properties: {
            location: {
              type: "string",
              description: "The city and state, e.g. San Francisco, CA",
            },
            unit: { type: "string", enum: ["celsius", "fahrenheit"] },
          },
          required: ["location"],
        },
      },
    },
  ],
  tool_choice: "auto",
});

const res2 = await llm2.invoke([
  ["human", "What's the weather like in San Francisco?"],
]);

console.dir(res2.tool_calls, { depth: null });
[
  {
    name: 'get_current_weather',
    args: { location: 'San Francisco, CA', unit: 'fahrenheit' },
    type: 'tool_call',
    id: 'call_v4dp'
  }
]

.withStructuredOutput({ ... })​

info

The .withStructuredOutput method is in beta. It is actively being worked on, so the API may change.

You can also use the .withStructuredOutput({ ... }) method to coerce ChatGroq into returning a structured output.

The method allows for passing in either a Zod object, or a valid JSON schema (like what is returned from zodToJsonSchema).

Using the method is simple. Just define your LLM and call .withStructuredOutput({ ... }) on it, passing the desired schema.

Here is an example using a Zod schema and the functionCalling mode (default mode):

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatGroq } from "@langchain/groq";
import { z } from "zod";

const calculatorSchema = z.object({
  operation: z.enum(["add", "subtract", "multiply", "divide"]),
  number1: z.number(),
  number2: z.number(),
});

const llm3 = new ChatGroq({
  temperature: 0,
  model: "mixtral-8x7b-32768",
});
const modelWithStructuredOutput3 = llm3.withStructuredOutput(calculatorSchema);

const prompt3 = ChatPromptTemplate.fromMessages([
  ["system", "You are VERY bad at math and must always use a calculator."],
  ["human", "Please help me!! What is 2 + 2?"],
]);
const chain3 = prompt3.pipe(modelWithStructuredOutput3);
const result3 = await chain3.invoke({});
console.log(result3);
{ operation: 'add', number1: 2, number2: 2 }

You can also specify ‘includeRaw’ to return the parsed and raw output in the result.

const includeRawModel3 = llm3.withStructuredOutput(calculatorSchema, {
  name: "calculator",
  includeRaw: true,
});

const includeRawChain3 = prompt3.pipe(includeRawModel3);
const includeRawResult3 = await includeRawModel3.invoke("");
console.dir(includeRawResult3, { depth: null });
{
  raw: AIMessage {
    lc_serializable: true,
    lc_kwargs: {
      content: '',
      additional_kwargs: {
        tool_calls: [
          {
            id: 'call_4cqx',
            type: 'function',
            function: {
              name: 'calculator',
              arguments: '{"number1":10,"number2":2,"operation":"divide"}'
            }
          }
        ]
      },
      tool_calls: [
        {
          name: 'calculator',
          args: { number1: 10, number2: 2, operation: 'divide' },
          type: 'tool_call',
          id: 'call_4cqx'
        }
      ],
      invalid_tool_calls: [],
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: '',
    name: undefined,
    additional_kwargs: {
      tool_calls: [
        {
          id: 'call_4cqx',
          type: 'function',
          function: {
            name: 'calculator',
            arguments: '{"number1":10,"number2":2,"operation":"divide"}'
          }
        }
      ]
    },
    response_metadata: {
      tokenUsage: { completionTokens: 113, promptTokens: 1227, totalTokens: 1340 },
      finish_reason: 'tool_calls'
    },
    id: undefined,
    tool_calls: [
      {
        name: 'calculator',
        args: { number1: 10, number2: 2, operation: 'divide' },
        type: 'tool_call',
        id: 'call_4cqx'
      }
    ],
    invalid_tool_calls: [],
    usage_metadata: undefined
  },
  parsed: { operation: 'divide', number1: 10, number2: 2 }
}

Streaming​

Groq’s API also supports streaming token responses. The example below demonstrates how to use this feature.

import { ChatGroq } from "@langchain/groq";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";

const model4 = new ChatGroq({
  apiKey: process.env.GROQ_API_KEY,
});
const prompt4 = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant"],
  ["human", "{input}"],
]);
const outputParser4 = new StringOutputParser();
const chain4 = prompt4.pipe(model4).pipe(outputParser4);
const response4 = await chain4.stream({
  input: "Hello",
});
let res4 = "";
for await (const item of response4) {
  res4 += item;
  console.log("stream:", res4);
}
stream:
stream: Hello
stream: Hello!
stream: Hello! I
stream: Hello! I'
stream: Hello! I'm
stream: Hello! I'm here
stream: Hello! I'm here to
stream: Hello! I'm here to help
stream: Hello! I'm here to help you
stream: Hello! I'm here to help you.
stream: Hello! I'm here to help you. How
stream: Hello! I'm here to help you. How can
stream: Hello! I'm here to help you. How can I
stream: Hello! I'm here to help you. How can I assist
stream: Hello! I'm here to help you. How can I assist you
stream: Hello! I'm here to help you. How can I assist you today
stream: Hello! I'm here to help you. How can I assist you today?
stream: Hello! I'm here to help you. How can I assist you today? If
stream: Hello! I'm here to help you. How can I assist you today? If you
stream: Hello! I'm here to help you. How can I assist you today? If you have
stream: Hello! I'm here to help you. How can I assist you today? If you have any
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic,
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask.
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers.
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let'
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let's
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let's get
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let's get started
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let's get started!
stream: Hello! I'm here to help you. How can I assist you today? If you have any questions or need information on a particular topic, feel free to ask. I'll do my best to provide you with accurate and relevant answers. Let's get started!

API reference​

For detailed documentation of all ChatGroq features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_groq.ChatGroq.html

Was this page helpful?


You can also leave detailed feedback on GitHub.