Effective management and scaling of data flows is a cornerstone of success for many organizations. Apache Kafka is a leading real-time data streaming platform that offers unmatched scalability and reliability. However, setting up and scaling Kafka clusters can be challenging and require significant time, expertise, and resources. Amazon Managed Streaming for Apache Kafka (MSK) helps you build and run production applications on Apache Kafka without the need for Kafka infrastructure management expertise or the complex overhead associated with custom Apache Kafka setup and operation.
Amazon MSK Provisioned supports both standard brokers and Express brokers. Express brokers are designed for higher throughput, faster scalability, and lower operational overhead, while standard brokers offer more granular control and configuration flexibility. While Amazon MSK significantly reduces cluster management overhead, teams still perform routine tasks such as managing topics, managing partitions, and implementing specific configurations to meet their business requirements.
To further simplify these day-to-day operations, you can use the Kiro Command Line Interface (CLI) together with the MSK Model Context Protocol (MCP) server for a more intuitive approach to cluster management. These tools enable teams to perform administrative tasks and operational activities using natural language commands. Whether you’re managing topics, monitoring cluster health, or implementing specific configurations, the ability to use plain English commands makes these tasks more accessible to both experienced administrators and developers new to Kafka.
In this post, we will show how Kiro CLI and MSK MCP server can streamline your Kafka management. Through practical examples and demonstrations, we’ll show you how to use these tools to efficiently perform common administrative tasks while maintaining robust security and reliability.
Understanding the benefits of the Model Context Protocol
MCP is an emerging open standard that defines how artificial intelligence (AI) agents can securely access and interact with external tools, data sources, and services. Instead of requiring developers to learn complex API syntax for multiple services, MCP allows AI assistants to contextually understand your environment and provide intelligent guidance. The Kiro CLI agent is a command-line AI assistant that understands your code and environment to perform tasks, generate code, and automate workflows through natural language interactions. Kiro CLI and Model Context Protocol (MCP) together enable teams to manage their MSK clusters using natural language, making cluster management more intuitive and accessible.
Amazon MSK MCP Server provides basic cluster management capabilities including describing clusters, updating configurations, and monitoring broker health. By combining these capabilities with the Kiro CLI’s ability to interact with Kafka’s native command-line tools, teams gain a comprehensive view of their Apache Kafka environment. Through this unified approach, users can manage both control plane operations through the MCP server and data plane operations, such as topic management, through the Kiro CLI with Kafka tools. This integration enables teams to monitor, manage, and optimize their clusters through conversational interactions while maintaining enterprise-grade security through AWS Identity and Access Management (IAM) and fine-grained access control.
Prerequisites
For this tutorial you should have the following prerequisites:
Configure Kiro CLI with Amazon MSK MCP server
Installation and configuration
The following section describes the steps required to install and configure the Amazon MSK MCP server.
Install the required dependencies
To install the required dependencies, follow these steps:
- Install the uv package manager if you haven’t already:
Configure the MCP server
Follow the instructions below to set up the Kiro CLI on your host machine and access the Amazon MSK MCP server. Configure the Amazon MSK MCP server in the Kiro CLI configuration. Edit the MCP configuration file to ~/.aws/.kiro/mcp.json:
Installing MacOS
For macOS, mcp.json file should be as follows:
Installing Windows
For Windows users, the MCP server configuration format is slightly different:
For more installation details, see Installing on Amazon MSK MCP README.md.
- Start the Kiro CLI and verify that the MCP server is configured correctly with the following command:
kiro-cli - Once you are logged into the Kiro CLI, enter the following command to check that all the MSK mcp server tools are available, as shown in the following screenshot:
/tools
Install and configure the Kafka CLI
To perform data plane operations on an Amazon MSK cluster, you need properly configured Kafka command-line tools. In the following video demonstration, we show how developers and administrators can use the Kiro CLI to streamline the installation and configuration of Kafka command-line tools, enabling seamless interaction with their MSK cluster through natural language commands.
Evaluate cluster best practices
Maintaining healthy and efficient adhesion to an Apache Kafka cluster requires established best practices, from proper replication factors to optimal resource utilization. Amazon MSK implements many of these best practices by default, but ongoing monitoring and adjustments are necessary for production workloads. In the following demo, we’ll show you how the Kiro CLI can help you evaluate your cluster configuration against recommended best practices, identify potential issues, and get actionable optimization recommendations. Watch the following video sample how we use natural language queries to assess MSK clusters against AWS best practices and identify and fix issues with replication factors that are not configured according to best practices.
Reacting to status notifications: Optimizing theme configurations for high availability
Amazon MSK status notifications serve as critical alerts to maintain optimal cluster performance and reliability. Customers using MSK Provisioned with Standard brokers may receive notifications about critical configuration parameters such as MinISR and replication factor settings that could affect application resiliency. In this section, we demonstrate how the Kiro CLI can help you quickly respond to status notifications regarding Minimum In-Sync Replicas (MiniISR) and replication factor configurations. Watch how we use natural language commands to identify topics with suboptimal settings, understand their current configuration, and implement recommended changes to confirm high availability during infrastructure maintenance or recovery. This real-world scenario shows how the Kiro CLI simplifies the process of maintaining robust Kafka operations while following AWS best practices.
Cluster-level configuration management: Simplifying parameter updates using natural language
Apache Kafka clusters often require configuration modifications to meet specific security requirements and use cases. While Amazon MSK provides default configurations, there are situations where you need to modify security parameters such as allow.everyone.if.no.acl.found implement proper access control and strengthen the security posture of your cluster. In this demo, we show how the Kiro CLI with the MSK MCP server simplifies the process of updating cluster-level configurations.
Instead of opening multiple CLI commands or console screens, you’ll see how natural language instructions can be used to understand your current security settings, evaluate the impact of changes, and seamlessly implement configuration updates across your MSK cluster. By setting allow.everyone.if.no.acl.found to false, we confirm that explicit ACL permissions are required for all operations, increasing the security of your Kafka deployment.
Conclusion
In this post, we showed how the Kiro CLI and MSK MCP server make Apache Kafka cluster management accessible through natural language commands. By transforming complex Kafka operations into simple conversational interactions, these tools enable both experienced administrators and novices to efficiently manage their MSK clusters. From routine tasks to troubleshooting configuration issues, this approach reduces operational complexity and allows teams to focus more on developing innovative streaming applications.
About the authors
Kalyan Janaki is a Senior Big Data & Analytics Specialist with Amazon Web Services. It helps customers design and build highly scalable, efficient and secure cloud solutions on AWS.
Aarvi Desai is a Technical Account Manager at Amazon Web Services based in the San Francisco Bay Area, where she helps customers solve cloud challenges and build scalable and reliable solutions. Her areas of focus include cloud technologies, architecture best practices and operational excellence.
Sandhya Kanderia is Sr. Technical Account Manager and data analysis specialist. Works with AWS customers to provide ongoing support and technical guidance to help plan and build best-practice solutions and proactively keep customers’ AWS environments up and running.
Ankit Mishra is a Senior Solutions Architect at Amazon Web Services, where he supports healthcare and life sciences customers worldwide. He is passionate about helping organizations design and build secure, scalable, reliable and cost-effective cloud solutions. Outside of work, Ankit enjoys spending time with his young daughter.