Operational Interface Compatibility with vLLM

Oct 13, 2025 by pinkandtrash.com 50 views

[RFC]: Make API Gateway interface OpenAI compatible

Summary

This RFC proposes making the API Gateway interface within AIBrix compatible with OpenAI. We have encountered a few issues in the past few days, and @gaocegege also suggested documenting compatible APIs earlier.

Motivation

Seamless Integration with OpenAI-related Ecosystems

Currently, there are configurations within AIBrix that are not fully compatible with the OpenAI API. As a result, some OpenAI SDKs may not sync with AIBrix deployments, which has been previously noted in issue #757. This lack of compatibility significantly impacts the adoption of AIBrix in scenarios where seamless integration with OpenAI-related ecosystems is desired.

Benefits of OpenAI Compatibility

Making the API Gateway interface compatible with OpenAI will enable AIBrix to:

Integrate seamlessly with OpenAI-related ecosystems
Support a wider range of use cases and applications
Enhance the overall user experience and adoption of AIBrix

Proposed Change

Inference Interface Compatibility

Update API Gateway Code: Update the API Gateway code to be fully compatible with OpenAI-compatible format, including interface, headers, error response format, etc.
Operational Interface Compatibility with vLLM: LLM has its own set of operational requirements and interfaces, such as handling model loading. AIBrix's operational interface should be made to match these requirements.
Extension of Metadata Service for New Interfaces: In cases where the Envoy gateway is not suitable, such as for batch APIs, the metadata service can be extended. The metadata service can store and provide additional information about the new interfaces, such as API version, input/output schemas, and usage limits.
Building Test Tools with OpenAI SDK for Compatibility Verification: Develop test scripts using the OpenAI SDK that can send requests to AIBrix's API Gateway and verify the responses. These tests should cover all the supported OpenAI-compatible endpoints and edge cases. Replace curl-based with SDK-based queries.
Documenting Supported Compatible APIs: Create a comprehensive document that details all the OpenAI-compatible APIs in AIBrix. This should include the API endpoints, request/response formats, authentication methods, and any limitations or caveats. Tracking issue: #757

Alternatives Considered

No alternatives were considered for this proposal.

Risks and Assumptions

Risks

The proposed changes may introduce new bugs or issues that need to be addressed.
The OpenAI SDK may not be compatible with AIBrix's API Gateway, which could impact the adoption of AIBrix.

Assumptions

The OpenAI SDK will be compatible with AIBrix's API Gateway.
The proposed changes will not introduce any significant performance or scalability issues.

Implementation Plan

Short-term (next 2 weeks)

Update API Gateway code to be fully compatible with OpenAI-compatible format
Develop test scripts using the OpenAI SDK to verify compatibility

Medium-term (next 4 weeks)

Implement operational interface compatibility with vLLM
Extend metadata service for new interfaces

Long-term (next 6 weeks)

Document supported compatible APIs
Replace curl-based with SDK-based queries

Conclusion

Making the API Gateway interface compatible with OpenAI will enable AIBrix to integrate seamlessly with OpenAI-related ecosystems, support a wider range of use cases and applications, and enhance the overall user experience and adoption of AIBrix. The proposed changes will be implemented in a phased manner to ensure that the API Gateway interface is fully compatible with OpenAI.
[RFC]: Make API Gateway interface OpenAI compatible - Q&A

Q: What is the motivation behind making the API Gateway interface compatible with OpenAI?

A: The motivation behind making the API Gateway interface compatible with OpenAI is to enable seamless integration with OpenAI-related ecosystems, support a wider range of use cases and applications, and enhance the overall user experience and adoption of AIBrix.

Q: What are the benefits of making the API Gateway interface compatible with OpenAI?

A: The benefits of making the API Gateway interface compatible with OpenAI include:

Seamless integration with OpenAI-related ecosystems
Support for a wider range of use cases and applications
Enhanced overall user experience and adoption of AIBrix

Q: What are the proposed changes to make the API Gateway interface compatible with OpenAI?

A: The proposed changes to make the API Gateway interface compatible with OpenAI include:

Updating the API Gateway code to be fully compatible with OpenAI-compatible format
Implementing operational interface compatibility with vLLM
Extending the metadata service for new interfaces
Building test tools with OpenAI SDK for compatibility verification
Documenting supported compatible APIs

Q: What are the risks associated with making the API Gateway interface compatible with OpenAI?

A: The risks associated with making the API Gateway interface compatible with OpenAI include:

Introducing new bugs or issues that need to be addressed
The OpenAI SDK may not be compatible with AIBrix's API Gateway, which could impact the adoption of AIBrix

Q: What are the assumptions made in making the API Gateway interface compatible with OpenAI?

A: The assumptions made in making the API Gateway interface compatible with OpenAI include:

The OpenAI SDK will be compatible with AIBrix's API Gateway
The proposed changes will not introduce any significant performance or scalability issues

Q: What is the implementation plan for making the API Gateway interface compatible with OpenAI?

A: The implementation plan for making the API Gateway interface compatible with OpenAI includes:

Short-term (next 2 weeks): Update API Gateway code to be fully compatible with OpenAI-compatible format and develop test scripts using the OpenAI SDK to verify compatibility
Medium-term (next 4 weeks): Implement operational interface compatibility with vLLM and extend metadata service for new interfaces
Long-term (next 6 weeks): Document supported compatible APIs and replace curl-based with SDK-based queries

Q: What is the expected outcome of making the API Gateway interface compatible with OpenAI?

A: The expected outcome of making the API Gateway interface compatible with OpenAI is to enable seamless integration with OpenAI-related ecosystems, support a wider range of use cases and applications, and enhance the overall user experience and adoption of AIBrix.

Q: What is the timeline for making the API Gateway interface compatible with OpenAI?

A: The timeline for making the API Gateway interface compatible with OpenAI is as follows:

Short-term (next 2 weeks): Update API Gateway code and develop test scripts
Medium-term (next 4 weeks): Implement operational interface compatibility and extend metadata service
Long-term (next 6 weeks): Document supported compatible APIs and replace curl-based with SDK-based queries

Q: Who is responsible for making the API Gateway interface compatible with OpenAI?

A: The responsibility for making the API Gateway interface compatible with OpenAI lies with the AIBrix development team.

Q: What is the expected impact of making the API Gateway interface compatible with OpenAI?

A: The expected impact of making the API Gateway interface compatible with OpenAI is to enhance the overall user experience and adoption of AIBrix, and to enable seamless integration with OpenAI-related ecosystems.