Advertising
Promote your product or service to developers and technologists.
Google, we had a solid AI tool (Gemini) to help us test, identify, and produce automated feedback.

In the end, we found that evaluating question quality and determining the appropriate feedback required some classic ML techniques in addition to our generative AI solution. This article will walk through how we considered, implemented, and measured the results of term frequency inverse document frequency (TF IDF) before passing in those features to logistic regression models.

Although we were building more traditional ML models to flag questions based on quality indicators, we still needed to pair it with an LLM in the workflow to provide actionable feedback. Once an indicator flags a question, it sends a preloaded response text with the question to Gemini, along with some system prompts. Gemini then synthesizes these to produce feedback that addresses the indicator, but is specific to the question.

This mermaid diagram shows the flow:

A sequence diagram illustrating the interaction between a Staging Ground User and a Question Feedback Service, which consists of Feedback Classifiers and an LLM (Large Language Model). The user sends a question title and body to the service. Feedback Classifiers assess the question draft against predefined feedback indicators. If indicators are positive, pre-loaded feedback text is retrieved and sent along with the question details to the LLM. The LLM generates specific feedback based on the positive indicators, which is then returned to the user. The process repeats for multiple edits until the question is refined. The diagram includes arrows showing the flow of interactions and a highlighted section indicating that this process repeats for multiple edits.Ask Question page with the Ask Wizard. This time, we wanted to confirm the results of the first experiment and see if Question Assistant could also help more experienced question-askers.

We saw a steady success rate of +12% across both experiments. With the meaningful success rates and consistency of our findings, we made Question Assistant available to all askers on Stack Overflow on March 6, 2025.

The next step forward

Changing course is not uncommon in research and early development. But realizing when you're on a path that won’t provide impact and pivoting to new logic is key to making sure all the puzzle pieces still fit together, just in a different way. With traditional ML and Gemini working together, we were able to fuse the suggested indicator feedback and the question text in order to provide more specific, contextual feedback that is actionable for the asker in improving their question, making it easier for users to find the knowledge they need. This is one step forward in our work to improve the core Q&A flows to make asking, answering, and contributing to knowledge easier for everyone. And we’re not done with Question Assistant just yet. Our Community Product teams are looking ahead to ways we can iterate on the indicator models and further optimize the question-asking experience with this feature.