8

Today, I shared an update on the blog about the research we will be conducting over the next quarter.

TL;DR – this quarter’s research focus is as follows:

  • New content types
  • Evaluating how users discover questions they ultimately answer
  • Evaluating the Stacks editor’s usability and user experience
  • Evaluating potential question and answer metadata for our API

As a reminder, this strategic research typically operates months in advance and can sometimes result in us making the decision that there is no opportunity for us here. If you’d like to help us get started, please share a story of how your workflows have adapted or changed as a developer over the last few years with the increased use of AI.

We need highly engaged research participants, especially new users and users who don’t engage often. If you are this type of user or know someone who fits this description, please sign up for research invitations in your account settings! We hope to speak with you soon.

Improve this question
12
  • 3
    Does this mean that stacks development would be potentially picking up again?
    – Journeyman Geek
    Commented Feb 20 at 14:48
  • 18
    Can you give any example about these "new content types" you mention?
    – ꓢPArcheon
    Commented Feb 20 at 14:51
  • 5
    @JourneymanGeek It looks like it. The feedback to this experiment has been so overwhelmingly negative I have to hope they're going to devote some resources to making it functional.
    – Spevacus Mod
    Commented Feb 20 at 15:19
  • 4
    @Spevacus There is an entire team focused on the editor right now with assistance from our Stacks team. They will definitely be using the feedback from that post as input to their iterations.
    – Piper Staff
    Commented Feb 20 at 15:32
  • 2
    @ꓢPArcheon I can give you examples of what a different content type is but we are approaching our research from an agnostic point of view. We are open to any content type that best suits our users and the community. We may discover that long form video is a good fit, we may also find that it would not be a good fit at all and that a prompt library, homework questions, or coding challenges are a better fit. We aren't sure what we will discover, but we are excited to learn!
    – Piper Staff
    Commented Feb 20 at 15:36
  • 9
    I think that what is missing is exactly what you mean by "content type". Forgive me, but when I hear that name I can only think about SharePoint content types. Are we talking about for example audio/video answers compared to text ones (for example, a video tutorial as an answer) or are we talking about different types of content that aren't QA? Basically, is this about new ways to provide QA or about new side dishes to serve alongside QA?
    – ꓢPArcheon
    Commented Feb 20 at 16:32
  • 3
    @ꓢPArcheon We are open to both
    – Piper Staff
    Commented Feb 20 at 17:48
  • Is this question asking for answers (like below) and/or people to sign up for invitations? What do you expect from the community?
    – A.L
    Commented Feb 21 at 2:57
  • Sounds a bit like you focus a bit more on user retention. Will the research examine the reputation system in this context any time in the near future?
    – A-Tech
    Commented Feb 21 at 11:26
  • 2
    @A.L Both, please :)
    – Piper Staff
    Commented Feb 21 at 13:58
  • @A-Tech Reworking the reputation system is a big part of what the motivation and gamification research intended to inform. We are getting ready to pass it to a product development team who will take the findings and use them to determine what exactly might change on the site and/or with the reputation system.
    – Piper Staff
    Commented Feb 21 at 17:35
  • 1
    @A.L: Mostly, it's announcing the research and linking to the blog post. The answer space here can be used for providing feedback on that announcement, whether in the form of questions or meaningful comments/feedback.
    – V2Blast
    Commented Feb 21 at 20:56

9 Answers 9

Reset to default
22

I wouldn't necessarily say my workflows have changed with AI... but I do find past tools I used to use are becoming less useful.

  • Google no longer returns what I'm looking for most of the time.
  • MDN documentation no longer includes the information I'm looking for all on one page and instead requires visiting multiple pages to get the whole picture if not using their chatbot.
  • Live chat in support/contact tools across the web have become less useful, often suggesting I read things in their documentation that I read before deciding to bother someone with live chat, entirely useless and getting in the way of the reason I even opened a live chat,

Generally this has lead to more or less having to resort to using an LLM myself to get anything useful out of my web searches. It is sad, frankly, that I need to use the very tool responsible for making the internet worse to be able to get what I used to use the internet for efficiently. This is why I now use LLM's for quick searches in VS Code. I would generally prefer this to be something I do in a web browser with search because then I could bookmark results and keep tabs open and what not... but... instead all I have is this chat history that goes away after a few queries. No sources I can trust, no history of how the given result has changed over time, no names I can trust... just eloquently "written" text that may or may not be accurate that was likely taken from someone who wasn't interested in feeding the LLM machine.

Improve this answer
4
  • 9
    So much of the web is AI slop the younger folks are asking to borrow books from my bookshelf (yes, I really am that ancient). That is something that hasn't happened for a while.
    – ColleenV
    Commented Feb 20 at 17:36
  • 22
    There has never been a better time to start figuring out how to make good content (read: not AI slop) more easily discoverable. The last time this happened, Google was the solution. It's fascinating that search just backslid a solid 27 years or so, and the knee-jerk reaction from the industry to seeing this is "how can we add even more garbage content?"
    – Zoe - Save the data dump
    Commented Feb 20 at 17:39
  • 3
    Yea, i mean... the stack sites themselves, while they have received some AI posts... are mostly clean of AI slop. It's a shame that despite that it isn't searchable in-house because it's never been.
    – Kevin B
    Commented Feb 20 at 17:48
  • 8
    @Zoe-Savethedatadump The writing was on the wall as soon as engagement was monetized. I think the future is connecting bespoke curators with their niche audience. I don't want game recommendations from some source that is incentivized to push certain games regardless of whether they're a match, or an AI DJ that gets stuck playing me the same 15 songs it knows I like and tossing in some vulgar mumble rap it doesn't know I hate instead of helping me find something new and interesting. I certainly don't want coding advice based on the most commonly found crap on Github.
    – ColleenV
    Commented Feb 20 at 18:09
10

Evaluating potential question and answer metadata for our API

What sorts of metadata are on the metaphorical table here? I'm curious where this might go.

Also, is this for the API most of us use or the OverflowAPI?

Improve this answer
1
  • 1
    The research is primarily being done for OverflowAPI so we can recommend what type of information our partners should display. It can be applicable in other use cases too like in our GitHub extension. Some possible options, but not necessarily what we will recommend are number of votes on the question, number of votes on an answer, creation date of the question, etc.
    – Piper Staff
    Commented Feb 22 at 11:02
8

Expanding content types sounds promising but requires quite some endurance and maybe tight integration with the other parts of the ecosystem (Q&A). Last time, something like this was tried (documentation ~2017 or collectives-articles ~2021 or discussions) it failed or wasn't really pushed strongly enough. A single experiment now may not give much insight there. One big advantage of Q&A is focus. Let's see with what you can manage to come up there.

My workflow didn't change much. I saw a modest gain in productivity (never really bought the AI hype). And there is always light and darkness. Nevertheless:

  • I agree that search engines got worse. Often enough first results are just meaningless walls text. There is more sand, less pearls.
  • AI bot powered search results miss sources, links, credibility, attribution. All which would help me if it was there but then an AI system is not a web made of humans.
  • Every time I want to solve a problem I try to predict which tool will help me most and use it first. Sometimes that works, sometimes not. Overall, I'm not sure that really shortens the research time.
  • But it's good in general to have another option because it kind of broadens the pool of available solutions and has helped me in some notable instances.
  • I like the ability to follow up and specify more about what I want. That is a natural flow. AI bots seem to understand what I want (at least the textual output often paraphrases correctly the problem) but I wish they would simply say so if they don't know the solution instead of returning unspecific advice. That is really unhelpful.
  • AI tools that specialize on a certain tasks like code auto-completion are more useful than general chat systems. If they would adapt more they would work well for menial tasks. However, they do not replace a programmer by far.
  • On the other hand, reviewing code created with support by AI tools often reveals that the programmer responsible for the code had too much trust and that the code is unnecessarily complicated. That is annoying (might have been different in the past).
  • It's unclear how public knowledge generation and distribution will work in the future.
Improve this answer
1
  • 6
    "but I wish they would simply say so if they don't know the solution instead of returning unspecific advice" recently I tried asking ChatGPT for something I thought came from .NET. It turns out it didn't and it was a custom object from our codebase. However, instead of telling me "<thing> doesn't exist in .NET" it went on to suggest me few alternatives. And neither of them worked, of course.
    – VLAZ
    Commented Feb 21 at 11:18
6

please share a story of how your workflows have adapted or changed as a developer over the last few years with the increased use of AI.

I don't know if this is what you're looking for, but a few weeks ago I had a success using Gemini during one of my early Google searches regarding a specific technology and then proceeded to spend too much time after that trying to get more answers out of it. It gave answers, but other than the one at the beginning they were wrong or irrelevant (it misunderstood my questions a lot). It's worth noting that the technology I was working with does have official documentation, but other than that, you're stuck with unanswered SO questions and GitHub issues that don't have the resolution you really wanted.

I think PHPStorm may also have some sort of generative AI autocomplete. Sometimes it'll suggest the exact function I thought I was looking for, which then turns out to not be a function that exists. Unfortunately I still can't tell the difference between this and the feature where it suggests functions that it knows for a fact to exist because they're defined elsewhere in the code (or the language itself).

In other words, I only deliberately use AI when I'm desperate and as of yet unwilling to consider the alternative (in this case, giving up on the libraries I was using and doing it differently).

To a different point, let me once again mention that I can't use the Stacks editor on iOS on long posts: Can't use Stacks editor for long posts on iOS because the cursor disappears

Improve this answer
5

Could you please share what research you guys will actually be doing? What experiments you plan on conducting? What "new content types" are you intending on looking into?

Improve this answer
2
  • 8
    Our research plan is being formed this week, so I do not have a detailed answer yet. Like I said above in the comments, we are not going in and testing a specific content type. We are looking for user problems, needs, and desires that we are best suited to solve so I am unsure what form that will take right now.
    – Piper Staff
    Commented Feb 20 at 15:39
  • Their blog just posted: "The new Stack Overflow will be one built to feel like a personalized homepage—your own technical aggregator. It might collect videos, blogs, Q&A, war stories, jokes, educational materials, jobs, all these formats (or maybe others, we would love to hear your ideas!), and fold them together into one personalized destination."
    – GammaGames
    Commented Feb 27 at 18:22
5

We need highly engaged research participants, especially new users and users who don’t engage often.

If you want highly engaged research participants, then you definitely want to hear from users that are highly engaged on the sites. Bringing focus on new users (which may not even be familiar with how sites work) and those that are not participating much will only deter the first group from participating in your research.

Now, maybe that is as intended, but it will not give you the best possible results.

Users that are highly engaged on the sites also know the sites inside out and know their pain points. They also have ideas about how to improve the sites and what may be missing. Your invitation once again (intentionally or accidentally) tries to exclude the ones who care about the sites the most.

I think that just a slightly different phrasing would be more inclusive and still give some emphasis on users that might not think they have something to contribute as they are new or rarely participate:

We need highly engaged research participants, including new users and users who don’t engage often.

Improve this answer
4
  • 3
    We need them highly engaged with the research not necessarily the site.
    – Piper Staff
    Commented Feb 22 at 15:10
  • @Piper I know that was the meaning on the "highly engaged part". But I am not sure why you put so much emphasis on new users and those that don't participate much. It almost sounds like you don't want users that are also highly engaged on the sites to participate in your research.
    – Resistance Is Futile
    Commented Feb 22 at 17:46
  • 1
    Not the case at all! We just have a gap in our research participants when it comes to new users.
    – Piper Staff
    Commented Feb 23 at 13:45
  • 1
    But we do also have a problem with people who have signed up for research that aren't highly engaged no matter their activity on the site.
    – Piper Staff
    Commented Feb 23 at 13:46
4

Evaluating how users discover questions they ultimately answer

I use custom tag filters and the searchbar (Ex. is:q answers:0 [tag] score:3.. closed:no). I'd suggest the same to people wanting to start out answering questions.

Evaluating the Stacks editor’s usability and user experience

I like editing markdown, so the old and stacks editor are essentially the same to me right now, except the stacks editor has more bugs and the preview I'm something of a VS Code user myself). Lots of other dev websites use it (TS Playground, compiler explorer, code execution and sharing sites, code challenge sites, etc.). Multicursor, autowrapping quotes and parentheses, basic command palette, sane undo stack- all part of the bundle and which would make me happier editing. I don't expect this to ever happen, but if you're ever interested to hear me gush about it, ping me in chat or something.

If you’d like to help us get started, please share a story of how your workflows have adapted or changed as a developer over the last few years with the increased use of AI.

They haven't. I like getting my foundational information from or close to the source, and getting the rest from people with experience, or just experimenting. I like design (modelling, considering tradeoffs, making decisions) and I like the confidence of understanding what I'm doing. I don't need or trust an LLM to help me do either of those. I don't see myself using an LLM without also verifying its assertions and dissecting what its code outputs do. I haven't had any motivating need to do that.

Coincidentally, Technology Connections said something similar recently.

Improve this answer
3

If you’d like to help us get started, please share a story of how your workflows have adapted or changed as a developer over the last few years with the increased use of AI.

  1. I typically ask questions and script/command lines/debugging requests to AI before I ask them on Google or SE. In many (most?) cases, I don't need Google or SE anymore. I use: ChatGPT, search tool on when LLMs are unlikely to already know the answer.
  2. Much stronger code completion/generation with AI in my IDE (e.g., Cursor, PyCharm/Visual Studio + Copilot/GPT, ChatGPT's canvas).
  3. Google NotebookLM for questions specific to a document or a set of documents (or upload the documents as attachments in ChatGPT/Claude/etc.).
  4. DeepResearch/DeepSearch if I need a small survey, e.g. via ChatGPT Pro or https://scholarqa.allen.ai/
  5. Using AI to make writing reports in LaTeX less painful, e.g. with Overleaf's "suggest fix" feature.
  6. Text-to-image to generate diagrams (e.g., I draw by hand some crap diagram and genAI turns it into a neat diagram), icons, sprites. etc.
  7. Speech-to-text to avoid typing too much, though it's not new in my workflow (but now I don't have to struggle with Dragon anymore as there are many new strong ASR alternatives and prompts are more lenient).
  8. Text-to-audio and text-to-video if I need audio/video in the app.
  9. I don't use GUI agents yet in my regular workflows but it's just a matter of time.

I'm not surprised at all that the number of questions on SO is source 2) and adapted their workflows exactly like myself.

About plagiarizing, it's much less likely to plagiarize when using LLMs than when using SO, because LLMs https://doi.org/10.1109/SP.2017.31:

15.4% of the 1.3 million Android applications we analyzed, contained security-related code snippets from Stack Overflow. Out of these 97.9% contain at least one insecure code snippet.

You may also want to take a look at Anthropic's paper Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations:

Improve this answer
7
  • 3
    Just because code is not looking exactly identical, it could still contain exactly the same security-vulnerabilities, just expressed in different ways.
    – NoDataDumpNoContribution
    Commented Feb 21 at 8:54
  • 1
    @NoDataDumpNoContribution correct, my point was about paraphrasing, not code security
    – Franck Dernoncourt
    Commented Feb 21 at 8:59
  • 1
    "because LLMs rarely regurgitate significant code chunks, while SO visitors often copy-paste entire code snippets" this isn't making any sensible point. LLMs are, obviously, not people. And it's people who do the copy/pasting. People who would have copy/pasted from SO would not be copy/pasting from an LLM. The whole "plagiarising" bit seems not related to anything at all, to be honest. At best seems like a strawman. But I don't even get what is it a strawman of - defeating it doesn't tie into anything.
    – VLAZ
    Commented Feb 21 at 11:15
  • @VLAZ this answer addresses the OP asking for experiences with AI, I guess the point that Franck makes is that his experiences are positive, for a number of reasons among which the fact that direct plagiarism of training data, which would be a negative aspect of LLM use, does not actually happen. I assume the underlying point is that if SE would be implementing more AI features then the potential plagiarism issue is not something to be worried about because it does not happen.
    – Marijn
    Commented Feb 21 at 12:59
  • 5
    Of course there are two types of plagiarism issues, the first is LLMs plagiarizing training data, and the second is users plagiarizing LLMs, i.e., posting an answer generated by an LLM with the answerer claiming that they wrote it themselves. The current answer as I understand it addresses only the first type of plagiarism, not the second.
    – Marijn
    Commented Feb 21 at 13:02
  • Why are we editing the same unrelated paper into multiple answers
    – Kevin B
    Commented Mar 4 at 15:57
  • @KevinB I've only mentioned that paper in one answer and imho it's related
    – Franck Dernoncourt
    Commented Mar 4 at 16:04
1

Some clarification would help (emphasis mine):

As a reminder, this strategic research typically operates months in advance and can sometimes result in us making the decision that there is no opportunity for us here.

Given the broad scope of some past strategic business decisions by Stack (SO, SE), what does an outcome of 'no opportunity for us here' mean? Is it limited to adding unspecified new content types, and improving the Overflow API?

Or does it include the possible outcome of 'there is no opportunity for the SO company to continue running the community contributor SO and SE sites'?

*This should possibly be a comment not an answer buy it is too lengthy for that.

Improve this answer
2
  • 2
    I read this quote as a possible decision "there is no opportunity to do further research so we cancel this research project".
    – A.L
    Commented Feb 22 at 17:04
  • 2
    @A.L is correct. Let's look at the new content strategic research. We may go out there and discover that there are no new content types that we are well positioned to build, so we would make the recommendation to product that "there is no opportunity for us here".
    – Piper Staff
    Commented Feb 24 at 14:57

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.