LLM Eval, LLMs for Market Research, and more
In this substack you will read about LLMs for Market Research, for software engineering, and evaluating LLMs. You will also read my take on the book "Give & Take".
In my substack, deep random thoughts, I share a randomly selected set of my writing every week. My posts will be related to LLMs (AI in general), product dev and UX, health (founders’ flavor), startup related topics, and of course the events we run!
LLM Stuff
In our recent #LLM workshop Josh Seltzer discussed the impact of language models on market research, including data collection, analysis, and the potential for AI transformation in the industry. He highlighted the benefits of combining qualitative and quantitative research methods and provided a case study on the use of language models in analyzing political associations. He also discussed the challenges and limitations of using language models in market research.
Topics:
-------
๏ Data Collection in Market Research
* Market research traditionally employs qualitative and quantitative approaches.
* Qualitative research involves methods like focus groups and interviews, while quantitative research is conducted through surveys.
* Language models enable automated conversations at scale.
* Conversational agents like Inca facilitate rich and semi-structured data collection.
๏ Analysis in Market Research
* Language models are being leveraged at all stages of the research lifecycle, including hypothesis generation, idea generation, knowledge management, and content creation.
* Language models enable researchers to ask questions about their data and explore it in new ways.
* Probing with language models can provide more detailed and valuable insights in market research analysis.
๏ Case Study: Political Associations
* Language models were used to analyze people's associations with political leaders.
* Probing with language models doubled the number of significant differences in associations.
* Probing provided richer insights into the specific policies that people associate with different leaders.
๏ Challenges and Limitations of Language Models in Market Research
* There are concerns about survey fraudsters using language models to generate fake survey responses.
* The adoption of AI in market research has been slow due to the lack of quality insights generated by AI tools.
* There is an emerging debate on the use of synthetic data in market research, with concerns about bias and the need for further research.
* Despite challenges, embracing AI technologies in market research can lead to significant financial gains.
At our Journal Club, we looked at the question: Can Language Models Solve Real-World Coding Challenges?"
Paper: https://arxiv.org/abs/2310.06770
Repo: https://github.com/princeton-nlp/SWE-benc
Add to your calendar: https://lu.ma/llm-journal-club
🌟 The Challenge: Language models are everywhere, from chatbots to coding helpers. However, our methods for evaluating their abilities are falling short. We need tougher tests that mimic real-world scenarios to improve these models.
🤔 Why It's a Challenge: The paper introduces SWE-bench, a fresh way to evaluate LMs. It offers 2,294 real software engineering problems taken from GitHub. The twist? Fixing these issues often requires understanding complex codebases and coordinating changes across multiple files. It's no walk in the park.
💡 Key Results:
1. The difficulty of the tasks varies across repositories, and different LLMs struggle to solve the same issues.
2. More context doesn't always help; LLMs can get distracted by too much information.
3. The date of the code changes doesn't significantly impact LLM performance which implied that there isn’t much memorizing and cheating.
4. Fine-tuned models are sensitive to changes in context.
5. Generating whole files is harder than patches.
6. LMs often produce shorter and simpler solutions compared to human-written ones.
We finalized the list of the speakers for our Dec 8 Workshop on LLM Eval
Add to your calendar: https://lu.ma/llm-eval
Finally, I’ll be speaking at Analytics Vidhya Data Hour about LLM Multi-agent Systems and some of our work at Aggregate Intellect!
Sign up here: https://community.analyticsvidhya.com/c/datahour/era-of-ai-assisted-innovation/
Related sources:
https://www.conversationagent.com/2017/09/successful-givers.html
https://www.conversationagent.com/2016/07/cognitive-empathy.html
Hit reply and let me know what you think about this format.