Specification Dilemma Experiment

Small empirical probe for the claim that sparse-specification prompts produce more homogeneous outputs across users than dense-specification prompts.

See experiment.pdf for the full specification.

Design note: matched pairs

experiment.tex describes a between-groups design with 30 independently-drawn sparse prompts and 30 independently-drawn dense prompts. The prompts in this repo follow a tighter variant: matched pairs. Each of 30 imagined users has a fixed underlying intent (audience, thesis, tone, voice, opening move, structural constraint). prompts/dense.json[i] expresses that user's full intent; prompts/sparse.json[i] is what the same user would type when underspecifying — topic only, in roughly their natural register. The sparse prompts carry no audience, thesis, tone, or structural specification.

The statistical comparison is unchanged — cross-user pairwise similarity in each condition — but the two conditions now sample the same population of underlying intents. This tests the sharper claim: when users with divergent intents underspecify, outputs converge (priors dominate); when they specify fully, outputs diverge (intents dominate).

Setup

1. Install LMStudio and download a strong instruction-tuned model (e.g. Qwen2.5-72B-Instruct or Llama-3.3-70B-Instruct).

2. Start the LMStudio local server (default: localhost:1234).

3. Create the environment and install dependencies with uv:

   uv sync
   

   (or pip install -r requirements.txt inside a venv if not using uv)

4. Edit config.yaml if your LMStudio model name or port differs from the defaults. If LMStudio is on a remote host, point lmstudio.base_url at that host (e.g. http://<host>:1234/v1).

5. Smoke-test the endpoint (checks connectivity, seed-honoring, and approximate per-generation latency):

   uv run python smoke_test.py
   

Running

Freeze your prompts in prompts/sparse.json and prompts/dense.json before generating anything.

Then run the full pipeline:

uv run python run_all.py

Or run steps individually:

uv run python generate.py     # LMStudio generations
uv run python embed.py        # sentence embeddings
uv run python similarity.py   # pairwise cosine similarities
uv run python stats.py        # t-test, Mann-Whitney, bootstrap, Cohen's d
uv run python plot.py         # violin plot

Outputs

• outputs/{sparse,dense}/NN.txt : raw model completions
• embeddings/{sparse,dense}.npy : L2-normalized embedding matrices
• results/pairwise.csv : all pairwise similarities
• results/stats.json : test statistics and summary
• results/plot.png : similarity distribution plot

Interpretation

A positive result: sparse-condition mean pairwise similarity is meaningfully higher than dense-condition mean similarity, the bootstrap 95% CI on the difference excludes 0, and Cohen's d is large (>0.8).

A null or inverted result is also interesting and should be reported honestly.