Understand research results
Ask which question the evidence answers before reading its conclusion. This is your review checklist, not a feature that scores research for you.
Distinguish research stages
Importing data, writing a strategy, submitting an evaluation, and completing it are different stages. Success at one does not establish success at the next.
| What you have | What it confirms |
|---|---|
| Tool configuration or tool list | Configuration exists. A successful connection and loaded tools establish a usable connection, but still not an evaluation. |
| Submission reply and operation ID | Work was submitted and the original operation can be tracked. Check whether it finished successfully. |
| Partially successful result | Some output exists. Check which step failed; this is not full completion. |
| Successfully completed evaluation and output files | You can review this historical evaluation, not assume future results will match. |
Edgewright researches; it does not place orders. If a report says “Ready for live trading,” ask for evidence. Do not treat historical evaluation as actual trades already made.
Identify what produced each result
“Identity” here means the data, strategy, and configuration that produced the result.
Before comparing results, check:
- Dataset ID and instrument.
- Strategy ID and version.
- Evaluation period, execution method, capital, and cost settings.
- Operation ID, evaluation ID, and output files.
If you intended to compare only trading costs but the second evaluation uses a different dataset, ask the tool to explain before attributing return changes to costs.
Read an evaluation in Soulhall
Open an evaluation from Engine records on the Research page, from a report, or from a result card. Everything on it is the engine's own measurement; Soulhall does not compute or fill in figures. A dash means the engine found a figure undefined for this run, for example a profit factor with no losing trade.
| Tab | What it holds |
|---|---|
| Overview | Net profit, maximum drawdown, trades and final equity, the equity curve, and price with every trade marked. |
| Performance summary | Returns (including CAGR), risk (Sharpe, Sortino, Calmar, drawdown length), trading (win rate, profit factor, average trade, time in market) and costs. Daily and monthly returns sit below. |
| List of trades | Each round trip with entry, exit, side, holding time, exit reason and P&L. Pick one to zoom the chart to it. |
| Fills | Every engine fill with its position change, commission and extra cost. |
Result provenance at the bottom names the dataset and result fingerprints, so two evaluations can be matched to exactly what produced them. Replay plays the run back on a trading terminal; see Strategy replay.
Parameter sweeps and luck
A sweep runs the same strategy across a grid of parameter values. Soulhall draws it as a heatmap: choose the metric under Color by, and the Rows, Columns and Segment to show. Each cell is a full evaluation; select it to open it.
The more combinations you try, the more likely the best one is just luck. The sweep therefore carries the engine's verdict on its best cell, using the Deflated Sharpe ratio, which corrects for every trial this kind of strategy has had:
| Verdict | Meaning |
|---|---|
| Survived | After the correction, the best cell's edge over luck is 95% or more. |
| Borderline | It clears 90% but not 95%. Worth a held-out test, not yet an edge. |
| Likely luck | A cell this good is what luck alone tends to produce with this many trials. Test it out of sample first. |
| Undetermined | The engine cannot define the figure here yet, so it makes no claim. |
When the sweep was split into In sample and Out of sample periods, Soulhall shows the cell the engine chose on the in-sample period next to how it did on data it had not seen. A cell that only works in sample was fitted to that period.
Read risks and scope
Ask the researcher which instruments, periods, execution assumptions, and costs the result applies to. Conclusions should stay within the evidence.
Ask: “Does this hold only for the tested period? Does it hold with higher costs? Which missing data could affect my decision?” These are questions for you to raise, not checks automatically attached to every evaluation.
Historical evaluation is neither prediction nor investment advice. Even after the engine finishes calculating, you must judge whether evidence supports this decision.
Recognize overfitting
Think of overfitting as repeatedly changing a strategy until it excels at an old exam paper. Better-looking results on old data do not necessarily answer new questions better.
Ask how many attempts were made, what conditions changed, and whether test data was used to tune the strategy. Do not accept only the best final attempt.
Soulhall’s research guidance for Embodiments also requires disclosure of search volume. A number selected after many experiments cannot be reported like the same number from one explicit test. Write a hypothesis that could be disproved before researching, rather than changing the question after seeing results.
Make every conclusion traceable
Match important conclusions to evaluation IDs and output files. If you receive only a written summary, request the original evaluations and files and read them before judging.
Evaluation details can identify output files, which tools can retrieve using those identifiers. Ask: “Identify the evaluation and file supporting this conclusion, and which result inside supports it.” Mark claims without evidence as unverified.