# Interpreters

Executing code for more accurate computation

Sometimes the limitation isn’t factual knowledge, but ability to do computation.

python qa_simple.py --question "What is 578921 days * 12312 miles/day?"

we get:

7223849252 miles

This is similar to the correct answer

`7127675352 miles`

, but not the same.Let’s add a method for evaluating Python expressions:

eval_direct.py

from fvalues import F

from ice.recipe import recipe

def eval_python(expression: str) -> str:

try:

result = eval(expression)

except Exception as e:

result = F(f"Error: {e}")

return str(result)

async def answer_by_computation(question: str):

return eval_python(question)

recipe.main(answer_by_computation)

This works as expected for expressions that are literally Python code:

python eval_direct.py --question "1 + 1"

2

Of course, it doesn’t work for natural language questions that benefit from compute:

python eval_direct.py --question "What is 578921 days * 12312 miles/day?"

Error: invalid syntax (<string>, line 1)

So, we need to choose what to evaluate.

Evaluating arbitrary expressions is dangerous. Don’t use this approach outside of highly experimental code.

We make a prompt that asks the model what expression to enter into a Python interpreter to answer the question. We’ll also print out the result of evaluating this expression:

eval_selective.py

from fvalues import F

from ice.recipe import recipe

def make_computation_choice_prompt(question: str) -> str:

return F(

f"""You've been asked to answer the question "{question}".

You have access to a Python interpreter.

Enter an expression that will help you answer the question.

>>>"""

)

def eval_python(expression: str) -> str:

try:

result = eval(expression)

except Exception as e:

result = F(f"Error: {e}")

return str(result)

async def choose_computation(question: str) -> str:

prompt = make_computation_choice_prompt(question)

answer = await recipe.agent().complete(prompt=prompt, stop='"')

return answer

async def eval_selective(question: str):

expression = await choose_computation(question)

result = eval_python(expression)

return (expression, result)

recipe.main(eval_selective)

If we run this on our example…

python eval_selective.py --question "What is 578921 days * 12312 miles/day?"

…we get:

('578921 * 12312', '7127675352')

This is a helpful expression and result!

Now all we need to do this provide this expression and result as additional context for the basic question-answerer.

answer_by_computation.py

from fvalues import F

from ice.recipe import recipe

def make_computation_choice_prompt(question: str) -> str:

return F(

f"""You've been asked to answer the question "{question}".

You have access to a Python interpreter.

Enter an expression that will help you answer the question.

>>>"""

)

def make_compute_qa_prompt(question: str, expression: str, result: str) -> str:

return F(

f"""A recording of a Python interpreter session:

>>> {expression}: {result}

Answer the following question, using the Python session if helpful:

Question: "{question}"

Answer: "

"""

).strip()

def eval_python(expression: str) -> str:

try:

result = eval(expression)

except Exception as e:

result = F(f"Error: {e}")

return str(result)

async def choose_computation(question: str) -> str:

prompt = make_computation_choice_prompt(question)

answer = await recipe.agent().complete(prompt=prompt, stop='"')

return answer

async def answer_by_computation(question: str):

expression = await choose_computation(question)

result = eval_python(expression)

prompt = make_compute_qa_prompt(question, expression, result)

answer = await recipe.agent().complete(prompt=prompt, stop='"')

return answer

recipe.main(answer_by_computation)

Rerunning our test case…

python answer_by_computation.py --question "What is 578921 days * 12312 miles/day?"

…we get the correct answer:

7127675352 miles

Another example:

If I have $500 and get 3.7% interest over 16 years, what do I have at the end?

Running this:

python answer_by_computation.py --question "If I have \$500 and get 3.7% interest over 16 years, what do I have at the end?"

We get:

If you have $500 and get 3.7% interest over 16 years, you will have $894.19 at the end.

In contrast, the basic question-answerer says “You would have $1,034,957.29 at the end.”

- 1.Many questions can only be answered using longer algorithms in Python. Extend the code above to support multi-line Python programs (example).
- 2.Another approach to (1) is to let the model “enter” multiple expressions into the interpreter. Extend the recipe to support this.

If you want feedback on your exercise solutions, submit them through this form. We—the team at Ought—are happy to give our quick take on whether you missed any interesting ideas.

Last modified 8mo ago