Don’t trust AI when coding!

Coding is evolving quickly. Tasks that can be described are now easily converted into code by LLM models. But is it safe to trust external tools that might lack awareness of syntax, best practices, and the pros & cons of different solutions?

Key Advice:
“Always test, write comprehensive unit tests, find errors, and suggest improvements.”

Let’s perform a simple test. Here’s the prompt:

Write typescript function which can interprete code. Input is multilingual string. Each command is in new line. Line contains command and parameter (except END command). Possible commands are: MUL which multiplies number by parameter value. ADD which adds parameter to previous value. FUN which parameter is string with function name. END ends function. Functions can include only MUL and ADD commands. Function is called when command INV is invoked with parameter which is function name. 

There is only one command per line. Commands may include trailing and leading spaces. Functions cannot call itself.

Assumption is that initial value is 0. 

Google Gemini and ChatGPT generate answer within 1 second, provide test case and assumption that code will work

Gemini

type Command = {
  command: string;
  parameter: string | undefined;
};

type FunctionDefinition = {
  name: string;
  commands: Command[];
};

function interpretCode(code: string): number {
  const commands: Command[] = code
    .split("\n")
    .map((line) => {
      const [command, parameter] = line.trim().split(" ");
      return { command, parameter };
    });

  const functionDefinitions: Record<string, FunctionDefinition> = {};
  let currentFunction: FunctionDefinition | undefined;
  let currentValue = 0;

  for (const command of commands) {
    console.log(command);
    switch (command.command) {
      case "MUL":
        currentValue *= parseFloat(command.parameter!);
        break;
      case "ADD":
        currentValue += parseFloat(command.parameter!);
        break;
      case "FUN":
        currentFunction = { name: command.parameter!, commands: [] };
        functionDefinitions[currentFunction.name] = currentFunction;
        break;
      case "END":
        currentFunction = undefined;
        break;
      case "INV":
        const functionToInvoke = functionDefinitions[command.parameter!];
        if (functionToInvoke) {
          currentValue = interpretCode(functionToInvoke.commands.map((cmd) => cmd.command + " " + cmd.parameter).join("\n"));
        }
        break;
      default:
        throw new Error(`Invalid command: ` + command.command);
    }
  }

  return currentValue;
}

// Example usage:
const code = `
FUN myFunction
MUL 2
ADD 3
END
INV myFunction
`;

const result = interpretCode(code);
console.log(result); // Output: 6

All looks good, it seems to cover simple scenarios of code without functions and with functions.
But what happens when we try to run this code?

We analyze code, check which command fails, and suggest to gemini, that empty lines needs to be ignored.

It adds filter method to ignore empty lines before mapping

.filter((line) => line.trim() !== “”)

We fix code, run again, but result is still 0.

What is wrong?

Experienced developer may need a few minutes to check how recursive function is called.
In line 22 we can see initial value set to 0 

let currentValue = 0;

Lets’ provide suggestions to Gemini.

Result is 0. An issue may be in line 22 – you are always setting initial value to 0, so current value is not changed by result of recursively called function.

Gemini is modifying code and this time it works correctly.
However if we don’t provide clear hint about initial value set to 0, it will provide subsequent solutions which are not working as expected.

We can imagine, what can be consequences of adding that type of code to our repository, especially if we don’t have proper unit testing in place. 

ChatGPT

For that task free edition of ChatGPT seems to produce different approach, covering more edge cases, but not responding as expected

type Command = 'MUL' | 'ADD' | 'FUN' | 'END' | 'INV';

interface FunctionCommand {
    type: 'FUN';
    name: string;
    operations: Array<{ command: 'MUL' | 'ADD'; param: number }>;
}

function interpretCode(input: string): Record<string, number> {
    const lines = input.split('\n');
    let currentValue = 0;
    const functions: Record<string, FunctionCommand> = {};
    let currentFunction: FunctionCommand | null = null;

    const applyFunction = (funcName: string, value: number): number => {
        const func = functions[funcName];
        if (!func) {
            throw new Error(`Function ${funcName} not found`);
        }
        for (const operation of func.operations) {
            if (operation.command === 'MUL') {
                value *= operation.param;
            } else if (operation.command === 'ADD') {
                value += operation.param;
            }
        }
        return value;
    };

    for (let line of lines) {
        // Trim leading/trailing spaces and split into command and parameter
        line = line.trim();
        if (line === '') continue;

        const [command, param] = line.split(/\s+/);

        switch (command as Command) {
            case 'MUL':
                const mulValue = parseFloat(param);
                if (currentFunction) {
                    currentFunction.operations.push({ command: 'MUL', param: mulValue });
                } else {
                    currentValue *= mulValue;
                }
                break;

            case 'ADD':
                const addValue = parseFloat(param);
                if (currentFunction) {
                    currentFunction.operations.push({ command: 'ADD', param: addValue });
                } else {
                    currentValue += addValue;
                }
                break;

            case 'FUN':
                currentFunction = { type: 'FUN', name: param, operations: [] };
                break;

            case 'END':
                if (currentFunction) {
                    functions[currentFunction.name] = currentFunction;
                    currentFunction = null;
                } else {
                    throw new Error("Unexpected END without a FUN command");
                }
                break;

            case 'INV':
                if (currentFunction) {
                    throw new Error("Cannot use INV within a function definition");
                }
                currentValue = applyFunction(param, currentValue);
                break;

            default:
                throw new Error(`Unknown command: ${command}`);
        }
    }

    // Compute and return the result for each function
    const result: Record<string, number> = {};
    for (const funcName in functions) {
        let funcValue = 0;
        funcValue = applyFunction(funcName, funcValue);
        result[funcName] = funcValue;
    }

    return result;
}

// Example usage
const code = `
ADD 3
FUN myFunction
MUL 2
ADD 3
END

INV myFunction
ADD 1
`;

const result = interpretCode(code);
console.log(result); // Output will depend on the sequence of operations

Let’s inform ChatGPT that code is not working as expected

This time we receive code and detailed explanation

type Command = 'MUL' | 'ADD' | 'FUN' | 'END' | 'INV';

interface FunctionCommand {
    type: 'FUN';
    name: string;
    operations: Array<{ command: 'MUL' | 'ADD'; param: number }>;
}

function interpretCode(input: string): number {
    const lines = input.split('\n');
    let currentValue = 0; // Global value for commands outside of functions
    const functions: Record<string, FunctionCommand> = {};
    let currentFunction: FunctionCommand | null = null; // Tracks the function being defined

    // Helper function to apply a function's operations to a given value
    const applyFunction = (funcName: string, value: number): number => {
        const func = functions[funcName];
        if (!func) {
            throw new Error(`Function ${funcName} not found`);
        }
        for (const operation of func.operations) {
            if (operation.command === 'MUL') {
                value *= operation.param;
            } else if (operation.command === 'ADD') {
                value += operation.param;
            }
        }
        return value;
    };

    for (let line of lines) {
        // Trim leading/trailing spaces and split into command and parameter
        line = line.trim();
        if (line === '') continue;

        const [command, param] = line.split(/\s+/);

        switch (command as Command) {
            case 'MUL':
                const mulValue = parseFloat(param);
                if (currentFunction) {
                    // If defining a function, add operation to the function's operations
                    currentFunction.operations.push({ command: 'MUL', param: mulValue });
                } else {
                    // If outside any function, apply operation to global value
                    currentValue *= mulValue;
                }
                break;

            case 'ADD':
                const addValue = parseFloat(param);
                if (currentFunction) {
                    // If defining a function, add operation to the function's operations
                    currentFunction.operations.push({ command: 'ADD', param: addValue });
                } else {
                    // If outside any function, apply operation to global value
                    currentValue += addValue;
                }
                break;

            case 'FUN':
                if (currentFunction) {
                    throw new Error("Nested functions are not supported");
                }
                // Start a new function definition
                currentFunction = { type: 'FUN', name: param, operations: [] };
                break;

            case 'END':
                if (!currentFunction) {
                    throw new Error("END command without a corresponding FUN");
                }
                // Save the function definition and reset the currentFunction tracker
                functions[currentFunction.name] = currentFunction;
                currentFunction = null;
                break;

            case 'INV':
                if (currentFunction) {
                    throw new Error("Cannot use INV within a function definition");
                }
                // Invoke a function's operations on the global value
                currentValue = applyFunction(param, currentValue);
                break;

            default:
                throw new Error(`Unknown command: ${command}`);
        }
    }

    return currentValue;
}

// Example usage
const code = `
ADD 2
FUN myFunc
ADD 5
MUL 3
END
INV myFunc
ADD 1
`;

const result = interpretCode(code);
console.log(result); // Expected output: 18 (2 + (5 * 3) + 1)

We should also remember that AI is learning quickly. Above tests were implemented with free versions of models. Repeating these tests in a few weeks may give much better responses, but it is always worth splitting tasks into smaller pieces.

Conclusion: AI can greatly boost productivity and inspire efficient solutions, but it requires careful supervision, proper testing, and a deep understanding of the responses to ensure code quality.