ollama code generation experiment

[Anon-E-moose]

I will say this, the models sometimes get updated and then need to be re-downloaded.
(not sure how ollama works, if it check for an updated model it may use the later one)

The other thing is, iif possible I like to use instruct models vs thinking ones, if they offer them
And for the qwen models, I use the coder variation if they offer it.

Goverp, you are at the low end (hardware wise), you're stuck with smallish models, due to ram constraints (video and system memory)
Not sure how tuned ollama that you use is.
I started out with llama.cpp instead of ollama or lm-studio, as both of them use llama.cpp underneath the hood.

[Goverp]

[quote="Anon-E-moose"]

Code: Select all

Prompt:
 Rust code to convert an i64 (8-byte signed integer) to a little-endian representation                                 
                                                                                                                       
Response:
 Here's a simple Rust function to convert an i64 to its little-endian byte representation:                             
                                                                                                                       
 ```rust                                                                                                               
   fn to_le_bytes(value: i64) -> [u8; 8] {                                                                             
       value.to_le_bytes()                                                                                             
   }                                                                                                                   
 ```                                                                                                                   
 ...
Didn't take long, a couple of minutes, but the llm is on a strix halo[/quote]
Right answer, but wrong question!  I was aware of to_le_bytes, and I'd hope that any self-respecting (?) AI would be too.  The bit of the requirement that makes it harder is to use the fewest bytes, so say 127 takes one byte, but 128 takes 2, 32767 also takes 2, but 32768 takes 3, etc. (and similarly but not identically for negative numbers; the difference is what requires some care).

My installed AI's solution was to iterate over the max/min numbers (expressed as a bit mask with shifts, which is a bit involved but logical) for each number of bytes, 1-8, and compare the number in question.

What I think is optimal is to locate the number's binary representation's first 1 from the left in +ve, first 0 in -ve.  The preceding bits are all just sign bits, and we need retain at least one, but not more that 7 of them, so you can directly calculate how many of the to_le_bytes are needed.  The first part of that, locating the first 1/0 is equivalent to counting the leading 0's/1s, which are the Rust method leading_zeros and leading_ones.  The useful feature is that on modern CPUs, that's a BSR instruction on a register, which is therefore pretty quick, followed by a division by 8 (a.k.a. >>3).  Hence me suggesting it to the AI.  What I didn't check is whether the AI was aware of said methods (I think they're relatively recent introductions to Rust - trouble with moving targets :-( ), perhaps not.

[Anon-E-moose]

Right answer, but wrong question! I was aware of to_le_bytes, and I'd hope that any self-respecting (?) AI would be too.

I kind of expect it to respond with the rust version, given the broadly ambiguous prompt.

Because the prompt was "Rust code to convert an i64 (8-byte signed integer) to a little-endian representation."

I think of talking to the ai/llm as if I were talking to a 3 year old, be very specific.

After I asked it the general question, then I asked "code something similar to value.to_le_bytes()"
then I got 2 valid responses, one a loop and the other using bitwise operators.

If the original prompt was this "Rust code to convert an i64 (8-byte signed integer) to a little-endian representation. Don't use built in functions, If multiple solutions exist, give me most efficient in terms of execution speed" then the response would be different.

This is the output, it also gave me test code to run the function, with timings, though I didn't ask it to do that, I also didn't tell it not to.

Code: Select all

/// Convert an i64 to its little-endian byte representation.
/// Most efficient implementation using bitwise operations.
#[inline]
pub fn i64_to_le(value: i64) -> [u8; 8] {
    // Extract each byte using bitwise operations
    // This is more efficient than loops or arithmetic for fixed-size integers
    [
        (value as u8),                                    // bits 0-7
        ((value >> 8) as u8),                             // bits 8-15
        ((value >> 16) as u8),                            // bits 16-23
        ((value >> 24) as u8),                            // bits 24-31
        ((value >> 32) as u8),                            // bits 32-39
        ((value >> 40) as u8),                            // bits 40-47
        ((value >> 48) as u8),                            // bits 48-55
        ((value >> 56) as u8),                            // bits 56-63
    ]
}

/// Alternative implementation using a union for zero-cost abstraction.
/// This is also highly efficient but requires unsafe code.
#[inline]
pub fn i64_to_le_union(value: i64) -> [u8; 8] {
    // On little-endian systems (most modern CPUs), this is essentially free
    // On big-endian systems, this would still be efficient with just a copy
    // Note: This relies on platform endianness at runtime
    unsafe {
        let mut result = [0u8; 8];
        std::ptr::copy_nonoverlapping(
            &value as *const i64 as *const u8,
            result.as_mut_ptr(),
            8,
        );
        result
    }
}

The more I play with llms the more I realize that the prompt is extremely important, not just for directions of what to do,
but more importantly, directions of what not to do.

[pingtoo]

The more I play with llms the more I realize that the prompt is extremely important, not just for directions of what to do,
but more importantly, directions of what not to do.

If I might add a little bit on this Not only how you express the questions, but also how long you interactive with the AI. (I mean same session) I found at lease for Gemini it is (smart?) always refer back to prior conversation and trying to bring the relevancy into latest and something do comparison. This is a feature I did not found on other AI chatbots. it is one that I always want to know how the local model handle this.

[Anon-E-moose]

For remembering previous conversations of the current session, it's more the llm instead of the model.
The llm being the software that handles input (conversation) as well as output.
The model tries to make sense of your prompt, and give you a response that it thinks you asked for. Basically it "reasons".

For example I use llama.cpp, with differing models, depending on what I'm asking the llm to do, ie. code vs produce art vs general knowledge questions.
Every model has strengths and weaknesses.

I am running pi-mono (AI agent toolkit: coding agent CLI) specifically the pi coding agent, on top of the llm, which gives me a framework for long running conversations, it also keeps track of the sessions that I have run, and though I haven't gotten to using that, it should allow one to continue a conversation
at a later date, by pulling in one or more previous session transcripts.

Note: I only run a local llm, I don't use any of the cloud based ones, as my interest lies more in what a local llm can do for me.

Monologue:
IMO it's a very interesting time to be involved in the whole ai growth phenomena. It's a continuation of the computer craze.

I've watched computers, evolve from basically a room size calculator to ones that can be held in ones hand, with more processing power than the original.
When I started programming it was on unixes like aix and sco, I watched the young upstart linux go from a toy to an extremely usable operating system.

Change is the nature of society.

[Goverp]

Right answer, but wrong question! I was aware of to_le_bytes, and I'd hope that any self-respecting (?) AI would be too.

I kind of expect it to respond with the rust version, given the broadly ambiguous prompt.

Because the prompt was "Rust code to convert an i64 (8-byte signed integer) to a little-endian representation."
...
The more I play with llms the more I realize that the prompt is extremely important, not just for directions of what to do,
but more importantly, directions of what not to do.

My prompt was:

Write Rust code to convert an i64 (8-byte signed integer) to a little-endian representation using the fewest bytes

later amended to

Using the Rust functions leading_zeros and leading_ones, write code to convert an i64 (8-byte signed integer) to a little-endian representation using the fewest bytes

Note the "fewest bytes", which rules out naive use of to_le_bytes.

FWIW, here's the AI's version in response to my prompt - correct, but could be smaller and faster:

Code: Select all

pub fn to_bytes(number: i64) -> Vec<u8> {
    if number == 0 {
        return Vec::new();
    }

    for n in 1..=8 {
        let min_val = -(1i64 << (8 * (n - 1)));
        let max_val = (1i64 << (8 * (n - 1))) - 1;
        if number >= min_val && number <= max_val {
            let num = number as u64;
            let mut bytes = Vec::with_capacity(n as usize);
            for i in 0..n {
                let byte = ((num >> (i * 8)) & 0xFF) as u8;
                bytes.push(byte);
            }
            return bytes;
        }
    }
    Vec::new()
}

<edit>I just noticed the "number as u64". Oops. I did actually care about the sign!

[Anon-E-moose]

using this prompt

Using the Rust functions leading_zeros and leading_ones, write code to convert an i64 (8-byte signed integer) to a little-endian representation using the fewest bytes, do not use built in functions, write your own.

Code: Select all

/// Convert an i64 to its minimal little-endian representation.
/// 
/// Uses leading_zeros/leading_ones to determine byte count.
/// Manually extracts bytes - no built-in byte conversion functions.
/// 
/// For positive numbers: MSB of last byte must be 0
/// For negative numbers (two's complement): MSB of last byte must be 1
pub fn to_le_min(value: i64) -> Vec<u8> {
    if value == 0 {
        return Vec::new();
    }

    // Extract raw little-endian bytes manually (8 bytes for i64)
    let mut raw_bytes = [0u8; 8];
    for i in 0..8 {
        raw_bytes[i] = ((value >> (i * 8)) & 0xFF) as u8;
    }

    if value >= 0 {
        // Positive: use leading_zeros to determine significant bits
        let leading = value.leading_zeros() as usize;
        let bits_needed = 64 - leading;
        let mut bytes_needed = (bits_needed + 7) / 8;

        // If last byte's MSB would be 1, we need a zero-padding byte
        if (bits_needed % 8) == 0 {
            bytes_needed += 1;
        }

        // Clamp to max 8 bytes
        let len = bytes_needed.min(8);
        raw_bytes[0..len].to_vec()
    } else {
        // Negative: use two's complement sign extension logic
        // Count leading ones (sign extension bits)
        let leading_ones = value.leading_ones() as usize;

        // The significant bits are 64 - leading_ones
        let bits_needed = 64 - leading_ones;

        // Calculate bytes needed
        let mut bytes_needed = (bits_needed + 7) / 8;

        // Need at least 1 byte for any negative number
        if bytes_needed == 0 {
            bytes_needed = 1;
        }

        // For negative numbers, if significant bits don't reach MSB of last byte,
        // we need sign extension (0xFF) byte
        if bits_needed > 0 && (bits_needed % 8) == 0 {
            bytes_needed += 1;
        }

        // Clamp to max 8 bytes
        let len = bytes_needed.min(8);
        raw_bytes[0..len].to_vec()
    }
}

It also compiled the code and ran tests on a variety of data, until it worked.
Created a reconstruct function, to make sure that the number converts properly both ways.

[Goverp]

That's better (though the bit about "if (bits_needed % == 0" can be optimized by a bit more thought about "bytes_needed = (bits_needed + 7) / 8".
I'd let it use to_le_bytes as well.

Which LLM?

[Anon-E-moose]

Using llama.cpp for the llm itself with pi-coding-agent as the agent harness
and qwen3-coder-next as the model (41gb, so needs a hefty box to run on, at least locally)
qwen3.5 should be out real soon, the next version is kind of a preview, and hopefully the models will get smaller with quantization.

It ran for a while, ~5 minutes, while it tried various cases, negative numbers, edge cases (exactly 64 for example)
It created the function, then a function to do the reverse, and rust code to test it out. All to make sure it worked before it gave it to me.


Edit to add: On a machine with limited resources, you could try Qwen2.5-Coder-7B-Instruct, not sure what quants ollama offers though.

[Goverp]

I had an interesting chat with my ollama qwen installation yesterday.

I asked it if it could help with one of my real-life projects, copy-editing a very large .docx file written by an author that doesn't understand styles. It said that it could, but it couldn't handle formatting, and that as it couldn't read .docx, I'd need to cut and paste the content. That was fine, so I asked if it would help if I converted it to Markdown first, which it liked, but said that for very large files, say 5 pages, I'd do better to handle the doc in small units.

I said that it might think 5 pages was very big; this document is 1,000 pages. It now explained that its context window would be far too small (1,000 pages = 2 M words ...), and that the best approach would be to just handle the worst parts. So if I were to give it say 10-15 problematic sentences, it would correct them, and give me some rules to apply to the rest of the document - by hand! Wow, any assistance short of actual help! And I've just been patronised by an AI!

I was actually impressed at the start where it's think-mode output showed it trying to guess my level of knowledge of AI to get its answer at the right level, and also careful to explain its own limitations.
Sadly, I'm left with 1,000 pages to handle the hard way, and no I don't need no AI telling me how to do that. (I can use languagetool, but the free version isn't very powerful. I could investigate Grammerly, but I'm not sure how much I can do locally.)