Building an Albanian AI
A five-part account of building a private, on-device Albanian translation model.
The vision
Why I’m Trying to Build an Albanian Translation Model
TLDR: Albanian is a low-resource language with around 7.5M speakers. I want to build a private, on-device translation model that (hopefully) runs in under 300ms on a modern iPhone. No servers, no data leaving your phone.
The Idea
I’ve been thinking a lot about what it means to translate Albanian well.
When I try to translate proverbs or idioms using whatever app is handy, whether that’s ChatGPT, Gemini, Google Translate, or any of the cloud-based services, something gets lost. The translation comes back technically correct but… flat? The cultural richness disappears.
Take “Më mirë shëndet, se mbret” which roughly means “Better health than king” (or more idiomatically, health is more valuable than power). These services handle it okay, but they don’t quite capture centuries of embedded Albanian wisdom about values and priorities.
I started wondering: could I build something better? Something that runs entirely on my phone, keeps my data private, and maybe handles Albanian nuance a little better?
I don’t know if I can. But I figured I’d try and document the journey.
Why On-Device Matters
Every time you use a cloud translation service, your text travels to someone else’s servers. (Why? More compute lives there, but I don’t like all my translations needing to go to someone else’s servers/computer.)
Maybe you’re translating a message from a family member. Or discussing something medical. Or working through a legal document. Having that text leave your phone and travel across the internet to be processed… it just feels different than keeping it local.
On-device translation means the text never leaves your pocket. The model lives on your phone. The processing happens on your phone. No internet connection needed. Complete privacy.
I think that matters, especially for diaspora communities who communicate across borders about sensitive family or business matters.
The Albanian Context
Here’s something I find fascinating about Albanian (though I should note I’m not a linguist - this is just what I’ve gathered from reading):
Albanian appears to be a linguistic isolate within the Indo-European family. When researchers draw the family tree of European languages, Albanian sits on its own branch with no close relatives. The languages closest to it apparently died out centuries ago, poorly documented.
This isolation seems to make Albanian uniquely interesting to linguists studying how languages evolve. It also seems to make translation harder: Albanian patterns don’t map as cleanly onto other European languages the way, say, Spanish relates to Italian.
The language has survived Ottoman occupation, decades of communist isolation, and ongoing globalization pressure. That resilience is remarkable.
What I’m Trying to Build
I set some goals for myself. Whether I can actually hit them remains to be seen:
| Goal | Target |
|---|---|
| Speed | Under 300ms per translation on recent iPhones |
| Size | Under 1GB (so it fits on phones without eating all your storage) |
| Accuracy | Better than I currently get with cloud services like ChatGPT, Google Translate, Gemini, or Claude. |
| Privacy | 100% on-device, no server calls |
These targets might be too aggressive. A 1GB model is tiny by modern LLM standards. But I’ve read that specialized models can sometimes outperform larger generalist models on narrow tasks. I want to test that hypothesis.
Why Proverbs Keep Coming Up
I keep returning to proverbs because they seem like the hardest test case.
Proverbs encode cultural knowledge in compressed form. “Fjala pa punë, si peshku pa lumë” translates literally as “Words without work, like fish without river.” But that misses the deeper meaning - the skepticism toward empty promises, the primacy of action over speech.
If I can get proverbs working reasonably well, I figure simpler translations should follow. If I can’t, at least I’ll learn something about the limits of small models.
These little compressed wisdom packets fascinate me. They’re also a brutal test for translation systems. If a model can handle “fish without river” correctly, it probably understands something about Albanian that goes beyond dictionary lookups. (See Part 5 on benchmarking for more on this.)
The Bigger Picture (Maybe)
I’ve been thinking about this project in a larger context.
There are thousands of languages spoken on Earth. Many are what researchers call “low-resource” - meaning there’s limited digital data available for training AI systems. Albanian, with its 7.5 million speakers, is somewhere in the middle. (Note on low resource languages: One excellent project mentioned in Part 4, that I managed to get running on an iPhone was OmniASR from Meta.)
If I can figure out a reasonable approach for Albanian: Synthetic data generation, efficient fine-tuning, on-device deployment, maybe the same techniques could help other language communities? I don’t want to overclaim here. I’m just one person messing around with models. But the possibility is exciting.
It could be cool to see Albania become a shining example of a nation leveraging local AI capabilities. Maybe that’s overly optimistic, but why not dream a little?
Technical Concepts (For Those New to This)
I’ll be using some technical terms throughout this series. Here’s my attempt to explain them in plain language (keeping in mind I’m learning too):
LLM (Large Language Model): Think of it as the engine. A neural network trained on massive amounts of text that learns to predict what words come next. GPT, Claude, Llama, Qwen - all LLMs.
Fine-tuning: Taking a general-purpose model and teaching it to specialize. The model already knows language; I’m trying to teach it specifically how Albanian maps to English. Like teaching a chef who knows cooking generally to make specifically Albanian food.
Quantization: Compressing the model to make it smaller. Neural networks usually use 32-bit numbers for their internal values. Quantization reduces this to 4-bit numbers. Result: roughly 8x smaller model, with (hopefully) minor quality loss.
On-device: Everything runs on your phone. Model weights stored locally. Processing happens locally. No internet, no servers, no data leaving the device.
LoRA (Low-Rank Adaptation): A fine-tuning technique that only updates about 2% of the model’s parameters. Much faster and cheaper than updating everything. Also helps prevent the model from forgetting what it already knows.
Tokens: Chunks of text the model processes. Roughly 0.75 words per token for English. Albanian tokenization seems slightly less efficient - more tokens per word on average.
SFT (Supervised Fine-Tuning): Teaching by example. You show the model thousands of input-output pairs (Albanian → English), and it learns the pattern. This is the bread-and-butter of making models do specific tasks.
Inference: Running the trained model to get output. When you type Albanian text and get English back, that’s “inference.” The training happens once; inference happens every time you use the app.
MLX: Apple’s machine learning framework for their chips. I ended up using this for running the model on iPhone. More on why in Part 4.
The Road Ahead
This series documents my journey:
Part 2: Sourcing the Beans - How I tried to find training data and eventually built a synthetic data pipeline. Spoiler: finding clean, licensed Albanian data was really hard.
Part 3: The Roast - All the training approaches I tried. Most failed! What eventually worked, and what I learned from the failures.
Part 4: The Pour - Getting the model onto an iPhone. Compression, MLX framework, the actual iOS app.
Part 5: The Taste Test - Current results (around 69% accuracy), why that’s both encouraging and frustrating, and what I want to try next.
A Note on Expertise
I want to be clear: I’m not an ML researcher. I’m not a linguist. And I’m not even Albanian lol.
What follows is my best attempt to document what I tried, what worked, what didn’t, and what I think it means. I’ve probably made mistakes. I’ve definitely made suboptimal choices. I’m sharing this anyway because I think the process of learning in public has value, and because maybe someone smarter than me will see this and do it better.
That said, I’ve tried to be careful about fair use and intellectual property throughout this project. The opinions expressed here are my own personal views based on my limited experience. I’m not trying to criticize any company or product - I’m just trying to solve a problem I personally care about.
Let’s see what happens.
First sip. The water’s heating. Let’s see if I can find the right beans.
Sourcing the data
Data Collection & Synthetic Generation
TLDR: I ended up building 22,369 training pairs from scratch using multiple AI models. Finding clean, open-source Albanian translation data was way harder than I expected.
I made a constraint for myself: only CC0 or MIT-licensed data. That led me down the synthetic data rabbit hole, which turned out to be pretty interesting.
The Data Problem
I started this project assuming I could just… find data somewhere. Albanian-English parallel text must exist, right? Dictionaries, textbooks, government documents, something?
Data exists. Lots of it, actually. But I quickly ran into a constraint I imposed on myself: I only wanted to use data that was clearly CC0, MIT-licensed, or otherwise unambiguously open source.
I’m not a lawyer, and I want to respect copyright. So I drew a hard line: if I couldn’t clearly verify the licensing, I wouldn’t use it.
This turned out to be a significant constraint. I looked at various resources online - dictionaries, translation corpora, benchmark datasets - and kept hitting uncertainty about whether I could actually use them for training a model. Some had restrictive licenses. Some had ambiguous terms. Some I just couldn’t verify at all.
The reality I landed on: there’s surprisingly little clearly-licensed Albanian-English training data I could find.
One Bright Spot
There was one exception I got really excited about.
Shoutout to Bonin. GitHub user bonin1 maintains a repository called Al-En-Ger-opensourcedataset with 630 high-quality Albanian-English-German triplets, properly licensed and clearly meant for this kind of use.
| Category | Approximate Count |
|---|---|
| Greetings | ~90 |
| Business | ~120 |
| Travel | ~160 |
| Education | ~110 |
| Healthcare | ~100 |
| Small talk | ~50 |
These are hand-curated, natural-sounding translations. Exactly the kind of clean data you want to build on.
630 examples isn’t enough to train a translation model. But it gave me an anchor - something I could trust completely. I ended up upsampling this dataset 5x in my final training mix, so the model sees these high-quality examples repeatedly.
Thanks again Bonin!
The Synthetic Data Approach
So here’s where I went: if clean data doesn’t exist, maybe I can create it.
The basic idea is to use multiple frontier AI models to generate translations, then filter and combine them into a training dataset. This approach has become pretty common in the ML world - I certainly didn’t invent it - but I was curious to see if it could work for Albanian.
The process ended up being more involved than I initially expected, but also more interesting.
Starting with Albanian Source Sentences
First, I needed Albanian text to translate. I curated sentences across several categories:
| Category | Rough Count | Where I Found Them |
|---|---|---|
| Proverbs | ~300 | sq.wikiquote.org (public domain) |
| Conversational | ~400 | Generated |
| Formal/business | ~300 | Generated |
| Technical | ~200 | Domain-specific terminology |
| Cultural references | ~150 | Historical, religious, traditional |
Total: around 1,350 unique Albanian source sentences.
Getting diversity here seemed important. A model trained only on casual conversation probably fails on formal text. A model trained only on simple sentences probably can’t handle complex grammar.
The Multi-Model Ensemble
I ran each source sentence through multiple different SOTA LLMs:
| Model | Why I Included It |
|---|---|
| Gemini 3 Flash | Fast, good multilingual reputation |
| Gemini 2.5 Flash Lite | Wanted to compare variants, this one benchmarks higher on some language specific tasks |
| GPT-5.2 | Figured the flagship would be strong if not a bit expensive |
| GPT-5.2-nano | Lightweight comparison |
| GPT-4o-mini | Another perspective, have heard from Native Albanian speakers that this was best-in-class. |
| Qwen3-next-80B | Open-weights model via OpenRouter |
Why multiple models? Different LLMs have different training data and different biases. When they all agree on a translation, that seems like a good signal. When they diverge, maybe that’s a flag for review.
Important note about Qwen: The Qwen 3 model documentation specifically mentions Tosk Albanian as one of the languages included in their training base dataset. This got me excited - having Albanian explicitly in the training data seemed like a good sign.
Quality Filtering
Not every generated translation is usable. I built a filtering pipeline to try to catch problems:
Length ratio checks - Albanian and English have roughly similar verbosity. If a translation is wildly different in length from the source, something might be wrong.
Character detection - Albanian uses Latin script with some specific characters (ë, ç). If English output contains Cyrillic, Chinese, or other unexpected characters, the model probably got confused.
Cross-model agreement - When 4+ models produce substantially similar translations, I weighted those higher.
I also ran everything through a safety filter (61-item blocklist for profanity, violence, explicit content). The flagged rate was pretty low - most flags were false positives like legitimate uses of words in proverbs.
What I Ended Up With
After all the filtering and deduplication:
| Component | Count |
|---|---|
| Bonin open-source (5x upsampled) | 3,150 |
| Synthetic high-quality | 14,872 |
| Synthetic medium-quality | 4,347 |
| Total training pairs | 22,369 |
I also created some preference data for potential DPO training later:
| Component | Count |
|---|---|
| CPO triplets (chosen/rejected pairs) | 1,155 |
22,369 translation pairs total, I wish this was 10x…maybe after initial launch :)
A Major Weakness: No Native Speaker Validation
Here’s something I need to be upfront about: I did not have a native Albanian speaker systematically validate this data.
This is a real weakness. Frontier LLMs are pretty good at Albanian, but they’re not perfect. They sometimes:
- Use archaic or unnatural phrasing
- Miss register (formal when casual is appropriate, or vice versa)
- Generate grammatically correct but semantically off translations
- Handle false cognates incorrectly
Without native speaker review, I’m sure there are errors in my training data that I haven’t caught. The automated filters help, but they can’t catch everything.
This is something I want to fix in future iterations, especially with less-known Albanian dialects…speaking of which:
The Dialect Challenge
Albanian has a lot of dialect complexity.
The two major dialect groups are Gheg (northern) and Tosk (southern, basis for standard Albanian). But within those, there appear to be many sub-dialects:
Northwest Gheg, Northeast Gheg, Central Gheg, Southern Gheg, Malsia Albanian, Upper Reka, Arbanasi, Transitional, Northern Tosk, Labërisht, Çam, Arvanitika, Arbëresh, Istrian Albanian
Important disclaimer: I found this list through research, but I’m not a linguist and can’t verify that all these classifications are accurate or current. Albanian dialectology seems to be a specialized field, and I’m definitely out of my depth here.
What I can say is that my training data skews heavily toward Tosk/standard Albanian. Speakers of other dialects may find the model less accurate. Building dialect-specific data would be valuable future work, though I honestly don’t know how I’d go about collecting it. (There’s also an incredible Instagram page, @projeki_ftillimi, that catalogs the massively different words for many objects/things.)
Wikiquote as a Quick Test
For informal testing during development, I used Albanian proverbs from sq.wikiquote.org/wiki/Fjalë_të_urta_shqiptare.
This Wikimedia page has around 200 Albanian proverbs with community translations. The quality varies - some translations are more literal than natural - but it’s public domain and gave me quick directional feedback.
This is definitely not a rigorous benchmark. Just a sanity check during iteration.
Why This Approach Might Matter
If you’re working on a well-resourced language like Spanish or French, you just download a dataset and start training.
For low-resource languages like Albanian, you have to build the dataset first.
The pipeline I ended up with is roughly:
- Find whatever small clean dataset exists (your “Bonin”)
- Curate diverse source sentences in the target language
- Translate via multiple frontier LLMs
- Filter: safety, deduplication, character detection
- Rank: length ratios, cross-model agreement
- Merge with upsampled clean data
The specific models will change over time. Gemini 4 will replace Gemini 3. GPT-6 will replace GPT-5. But the general approach should remain valid.
I’m hopeful this could be a template. Any language with at least some native speakers willing to curate source sentences and basic Unicode support in frontier LLMs could potentially follow this approach.
Maybe Albanian could become a reference case - the best-documented example of building AI capabilities for a low-resource language? That would be pretty cool.
What I Didn’t Solve
Dialect coverage - As mentioned, I’m heavily skewed toward Tosk. Gheg and other variants need attention.
Register diversity - I have conversational and formal text, but I’m light on specialized domains like legal, medical, or technical content.
Native speaker validation - This is my biggest weakness and the thing I most want to address.
Adversarial testing - I haven’t tested how the model handles intentionally confusing inputs, code-switching, or non-standard orthography.
These are all future work. The foundation is laid, but there’s a lot more to do.
Beans sourced. 22,369 of them. Let’s see if I can roast them without burning the batch.
The roast
Training Approaches & Failures
TLDR: I tried a bunch of different approaches. Most failed! Tried M2M-100, full fine-tuning, DPO preference learning (which broke spectacularly due to framework incompatibility). What eventually worked: MLX LoRA on Qwen3-1.7B with my 22k samples. After 15+ model variants, I landed at around 69% accuracy. Not great, not terrible - but I learned a ton.
Honest Accounting
I want to be upfront: I don’t fully know if the approaches I tried were the right ones. I’m not an ML researcher. I read papers, followed tutorials, asked for help, and experimented. A lot.
Most of what I tried didn’t work. That’s actually the story of this project - failure after failure, with occasional glimmers of hope that kept me going.
I’m sharing the failures because I think they’re at least entertaining.
The Training Environment
Let me describe what I was working with:
My main dev machine: MacBook Air M2 with 24GB unified RAM
Testing device: iPhone 14 Pro Max
Cloud compute: I used modal.com for most of my GPU training. I also tried to get set up on canopywave and hyperbolic, but ran into various issues and ended up sticking with Modal.
One thing I found myself wishing for: an Albanian cloud compute provider. It would be cool to keep the money local, support the Albanian tech ecosystem, and maybe even get better support for language-specific needs. If anyone knows of one, let me know?
Models I Explored
Before settling on my final approach, I looked at a bunch of different models:
Translation-specific models:
- M2M-100 (Meta’s multilingual translation model)
- Various seq2seq architectures
- OmniASR, but doesn’t have S2TT built-in
General LLMs I considered for fine-tuning:
- Qwen3-1.7B (what I ended up using)
- Qwen3-4B (larger, more expensive to run)
- Gemma3 (Google’s open model)
- EuroLLM-9B-Instruct (investigated, but Albanian doesn’t appear to be in their documented language list)
ASR models (for speech-to-text, which connects to my larger project):
- Whisper Turbo
- Whisper Tiny
- Whisper Flutra
- Meta’s OmniASR (got this working locally on iPhone! but it can’t do translation in the pipeline - needs more testing)
I should mention the ASR work is part of a bigger vision where speech could flow through to translation, but that’s a whole other blog series.
Why I Picked Qwen3-1.7B
Here’s my reasoning, such as it was:
Qwen explicitly mentions Albanian. In their model documentation, Qwen lists Albanian (specifically Tosk Albanian) as one of the languages included in their training data. Most other open-source models don’t mention Albanian at all.
My assumption was: a model that was deliberately trained on Albanian will probably perform better at Albanian than a model where Albanian is just incidental web scraping debris.
Is this assumption correct? Honestly, I don’t know. And this points to a bigger problem: there’s no benchmark for Albanian language capabilities in LLMs.
Think about it. If I want to know whether GPT-5 or Claude or Qwen handles Albanian better, where do I look? There’s no leaderboard. No standardized test. No grades.
I had to basically guess which model to use as my base, based on what the model creators say in their documentation and my own informal testing. That’s… not great.
Building an Albanian benchmark is high on my future work list. (I’m in talks with LLM-Stats and ready to send the benchmark.) More on that in Part 5.
What I Tried (The Failures)
There’s more attempts than listed here, but I didn’t properly document them all.
Attempt 1: M2M-100
Meta’s M2M-100 is designed for translation. It handles 100 languages, including Albanian.
Out of the box, it does… okay? Simple phrases worked. “Hello” translates fine. “Where is the train station?” comes through.
But I wasn’t happy with the outputs on more complex text. I also ran into various issues with the architecture. Fundamentally, I am also chasing less MT and more LLM.
I moved on.
Attempt 2: Full Fine-Tuning
Maybe I should just fine-tune all the parameters of a smaller model?
I tried this with Qwen3-1.7B. Training started okay, but around iteration 800, things went sideways. The loss spiked. The outputs became repetitive garbage.
My guess is that 22k examples isn’t enough to stably update 1.7 billion parameters. The model wandered into bad local minima.
Lesson learned: LoRA isn’t just more efficient - it seems to provide stability that full fine-tuning lacks on small datasets.
Attempt 3: DPO on Base Model
I’d read about Direct Preference Optimization (DPO) and thought it sounded promising. Show the model pairs of translations, tell it which one is better, let it learn preferences.
I tried applying DPO directly to the base Qwen3-1.7B using my 1,155 CPO triplets.
Result: 19% accuracy. Worse than the unmodified model.
In retrospect, this makes sense. DPO refines existing capabilities. If the model can barely translate Albanian to begin with, there’s nothing to refine. I was trying to optimize a skill the model didn’t have. (I think? Again, no benchmarks.)
Attempt 4: DPO on Fine-Tuned Model (The Disaster)
Okay, so DPO needs a model that can already do the task. I had a fine-tuned model from my LoRA experiments that could translate Albanian. Let me apply DPO to that!
This is where things got bad.
I trained my SFT (supervised fine-tuned) model using MLX, Apple’s framework for their chips. But the DPO library I was using (TRL from HuggingFace) requires PyTorch.
“No problem,” I thought. “I’ll just load the MLX weights into PyTorch.”
Result: 0% accuracy. Complete garbage. Random characters. Chinese text. Random symbols.
TLDR - MLX and PyTorch store tensor weights differently. When I loaded MLX weights into PyTorch, the values got misinterpreted. Imagine loading a JPEG as raw bitmap data - you get something, but it’s noise.
The model was outputting garbage before DPO training even started. I trained DPO on a corrupted model. Oops.
Money spent on that training run: down the drain.
Attempt 5: Targeted Overfitting
I had 18 test phrases that kept failing. What if I just trained really hard on exactly those phrases?
I created a tiny dataset of just these 18 examples and trained with a high learning rate.
Result: 23% accuracy. Even worse than before.
The model memorized (sort of) those 18 phrases and forgot everything else. Classic catastrophic forgetting.
You can’t shortcut your way to capability by training intensively on evaluation data. Who knew? (Everyone knew. I should have known.)
What Actually Worked
SFT + LoRA
After all those failures, here’s what eventually produced reasonable results:
| Setting | Value |
|---|---|
| Base model | Qwen3-1.7B |
| Method | LoRA (Low-Rank Adaptation) |
| LoRA rank | 16 |
| Training data | 22,369 SFT pairs |
| Iterations | ~3,000 |
| Framework | MLX |
LoRA only updates about 2% of the model’s parameters. This turned out to be important for stability.
One crucial discovery: the “no think” directive.
Qwen3 has a “thinking” mode where it reasons through problems before answering. Great for math problems. Bad for translation - the thinking tokens add latency and sometimes confuse the output.
I found that adding /no_think to my prompts dramatically improved both speed and consistency. The model just outputs the translation instead of “thinking” about it first.
This seems obvious in retrospect, but it took me a while to figure out. (I didn’t know the smaller Qwen models had this on by default.)
Result: 69.2% Accuracy
On my test set of 13 phrases (more on why that’s a problem in Part 5), the final model gets about 69% correct.
Basic phrases: near-perfect.
Simple sentences: very good.
Proverbs: not great.
Is 69% good? I genuinely don’t know. There’s no Albanian translation benchmark to compare against. I don’t know if my test set is representative. I don’t know if other approaches would do better.
It feels like a reasonable start. Room to improve.
The Model Zoo
I trained 15+ variants over the course of this project. Here’s a partial list:
| Model | Method | Accuracy | Notes |
|---|---|---|---|
| albanian_mt_final_4bit | SFT + LoRA | ~69% | Best so far |
| adapters_qwen_final | LoRA adapters only | ~69% | Equivalent |
| albanian_mt_dpo | DPO on base | ~19% | Failed |
| albanian_mt_dpo_v2 | DPO on SFT (corrupted) | ~0% | Disaster |
| albanian_mt_targeted | Targeted FT | ~23% | Forgetting |
| adapters_gemma | Gemma3 base | Didn’t log | Not as good |
| various others | various | various | Learning experiences |
Each training run cost money. Each failure taught me something. Some taught me more than the successes did.
Compute Costs and Learnings
Every time something failed, I had to decide whether to debug or try something different. Debugging often meant more training runs. More training runs meant more cost.
This is one reason I’m so interested in seeing an Albanian benchmark created. If I’d had a clear way to evaluate models before fine-tuning, I might have made better choices about which base model to use. I might have caught the MLX↔PyTorch incompatibility earlier. I might have spent less money on dead ends.
What I’d Do Differently
If I were starting over:
- Stay in one framework. I’d do everything in PyTorch if I needed DPO, then convert to MLX only at the end. (I thought I’d succeed sooner in the pipeline)
- Test model outputs immediately after any weight loading. I’d never train on a model without first verifying it produces sensible outputs.
- Build evaluation infrastructure first. Having a good test set from the start would have helped me catch problems earlier.
- Talk to more Albanian speakers. Native speaker feedback would have helped validate my approach earlier in the process.
The roast is done. Some beans burned. But I think there’s something usable here. Let’s see if it pours well.
Quantization and iOS deployment
TLDR: I got a 7GB model down to 934MB using 4-bit quantization. Then I used Apple’s MLX framework to run it on iPhone. The app itself is pretty simple - around 200 lines of Swift. Model loads in about 5 seconds, translations happen in under 300ms. I don’t know if this is the best approach, but it works!
The Size Problem
So I had a trained model. Exciting! But it was way too big to fit on a phone.
Qwen3-1.7B in full precision comes out to around 7GB. That’s… a lot. More than most apps. More than you want to ask someone to download. And when you load it into memory, you’re eating up significant resources.
I needed to make it smaller.
Quantization
Quantization is basically compression for neural networks. The idea is to reduce the precision of the numbers the model uses internally. Fortunately, I’ve quantised several models before, so this was somewhat familiar territory.
Normal neural networks (typically) use 32-bit floating point numbers (FP32) for their weights. Quantization reduces this:
| Format | Bits per Weight | Approximate Size |
|---|---|---|
| FP32 | 32 | ~7GB |
| FP16/BF16 | 16 | ~3.4GB |
| INT8 | 8 | ~1.7GB |
| INT4 | 4 | ~850MB |
The trade-off: smaller usually means some quality loss. The question is how much.
What I Found
I tested several quantization levels to see what happened:
| Quantization | Model Size | Accuracy | My Notes |
|---|---|---|---|
| BF16 (baseline) | ~3.4GB | ~70% | Too big for comfort |
| 8-bit | ~1.7GB | ~70% | Still pretty big |
| 4-bit | ~934MB | ~69% | Sweet spot? |
| 3-bit | ~680MB | ~64% | Quality starts dropping |
| 2-bit | ~520MB | ~47% | Unusable |
4-bit seemed like the right choice. I lost maybe 1% accuracy compared to the full model, but gained a huge size reduction. The model fits comfortably under 1GB.
Why MLX?
Apple has this framework called MLX for running machine learning on their chips. I ended up using it, though I’m honestly not 100% sure it was the right choice, but it’s working better than CoreML experiments in the past.
Things I liked about MLX:
- It’s designed specifically for Apple Silicon
- The CPU and GPU share memory, which seems to help with performance
- Variable-length inputs work naturally (unlike some frameworks that want fixed sizes)
- There’s both Python and Swift support
Comparison with alternatives:
I tried a few other approaches:
CoreML - Apple’s older ML framework. It works, but it felt clunkier for transformer models. I had to deal with fixed shape requirements that made variable-length translation annoying.
llama.cpp - This is a popular project for running LLMs efficiently. It’s really good! But I ran into issues with Qwen3’s tokenizer. Some special tokens weren’t handled correctly, leading to weird outputs.
In my (very unscientific) testing, MLX gave me better performance on my Mac and iPhone than the alternatives. Your mileage may vary.
The Learning Curve
I should be honest: getting all this working was not straightforward.
Some things that tripped me up:
Model format conversions - You can’t just take any model and load it with MLX. There’s a specific format with specific files (config.json, model.safetensors, tokenizer files). I had to learn how to convert my trained model into this format correctly.
Memory management - On the first few attempts, the app would crash shortly after loading the model. I was holding references wrong, causing memory to balloon. Took a while to figure out the right patterns.
Tokenizer quirks - Qwen3’s tokenizer has some special tokens that need to be handled carefully. I had a bug where the model would emit special tokens in the output, which looked like garbage. Fixing this required understanding the tokenizer configuration better than I wanted to.
None of this is insurmountable. But I want to be clear: if you’re thinking “I’ll just throw a model on an iPhone,” expect some small friction.
The iOS App
The app itself is pretty simple. I’m not a professional iOS developer, so the code might not be idiomatic, but it works.
Here’s the core of the translation logic:
// Setting up the model container
// MLX handles loading the quantized model from disk
let configuration = ModelConfiguration(
id: modelPath, // Path to our 4-bit model
defaultPrompt: "Translate Albanian to English"
)
// Load model (this takes ~5 seconds on my iPhone 14 Pro Max)
modelContainer = try await ModelContainer.load(configuration: configuration)And the translation session:
// Create a chat session with our translation instructions
// The /no_think flag tells Qwen to skip reasoning and just output
chatSession = ChatSession(
container,
instructions: """
You are a translator. Translate from Albanian to English.
Output only the English translation, nothing else.
/no_think
""",
generateParameters: GenerateParameters(
temperature: 0.1, // Low = more consistent outputs
maxTokens: 256 // Albanian sentences are rarely this long
)
)The actual translation is straightforward:
// Stream tokens as they're generated
// This makes the UI feel more responsive
for await token in session.generate(prompt: "Translate: \(albanianText)") {
result += token
}That’s… basically it. The MLX framework does the heavy lifting. My code just coordinates loading and calling the model. (Obviously, will make the app better after the model is better.)
Performance Numbers
Here’s what I measured on my iPhone 14 Pro Max:
| Metric | Value |
|---|---|
| Model load time | ~5 seconds |
| Tokens per second | ~10-90 tok/s |
| Short phrase translation | <150ms |
I don’t know if these numbers are good compared to other approaches. I don’t have a baseline to compare against. But they feel fast enough - when I type something and tap translate, the result appears quickly enough that it feels responsive.
Model Loading Strategy
I wanted flexibility during development, so I set up the app to look for the model in multiple places:
func findModelPath() -> String {
// First: check Documents folder (for testing new models)
// I can drop a new model via Finder without rebuilding the app
let documentsPath = FileManager.default
.urls(for: .documentDirectory, in: .userDomainMask)[0]
.appendingPathComponent("albanian-translator").path
if FileManager.default.fileExists(atPath: documentsPath) {
return documentsPath
}
// Second: check app bundle (for production)
if let bundlePath = Bundle.main.path(
forResource: "albanian-translator", ofType: nil
) {
return bundlePath
}
// Last resort: download from HuggingFace Hub
// This should never happen in production, but it's a safety net
return "mlx-community/albanian-translator-4bit"
}The Documents folder trick was really useful during development. I could test new model versions by just dragging files via Finder, without going through the whole Xcode build cycle.
What I Don’t Know
There’s a lot about iOS deployment that I’m unsure about:
Battery impact - I haven’t done rigorous battery testing. Running a 934MB model definitely uses power, but I don’t know how much compared to alternatives.
Memory pressure - The model uses up to ~900MB of RAM. On newer iPhones with 6GB+, that’s fine. On older devices with 4GB, it might cause issues. I haven’t tested extensively.
Background behavior - What happens when the app goes to background? I think the model gets unloaded, but I’m not certain about the details.
App Store approval - I haven’t submitted this to the App Store yet. I don’t know if there will be issues with the model size or performance requirements.
These are all things I need to figure out before a real release.
App Size
The final app breakdown is roughly:
| Component | Size |
|---|---|
| Model weights | ~934MB |
| Swift binary | ~12MB |
| MLX framework | ~8MB |
| Assets | ~2MB |
| Total | ~956MB |
Just under 1GB, which hits my target. But it’s still a hefty download. Users on cellular might not be thrilled.
The trade-off: you download once, and then it works forever offline. No ongoing data usage. No privacy concerns about your text going to servers.
I think that’s worth it for the right users, but it’s definitely a barrier.
What I Skipped
Some features I deliberately didn’t build:
Cloud sync - Your translations stay on your device. No iCloud, no server. Privacy-first means your data doesn’t leave, period.
Batch translation - One sentence at a time for now. Document translation would be cool but adds complexity.
Bidirectional translation - This model only does Albanian → English. English → Albanian would need a separate model or a different architecture.
Offline dictionary - When translation fails, a simple dictionary lookup might help. Future work.
Better streaming/Touch ups – Needs some work.
I wanted to ship something that worked before adding features. Maybe some of these come later.
Lessons for Others
If someone else wants to deploy a model to iOS, here’s what I learned:
- Start with MLX examples. There are sample projects in the mlx-swift repository. Study them before trying to build your own thing.
- Test on device early. The simulator doesn’t behave the same as real hardware. Memory constraints are different. Performance is different. Test on a real iPhone as soon as possible.
- Budget for model conversion headaches. Getting your trained model into the right format for MLX will probably take longer than you expect. Plan for it.
- Keep the UI simple at first. I spent too much time on UI polish early on, when I should have been focused on getting the model working correctly. (Oops)
- Document your model loading path. Where does the model live? How does it get there? How do you update it? These questions are surprisingly annoying if you don’t think about them upfront.
I hope this helps someone. I’m definitely not an expert - I’m just sharing what worked for me, mistakes and all.
The coffee is poured. Let’s see how it tastes.
Results and future plans
Results & Future Plans
TLDR: 69.2% accuracy on my tiny 13-phrase test set. Basic phrases work. Proverbs mostly fail.
The big challenge: I can’t actually tell if this is good because there’s no Albanian benchmark! Building that benchmark is now priority #1.
The Frustrating Part
Let me show you my test results:
| Albanian | Expected | Model Output | Result |
|---|---|---|---|
| Sa kushton? | How much? | How much? | Pass |
| plazh | beach | beach | Pass |
| Faleminderit | Thank you | Thank you | Pass |
| Mirëmëngjes! | Good morning! | Good evening! | Miss |
| Po | Yes | Yes | Pass |
| Jo | No | No | Pass |
| Mirupafshim | Goodbye | Goodbye | Pass |
| Si jeni? | How are you? | How are you? | Pass |
| Ku është stacioni? | Where is the station? | Where is the station? | Pass |
| Më mirë shëndet, se mbret | Better health than king | Better health, than a priest | Miss |
| Fjala pa punë, si peshku pa lumë | Words without work, like fish without river | A word without work, like a stone without a face | Miss |
| Kush punon, ha bukë | Who works, eats bread | Who works, eats | Miss |
| Nuk ka tym pa zjarr | No smoke without fire | There’s no smoke without fire | Pass |
Score: 9/13 = 69.2%
That looks decent, right? The simple stuff works. Proverbs struggle.
But here’s my frustration: I have no idea if 69% is good.
Thirteen phrases isn’t a real benchmark. I picked these phrases somewhat arbitrarily based on what I thought would be interesting test cases. Maybe I picked too many hard ones. Maybe I picked too many easy ones. Maybe my expected translations are wrong.
Is Qwen the right base model? I don’t know. I can’t compare because there’s no benchmark showing how GPT-5, Claude, Gemini, or any other model performs on Albanian.
Was my training data good? I don’t know. I have no baseline to compare against.
Did I choose the right training approach? I don’t know! Who knows!
It’s like going to school and the teacher never gives you grades. You hand in your homework and… nothing. Did you pass? Did you fail? Unclear.
This Is Why We Need an Albanian Benchmark
Let me say this louder for the people in the back:
We’re flying blind.
If a researcher wants to improve Albanian language technology, they have no way to measure progress. If a company wants to know if their translation is good enough, they have no standard to test against. If I want to know whether I should use Qwen or Gemma or Llama as my base model, I have to guess.
I’m not an expert. I’m not a linguist. But I can see that this gap needs to be filled. So I’m going to try to build something.
What An Albanian NMT Benchmark Might Look Like
Here’s my rough thinking (and I’d love feedback from people who know more than me):
| Domain | Target Sentences | Why |
|---|---|---|
| Conversational | 150 | Basic usage, most common need |
| Formal/business | 100 | Professional communications |
| News/factual | 100 | Information content |
| Proverbs | 200 | Cultural nuance, hardest test |
| Idioms | 50 | Figurative language |
| Technical | 50 | Domain-specific vocabulary |
That would be around 650 sentences. Each would need:
- Human-validated reference translations (ideally from multiple native speakers)
- Quality scores from established metrics like COMET
- Difficulty ratings so we can track performance by category
I’d want to make it open-source, with a public leaderboard where anyone can submit their model’s performance.
Is this the right approach? I’m genuinely not sure. But it seems better than what exists now, which is nothing.
The Dialect Problem (Again)
Any benchmark should probably address dialects, but this is hard.
Albanian has numerous dialects:
- Northwest Gheg, Northeast Gheg, Central Gheg, Southern Gheg
- Malsia Albanian, Upper Reka, Arbanasi
- Transitional dialects between Gheg and Tosk
- Northern Tosk, Labërisht, Çam
- Arvanitika (Greece), Arbëresh (Italy), Istrian Albanian
(Disclaimer: I compiled this list from various sources but I’m not a dialectologist. Some of these classifications might be outdated or disputed.)
My model is trained almost entirely on Tosk/standard Albanian. Testing on other dialects would probably show worse performance. But getting training data for minority dialects is difficult.
This is an unsolved problem. I’m noting it here because I think it matters, but I don’t have a solution.
Money Down the Drain
Let me be real about the cost of failures.
Every time I trained a model that didn’t work, that cost money. Cloud GPU time isn’t free. The DPO disaster where I trained on corrupted weights? Money gone. The targeted overfitting experiment? Money gone. The full fine-tuning that destabilized? Money gone.
If I’d had a benchmark from the start, if I could have evaluated base models before training, tested outputs along the way, caught problems earlier - I probably could have saved half that cost.
The benchmark isn’t just academically nice. It has practical value. It saves money and time.
Future Approach: SERA and Soft Verification
A paper came out literally today (January 28, 2026) that has me rethinking my whole approach.
SERA (Soft Extraction for Reproducibility Assessment) is a technique for generating training data using what they call “soft verification.” The basic idea, applied to my problem, would work something like this:
Back-Translation with Automatic Quality Checking
- Forward translation: Take Albanian sentence, translate to English with a good model
- Back-translation: Translate that English back to Albanian with a different model (Similar to my approach from Part 3, with a bit of extra flair.)
- Forward again: Translate the back-translated Albanian to English again
- Compare: If the two English translations are similar, the original translation was probably good
The insight is that consistency implies quality. If I translate A→B→A’→B’, and B ≈ B’, then the translation is probably reliable - even without a human checking.
This could let me:
- Generate way more training data (50k+ pairs)
- Automatically filter for quality without native speaker review (huge)
- Focus human effort on the hard cases where models disagree
Why This Might Fix My DPO Problem
My DPO training failed because of framework incompatibility (MLX → PyTorch weight corruption). But SERA suggests I might not even need DPO - their results show that pure SFT with well-verified data can match models trained with RL or preference learning.
If true, I could skip the whole DPO nightmare and just generate better training data.
Vague Instructions for Diversity
SERA also suggests using vague prompts to get diverse outputs. Instead of “translate this Albanian proverb,” try:
- “Express this Albanian concept in English”
- “Convey the meaning of this phrase”
- “Rephrase this for English speakers”
Different prompts elicit different translation styles, all potentially useful as training signal.
My Tentative Plan
- Implement soft back-translation verification
- Generate 50k+ verified training pairs
- Train SFT-only (skip DPO entirely)
- See if this beats my current 69% model
I’m excited about this but also uncertain. The paper is brand new. I haven’t tried it yet. It might not work for translation the way it works for code. But it feels like a promising direction.
The Vision: Albania as a Inspirational Case
Here’s my concept:
What if Albanian became the best-documented example of building AI capabilities for a low-resource language?
Most AI research focuses on English, Chinese, Spanish - languages with huge speaker populations and massive data availability. Low-resource languages get ignored. :(
But someone has to go first. Someone has to figure out the techniques, document the failures, build the benchmarks, publish the data.
What if that someone is the Albanian community?
Albania is small but tech-savvy. There’s a diaspora of Albanian software engineers around the world. The language is unique and valuable.
I’d love to see Albania become known for having the best low-resource language AI infrastructure in the world. A reference implementation that other language communities can learn from. A proof that you don’t need FAANG-scale resources to build something meaningful.
Maybe that’s naive. Maybe it’s impossible. But I think it’s worth trying.
Building More.
ML researchers: I’m an amateur. If you see obvious mistakes in my approach, please tell me. I’d rather be embarrassed than wrong.
Other low-resource language communities: I’m trying to document everything I learn. If this helps you build something for your language, that would make this project worthwhile.
Standing on Shoulders
I’ve read so many papers over the course of this project. Tried to understand so many techniques. Watched so many tutorials. Asked so many questions in Discord servers and forums.
I’m truly standing on the shoulders of giants here, and I couldn’t possibly name everyone who contributed to the knowledge I built on. But thank you - to the researchers who publish their work openly, to the engineers who open-source their code, to the community members who answer questions from confused beginners like me.
Specific thanks to:
- Bonin (GitHub: bonin1) for the open-source Albanian-English dataset that anchored my training
- The MLX team at Apple for building a framework that made on-device deployment possible
- The Albanian Wikipedia and Wikiquote communities for public domain content
- The authors of the SERA paper for giving me a new direction to explore
What’s Next
Short term:
- Build a proper Albanian NMT benchmark
- Submit that to LLM-stats.com and pay to run SOTA models against it
- Try the SERA approach with soft verification
- Get more native speaker involvement
Medium term:
- Scale to 50k+ training pairs
- Investigate dialect support (this is hard but important)
- Get the app to a state where I’d be comfortable releasing it
Long term (dreams):
- Public benchmark with leaderboard
- Albanian becoming a reference case for low-resource language AI
First sip of a new blend. The extraction could be better. The grind needs adjustment. Some notes are there that shouldn’t be, and some are missing that should be.
But the foundation is poured. The process is documented. The problems are identified.
Now I iterate.
Series Navigation:
- Part 1: The Vision - Why build this
- Part 2: Sourcing the Beans - Data collection
- Part 3: The Roast - Training approaches
- Part 4: The Pour - iOS deployment
- Part 5: The Taste Test - Results & future (you are here)